Video frame compression method, video frame expansion method and device

The use of multiple neural networks in video frame compression methods addresses the issue of error accumulation by independently generating compression information, improving frame quality and reducing data transmission.

JP7589427B2Active Publication Date: 2024-11-26HUAWEI TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2023528362
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-11-13
Filing Date
2021-08-11
Publication Date
2024-11-26
Estimated Expiration
2041-08-11

AI Technical Summary

Technical Problem

Existing video frame compression methods using neural networks rely heavily on the quality of decompressed reference frames, leading to error accumulation and reduced quality of reconstructed frames.

Method used

A method involving multiple neural networks, where the encoder selects a target neural network based on a network selection policy to generate compression information independently of the reference frame quality, combining the advantages of different networks to minimize data transmission and improve frame quality.

Benefits of technology

Prevents error accumulation in reconstructed frames by using multiple neural networks, ensuring frame quality and reducing data transmission, thereby enhancing the overall efficiency of video frame encoding and decoding.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007589427000013
    Figure 0007589427000013
  • Figure 0007589427000014
    Figure 0007589427000014
  • Figure 0007589427000015
    Figure 0007589427000015
Patent Text Reader

Abstract

The present application discloses a video frame compression method, a video frame decompression method, and an apparatus therefor. The method includes: determining a target neural network from a plurality of neural networks according to a network selection policy; and generating compressed information corresponding to a current video frame using the target neural network. When the compressed information is obtained using a first neural network, the compressed information includes first compressed information of a first feature of the current video frame, and a reference frame of the current video frame is used in the compression process of the first feature of the current video frame. When the compressed information is obtained using a second neural network, the compressed information includes second compressed information of a second feature of the current video frame, and a reference frame of the current video frame is used in the generation process of the second feature of the current video frame. When the compressed information is obtained using the first neural network, the accumulation of errors per frame is prevented, improving the quality of the reconstructed video frame. The method combines the advantages of the first neural network and the second neural network to minimize the amount of data that needs to be transmitted.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] [Related Applications] This application claims priority to Chinese Patent Application No. 202011271217.8, filed with the State Intellectual Property Office of the People's Republic of China on November 13, 2020, entitled "COMPRESSION ENCODING METHOD, DECOMPRESSION METHOD, AND APPARATUS," which is incorporated herein by reference in its entirety.

[0002] [Technical field] The present application relates to the field of artificial intelligence, and in particular to a video frame compression method, a video frame decompression method and an apparatus. [Background technology]

[0003] Artificial Intelligence (AI) is the theory, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, expand, and extend human intelligence, and use that knowledge to perceive the environment, obtain knowledge, and obtain optimal results. In other words, AI is a field of computer science that aims to understand the nature of intelligence and create new intelligent machines that can respond in a similar way to human intelligence. AI is the study of the design principles and implementation methods of various intelligent machines, and to give machines the capabilities of perception, reasoning, and decision-making.

[0004] Currently, a common application method of artificial intelligence is to compress video frames using a neural network based on deep learning. Specifically, an encoder uses a neural network to calculate the optical flow of the current video frame relative to a reference frame of the current video frame, and generates an optical flow of the original current video frame relative to the reference frame. The encoder performs compression encoding on the optical flow to obtain a compressed optical flow. Both the reference frame of the current video frame and the current video frame belong to the current video sequence, and the reference frame of the current video frame is a video frame that needs to be referred to when performing compression encoding on the current video frame. The compressed optical flow is decompressed to obtain a decompressed optical flow, and a predicted current video frame is generated based on the decompressed optical flow and the reference frame. A neural network is used to calculate a residual between the original current video frame and the predicted current video frame, and compression encoding is performed on the residual. The compressed optical flow and the compressed residual are sent to a decoder. Therefore, the decoder can use the neural network to obtain the decompressed current video frame based on the decompressed reference frame, the decompressed optical flow, and the decompressed residual.

[0005] The process of obtaining decompressed video frames using neural networks is overly dependent on the quality of the decompressed reference frames and accumulates errors from frame to frame, so there is an urgent need for a solution to improve the quality of the reconstructed frames of a video frame. Summary of the Invention

[0006] This application discloses a video frame compression method, a video frame decompression method and an apparatus. When a first neural network is used to obtain compression information, the quality of the reconstructed frame of a current video frame does not depend on the quality of the reconstructed frame of a reference frame of the current video frame, so that the accumulation of errors from frame to frame can be prevented and the quality of the reconstructed frame of a video frame can be improved. In addition, the advantages of the first neural network and the second neural network can be combined to minimize the amount of data that needs to be transmitted and improve the quality of the reconstructed frame of a video frame.

[0007] In order to solve the above technical problems, the present application provides the following technical solutions:

[0008] According to a first aspect, the present application provides a video frame compression method, in which artificial intelligence technology can be applied to the field of video frame encoding / decoding. The method may include: an encoder determines a target neural network from a plurality of neural networks according to a network selection policy, where the plurality of neural networks includes a first neural network and a second neural network, and performs compression encoding on a current video frame using the target neural network to obtain compression information corresponding to the current video frame.

[0009] When the compression information is obtained using the first neural network, the compression information includes first compression information of a first feature of the current video frame, a reference frame of the current video frame is used in a compression process of the first feature of the current video frame, and the reference frame of the current video frame is not used in a generation process of the first feature of the current video frame, that is, the first feature of the current video frame can be obtained based only on the current video frame, and the reference frame of the current video frame is not required in a generation process of the first feature of the current video frame. When the compression information is obtained using the second neural network, the compression information includes second compression information of a second feature of the current video frame, and the reference frame of the current video frame is used in a generation process of the second feature of the current video frame.

[0010] The current video frame is an original video frame included in the current video sequence. The reference frame of the current video frame may or may not be an original video frame in the current video sequence. The reference frame of the current video frame may be a video frame obtained after transform coding the original reference frame using an encoding network and then inverse transform decoding using a decoding network. Alternatively, the reference frame of the current video frame is a video frame obtained after an encoder performs compression coding and decompression on the original reference frame.

[0011] In this implementation, when the compression information is obtained using the first neural network, the compression information carries compression information of the first feature of the current video frame, and the reference frame of the current video frame is only used for the compression process of the first feature of the current video frame, and is not used for the generation process of the first feature of the current video frame. Therefore, after the decoder performs decompression based on the first compression information, a reconstructed frame of the current video frame can be obtained without the reference frame of the current video frame to obtain the first feature of the current video frame. Therefore, when the compression information is obtained using the first neural network, the quality of the reconstructed frame of the current video frame does not depend on the quality of the reconstructed frame of the reference frame of the current video frame, so that the accumulation of errors for each frame can be prevented and the quality of the reconstructed frame of the video frame can be improved. In addition, since a second feature of the current video frame is generated based on the reference frame of the current video frame, and an amount of data corresponding to the second compressed information of the second feature is less than an amount of data corresponding to the first compressed information of the first feature, the encoder can use the first neural network and the second neural network to process different video frames of the current video sequence, thereby combining the advantages of the first neural network and the second neural network to minimize the amount of data that needs to be transmitted and improve the quality of the reconstructed frame of the video sequence.

[0012] In a possible implementation of the first aspect, the first neural network includes an encoding network and an entropy coding layer, and obtains a first feature of the current video frame from the current video frame using the encoding network, performs entropy coding on the first feature of the current video frame using the entropy coding layer, and outputs the first compressed information. Furthermore, the first feature of the current video frame is obtained after transform coding and quantization are performed on the current video frame using the first encoding network, and the quantization is performed after the transform coding.

[0013] In a possible implementation of the first aspect, the second neural network includes a convolutional network and an entropy coding layer, the convolutional network includes a plurality of convolutional layers and an activated ReLU layer, the convolutional network is used to obtain a residual of a current video frame based on a reference frame of the current video frame, and the entropy coding layer is used to perform entropy coding on the residual of the current video frame to output second compressed information.

[0014] In a possible implementation of the first aspect, when the compression information is obtained using a second neural network, the encoder performing compression encoding on the current video frame using the target neural network to obtain compression information corresponding to the current video frame may include: the encoder generating an optical flow of the original current video frame relative to a reference frame of the current video frame, and performing compression encoding on the optical flow to obtain a compressed optical flow. The second feature of the current video frame includes the optical flow of the original current video frame relative to the reference frame of the current video frame.

[0015] Optionally, the encoder may further decompress the compressed optical flow to obtain a decompressed optical flow, generate a predicted current video frame based on the decompressed optical flow and the reference frame of the current video frame, and calculate a residual between the original current video frame and the predicted current video frame. The second feature of the current video frame includes an optical flow of the original current video frame relative to the reference frame of the current video frame, and a residual between the original current video frame and the predicted current video frame.

[0016] In a possible implementation of the first aspect, the network selection policy relates to one or more of either position information of the current video frame or the amount of data carried in the current video frame.

[0017] In a possible implementation of the first aspect, the encoder determining a target neural network from the plurality of neural networks according to the network selection policy includes: the encoder obtains position information of a current video frame in a current video sequence. The position information indicates that the current video frame is an Xth frame in the current video sequence, and the position information of the current video frame in the current video sequence may be specifically represented as an index number, and the index number may be specifically represented as a string. The encoder selects the target neural network from the plurality of neural networks based on the position information. Alternatively, the encoder determining a target neural network from the plurality of neural networks according to the network selection policy includes: the encoder selects the target neural network from the plurality of neural networks based on an attribute of the current video frame. The attribute of the current video frame indicates an amount of data to be transmitted in the current video frame, and the attribute of the current video frame includes any one or any combination of entropy, contrast, and saturation of the current video frame.

[0018] In this implementation, a target neural network is selected from the multiple neural networks based on position information of the current video frame in the current video sequence. Alternatively, a target neural network can be selected from the multiple neural networks based on at least one attribute of the current video, and the target neural network can be used to generate the compression information of the current video frame. This provides multiple implementation solutions that are simple and easy to operate, and improves the flexibility of the implementation of the solution.

[0019] In a possible implementation of the first aspect, the method further includes: the encoder generates and transmits at least one indication information in one-to-one correspondence with one or more pieces of compressed information, each indication information indicating that one piece of compressed information is obtained using a target neural network, which is either the first neural network or the second neural network, i.e., the indication information indicates that the compressed information is obtained using either the first neural network or the second neural network.

[0020] In this implementation, the decoder can obtain multiple indication information corresponding to multiple compression information, so that the decoder can know to use either the first neural network or the second neural network to decompress each video frame of the current video sequence, which helps the decoder to reduce the time it takes to decode the compression information, that is, this helps to improve the overall efficiency of the video frame transmission performed by the encoder and the decoder.

[0021] In a possible implementation of the first aspect, if the compression information is obtained using a first neural network, the encoder performing compression encoding on the current video frame using the target neural network to obtain compression information corresponding to the current video frame may include: the encoder obtaining a first feature of the current video frame from the current video frame using the encoding network, and predicting a feature of the current video frame based on a reference frame of the current video frame using an entropy coding layer to generate a predicted feature of the current video frame. The predicted feature of the current video frame is a prediction result of the first feature of the current video frame, and a data shape of the predicted feature of the current video frame is the same as a data shape of the first feature of the current video frame. The encoder generating a probability distribution of the first feature of the current video frame using the entropy coding layer based on the predicted feature of the current video frame. The probability distribution of the first feature of the current video frame includes a mean of the first feature of the current video frame and a variance of the first feature of the current video frame. The encoder performing entropy encoding using the entropy coding layer on the first feature of the current video frame based on the probability distribution of the first feature of the current video frame to obtain the first compression information.

[0022] In this implementation, the encoder generates a probability distribution of the first feature of the current video frame based on the predicted feature of the current video frame, and then performs compression encoding on the first feature of the current video frame based on the probability distribution of the first feature of the current video frame to obtain the first compressed information of the current video frame. A high similarity between the predicted feature of the current video frame and the first feature indicates a large compression ratio of the first feature and a small first compressed information finally obtained. The predicted feature of the current video frame is obtained by predicting the feature of the current video frame based on the reference frame of the current video frame, and improves the similarity between the predicted feature of the current video frame and the first feature of the current video frame. This reduces the size of the compressed first compressed information. That is, the quality of the reconstructed frame obtained by the decoder can be ensured and the amount of data transmitted between the encoder and the decoder can be reduced.

[0023] In a possible implementation of the first aspect, both the first neural network and the second neural network are trained neural networks, and the model parameters of the first neural network are updated according to a first loss function of the first neural network. The first loss function includes a loss term of similarity between the first training video frame and the first training reconstructed frame and a loss term of data size of compressed information of the first training video frame, and the first training reconstructed frame is a reconstructed frame of the first training video frame. The training objective of the first loss function includes increasing the similarity between the first training video frame and the first training reconstructed frame, and further includes reducing the size of the first compressed information of the first training video frame. During training of the second neural network based on one or more second training video frames, the reference frames of the second training video frames, and the second loss function, the second loss function includes a loss term of similarity between the second training video frame and the second training reconstructed frame and a loss term of data size of compressed information of the second training video frame. The second training reconstructed frame is a reconstructed frame of the second training video frame, and the reference frame of the second training video frame is a video frame processed by the first neural network. The training objective of the second loss function includes increasing the similarity between the second training video frame and the second training reconstructed frame, and further includes reducing the size of the second compressed information of the second training video frame.

[0024] In this implementation, in the execution phase, the reference frames used by the second neural network may be processed by the first neural network, and thus the reference frames processed by the first neural network are used to train the second neural network, which helps maintain consistency between the training and execution phases and improve the accuracy of the execution phase.

[0025] According to a second aspect, an embodiment of the present application provides a video frame compression method, in which artificial intelligence technology can be applied to the field of video frame encoding / decoding. An encoder performs compression encoding on a current video frame using a first neural network, obtains first compression information of a first feature of the current video frame, a reference frame of the current video frame is used in the compression process of the first feature of the current video frame, and generates a first video frame using the first neural network, where the first video frame is a reconstructed frame of the current video frame.

[0026] The encoder performs compression encoding on the current video frame using a second neural network to obtain second compression information of a second feature of the current video frame, a reference frame of the current video frame is used in the generation process of the second feature of the current video frame, and generates a second video frame using the second neural network, where the second video frame is a reconstructed frame of the current video frame.

[0027] The encoder determines compression information corresponding to the current video frame based on the first compression information, the first video frame, the second compression information, and the second video frame. The determined compression information is obtained using a first neural network, and the determined compression information is the first compression information. Alternatively, the determined compression information is obtained using a second neural network, and the determined compression information is the second compression information.

[0028] In this implementation, the compressed information that needs to be finally transmitted is selected from the first compressed information and the second compressed information based on at least one first compressed information of a current video frame, the first video frame, the second compressed information of the current video frame, and the second video frame. In this implementation, compared with a method of determining a target neural network from multiple neural networks according to a network selection policy and then generating target compressed information using the target neural network, the performance of the compressed information corresponding to the entire current video sequence can be improved as much as possible.

[0029] In a possible implementation of the second aspect, for different video frames in a current video sequence, the encoder can use the same method to select target compression information. Specifically, the encoder calculates a first score (i.e., a first score corresponding to a first neural network) corresponding to the first compression information based on the first compression information and the first video frame, calculates a second score (i.e., a second score corresponding to a second neural network) corresponding to the second compression information based on the second compression information and the second video frame, selects a smaller score among the first score and the second score, and determines the compression information corresponding to the smaller score from the first compression information and the second compression information as the compression information of the current video frame, that is, determines the neural network corresponding to the smaller score as the target neural network.

[0030] In this implementation, for each video frame of the current video sequence, the encoder first compresses the current video frame using a first neural network and a second neural network, obtains a first score corresponding to the first compressed information and a second score corresponding to the second compressed information, and determines the smaller score of the first score and the second score to minimize the scores of all video frames in the entire video sequence, thereby improving the performance of the compressed information corresponding to the entire current video sequence.

[0031] In a possible implementation of the second aspect, the encoder may use one period as a calculation unit, and generate coefficients and offsets of a first fitting equation corresponding to a plurality of first scores in the period based on two first scores corresponding to the twelfth current video frame in the period, and generate coefficients and offsets of a second fitting equation corresponding to a plurality of second scores in the period based on two second scores corresponding to the twelfth current video frame in the period. The encoder determines compression information of the current video frame from the first compression information and the second compression information according to the first fitting equation and the second fitting equation. The optimization objective is to minimize the average value of the total score in the period, i.e., the optimization objective is to minimize the total score in the period.

[0032] In the embodiment of the present application, a person skilled in the art will find the change rule of the first score and the second score in a period during research, and use the minimum average value of all scores in a period as the optimization objective. That is, when determining the target compression information corresponding to each current video frame, it is necessary to consider both the score of the current video frame and the average value of the score in the entire period to further reduce the scores corresponding to all video frames in the entire current video sequence. This will further improve the performance of the compression information corresponding to the entire current video sequence.

[0033] In a second aspect of this embodiment of the present application, the encoder can further perform the steps performed by the encoder in the possible implementation of the first aspect. For the second aspect of this embodiment of the present application, the meaning of the terms, the specific implementation steps of the second aspect and the possible implementations of the second aspect of the present application, and the beneficial effects brought about by each possible implementation, please refer to the description of the possible implementations of the first aspect. The details will not be described again here.

[0034] According to a third aspect, an embodiment of the present application provides a video frame compression method, in which artificial intelligence techniques can be applied to the field of video frame encoding / decoding. The method may include: an encoder; performing compression encoding on a third video frame using a first neural network to obtain first compression information corresponding to the third video frame, the first compression information including compression information of a first feature of the third video frame, and a reference frame of the third video frame being used for compression processing of the first feature of the third video frame; performing compression encoding on a fourth video frame using a second neural network to obtain second compression information corresponding to the fourth video frame, the second compression information including compression information of a second feature of the fourth video frame, a reference frame of the fourth video frame being used in the generation process of the second feature of the fourth video frame, and the third video frame and the fourth video frame being different video frames of a same video sequence; The method includes:

[0035] In the third aspect of this embodiment of the present application, the encoder can further perform the steps performed by the encoder in the possible implementation of the first aspect. For the third aspect of this embodiment of the present application, the meaning of the terms in the third aspect, the specific implementation steps in the possible implementations, and the beneficial effects brought about by each possible implementation, please refer to the description in the possible implementations of the first aspect. Details will not be described again here.

[0036] According to a fourth aspect, an embodiment of the present application provides a video frame decompression method. In this method, artificial intelligence technology can be applied to the field of video frame encoding / decoding. A decoder obtains compression information of a current video frame, and performs decompression using a target neural network based on the compression information of the current video frame to obtain a reconstructed frame of the current video frame. The target neural network is a neural network selected from a plurality of neural networks, where the plurality of neural networks includes a third neural network and a fourth neural network. When the target neural network is the third neural network, the compression information includes first compression information of a first feature of the current video frame. A reference frame of the current video frame is used in a decompression process of the first compression information to obtain the first feature of the current video frame. The first feature of the current video frame is used in a generation process of a reconstructed frame of the current video frame. When the target neural network is the fourth neural network, the compression information includes second compression information of a second feature of the current video frame, and the second compression information is used by the decoder to perform decompression to obtain a second feature of the current video frame. The reference frame of the current video frame and the second feature of the current video frame are used in a process of generating a reconstructed frame of the current video frame, and the current video sequence includes the reconstructed frame of the current video frame and the reference frame of the current video frame.

[0037] In a possible implementation of the fourth aspect, the third neural network includes an entropy decoding layer and a decoding network, where the entropy decoding layer is used to perform entropy decoding on the first compressed information of the current video frame based on a reference frame of the current video frame, and the decoding network is used to generate a reconstructed frame of the current video frame based on a first feature of the current video frame.

[0038] Furthermore, if the compressed information is decompressed using a third neural network, the decoder performs decompression using the target neural network based on the compressed information of the current video frame, and obtaining a reconstructed frame of the current video frame may include: the decoder generates a probability distribution of a first feature based on the predicted feature of the current video frame. The predicted feature of the current video frame is obtained by predicting the first feature based on a reference frame of the current video frame. The decoder performs entropy decoding on the compressed information based on the probability distribution of the first feature to obtain the first feature, and performs inverse transform decoding on the first feature to obtain a reconstructed frame of the current video frame.

[0039] In a possible implementation of the fourth aspect, the fourth neural network includes an entropy decoding layer and a convolutional network, where the entropy decoding layer is used to perform entropy decoding on the second compressed information, and the convolutional network is used to perform a process for generating a reconstructed frame of the current video frame based on a reference frame of the current video frame and a second feature of the current video frame.

[0040] Further, if the compressed information is decompressed using a fourth neural network, the decoder performs decompression using the target neural network based on the compressed information of the current video frame, and obtaining a reconstructed frame of the current video frame may include: the decoder decompresses the second compressed information and obtains a second feature of the fourth video frame, i.e., obtains the second feature of the current video frame, i.e., obtains an optical flow of the original current video frame relative to a reference frame of the current video frame and a residual between the original current video frame and the predicted current video frame; the encoder predicts the current video frame based on the reference frame of the current video frame and the optical flow of the original current video frame relative to the reference frame of the current video frame to obtain a predicted current video frame, and generates a reconstructed frame of the current video frame based on the predicted current video frame and the residual between the original current video frame and the predicted current video frame.

[0041] In a possible implementation of the fourth aspect, the method may further include: the decoder obtains at least one indication information having a one-to-one correspondence with the at least one compression information, and determines a target neural network corresponding to the current video frame from a plurality of neural networks including a third neural network and a fourth neural network based on the at least one indication information and the compression information of the current video frame.

[0042] For the fourth aspect of this embodiment of the present application, the meaning of the terms in the fourth aspect, and the beneficial effects brought about by each possible implementation, please refer to the description in the possible implementation of the first aspect. Details will not be described again here.

[0043] According to a fifth aspect, an embodiment of the present application provides a video frame decompression method. In this method, artificial intelligence technology can be applied to the field of video frame encoding / decoding. The decoder uses a third neural network to decompress first compressed information of a third video frame to obtain a reconstructed frame of the third video frame. The first compressed information includes compressed information of a first feature of the third video frame, and a reference frame of the third video frame is used in a decompression process of the first compression process to obtain a first feature of the third video frame, and the first feature of the third video frame is used in a generation process of a reconstructed frame of the third video frame. The decoder uses a fourth neural network to decompress second compressed information of a fourth video frame to obtain a decompressed fourth video frame. The second compressed information includes compressed information of a second feature of the fourth video frame, and the second compressed information is used by the decoder to perform decompression to obtain a second feature of the fourth video frame, and the reference frame of the fourth video frame and the second feature of the fourth video frame are used in a generation process of a reconstructed frame of the fourth video frame.

[0044] In the fifth aspect of this embodiment of the present application, the decoder can further perform the steps performed by the decoder in the possible implementation of the fourth aspect. For the fifth aspect of this embodiment of the present application, the meaning of the terms in the fifth aspect, the specific implementation steps in the possible implementations, and the beneficial effects brought about by each possible implementation, please refer to the description in the possible implementations of the fourth aspect. Details will not be described again here.

[0045] According to a sixth aspect, an embodiment of the present application provides an encoder including a processing circuit configured to perform a method according to any of the first, second, third, fourth or fifth aspects.

[0046] According to a seventh aspect, an embodiment of the present application provides a decoder including a processing circuit configured to perform a method of any of the first, second, third, fourth or fifth aspects.

[0047] According to an eighth aspect, an embodiment of the present application provides a computer program product which, when executed on a computer, enables the computer to carry out a method of any of the first, second, third, fourth or fifth aspects.

[0048] According to a ninth aspect, an embodiment of the present application provides an encoder comprising one or more processors and a non-transitory computer readable storage medium coupled to the processors and storing program instructions executed by the processors, the program instructions when executed by the processors enabling the encoder to perform a video frame compression method according to any of the first, second or third aspects.

[0049] According to a tenth aspect, an embodiment of the present application provides a decoder, the decoder comprising one or more non-transitory computer readable storage media coupled to a processor and storing program instructions for execution by the processor, the program instructions when executed by the processor enabling the decoder to perform the method for decompressing a video frame of either the fourth or fifth aspect.

[0050] According to an eleventh aspect, an embodiment of the present application provides a non-transitory computer readable storage medium comprising program code which, when executed on a computer, enables the computer to perform any of the methods of the first, second, third, fourth or fifth aspects.

[0051] According to a twelfth aspect, an embodiment of the present application further provides a circuit system including a processing circuit, the processing circuit configured to perform a method according to any of the first, second, third, fourth or fifth aspects.

[0052] According to a thirteenth aspect, an embodiment of the present application provides a chip system. The chip system includes a processor and is configured to implement the functionality of the aforementioned aspects, such as, for example, transmitting or processing data and / or information in the manner described above. In a possible design, the chip system further includes a memory. The memory is configured to store program instructions and data required for the server or communication device. The chip system may include the chip, or may include the chip and another discrete component. [Brief description of the drawings]

[0053] [Figure 1A] 1 is a schematic diagram of the structure of the main framework of artificial intelligence according to an embodiment of the present application;

[0054] [Figure 1B] FIG. 2 is a diagram of an application scenario of the video frame compression method and the video frame decompression method according to an embodiment of the present application;

[0055] [Figure 1C] FIG. 2 is a diagram of another application scenario of the video frame compression method and the video frame decompression method according to an embodiment of the present application;

[0056] [Diagram 2] FIG. 1 is a schematic diagram of the principle of a video frame compression method according to an embodiment of the present application;

[0057] [Diagram 3] 1 is a schematic flow chart of a video frame compression method according to an embodiment of the present application;

[0058] [Figure 4] 2 is a schematic diagram of the correspondence between the position of the current video frame and the target neural network used in the video frame compression method according to an embodiment of the present application;

[0059] [Figure 5A] FIG. 2 is a schematic diagram of the structure of a first neural network according to an embodiment of the present application;

[0060] [Figure 5B] FIG. 2 is a schematic diagram of the structure of a second neural network according to an embodiment of the present application.

[0061] [Figure 5C] FIG. 2 is a schematic diagram of a comparison between a first feature and a second feature in a video frame compression method according to an embodiment of the present application;

[0062] [Figure 6] FIG. 2 is a schematic diagram of another principle of a video frame compression method according to an embodiment of the present application;

[0063] [Figure 7A] 4 is another schematic flowchart of a video frame compression method according to an embodiment of the present application;

[0064] [Figure 7B] FIG. 2 is a schematic diagram of a first score and a second score in the video frame compression method according to an embodiment of the present application;

[0065] [Figure 7C] FIG. 2 is a schematic diagram of the calculation of coefficients and offsets of a first fitting equation and coefficients and offsets of a second fitting equation of a video frame compression method according to an embodiment of the present application;

[0066] [Figure 8] 4 is another schematic flowchart of a video frame compression method according to an embodiment of the present application;

[0067] [Figure 9] 1 is a schematic diagram of a video frame compression method according to an embodiment of the present application;

[0068] [Figure 10A] 1 is a schematic flowchart of a video frame decompression method according to an embodiment of the present application;

[0069] [Figure 10B] 4 is another schematic flowchart of a video frame decompression method according to an embodiment of the present application;

[0070] [Figure 11] 4 is another schematic flowchart of a video frame decompression method according to an embodiment of the present application;

[0071] [Figure 12] 1 is a schematic flowchart of a method for training a video frame compression and decompression system according to an embodiment of the present application;

[0072] [Figure 13] FIG. 1 is a diagram of a system architecture of a video encoding / decoding system according to an embodiment of the present application.

[0073] [Figure 14] FIG. 2 is another system architecture diagram of a video encoding / decoding system according to an embodiment of the present application.

[0074] [Figure 15] 1 is a schematic diagram of a video coding device according to an embodiment of the present application;

[0075] [Figure 16] FIG. 1 is a simplified block diagram of an apparatus according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0076] In the specification, claims, and accompanying drawings of this application, the terms "first", "second", etc. are intended to distinguish between similar objects and do not necessarily indicate a particular order or sequence. It should be understood that terms used in such a manner are interchangeable in the appropriate environment, and this is merely a way of distinguishing when objects having the same attributes are described in the embodiments of this application. Furthermore, the terms "comprise", "have", and any other variations are meant to cover a non-exclusive inclusion, and a process, method, system, product, or apparatus that includes a series of units is not limited to those units and may include other units that are not expressly listed or that are inherent to such process, method, system, product, or apparatus.

[0077] The following describes embodiments of the present application with reference to the accompanying drawings. Those skilled in the art can understand that the technical solutions provided in the embodiments of the present application can also be applied to similar technical problems as technology evolves and new scenarios emerge.

[0078] First, we will explain the overall operating procedure of the artificial intelligence system. Figure 1A is a schematic diagram of the structure of the main framework of artificial intelligence. In the following, the main framework of artificial intelligence is described from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). The "intelligent information chain" reflects a series of processing from data acquisition to processing. For example, the processing may be the general processing of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In these processing, the data goes through the refinement processing of "data-information-knowledge-intelligence". The "IT value chain" reflects the value brought by artificial intelligence to the information technology industry, from the underlying infrastructure and information of human perception (technology that provides and processes implementation) to the industrial ecological processing of the system.

[0079] (1) Infrastructure

[0080] The infrastructure supports the computing power of the artificial intelligence system, implements communication with the outside world, and implements the support using a basic platform. The infrastructure communicates with the outside world using sensors. The computing power is provided by smart chips. For example, smart chips include a central processing unit (CPU), a neural network processing unit (NPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a hardware acceleration chip such as a field programmable gate array (FPGA). The basic platform includes related platforms for guarantee and support, such as distributed computing frameworks and networks, including cloud storage and computing, interconnection networks, etc. For example, sensors communicate with the outside world to obtain data, and the data is provided to the smart chips of the distributed computing system provided by the basic platform for computing.

[0081] (2) Data

[0082] The data in the upper layer of the infrastructure represents data sources in the field of artificial intelligence. The data relates to graphs, pictures, voice, text, and also to IoT (Internet of things) data of traditional devices, service data of existing systems, and sensory data such as force, displacement, liquid level, temperature, humidity, etc.

[0083] (3) Data processing

[0084] Data processing typically includes methods such as data training, machine learning, deep learning, search, inference, or decision-making.

[0085] Machine learning and deep learning can mean performing symbolic and formalized intelligent information modeling, extraction, pre-processing, training, etc. on data.

[0086] Inference is the process of simulating the human intellectual reasoning method in a computer or intelligent system, and using formal information to perform mechanical thinking and problem solving according to an inference control policy. The main functions are searching and matching.

[0087] Decision making is the process of making decisions based on intelligent information inference, usually providing functions such as classification, ranking, and prediction.

[0088] (4) General Ability

[0089] After the above data processing is performed on the data, some further general capabilities may be formed based on the data processing results, such as algorithms and general systems, such as translation, text analysis, computer vision processing, voice recognition, image recognition, etc.

[0090] (5)Smart products and industrial applications

[0091] Smart products and industrial applications are the products and applications of artificial intelligence systems in various fields, and are the overall solution package of artificial intelligence, so that intelligent information decision-making is commercialized and applications are implemented. The application fields mainly include smart terminals, smart manufacturing, smart transportation, smart homes, smart healthcare, smart security protection, autonomous driving, smart cities, etc.

[0092] The embodiments of the present application are mainly applied to scenarios in which encoding / decoding needs to be performed on video frames of a video in the above various application fields. Specifically, in order to more intuitively understand the application scenario of the solution, FIG. 1B is a diagram of the application scenario of the video frame compression method and the video frame decompression method according to the embodiment of the present application. Please refer to FIG. 1B. For example, when a video is stored in a client's album, a requirement for sending the video in the album to a cloud server is stored. Then, the client (i.e., the encoder) may use AI technology to perform compression encoding on the video frames and obtain compression information corresponding to each video frame. The compression information corresponding to each video frame is sent to the cloud server. In response, the cloud server (i.e., the decoder) can use AI technology to perform decompression and obtain a reconstructed frame of the video frame. It should be understood that the example of FIG. 1B is merely for the purpose of making the solution easier to understand, and does not limit the solution.

[0093] In another example, in the smart city field, a monitor needs to send collected videos to a management center. Before sending the videos to the management center, the monitor (i.e., an encoder) needs to compress the video frames of the video. Correspondingly, the management center (i.e., a decoder) needs to decompress the video frames of the video to obtain the video frames.

[0094] As another example, in order to more intuitively understand the application scenario of the solution, FIG. 1C is a diagram of another application scenario of the video frame compression method and the video frame decompression method according to an embodiment of the present application. FIG. 1C is explained by using an example of applying the embodiment of the present application to a live broadcast scenario. An anchor uses a client to collect a video, and the client needs to send the collected video to a server, and the server distributes the video to a user. Before the client (i.e., the encoder) sends the video to the server, it is necessary to perform compression encoding on the video frames of the video using AI technology. Correspondingly, the client (i.e., the decoder) can perform decompression using AI technology to obtain a reconstructed frame of the video frame, and so on. It should be understood that the example of FIG. 1C is merely for easy understanding of the solution, and does not limit the solution.

[0095] Note that this example is provided to facilitate understanding of application scenarios of the embodiments of the present application, and does not enumerate the application scenarios of the embodiments of the present application.

[0096] In the embodiment of the present application, AI technology (i.e., neural network) is used to perform compression coding and decompression on video frames. Therefore, the embodiment of the present application includes a neural network inference phase and a neural network training phase. The processes of the neural network inference phase and the neural network training phase are different. In the following, the inference phase and the training phase will be described separately.

[0097] 1. Inference Phase

[0098] From the above description, it can be seen that in the compression encoding method and the decompression method provided in the embodiment of the present application, the encoder performs compression encoding, and the decoder performs decompression. In the following, the operation of the encoder and the decoder will be described separately. In addition, in the case where multiple neural networks are configured in the encoder, a process in which the encoder generates target compression information corresponding to the current video will be described. In the implementation, the encoder first determines a target neural network from multiple neural networks according to a network selection policy, and generates target compression information corresponding to the current video frame using the target neural network. In another implementation, the encoder uses multiple neural networks to individually generate multiple compression information of the current video frame, and determines target compression information corresponding to the current video frame based on the generated multiple compression information. Since the above two implementations have different implementation procedures, the two implementations will be described separately below.

[0099] 1. The encoder first selects a target neural network from a number of neural networks.

[0100] In some embodiments of the present application, the encoder first selects a target neural network to be used to process a current video frame from a plurality of neural networks according to a network selection policy. To more intuitively understand this solution, FIG. 2 is a schematic diagram of the principle of a video frame compression method according to an embodiment of the present application. As shown in FIG. 2, for any video frame in a current video sequence (i.e., the current video frame in FIG. 2), the encoder selects a target neural network from a plurality of neural networks according to a network selection policy, and performs compression encoding on the current video frame using the target neural network to obtain target compression information corresponding to the current video frame. It should be understood that the example of FIG. 2 is merely for ease of understanding the solution, and does not limit the solution. Specifically, FIG. 3 is a schematic flowchart of a video frame compression method according to an embodiment of the present application. The video frame compression method according to an embodiment of the present application may include the following steps:

[0101] 301: An encoder determines a target neural network from a plurality of neural networks according to a network selection policy.

[0102] In an embodiment of the present application, a plurality of neural networks are configured for an encoder. The plurality of neural networks includes at least a first neural network, a second neural network, or another type of neural network used to perform compression. The first neural network, the second neural network, and the other type of neural network are all trained neural networks. When the encoder processes any video frame in a current video sequence, the encoder can determine a target neural network from the plurality of neural networks according to a network selection policy, and use the target neural network to perform compression encoding on the current video frame to obtain target compression information corresponding to the current video frame. The target compression information is the compression information that the encoder determines to finally send to the decoder. That is, the target compression information is generated by one target neural network among the plurality of neural networks.

[0103] In the following embodiments of the present application, an example will be described in which the multiple neural networks include only the first neural network and the second neural network. For a case in which the multiple neural networks include three or more neural networks, please refer to the description of the case in which the multiple neural networks include the first neural network and the second neural network in the embodiment of the present application. Details will not be described again in this embodiment of the present application.

[0104] Video coding usually means processing a sequence of pictures that form a video or a video sequence. In the field of video coding, the terms "picture", "video frame" or "image" may be used synonymously. Video coding is performed at the source side and usually involves processing (e.g., compression) of the original video frame to reduce the amount of data required to represent the video frame (thus improving storage and / or transmission efficiency). Video decoding is performed at the destination side and usually involves the reverse processing compared to the encoder to reconstruct the video frame. The encoder and decoder are also called encoding and decoding (CODEC).

[0105] A network selection policy relates to one or more of the position information of the current video frame or the amount of data carried in the current video frame.

[0106] Specifically, the following describes a process of selecting a target neural network from a plurality of neural networks according to a network selection policy. In implementation, step 301 may include: an encoder can obtain position information of a current video frame in a current video sequence, where the position information indicates that the current video frame is an Xth frame in the current video sequence, and can select a target neural network corresponding to the position information of the current video sequence from a plurality of neural networks including a first neural network and a second neural network according to a network selection policy.

[0107] The position information of the current video frame within the current video sequence may be specifically represented as an index number, and the index number may be specifically represented as a string. For example, the index number of the current video frame may be specifically 00000223, 00000368, or another string, not enumerated here.

[0108] The network selection policy may alternately select the first neural network or the second neural network according to a certain rule. That is, the encoder performs compression encoding on n video frames of the current video frame using the first neural network, and then performs compression encoding on m video frames of the current video frame using the second neural network. Alternatively, after performing compression encoding on m video frames of the current video frame using the second neural network, the encoder performs compression encoding on n video frames of the current video frame using the first neural network. The values ​​of both n and m may be integers equal to or greater than 1, and the values ​​of n and m may be the same or different.

[0109] For example, when the values ​​of n and m are 1, the network selection policy may perform compression encoding on odd frames of the current video sequence using a first neural network and perform compression encoding on even frames of the current video sequence using a second neural network. Alternatively, the network selection policy may perform compression encoding on odd frames of the current video sequence using a second neural network and perform compression encoding on even frames of the current video sequence using the first neural network. In another example, for example, the value of n is 1 and the value of m is 3. The network selection policy may perform compression encoding on one video frame of the current video sequence using the first neural network, followed by compression encoding on three consecutive video frames of the current video sequence using the second neural network, etc., which are not enumerated here.

[0110] In order to understand the solution more intuitively, FIG. 4 is a schematic diagram of the correspondence between the position of the current video frame and the target neural network used in the video frame compression method according to the embodiment of the present application. FIG. 3 is explained using an example where n is 1 and m is 3. As shown in FIG. 4, after performing compression encoding on the t-th video frame using the first neural network, the encoder performs compression encoding on the (t+1)-th video frame, the (t+2)-th video frame, and the (t+3)-th video frame separately using the second neural network, and performs compression encoding on the (t+4)-th video frame using the first neural network at another time. That is, after performing compression encoding on one current video frame using the first neural network, the encoder performs compression encoding on three current video frames using the second neural network. It should be understood that the example of FIG. 4 is merely for the purpose of making the solution easier to understand, and does not limit the solution.

[0111] In another implementation, step 301 may include: the encoder obtains attributes of a current video frame, and selects a target neural network from the first neural network and the second neural network. The attributes of the current video frame indicate an amount of data to be transmitted in the current video frame, and the attributes of the current video frame include any one or any combination of entropy, contrast, saturation, and other types of attributes of the current video frame, which are not enumerated here.

[0112] Furthermore, a larger entropy of the current video frame indicates a larger amount of data carried in the current video frame and a larger probability that the target neural network is the second neural network. A smaller entropy of the current video frame indicates a smaller probability that the target neural network is the second neural network. A larger contrast of the current video frame indicates a larger amount of data carried in the current video frame and a larger probability that the target neural network is the second neural network. A smaller contrast of the current video frame indicates a smaller probability that the target neural network is the second neural network.

[0113] In this implementation of the present application, a target neural network is selected from the multiple neural networks based on the position information of the current video frame in the current video sequence. Alternatively, a target neural network can be selected from the multiple neural networks based on at least one attribute of the current video, and the target neural network can be used to generate the compression information of the current video frame. This provides multiple implementation solutions that are simple and easy to operate, and improves the flexibility of the implementation of the solution.

[0114] In another implementation, the encoder can select either the first neural network or the second neural network as the target neural network, and generate the target compressed information of the current video frame using the target neural network. Optionally, the encoder can separately configure a first selection probability of the first neural network and a second selection probability of the second neural network, where the value of the second selection probability is equal to or greater than the value of the first selection probability, and can select the target neural network based on the first selection probability and the second selection probability. For example, the value of the first selection probability is 0.2, and the value of the second selection probability is 0.8. In another example, the value of the first selection probability is 0.3, and the value of the second selection probability is 0.7. The values ​​of the first selection probability and the second selection probability are not enumerated here.

[0115] 302: The encoder performs compression encoding on the current video frame using the target neural network to obtain target compression information corresponding to the current video frame.

[0116] In this embodiment of the present application, the target neural network may be a first neural network, a second neural network, another network used to compress the video frame, etc. When the first neural network is used to obtain the compression information, the target compression information includes first compression information of the first feature of the current video frame, a reference frame of the current video frame is used in the compression process of the first feature of the current video frame, and a reference frame of the current video frame is not used in the generation process of the first feature of the current video frame.

[0117] Both the reference frame of the current video frame and the current video frame are from the current video sequence. The current video frame is an original video frame included in the current video sequence. In the implementation, the reference frame of the current video frame may be an original video frame of the current video sequence, and the sorting position of the reference frame of the current video sequence may be located before or after the current video frame. That is, when playing the current video sequence, the reference frame may be displayed before the current video frame or after the current video frame.

[0118] In another implementation, the reference frame of the current video frame may not be an original video frame of the current video sequence, and the sorting position of the original reference frame in the current video sequence corresponding to the reference frame of the current video frame may be located before or after the current video frame. The reference frame of the current video frame may be a video frame obtained after the encoder performs transform coding and inverse transform decoding on the original reference frame. Alternatively, the reference frame of the current video frame is a video frame obtained after the encoder performs compression coding and decompression on the original reference frame. Furthermore, the compression can be performed using the first neural network or the second neural network.

[0119] See the description in Patent Application No. CN202011271217.8. The first neural network includes at least an encoding network and an entropy coding layer, and uses the encoding network to obtain a first feature of the current video frame from the current video frame, and uses the entropy coding layer to perform a compression process of the first feature of the current video frame according to a reference frame of the current video frame, and output first compression information corresponding to the current video frame.

[0120] In order to more intuitively understand this solution, Figure 5A is a schematic diagram of the structure of a first neural network according to an embodiment of the present application. As shown in Figure 5A, the encoding network is used to encode a current video frame and perform quantization to obtain a first feature of the current video frame. The entropy encoding layer is used to compress the first feature of the current video frame based on a reference frame of the current video frame, and output the first compressed information corresponding to the current video frame (i.e., an example of the target compressed information corresponding to the current video frame). It should be understood that the example of Figure 5A is merely for ease of understanding the solution, and is not intended to limit the solution.

[0121] Specifically, the following describes a process in which an encoder uses a first neural network to generate first compressed information corresponding to a current video frame. The encoder uses a first encoding network to perform transform coding on the current video frame, and performs quantization after the transform coding is performed to obtain a first feature of the current video frame. That is, the first feature of the current video frame can be obtained based only on the current video frame, and the reference frame of the current video frame is not needed in the first feature generation process.

[0122] Furthermore, the first encoding network may be specifically represented as a multi-layer convolutional network. The first feature includes features of M pixels and may be specifically represented as an L-dimensional tensor, such as a 1-dimensional tensor (i.e., a vector), a 2-dimensional tensor (i.e., a matrix), a 3-dimensional tensor, or a higher dimensional tensor, which will not be enumerated here.

[0123] The encoder predicts a feature of the current video frame based on the N reference frames of the current video frame, generates a first predicted feature of the current video frame, and generates a probability distribution of the first feature of the current video frame based on the first predicted feature of the current video frame. The encoder performs entropy encoding on the first feature of the current video frame based on the probability distribution of the first feature of the current video frame to obtain first compressed information.

[0124] The first predicted feature of the current video frame is a prediction result of the first feature of the current video frame, where the first predicted feature of the current video frame also includes features of M pixels, and the first predicted feature of the current video frame may be specifically represented as a tensor. The data shape of the first predicted feature of the current video frame is the same as the data shape of the first feature of the current video frame, and the shape of the first predicted feature being the same as the shape of the first feature means that both the first predicted feature and the first feature are L-dimensional tensors, and the size of a first dimension among the L dimensions of the first predicted feature is the same as the size of a second dimension among the L dimensions of the first feature, where L is an integer equal to or greater than 1, the first dimension is any dimension among the L dimensions of the first predicted feature, and the second dimension is a dimension among the L dimensions of the first feature and is the same dimension as the first dimension.

[0125] The probability distribution of the first feature of the current video frame includes a mean of the first feature of the current video frame and a variance of the first feature of the current video frame. Furthermore, both the mean of the first feature and the variance of the first feature can be represented as an L-dimensional tensor, where the data shape of the mean of the first feature is the same as the data shape of the first feature and the shape of the variance of the first feature is the same as the data shape of the first feature. Thus, the mean of the first feature includes values ​​corresponding to each of the M pixels and the variance of the first feature includes values ​​corresponding to each of the M pixels.

[0126] For a specific implementation in which an encoder predicts features of a current video frame based on N reference frames of the current video frame to generate a first predicted feature of the current video frame, and for a specific implementation in which an encoder generates a probability distribution of a first feature of the current video frame based on the first predicted feature of the current video frame, see the description in Patent Application No. CN202011271217.8.

[0127] The difference is that in patent application No. CN202011271217.8, the features of the first video frame are predicted based on N second video frames, a first predicted feature of the first video frame is generated, and a probability distribution of the first feature of the first video frame is generated based on the first predicted feature of the first video frame. In the embodiment of the present application, the current video frame is predicted based on N reference frames of the current video frame, a first predicted feature of the current video frame is generated, and a probability distribution of the first feature of the current video frame is generated based on the first predicted feature of the current video frame. That is, in the embodiment of the present application, the "first video frame" in patent application No. CN202011271217.8 is replaced with the "current video frame", and the "second video frame" in patent application No. CN202011271217.8 is replaced with the "reference frame of the current video frame". For the specific implementation, please refer to the description of patent application No. CN202011271217.8. Details will not be described here.

[0128] In this implementation of the present application, the encoder generates a probability distribution of the first feature of the current video frame based on the first predicted feature corresponding to the current video frame, and then performs compression encoding on the first feature of the current video frame based on the probability distribution of the first feature of the current video frame to obtain the first compressed information of the current video frame. A high similarity between the first predicted feature and the first feature indicates a large compression ratio of the first feature and a small first compressed information finally obtained. The first predicted feature of the current video frame is obtained by predicting the feature of the current video frame based on N reference frames of the current video frame, thereby improving the similarity between the first predicted feature of the current video frame and the first feature of the current video frame. This reduces the size of the compressed first compressed information. That is, the quality of the reconstructed frame obtained by the decoder can be ensured and the amount of data transmitted between the encoder and the decoder can be reduced.

[0129] When the target compression information is obtained using the second neural network, the target compression information includes second compression information of the second feature of the current video frame, and a reference frame of the current video frame is used for the generation process of the second feature of the current video frame. The second neural network includes a convolutional network and an entropy coding layer, the convolutional network includes a plurality of convolutional layers and an activation ReLU layer, the convolutional network is used to perform the generation process of the second feature of the current video frame based on the reference frame of the current video frame, and the entropy coding layer is used to compress the second feature of the current video frame and output the second compression information corresponding to the current video frame.

[0130] In this embodiment of the present application, a specific network structure of the first neural network and the second neural network is provided, which improves the integration of the solution with a specific application scenario.

[0131] Specifically, after generating the optical flow of the original current video frame relative to the reference frame of the current video frame, the encoder can perform compression encoding on the optical flow to obtain a compressed optical flow. The second feature of the current video frame only includes the optical flow of the original current video frame relative to the reference frame of the current video frame.

[0132] Optionally, the encoder may alternatively generate a predicted current video frame based on a reference frame of the current video frame and an optical flow of the original current video frame relative to the reference frame of the current video frame, calculate a residual between the original current video frame and the predicted current video frame, perform compression encoding on the optical flow of the original current video frame relative to the reference frame of the current video frame and the residual between the original current video frame and the predicted current video frame, and output second compression information corresponding to the current video frame. The second feature of the current video frame may include the optical flow of the original current video frame relative to the reference frame of the current video frame and the residual between the original current video frame and the predicted current video frame.

[0133] In addition, after the encoder obtains the second feature of the current video frame, because the data amount of the second feature of the current video frame is small, the encoder can directly compress the second feature of the current video frame to obtain the second compressed information corresponding to the current video frame. The compression can be implemented using a neural network or a non-neural network method. For example, the compression encoding can be entropy encoding.

[0134] In order to more intuitively understand this solution, FIG. 5B is a schematic diagram of the structure of the second neural network according to an embodiment of the present application. As shown in FIG. 5B, the encoder inputs the current video frame and the reference frame of the current video frame into a convolutional network, and uses the convolutional network to perform optical flow estimation to obtain the optical flow of the current video frame relative to the reference frame of the current video frame. The encoder uses the convolutional network to generate a reconstructed frame of the current video frame based on the reference frame of the current video frame and the optical flow of the current video frame relative to the reference frame of the current video frame, and obtains a residual between the reconstructed frame of the current video and the current video frame. The encoder can compress the optical flow of the current video frame relative to the reference frame of the current video frame and the residual between the reconstructed frame of the current video and the current video frame using an entropy coding layer, and output the second compressed information of the current video frame. It should be understood that the example of FIG. 5B is merely for ease of understanding the solution, and does not limit the solution.

[0135] In order to more intuitively understand the difference between the first feature and the second feature, FIG. 5C is a schematic diagram of a comparison between the first feature and the second feature in a video frame compression method according to an embodiment of the present application. FIG. 5C includes two subgraphs, (a) and (b). (a) of FIG. 5C is a schematic diagram of generating a first feature of a current video frame, and (b) of FIG. 5C is a schematic diagram of generating a second feature of a current video frame. Please refer to (a) of FIG. 5A. The current video frame is input to a coding network, and transform coding and quantization (Q) are performed using the coding network to obtain the first feature of the current video frame. Quantization is performed after transform coding.

[0136] Please refer to FIG. 5C (b). The contents of the dashed box in FIG. 5C (b) show the second feature of the current video frame. FIG. 5C (b) shows in detail that the second feature of the current video frame includes both the optical flow of the original current video frame with respect to the reference frame of the current video frame, and the residual between the original current video frame and the predicted current video frame. Here, the generation process of the second feature of the current video frame is not described. Comparing FIG. 5C (a) and FIG. 5C (b), it can be seen that the reference frame of the current video frame is not necessary in the generation process of the first feature of the current video frame, but is necessary in the generation process of the second feature of the current video frame. It should be understood that the example of FIG. 5C is merely for making the concept of the first feature and the second feature easier to understand, and is not intended to limit the solution.

[0137] It should be noted that the encoder is configured with at least the first and second neural networks, although another neural network (called a "fifth neural network" for ease of explanation) used to perform compression encoding on the video frame can also be configured in the encoder. For detailed processing of performing compression encoding using the first and second neural networks, please refer to the description of the following embodiments. Details will not be described here. For example, the fifth neural network may be a neural network that directly compresses the current video frame. That is, the encoder may input the current video frame into the fifth neural network, and directly compress the current video frame using the fifth neural network to obtain the third compression information corresponding to the current video frame, which is output by the fifth neural network. Furthermore, the fifth neural network may specifically be a convolutional neural network.

[0138] 303: The encoder generates indication information corresponding to the target compressed information, where the indication information indicates that the target compressed information was obtained using a target neural network, which is either the first neural network or the second neural network.

[0139] In this embodiment of the present application, after obtaining the target compression information of one or more current video frames, the encoder can further generate at least one indication information corresponding one-to-one to the target compression information of the at least one current video frame. The at least one indication information indicates that each target compression information is obtained using a target neural network, which is either a first neural network or a second neural network. That is, the indication information indicates that the target compression information is obtained using either the first neural network or the second neural network.

[0140] The plurality of indication information corresponding to the target compression information of the plurality of video frames in the current video sequence may be specifically represented as a string or in another format. For example, the plurality of indication information corresponding to the target compression information of the plurality of video frames in the current video sequence may be specifically 0010110101. One character in the string indicates one indication information. When one indication information is 0, it indicates that the current video frame corresponding to the indication information is compressed using a first neural network. When one indication information is 1, it indicates that the current video frame corresponding to the indication information is compressed using a second neural network.

[0141] Specifically, in the implementation, each time the encoder obtains the target compression information of one current video frame, the encoder may generate one indication information corresponding to the target compression information of the current video frame, that is, the encoder may alternately perform step 303, step 301, and step 302.

[0142] In another implementation, the encoder may alternatively use step 301 to generate target compression information for a preset number of current video frames, and then generate a preset number of indication information corresponding to the preset number of current video frames, where the preset number is an integer greater than 1, for example, 3, 4, 5, 6, or another value, which is not limited herein.

[0143] In another implementation, the encoder may alternatively use steps 301 and 302 to generate multiple target compression information corresponding to the entire current video sequence, and then use step 303 to generate multiple indication information corresponding to the entire current video sequence. The specific implementation is not limited herein.

[0144] 304: The encoder sends target compression information for the current video frame.

[0145] In this embodiment of the present application, the encoder may transmit target compression information of at least one current video frame in a current video sequence to the decoder under the constraints of a file transfer protocol (FTP).

[0146] Specifically, in some implementations, the encoder can send at least one target compression information directly to the decoder, while in other implementations, the encoder can alternatively send at least one target compression information to an intermediate device, such as a server or management center, which sends the target compression information to the decoder.

[0147] Optionally, when using a first neural network to generate the target compressed information, refer to the description of patent application No. CN202011271217.8. When sending the first compressed information of the current video frame to the decoder, the encoder can further send one or two types of first inter-side information, second inter-side information, first intra-side information, and second intra-side information corresponding to the current video frame to the decoder in a manner of generating a first predicted feature of the current video frame. Correspondingly, the decoder can receive one or two types of first inter-side information, second inter-side information, first intra-side information, and second intra-side information corresponding to the current video frame. The specific type of information to be sent should be determined based on the type of information required during the decompression of the first compressed information of the current video frame.

[0148] In addition, for the meaning and function of the first interside information, the second interside information, the first intraside information, and the second intraside information, please refer to the description in patent application number CN2011271217.8, which will not be described in detail here.

[0149] 305: The encoder sends indication information corresponding to the target compression information of the current video frame.

[0150] In the present embodiment of the present application, step 305 is an optional step. If step 303 is not performed, step 305 is not performed. If step 303 is performed, step 305 is performed. If step 305 is performed, step 305 and step 304 may be performed simultaneously. That is, the encoder sends to the decoder, under the constraint of the FTP protocol (short for File Transfer Protocol), the target compression information of at least one current video frame in the current video sequence and at least one indication information that corresponds one-to-one to the target compression information of the at least one current video frame. Alternatively, step 304 and step 305 may be performed separately. In the present embodiment, the order of performing step 304 and step 305 is not limited.

[0151] Therefore, the decoder can obtain multiple indication information corresponding to multiple target compressed information, so that the decoder can know to use either the first neural network or the second neural network to decompress each video frame of the current video sequence, which helps the decoder to reduce the time for decoding the compressed information, that is, this helps to improve the overall efficiency of the video frame transmission performed by the encoder and the decoder.

[0152] In this implementation of the present application, when the compression information is obtained using the first neural network, the compression information carries compression information of the first feature of the current video frame, and the reference frame of the current video frame is only used for the compression process of the first feature of the current video frame, and is not used for the generation process of the first feature of the current video frame. Therefore, after the decoder performs decompression based on the first compression information, a reconstructed frame of the current video frame can be obtained without the reference frame of the current video frame to obtain the first feature of the current video frame. Therefore, when the compression information is obtained using the first neural network, the quality of the reconstructed frame of the current video frame does not depend on the quality of the reconstructed frame of the reference frame of the current video frame, so that the accumulation of errors for each frame can be prevented and the quality of the reconstructed frame of the video frame can be improved. In addition, since a second feature of the current video frame is generated based on the reference frame of the current video frame, and an amount of data corresponding to the second compressed information of the second feature is less than an amount of data corresponding to the first compressed information of the first feature, the encoder can use the first neural network and the second neural network to process different video frames of the current video sequence, thereby combining the advantages of the first neural network and the second neural network to minimize the amount of data that needs to be transmitted and improve the quality of the reconstructed frame of the video sequence.

[0153] 2. The encoder performs compression encoding individually using multiple neural networks and determines the target compression information.

[0154] In some embodiments of the present application, the encoder uses multiple neural networks to perform compression encoding on the current video frame separately, and then determines target compression information corresponding to the current video frame. To more intuitively understand this solution, FIG. 6 is a schematic diagram of another principle of the video frame compression method according to an embodiment of the present application. FIG. 6 is explained using an example in which the multiple neural networks only include a first neural network and a second neural network. The encoder uses the first neural network to perform compression encoding on the current video frame, and determines the first compression information of the first feature of the current video frame (i.e., r in FIG. 6 ). p ) and obtain a reconstructed frame of the current video frame based on the first compression information (i.e., d in FIG. 6 ). p 6 ). The encoder performs compression encoding on the current video frame using the second neural network to generate second compressed information of the second feature of the current video frame (i.e., r r ) and obtain a reconstructed frame of the current video frame based on the second compressed information (i.e., d in FIG. 6 ). r ) The encoder generates p , d p , r r , d r , and determine target compressed information corresponding to the current video frame from the first compressed information and the second compressed information based on the network selection policy. It should be understood that the example in FIG. 6 is merely for ease of understanding the solution, and is not intended to limit the solution.

[0155] Specifically, Figure 7A is another schematic flowchart of a video frame compression method according to an embodiment of the present application. The video frame compression method according to an embodiment of the present application may include the following steps:

[0156] 701: An encoder performs compression encoding on a current video frame using a first neural network to obtain first compression information of a first feature of the current video frame, where a reference frame of the current video frame is used in the compression process of the first feature of the current video frame.

[0157] In this embodiment of the present application, after obtaining the current video frame, the encoder uses a first neural network of the multiple neural networks to perform compression encoding on the current video frame to obtain first compression information of the first feature of the current video frame. For the meaning of the first feature of the current video frame, the meaning of the first compression information of the first feature of the current video frame, and the specific implementation of step 701, please refer to the description of the embodiment corresponding to Figure 3. The details will not be described again here.

[0158] 702: The encoder generates a first video frame using a first neural network, where the first video frame is a reconstructed frame of a current video frame.

[0159] In some embodiments of the present application, after generating the first compressed information of the first feature of the current video frame using the first neural network, the encoder further performs decompression using the first neural network to generate a first video frame, where the first video frame is a reconstructed frame of the current video frame.

[0160] The first compressed information includes compressed information of a first feature of the current video frame, a reference frame of the current video frame is used in a decompression process of the first compressed information to obtain the first feature of the current video frame, and the first feature of the current video frame is used in a generation process of a reconstructed frame of the current video frame, that is, after decompressing the first compressed information, the encoder can obtain a reconstructed frame of the current video frame without a reference frame of the current video frame.

[0161] The first neural network further includes an entropy decoding layer and a decoding network, where the entropy decoding layer is used to decompress the first compressed information of the current video frame based on a reference frame of the current video frame, and the decoding network is used to generate a reconstructed frame of the current video frame based on a first feature of the current video frame.

[0162] Specifically, the encoder can predict features of the current video frame based on the reconstructed frames of the N reference frames of the current video frame by using an entropy decoding layer to obtain a first predicted feature of the current video frame, and can generate a probability distribution of the first feature of the current video frame based on the first predicted feature of the current video frame by using the entropy decoding layer. The encoder performs entropy decoding on the first compressed information of the current video frame based on the probability distribution of the first feature of the current video frame, using the entropy decoding layer, to obtain the first feature of the current video frame. The encoder further performs inverse transform decoding on the first feature of the current video frame using a first decoding network, to obtain a reconstructed frame of the current video frame. The first decoding network corresponds to a first encoding network, and the first decoding network can also be represented as a multi-layer convolutional network.

[0163] More specifically, a specific implementation in which the encoder generates a first predicted feature of the current video frame based on a reconstructed frame of the N reference frames of the current video frame is similar to a specific implementation in which the encoder generates a first predicted feature of the current video frame based on a reconstructed frame of the N reference frames of the current video frame. A specific implementation in which the encoder generates a probability distribution of the first feature of the current video frame based on the first predicted feature of the current video frame is similar to a specific implementation in which the encoder generates a probability distribution of the first feature of the current video frame based on the first predicted feature of the current video frame. For specific implementations of the above steps, please refer to the description of step 302 in the embodiment corresponding to FIG. 3. Details will not be described again here.

[0164] 703: The encoder performs compression encoding on the current video frame using a second neural network to obtain second compression information of a second feature of the current video frame, where a reference frame of the current video frame is used in the generation process of the second feature of the current video frame.

[0165] In this embodiment of the present application, after obtaining the current video frame, the encoder performs compression encoding on the current video frame using a second neural network of the multiple neural networks to obtain second compression information of the second feature of the current video frame. For the meaning of the second feature of the current video frame, the meaning of the second compression information of the second feature of the current video frame, and the specific implementation of step 701, please refer to the description of the embodiment corresponding to FIG. 3. The details will not be described again here.

[0166] 704: The encoder generates a second video frame using a second neural network, where the second video frame is a reconstructed frame of the current video frame.

[0167] In some embodiments of the present application, after generating the second compressed information of the second feature of the current video frame using the second neural network, the encoder further performs decompression using the second neural network to generate a second video frame, where the second video frame is a reconstructed frame of the current video frame.

[0168] The second neural network may further include an entropy decoding layer and a convolutional network, where the entropy decoding layer is used to perform entropy decoding on the second compressed information, and the convolutional network is used to perform a process of generating a reconstructed frame of the current video frame based on a reference frame of the current video frame and a second feature of the current video frame.

[0169] Specifically, the encoder may use an entropy decoding layer to perform entropy decoding on the second compressed information to obtain a second feature of the current video frame, i.e., an optical flow of the original current video frame relative to the reference frame of the current video frame. Optionally, the second feature of the current video frame further includes a residual between the original current video frame and the predicted current video frame.

[0170] The encoder predicts the current video frame based on a reference frame of the current video frame and an optical flow of the original current video frame relative to the reference frame of the current video frame to obtain a predicted current video frame, and further generates a second video frame (i.e., a reconstructed frame of the current video frame) based on the predicted current video frame and a residual between the original current video frame and the predicted current video frame.

[0171] 705: The encoder determines target compression information corresponding to the current video frame based on the first compression information, the first video frame, the second compression information, and the second video frame, where the determined target compression information is obtained using a first neural network and the determined target compression information is the first compression information, or the determined target compression information is obtained using a second neural network and the determined target compression information is the second compression information.

[0172] In this embodiment of the present application, the encoder can calculate a first score corresponding to the first compression information (i.e., a first score corresponding to the first neural network) based on the first compression information and the first video frame, calculate a second score corresponding to the second compression information (i.e., a second score corresponding to the second neural network) based on the second compression information and the second video frame, and determine target compression information corresponding to the current video frame based on the first score and the second score. If the determined target compression information is the first compression information obtained using the first neural network, the target neural network is the first neural network. Alternatively, if the determined target compression information is the second compression information obtained using the second neural network, the target neural network is the second neural network.

[0173] The first score indicates a performance for compressing the current video frame using the first neural network, and the second score indicates a performance for compressing the current video frame using the second neural network. Furthermore, a smaller first score indicates a better performance for processing the current video frame using the first neural network, and a larger first score indicates a worse performance for processing the current video frame using the first neural network. A smaller second score indicates a better performance for processing the current video frame using the second neural network, and a larger second score indicates a worse performance for processing the current video frame using the second neural network.

[0174] The calculation process of the first score and the second score will be described below. Specifically, after obtaining the first compression information and the first video frame, the encoder can obtain the data amount of the first compression information, calculate the first compression ratio of the first compression information for the current video frame, calculate the image quality of the first video frame, and generate the first score based on the first compression ratio of the first compression information for the current video frame and the image quality of the first video frame. The larger the data amount of the first compression information, the larger the first score, and the smaller the data amount of the first compression information, the smaller the first score. The lower the image quality of the first video frame, the larger the first score, and the higher the image quality of the first video frame, the smaller the first score.

[0175] Moreover, the first compression ratio of the first compressed information for the current video frame may be a ratio between the data amount of the first compressed information and the data amount of the current video frame.

[0176] The encoder may calculate a structural similarity index (SSIM) between the current video frame and the first video frame, where the "structural similarity index" indicates the image quality of the first video frame. Note that the encoder may use another index to further measure the image quality of the first video frame. For example, the "structural similarity index" may be replaced with a multi-scale structural similarity index (MS-SSIM), a peak signal-to-noise ratio (PSNR), another index, etc. The indexes are not enumerated here.

[0177] After obtaining the first compression ratio of the first compression information for the current video frame and the image quality of the first video frame, the encoder may perform a weighted sum on the first compression ratio and the image quality of the first video frame to generate a first score corresponding to the first neural network. It should be noted that, after obtaining the first compression ratio and the image quality of the first video frame, the encoder may alternatively obtain the first score in another manner, such as multiplying the first compression ratio by the image quality of the first video frame. A specific manner of obtaining the first score based on the first compression ratio and the image quality of the first video frame may be flexibly determined based on an actual application scenario, which will not be enumerated here.

[0178] Correspondingly, after obtaining the second compression information and the second video frame, the encoder may calculate the data amount of the second compression information and the image quality of the second video frame, and then generate a second score according to the data amount of the second compression information and the image quality of the second video frame. The method of generating the second score is similar to the method of generating the first score. For details, please refer to the above description. The details will not be described again here.

[0179] Here, a process of determining target compression information corresponding to a current video frame based on the first score and the second score will be described. Specifically, in implementation, the encoder may obtain a calculated first score corresponding to the first compression information and a calculated second score corresponding to the second compression information, and then select the smaller score of the first score and the second score as the target score, and determine the compression information corresponding to the target score as the target compression information. The encoder performs the above operation on each video frame in the video sequence to obtain target compression information corresponding to each video frame.

[0180] For the knowledge that a person skilled in the art has obtained during research, please refer to FIG. 7B. In another implementation, FIG. 7B is a schematic diagram of the first score and the second score in the video frame compression method according to an embodiment of the present application. In FIG. 7B, the horizontal coordinate indicates the position information of one video frame in the current video sequence, the vertical coordinate indicates the score corresponding to each video frame, A1 indicates the dashed line corresponding to the first score in the process of compressing multiple video frames in the current video sequence, and A2 indicates the dashed line corresponding to the second score in the process of compressing multiple video frames in the current video sequence. A3 indicates the first score and the second score obtained by compressing video frame 1 separately using the first neural network and the second neural network. From FIG. 7B, it can be seen that the score obtained by processing video frame 1 using the first neural network is smaller. Therefore, the encoder processes video frame 1 using the first neural network. After video frame 1 is processed using the first neural network, both the first score and the second score corresponding to video frame 2 (i.e., the next video frame of video frame 1 in the current video sequence) are greatly reduced. That is, each time a video frame is compressed using the first neural network, the start of a new period is triggered. In a period, the value of the first score increases linearly, the value of the second score also increases linearly, and the growth rate of the second score is greater than the growth rate of the first score. It should be understood that the example of FIG. 7B is merely for ease of understanding the solution, and is not intended to limit the solution.

[0181] To get a more intuitive understanding of the solution, we can plug the first scores for a period into the following formula:

number

[0182] L pidenotes the starting point of the straight line corresponding to the first scores in one period, i.e., the offset of the first fitting equation corresponding to the first scores, and k pi denotes a slope of the line corresponding to the first scores for a period, i.e., a coefficient of the first fitting equation corresponding to the first scores, and t denotes the number of video frames between the first video frame for a period and any current video frame for that period. For example, the value of t corresponding to the second video frame for a period is 1.

[0183] In one period, multiple secondary scores can be applied to the following formula:

number

[0184] L pi represents the starting point of the line corresponding to the second scores in one period, i.e., the offset of the second fitting equation corresponding to the second scores, and k pi indicates the slope of the line corresponding to the multiple second scores for one period, i.e., the coefficient of the second fitting equation corresponding to the multiple second scores. For the meaning of t, see the explanation of equation (1).

[0185] The overall score for a period can be calculated using the following formula:

number

[0186] loss denotes the sum of all scores in one period, and T denotes the total number of video frames in one period. Compressing one video frame using the first neural network triggers a new period to be entered, where the first (T-1) video frames of one period are compressed using the second neural network, and the last video frame is compressed using the first neural network. Thus, the following formula denotes the sum of at least the second scores corresponding to all video frames compressed using the second neural network in one period:

number

number

[0187] The encoder then uses one period as a computation unit and aims to minimize the average total score of each period. To understand the solution more intuitively, we use the following formula:

number

[0188] For the meaning of T and loss, see the explanation of equation (3). The details will not be repeated here. The following equation shows that we aim to minimize the average total score for one period:

number

[0189] Substituting equation (3) into equation (4), we obtain:

number

[0190] Those skilled in the art discovered during their research that:

number

number

[0191] Inference is performed according to a mathematical formula. Specifically, in the implementation, for any one of a plurality of periods corresponding to a current video sequence, the encoder first obtains two first scores corresponding to the 12th current video frame of a period, and obtains two second scores corresponding to the 12th current video frame of the period. For the method of obtaining the first score corresponding to the current video frame and the second score corresponding to the current video frame, please refer to the above description. The details will not be described again here.

[0192] The encoder calculates coefficients and offsets of a first fitting equation corresponding to a plurality of first scores in a period, i.e., l, based on two first scores corresponding to the 12th current video frame in the period. pi and k pi The encoder may generate a coefficient and offset of a second fitting equation corresponding to the plurality of second scores in a period, i.e., l based on the two second scores corresponding to the twelfth current video frame in a period. pr and k pr Generates a value of

[0193] The following describes the process in which the encoder determines the target compression information of the current video frame after obtaining the coefficients and offsets of the first fitting equation and the coefficients and offsets of the second fitting equation. In the implementation, when t is equal to 0, the encoder determines the second compression information corresponding to the first video frame of the period as the target compression information of the current video frame (i.e., the first video frame of the period), that is, determines the target neural network corresponding to the first video frame of the period as the second neural network, and continues the process when t is equal to 1.

[0194] When t is equal to 1, that is, after obtaining the two first scores corresponding to the 12th current video frame in the period and obtaining the two second scores corresponding to the 12th current video frame in the period, the encoder can calculate the value of T according to formula (5). When T<3, the encoder determines the first compression information corresponding to the second video frame in the period as the target compression information of the current video frame (i.e., the second video frame in the period), that is, determines the target neural network corresponding to the second video frame in the period as the first neural network. Triggered to enter the next period.

[0195] If T≧3, the encoder determines the second compressed information corresponding to the second video frame of the period as the target compressed information of the current video frame (i.e., the second video frame of the period), that is, determines the target neural network corresponding to the second video frame of the period as the second neural network, and continues the process when t is equal to 2.

[0196] When t is equal to 2, the encoder obtains a first score and a second score corresponding to the third video frame (i.e., an example of the current video frame) in one period. For the specific method of generating the first score and the second score corresponding to the current video frame, refer to the foregoing description. Details are not described again here. The encoder recalculates the coefficients and offsets of the first fitting formula based on the three first scores corresponding to the 13th video frame in one period (i.e., recalculates the values of l pi and k pi ), recalculates the coefficients and offsets of the second fitting formula based on the three second scores corresponding to the 13th video frame in the period (i.e., recalculates the values of l pr and k pr ), and recalculates the value of T based on the recalculated coefficients and recalculated offsets of the first fitting formula and the recalculated coefficients and recalculated offsets of the second fitting formula.

[0197] When T < t + 2, the encoder determines the first compression information corresponding to the third video frame in that period as the target compression information of the current video frame (i.e., the third video frame in that period), that is, determines the target neural network corresponding to the third video frame in that period as the first neural network. It is triggered to enter the next period.

[0198] When T ≥ t + 2, the encoder determines the second compression information corresponding to the third video frame in that period as the target compression information of the current video frame (i.e., the third video frame in that period), that is, determines the target neural network corresponding to the third video frame in that period as the second neural network, and continues the processing when t is equal to 3.

[0199] When the value of t is 3, 4, or more, the processing method of the encoder is the same as the processing method when t is equal to 2. Details are not described again here.

[0200] In another implementation, when t is equal to 0, the encoder determines the second compressed information corresponding to the first video frame of the period as the target compressed information of the current video frame (i.e., the first video frame of the period), i.e., determines the target neural network corresponding to the first video frame of the period as the second neural network, and continues the process when t is equal to 1.

[0201] When t is equal to 1, after obtaining two first scores corresponding to the 12th current video frame in a period, and obtaining two second scores corresponding to the 12th current video frame in a period, the encoder calculates the coefficients and offsets (i.e., l pi and k pi ), and the coefficients and offsets of the second fitting equation (i.e., l pr and k pr (value of ), calculate a first average value of the total score for the period obtained by compressing a second video frame in the period (i.e., an example of a current video frame) using the first neural network according to equation (5), and calculate a second average value of the total score for the period obtained by compressing a second video frame in the period (i.e., an example of a current video frame) using the second neural network and compressing a third video frame in the period using the first neural network.

[0202] If the first average value is greater than the second average value, the encoder determines the target compression information corresponding to the second video frame in the period as the first compression information of the current video frame, that is, determines the target compression information corresponding to the second video frame in the period as the first neural network is triggered to enter a new period.

[0203] If the first average value is equal to the second average value, the encoder determines the first compression information corresponding to the second video frame in the period as the target compression information of the current video frame, i.e., determines the target neural network corresponding to the second video frame in the period as the first neural network. A new period is triggered to enter. Alternatively, the encoder can determine the second compression information corresponding to the second video frame in the period as the target compression information of the current video frame, i.e., determines the target neural network corresponding to the second video frame in the period as the second neural network, and continue the process when t is equal to 2.

[0204] If the first average value is smaller than the second average value, the encoder determines the second compressed information corresponding to the second video frame of the period as the target compressed information of the current video frame, that is, determines the target neural network corresponding to the second video frame of the period as the second neural network, and can continue processing when t is equal to 2.

[0205] When t is equal to 2, the encoder can obtain a first score corresponding to the third video frame in the period, and obtain a second score corresponding to the twelfth current video frame in the period. The specific method of generating the first score and the second score corresponding to the current video frame is not described again here. The encoder recalculates the coefficients and offsets of the first fitting equation based on the three first scores corresponding to the thirteenth video frame in a period (i.e., l pi and k pi recalculate the value of l ), ​​and recalculate the coefficients and offsets of the second fitting equation based on the three second scores corresponding to the 13th video frame of the period (i.e., l pr and k prand calculating an updated first average value and an updated second average value based on the recalculated coefficients and the recalculated offsets of the first fitting equation and the recalculated coefficients and the recalculated offsets of the second fitting equation, where the updated first average value is an average value of the total scores for the period obtained by compressing a third video frame of the period (i.e., the current example video frame) using the first neural network, and the updated second average value is an average value of the total scores for the period obtained by compressing a third video frame of the period (i.e., the current example video frame) using the second neural network and compressing a fourth video frame of the period using the first neural network.

[0206] If the updated first average value is greater than the updated second average value, the encoder determines the target compression information corresponding to the third video frame in the period as the first compression information of the current video frame, that is, determines the target compression information corresponding to the third video frame in the period as the first neural network is triggered to enter a new period.

[0207] If the updated first average value is equal to the updated second average value, the encoder determines the first compression information corresponding to the third video frame in the period as the target compression information of the current video frame, i.e., determines the target neural network corresponding to the third video frame in the period as the first neural network. A new period is triggered to enter. Alternatively, the encoder can determine the second compression information corresponding to the third video frame in the period as the target compression information of the current video frame, i.e., determines the target neural network corresponding to the third video frame in the period as the second neural network, and continue the process when t is equal to 3.

[0208] If the updated first average value is less than the updated second average value, the encoder determines the second compression information corresponding to the third video frame of the period as the target compression information of the current video frame, that is, determines the target neural network corresponding to the third video frame of the period as the second neural network, and can continue processing when t is equal to 3.

[0209] If the value of t is 3, 4, or more, the encoder operates in a similar manner to the case where t is equal to 2. The details will not be described again here.

[0210] In the embodiment of the present application, a person skilled in the art will find the change rule of the first score and the second score in one period during research, and use the minimum average value of all scores in one period as the optimization objective. That is, when determining the target compression information corresponding to each current video frame, it is necessary to consider both the score of the current video frame and the average value of the score in the entire period to further reduce the scores corresponding to all video frames in the entire current video sequence. This will further improve the performance of the compression information corresponding to the entire current video sequence. In addition, two different implementations are provided to improve the flexibility of the implementation of the solution.

[0211] In another implementation, the encoder can alternatively use one period as a calculation unit, aiming to minimize the average value of the total score for each period. For a specific implementation when t is equal to 0 and a specific implementation when t is equal to 1, see the description of the first implementation in case B. The details will not be described again here.

[0212] When t = 2, the encoder does not obtain the first score and the second score corresponding to the third video frame within the period (i.e., an example of the current video frame), and does not recalculate the coefficients and offsets of the first fitting formula and the coefficients and offsets of the second fitting formula. However, when t = 1, the encoder directly obtains the value of T obtained by calculation. When T < t + 2, the encoder determines the first compression information corresponding to the third video frame of that period as the target compression information of the current video frame (i.e., the third video frame of that period), that is, determines the target neural network corresponding to the third video frame of that period as the first neural network, and is triggered to enter the next period.

[0213] When T ≥ t + 2, the encoder determines the second compression information corresponding to the third video frame of that period as the target compression information of the current video frame (i.e., the third video frame of that period), that is, determines the target neural network corresponding to the third video frame of that period as the second neural network, and continues the processing when t is equal to 3.

[0214] When the value of t is 3, 4, or more, the processing method of the encoder is the same as the processing method when t is equal to 2. Details are not described here again.

[0215] In order to understand the solution more intuitively, FIG. 7C is a schematic diagram of the calculation of the coefficients and offsets of the first fitting equation and the coefficients and offsets of the second fitting equation in the video frame compression method according to an embodiment of the present application. As shown in FIG. 7C, the video frames of one period are processed between two vertical dashed lines. During the period, the second neural network is used to perform compression encoding on multiple video frames, and the first neural network is used to perform compression encoding on the last video frame in the period. First, the encoder obtains two first scores corresponding to the 12th current video frame (i.e., the first video frame and the second video frame) in one period, obtains two second scores corresponding to the 12th current video frame in the period, and calculates the coefficients and offsets of the first fitting equation (i.e., l pi and k pi ), and the coefficients and offsets of the second fitting equation (i.e., l pr and k pr (value of T) and calculate the optimal value of T in the period according to equation (5). When t=2, the encoder does not obtain the first score and the second score corresponding to the third video frame in the period, and does not recalculate the coefficients and offsets of the first fitting equation and the coefficients and offsets of the second fitting equation. It should be understood that the example of FIG. 7C is merely for ease of understanding the solution, and is not intended to limit the solution.

[0216] In this embodiment of the present application, in one period, the coefficients and offsets of the first fitting formula and the coefficients and offsets of the second fitting formula are obtained by calculation based only on the two first scores and the two second scores corresponding to the twelfth video frame in the period. Then, the minimum average value of the total scores in the period is used as the optimization objective to obtain the optimal number of video frames in the current period. The minimum average value of the total scores in the period continues to be used as the optimization objective, so that the scores corresponding to all video frames in the current video sequence can be further reduced. In addition, when t is 2 or more, the coefficients and offsets of the first fitting formula and the coefficients and offsets of the second fitting formula are not updated. This saves the time of calculating the parameters of the first fitting formula and the second fitting formula, and further improves the efficiency of generating compressed information of the current video sequence.

[0217] In another implementation, the encoder can alternatively use one period as a calculation unit, aiming to minimize the average value of the total score for each period. For a specific implementation when t is equal to 0 and a specific implementation when t is equal to 1, see the description of the first implementation in case B. The details will not be described again here.

[0218] When t = 2, the encoder only obtains the second score corresponding to the third video frame within the period (i.e., an example of the current video frame), and does not obtain the first score corresponding to the third video frame within the period (i.e., an example of the current video frame). Also, the encoder only recalculates the coefficients and offset of the second fitting formula, and does not recalculate the coefficients and offset of the first fitting formula. The encoder calculates the value of T at t = 2 according to the unupdated first fitting formula and the updated second fitting formula. When T < t + 2, the encoder determines the first compression information corresponding to the third video frame of that period as the target compression information of the current video frame (i.e., the third video frame of that period), that is, determines the target neural network corresponding to the third video frame of that period as the first neural network, and is then triggered to enter the next period.

[0219] When T ≥ t + 2, the encoder determines the second compression information corresponding to the third video frame of that period as the target compression information of the current video frame (i.e., the third video frame of that period), that is, determines the target neural network corresponding to the third video frame of that period as the second neural network, and continues with the processing when t is equal to 3.

[0220] When the value of t is 3, 4, or more, the processing method of the encoder is the same as the processing method when t is equal to 2. Details are not described again here.

[0221] In this implementation, the compressed information that needs to be finally transmitted is selected based on at least one first compressed information of a current video frame, the first video frame, the second compressed information of the current video frame, and the second video frame. In this implementation, compared with a method of determining a target neural network from the first neural network and the second neural network according to a preset network selection policy, and then generating target compressed information using the target neural network, the performance of the compressed information corresponding to the entire current video sequence can be improved as much as possible.

[0222] 706: The encoder generates indication information corresponding to the target compression information, where the indication information indicates that the target compression information was obtained using a target neural network, which is either the first neural network or the second neural network.

[0223] 707: The encoder sends target compression information for the current video frame.

[0224] 708: The encoder sends indication information corresponding to the target compression information for the current video frame.

[0225] In this embodiment of the present application, step 706 and step 708 are essential steps. For specific implementation of step 706 to step 708, please refer to the description of step 303 to step 305 in the embodiment corresponding to FIG. 3. The details will not be described again here. It should be noted that in the embodiment of the present application, the execution order of step 707 and step 708 is not limited. Step 707 and step 708 may be executed simultaneously, step 707 may be executed before step 708, and step 708 may be executed before step 707.

[0226] In this embodiment of the present application, the compressed information that needs to be finally transmitted is selected from the first compressed information and the second compressed information according to at least one first compressed information of a current video frame, the first video frame, the second compressed information of the current video frame, and the second video frame. In this implementation, compared with the method of determining a target neural network from multiple neural networks according to a network selection policy, and then using the target neural network to generate target compressed information, the performance of the compressed information corresponding to the entire current video sequence can be improved as much as possible.

[0227] In an embodiment of the present application, Figure 8 is another schematic flowchart of a video frame compression method according to an embodiment of the present application. The video frame compression method according to an embodiment of the present application may include the following steps:

[0228] 801: An encoder performs compression encoding on a third video frame using a first neural network, and obtains first compression information corresponding to the third video frame, where the first compression information includes compression information of a first feature of the third video frame, and a reference frame of the third video frame is used for the compression process of the first feature of the third video frame.

[0229] In this embodiment of the present application, when processing the third video frame of the current video frame, the encoder determines that the target compressed information of the third video frame is the first compressed information corresponding to the third video frame generated by the first neural network. The third video frame is a video frame of the current video sequence, and the concept of the third video frame is similar to that of the current video frame. For the meaning of the first feature of the third video frame, please refer to the description of the meaning of "first feature of the current video frame" in the embodiment corresponding to FIG. 3. For the meaning of "reference frame of the third video frame", the specific implementation in which the encoder generates the first compressed information corresponding to the third video frame, and the specific implementation in which the encoder determines the compressed information of the third video frame that needs to be finally sent to the decoder, please refer to the description of the embodiment corresponding to FIG. 3. Details will not be described again here.

[0230] 802: The encoder performs compression encoding on a fourth video frame using a second neural network to obtain second compressed information corresponding to the fourth video frame, the second compressed information including compressed information of a second feature of the fourth video frame, a reference frame of the fourth video frame is used in the generation process of the second feature of the fourth video frame, and the third video frame and the fourth video frame are different video frames of the same video sequence.

[0231] In this embodiment of the present application, when processing the fourth video frame of the current video frame, the encoder determines that the target compressed information of the fourth video frame is the second compressed information generated by the second neural network corresponding to the fourth video frame, the fourth video frame is a video frame of the current video sequence, the concept of the fourth video frame is similar to the concept of the current video frame, and the third video frame and the fourth video frame are different video frames of the same current video sequence.

[0232] For the meaning of the second feature of the fourth video frame, see the explanation of the meaning of "second feature of the current video frame" in the embodiment corresponding to Fig. 3. For the meaning of "reference frame of the fourth video frame", a specific implementation in which the encoder generates the second compressed information corresponding to the fourth video frame, and a specific implementation in which the encoder determines the compressed information of the fourth video frame that needs to be finally transmitted to the decoder, see the explanation of the embodiment corresponding to Fig. 3. The details will not be described again here.

[0233] It should be noted that in this embodiment of the present application, the specific implementation order of step 801 and step 802 is not limited. Step 801 may be performed before step 802, or step 802 may be performed before step 801. The specific order needs to be determined based on the actual application scenario, and is not limited here.

[0234] 803: The encoder generates indication information, the indication information indicating that the first compressed information is obtained using a first neural network and the second compressed information is obtained using a second neural network.

[0235] In this embodiment of the present application, step 803 is similar to step 303 in the embodiment corresponding to Fig. 3. After generating the target compression information of one or more current video frames in a current video sequence, the encoder may generate indication information that corresponds one-to-one to the one or more target compression information. The target compression information is specifically the first compression information or the second compression information. For the meaning of the target compression information and the indication information, please refer to the description of step 303 in the embodiment corresponding to Fig. 3. The details will not be described again here.

[0236] Specifically, the encoder first performs step 801 and step 802 multiple times, and then uses step 803 to generate indication information that corresponds one-to-one to the target compression information of each video frame of the current video sequence. Alternatively, the encoder can also perform step 803 every time step 801 is performed or step 802 is performed. Alternatively, the encoder can also perform step 803 once after step 801 and / or step 802 are performed a preset number of times. The preset number is an integer greater than 1, for example, 3, 4, 5, 6, or another value. This is not limited here.

[0237] It should be noted that, in step 801 or step 802, if the encoder determines the target compression information of the current video frame (i.e., the third video frame or the fourth video frame) in the manner shown in the embodiment corresponding to FIG. 7A, step 803 is a required step. However, in step 801 or step 802, if the encoder obtains the target compression information of the current video frame (i.e., the third video frame or the fourth video frame) in the manner shown in the embodiment corresponding to FIG. 3, step 803 is an optional step. For a specific implementation of step 803, please refer to the description of step 303 in the embodiment corresponding to FIG. 3. The details will not be described again here.

[0238] 804: The encoder sends target compressed information corresponding to the current video frame, where the target compressed information is the first compressed information or the second compressed information.

[0239] In this embodiment of the present application, after generating at least one first compressed information that corresponds one-to-one to at least one third video frame, and / or generating at least one second compressed information that corresponds one-to-one to at least one fourth video frame, the encoder can send at least one target compressed information (i.e., the first compressed information and / or the second compressed information) that corresponds one-to-one to at least one current video frame (i.e., the third video frame and / or the fourth video frame) to the decoder under the constraint of the FTP protocol. For the specific implementation of step 804, please refer to the description of step 304 in the embodiment corresponding to FIG. 3. The details will not be described again here.

[0240] In order to understand this solution more intuitively, Fig. 9 is a schematic diagram of a video frame compression method according to an embodiment of the present application. As shown in Fig. 9, the encoder uses a third neural network to perform compression encoding on some video frames of a current video sequence, and uses a fourth neural network to perform compression encoding on other video frames of the current video sequence, and then transmits target compression information corresponding to all current video frames of the current video sequence. The target compression information is the first compression information or the second compression information. It should be understood that the example in Fig. 9 is just for easy understanding of the solution, and is not intended to limit the solution.

[0241] 805: The encoder sends the indication information corresponding to the current video frame.

[0242] In this embodiment of the present application, step 805 is an optional step. If step 803 is not performed, step 805 is not performed. If step 803 is performed, step 805 is performed. If step 805 is performed, step 805 and step 804 may be performed simultaneously. For a specific implementation of step 805, please refer to the description of step 305 in the embodiment corresponding to FIG. 3. The details will not be described again here.

[0243] In this embodiment of the present application, when the first neural network is used to perform compression encoding on the third video frame of the current video sequence, the first compression information conveys compression information of the first feature of the current video frame, and the reference frame of the current video frame is only used for the compression process of the first feature of the current video frame, but is not used for the generation process of the first feature of the current video frame. Therefore, after the decoder performs decompression based on the first compression information, a reconstructed frame of the current video frame can be obtained without the reference frame of the current video frame to obtain the first feature of the current video frame. Therefore, when the first neural network is used to obtain the target compression information, the quality of the reconstructed frame of the current video frame does not depend on the quality of the reconstructed frame of the reference frame of the current video frame, so that the accumulation of errors from frame to frame can be prevented and the quality of the reconstructed frame of the video frame can be improved. When the second neural network is used to perform compression encoding on the fourth video frame, the second feature of the fourth video frame is generated based on the reference frame of the fourth video frame, so that the amount of data corresponding to the second compression information is less than the amount of data corresponding to the first compression information. Also, both the first neural network and the second neural network are used to process different video frames of the current video sequence, combining the advantages of the first neural network and the second neural network to minimize the amount of data that needs to be transmitted and improve the quality of the reconstructed frames of the video sequence.

[0244] Next, the steps performed by the decoder will be described in detail with reference to Figures 10A to 12. Figure 10A is a schematic flow chart of a video frame decompression method according to an embodiment of the present application. The video frame decompression method according to this embodiment of the present application may include the following steps:

[0245] 1001: A decoder receives target compression information corresponding to at least one current video frame.

[0246] In this embodiment of the present application, the encoder can transmit at least one target compression information corresponding to at least one current video frame of the current video frame to the decoder under the constraints of the FTP protocol. Correspondingly, the decoder can receive at least one target compression information corresponding to at least one current video frame of the current video frame.

[0247] Specifically, in an implementation, the decoder may receive the target compression information corresponding to the at least one current video frame directly from the encoder. In another implementation, the decoder may alternatively receive the target compression information corresponding to the at least one current video frame from an intermediate device, such as a server or management center.

[0248] 1002: A decoder receives indication information corresponding to target compressed information.

[0249] In some embodiments of the present application, the encoder sends at least one indication information that corresponds one-to-one to the at least one target compression information. Correspondingly, the decoder can receive at least one indication information that corresponds one-to-one to the at least one target compression information. For the meaning of the indication information, please refer to the description of the embodiment corresponding to FIG. 3. The details will not be described again here.

[0250] It should be noted that step 1002 is an optional step. When step 1002 is executed, the execution order of step 1001 and step 1002 is not limited in this embodiment of the present application, and step 1001 and step 1002 may be executed simultaneously.

[0251] 1003: The decoder selects a target neural network corresponding to the current video frame from a plurality of neural networks, where the plurality of neural networks includes a third neural network and a fourth neural network.

[0252] In this embodiment of the present application, after obtaining at least one target compression information corresponding to at least one current video frame, the decoder needs to select a target neural network from the multiple neural networks to perform decompression and obtain a reconstructed frame of each current video frame, where the multiple neural networks include a third neural network and a fourth neural network, and the third neural network and the fourth neural network are both neural networks used to perform decompression.

[0253] And the third neural network corresponds to the first neural network, that is, when the target compressed information of the current video frame is the first compressed information of the current video frame obtained by using the first neural network, the decoder needs to decompress the first compressed information of the current video frame by using the third neural network to obtain a reconstructed frame of the current video frame.

[0254] The fourth neural network corresponds to the second neural network, that is, if the target compressed information of the current video frame is the second compressed information of the current video frame obtained by using the second neural network, the decoder needs to decompress the second compressed information of the current video frame by using the fourth neural network to obtain a reconstructed frame of the current video frame.

[0255] Note that the specific implementation of the decoder using the third neural network or the fourth neural network to decompress the target compressed information will be described in the following embodiments, and will not be described in detail here.

[0256] The following describes the process that the decoder determines the target neural network. Specifically, in implementation, when performing step 1002, the decoder can directly determine that the target neural network corresponding to each target compressed information is the first neural network or the second neural network according to a plurality of indication information that corresponds one-to-one to a plurality of target compressed information.

[0257] In order to understand the solution more intuitively, Fig. 10B is another schematic flow chart of a video frame decompression method according to an embodiment of the present application. As shown in Fig. 10B, after the decoder obtains the target compression information corresponding to the current video frame and the instruction information corresponding to the target compression information, it can determine a target neural network from the third neural network and the fourth neural network according to the instruction information corresponding to the target compression information, and use the target neural network to decompress the target compression information corresponding to the current video frame, so as to obtain a reconstructed frame of the current video frame. It should be understood that the example of Fig. 10B is just for easy understanding of the solution, and is not intended to limit the solution.

[0258] In another implementation, if step 1002 is not performed, the decoder obtains position information of a current video frame in the current video sequence that corresponds one-to-one to each target compression information, and the position information indicates that the current video frame that corresponds one-to-one to each target compression information is the Xth frame in the current video sequence, and can select a target neural network corresponding to the position information of the current video sequence from the third neural network and the fourth neural network according to a preset rule.

[0259] For the meaning of the position information, please refer to the description of the embodiment corresponding to FIG. 3. The details will not be described again here. The pre-set rule may alternately select the third neural network or the fourth neural network according to a specific rule. That is, the decoder performs compression encoding on n video frames of the current video frame using the third neural network, and then performs compression encoding on m video frames of the current video frame using the fourth neural network. Alternatively, after performing compression encoding on m video frames of the current video frame using the fourth neural network, the encoder performs compression encoding on n video frames of the current video frame using the third neural network. The values ​​of both n and m may be integers equal to or greater than 1, and the values ​​of n and m may be the same or different.

[0260] A specific implementation in which the decoder selects a target neural network corresponding to the position information of the current video sequence from a plurality of neural networks including a third neural network and a fourth neural network according to a preset rule is similar to a specific implementation in which the encoder selects a target neural network corresponding to the position information of the current video sequence from a plurality of neural networks including a first neural network and a second neural network according to a network selection policy. The difference is that the "first neural network" in the embodiment corresponding to Fig. 3 is replaced by the "third neural network" in this embodiment, and the "second neural network" in the embodiment corresponding to Fig. 3 is replaced by the "fourth neural network" in this embodiment. For details, please refer to the description in the embodiment corresponding to Fig. 3. The details will not be described again here.

[0261] 1004: The decoder performs decompression using the target neural network based on the target compression information to obtain a reconstructed frame of the current video frame. When the target neural network is a third neural network, the target compression information includes first compression information of a first feature of the current video frame. A reference frame of the current video frame is used in the decompression process of the first compression information to obtain the first feature of the current video frame. The first feature of the current video frame is used in the generation process of the reconstructed frame of the current video frame. When the target neural network is a fourth neural network, the target compression information includes second compression information of a second feature of the current video frame, and the second compression information is used by the decoder to perform decompression to obtain a second feature of the current video frame. The reference frame of the current video frame and the second feature of the current video frame are used in the generation process of the reconstructed frame of the current video frame.

[0262] In this embodiment of the present application, when the target neural network is a third neural network, the target compressed information includes first compressed information of a first feature of a current video frame, and the third neural network includes an entropy decoding layer and a decoding network, where the entropy decoding layer is used to decompress the first compressed information of the current video frame based on a reference frame of the current video frame, and the decoding network is used to generate a reconstructed frame of the current video frame based on the first feature of the current video frame.

[0263] Specifically, when the target neural network is a third neural network, for a specific implementation in which the decoder performs step 1004, see the description of step 702 in the embodiment corresponding to FIG. 7A. The difference is that in step 702, the encoder performs decompression using the first neural network based on the first compression information corresponding to the current video frame to obtain a reconstructed frame of the current video frame. However, in step 1004, the decoder performs decompression using the third neural network based on the first compression information corresponding to the current video frame to obtain a reconstructed frame of the current video frame.

[0264] When the target neural network is a fourth neural network, the target compressed information includes second compressed information of a second feature of the current video frame, and the fourth neural network includes an entropy decoding layer and a convolutional network, where the entropy decoding layer is used to perform entropy decoding on the second compressed information, and the convolutional network is used to perform a process of generating a reconstructed frame of the current video frame based on a reference frame of the current video frame and the second feature of the current video frame.

[0265] Specifically, when the target neural network is a fourth neural network, for a specific implementation in which the decoder performs step 1004, see the description of step 704 in the embodiment corresponding to FIG. 7A. The difference is that in step 704, the encoder performs decompression using the second neural network based on the second compression information corresponding to the current video frame to obtain a reconstructed frame of the current video frame. However, in step 1004, the decoder performs decompression using the fourth neural network based on the second compression information corresponding to the current video frame to obtain a reconstructed frame of the current video frame.

[0266] An embodiment of the present application further provides a video frame decompression method. Figure 11 is another schematic flowchart of a video frame decompression method according to an embodiment of the present application. The video frame decompression method according to this embodiment of the present application may include the following steps:

[0267] 1101: A decoder receives target compressed information corresponding to a current video frame, the target compressed information being the first compressed information or the second compressed information.

[0268] 1102: A decoder receives indication information corresponding to a current video frame, the indication information indicating that the first compressed information is obtained by decompression using a third neural network, and the second compressed information is obtained by decompression using a fourth neural network.

[0269] For specific implementation of step 1101 and step 1102 in this embodiment of the present application, please refer to the description of step 1001 and step 1002 in the embodiment corresponding to Fig. 10A, and the details will not be described again here.

[0270] 1103: The decoder decompresses the first compressed information of the third video frame using a third neural network to obtain a reconstructed frame of the third video frame.

[0271] In this embodiment of the present application, the decoder selects a third neural network from the plurality of neural networks, and decompresses the first compressed information of the third video frame. For the specific implementation process of "selecting a third neural network corresponding to the third video frame from the plurality of neural networks", please refer to the description of step 1003 in the embodiment corresponding to Figure 10A. The details will not be described again here.

[0272] , the third neural network includes an entropy decoding layer and a decoding network, the entropy decoding layer is used to perform entropy decoding on the first compressed information of the current video frame according to the reference frame of the current video frame, and the decoding network is used to generate a reconstructed frame of the current video frame according to the first feature of the current video frame. For a specific implementation of the decoder using the third neural network to decompress the first compressed information of the third video frame, please refer to the description of step 702 in the embodiment corresponding to FIG. 7A. The details will not be described again here.

[0273] The first compression information includes compression information of the first feature of the third video frame, the reference frame of the third video frame is used in the decompression process of the first compression process to obtain the first feature of the third video frame, and the first feature of the third video frame is used in the generation process of the reconstructed frame of the third video frame. The current video sequence includes both the reconstructed frame of the third video frame and the reference frame of the third video frame. That is, the decoder can obtain the reconstructed frame of the third video frame without the reference frame of the third video frame after decompressing the first compression information.

[0274] Furthermore, for the meaning of "the first feature of the third video frame", please refer to the explanation of the meaning of "the first feature of the current video frame". For the meaning of "the reference frame of the third video frame", please refer to the explanation of the meaning of "the reference frame of the current video frame". Details will not be explained again here. The reconstructed frame of the third video frame is a video frame corresponding to the third video frame, obtained by decompressing the first compressed information.

[0275] 1104: The decoder decompresses the second compressed information of the fourth video frame using a fourth neural network to obtain a reconstructed frame of the fourth video frame.

[0276] In this embodiment of the present application, the decoder selects a fourth neural network from the plurality of neural networks, and decompresses the first compressed information of the fourth video frame. For the specific implementation process of "selecting a fourth neural network corresponding to the fourth video frame from the plurality of neural networks", please refer to the description of step 1003 in the embodiment corresponding to Figure 10A. The details will not be described again here.

[0277] The fourth neural network further includes an entropy decoding layer and a convolutional network, the entropy decoding layer is used to perform entropy decoding on the second compressed information, and the convolutional network is used to perform a process of generating a reconstructed frame of the current video frame according to a reference frame of the current video frame and a second feature of the current video frame. For a specific implementation of the decoder using the fourth neural network to decompress the second compressed information of the fourth video frame, please refer to the description of step 704 in the embodiment corresponding to Figure 7A. The details will not be described again here.

[0278] The second compressed information includes compressed information of the second feature of the fourth video frame, the second compressed information is used by a decoder to perform decompression to obtain the second feature of the fourth video frame, and the reference frame of the fourth video frame and the second feature of the fourth video frame are used in a process of generating a reconstructed frame of the fourth video frame. The current video sequence includes both the reconstructed frame of the fourth video frame and the reference frame of the fourth video frame.

[0279] Furthermore, for the meaning of "the second feature of the fourth video frame", please refer to the explanation of the meaning of "the second feature of the current video frame". For the meaning of "the reference frame of the fourth video frame", please refer to the explanation of the meaning of "the reference frame of the current video frame". Details will not be explained again here. The reconstructed frame of the fourth video frame is a video frame corresponding to the fourth video frame, which is obtained by decompressing the second compressed information.

[0280] 2. Training Phase

[0281] 12 is a schematic flowchart of a method for training a video frame compression and decompression system according to an embodiment. The method for training a video frame compression and decompression system according to this embodiment of the present application may include the following steps:

[0282] 1201: For a current video frame, the training device performs compression encoding on a first training video frame using a first neural network, and obtains first compression information corresponding to the first training frame.

[0283] In this embodiment of the present application, the training device pre-stores a training dataset, and the training dataset includes a number of first training video frames. For the specific implementation of step 1201, please refer to the description of step 801 in the embodiment corresponding to FIG. 8. The details will not be described again here. The difference is that the "third video frame" in step 801 is replaced by the "first training video frame" in this embodiment, and in step 1201, the training device does not need to select a target neural network from the first neural network and the second neural network. That is, in step 1201, the training device does not need to select target compressed information from the first compressed information and the second compressed information.

[0284] 1202: The training device decompresses the first compressed information of the first training video frame using a third neural network to obtain a first training reconstructed frame.

[0285] For the specific implementation of the training device performing step 1202 in this embodiment of the present application, please refer to the description of step 1103 in the embodiment corresponding to Fig. 11. The details will not be described again here. The difference is that the "third video frame" in step 1103 is replaced with the "first training video frame" in this embodiment, and in step 1202, the training device does not need to select a target neural network from the third neural network and the fourth neural network.

[0286] 1203: The training device trains the first neural network and the third neural network based on the first training video frame, the first training reconstructed frame, the first compressed information, and the first loss function until a preset condition is met.

[0287] In this embodiment of the present application, the training device can perform iterative training on the first neural network and the third neural network based on the first training video frame, the first training reconstructed frame, the first compressed information corresponding to the first training video frame, and the first loss function until a convergence condition of the first loss function is satisfied.

[0288] The first loss function includes a loss term of similarity between the first training video frame and the first training reconstructed frame and a loss term of data size of the first compressed information of the first training video frame, the first training reconstructed frame being a reconstructed frame of the first training video frame. A training objective of the first loss function includes increasing the similarity between the first training video frame and the first training reconstructed frame. The training objective of the first loss function further includes reducing a size of the first compressed information of the first training video frame. The first neural network is a neural network used to perform compression encoding on the video frames. The first neural network is a neural network used to perform decompression based on the compression information.

[0289] Specifically, the training device can calculate a function value of a first loss function according to the first training video frame, the first training reconstructed frame, and the first compressed information corresponding to the first training video frame, generate a gradient value according to the function value of the first loss function, and perform inverse updating on the weight parameters of the first neural network and the third neural network to complete one training of the first neural network and the third neural network. The training device repeatedly executes steps 1201 to 1203 to implement iterative training of the first neural network and the third neural network.

[0290] 1204: The training device uses a second neural network to perform compression encoding on the second training video frame based on a reference frame of the second training video frame, and obtains second compression information corresponding to the second training video frame, where the reference frame of the second training video frame is a video frame processed by the trained first neural network.

[0291] For the specific implementation of the training device performing step 1202 in this embodiment of the present application, please refer to the description of step 802 in the embodiment corresponding to FIG. 8. The details will not be described again here. The difference is that the "fourth video frame" in step 802 is replaced with the "second training video frame" in this embodiment, and in step 1204, the training device does not need to select the target neural network from the first neural network and the second neural network. That is, in step 1204, the training device does not need to select the target compressed information from the first compressed information and the second compressed information.

[0292] The reference frame for the second training video frame may be an original video frame in the training dataset or may be a video frame processed by the mature first neural network (i.e., the trained first neural network).

[0293] Specifically, in the implementation, the first neural network includes a first encoding network, and the third neural network includes a first decoding network, so that the training device can input the original reference frame of the second training video frame into the first encoding network in the mature first neural network (i.e., the trained first neural network) to encode the second training video frame and obtain an encoding result, and input the encoding result into the first decoding network in the mature third neural network (i.e., the trained third neural network) to decode the encoding result and obtain a processed reference frame of the second training video frame. Furthermore, the training device inputs the second training video frame and the processed reference frame of the second training video frame into the second neural network, and uses the second neural network to generate second compressed information corresponding to the second training video frame.

[0294] In another implementation, the training device can input the original reference frame of the second training video frame to a mature first neural network, use the mature first neural network to generate first compressed information corresponding to the original reference frame of the second training video frame, and use the mature third neural network to perform decompression based on the first compressed information corresponding to the original reference frame of the second training video frame to obtain a processed reference frame of the second training video frame. Further, the training device can input the second training video frame and the processed reference frame of the second training video frame to a second neural network, and use the second neural network to generate second compressed information corresponding to the second training video frame.

[0295] In this implementation of the present application, in the execution phase, the reference frame used by the second neural network may be processed by the first neural network, so that the reference frame processed by the first neural network is used to train the second neural network, which helps to maintain consistency between the training phase and the execution phase and improve the accuracy of the execution phase.

[0296] 1205: The training device decompresses the second compressed information of the second training video frame using a fourth neural network to obtain a second training reconstructed frame.

[0297] For a specific implementation of the training device performing step 1202 in this embodiment of the present application, please refer to the description of step 1104 in the embodiment corresponding to Fig. 11. The details will not be described again here. The difference is that in this embodiment, the "fourth video frame" in step 1104 is replaced with the "second training video frame", and in step 1205, the training device does not need to select a target neural network from the third neural network and the fourth neural network.

[0298] 1206: The training device trains the second neural network and the fourth neural network based on the second training video frame, the second training reconstructed frame, the second compressed information, and the second loss function until a preset condition is met.

[0299] In this embodiment of the present application, the training device can perform iterative training on the second neural network and the fourth neural network based on the second training video frame, the second training reconstructed frame, the second compressed information corresponding to the second training video frame, and the second loss function until a convergence condition of the second loss function is satisfied.

[0300] The second loss function includes a loss term of similarity between the second training video frame and the second training reconstructed frame and a loss term of data size of the second compressed information of the second training video frame, the second training reconstructed frame being a reconstructed frame of the second training video frame. A training objective of the second loss function includes increasing the similarity between the second training video frame and the second training reconstructed frame. The training objective of the second loss function further includes reducing the size of the second compressed information of the second training video frame. The second neural network is a neural network used to perform compression encoding on the video frames. The fourth neural network is a neural network used to perform decompression based on the compression information.

[0301] Specifically, the training device can calculate a function value of a second loss function according to the second training video frame, the second training reconstructed frame, and the second compressed information corresponding to the second training video frame, generate a gradient value according to the function value of the second loss function, and perform an inverse update on the weight parameters of the second neural network and the fourth neural network to complete one training of the second neural network and the fourth neural network. The training device repeatedly performs steps 1204 to 1206 to implement iterative training of the second neural network and the fourth neural network.

[0302] Since both the first neural network and the third neural network include a plurality of independent neural network modules, correspondingly, the second neural network and the fourth neural network also include a plurality of independent neural network modules. An independent neural network module is a neural network module having an independent function. For example, the first encoding network in the first neural network is an independent neural network module. As another example, the first decoding network in the second neural network is an independent neural network module.

[0303] Optionally, when the second neural network and the fourth neural network have the same neural network modules as the first neural network and the third neural network, the parameters of the second neural network and the fourth neural network may be initially initialized based on the trained first neural network and the trained third neural network, that is, the parameters of the trained first neural network and the trained third neural network are assigned to the same neural network modules, and the parameters of the same neural network modules do not change in the training process of the second neural network and the fourth neural network. In order to shorten the total time of the training process of the second neural network and the fourth neural network and improve the training efficiency of the second neural network and the fourth neural network, the parameters of the remaining neural network modules of the second neural network and the fourth neural network are adjusted.

[0304] The embodiment of the present application provides both a neural network execution process and a neural network training process, which expands the application scenarios of the solution and improves the comprehensiveness of the solution.

[0305] In order to more intuitively understand the beneficial effects brought by the embodiments of the present application, the beneficial effects brought by the embodiments of the present application are further described below with reference to the accompanying drawings. In this experiment, for example, every time a video frame is compressed using a first neural network, a video frame is compressed using a second neural network. Table 1 shows the experimental data. [Table 1] Table 1

[0306] See Table 1. For three groups of video sequences with different resolutions, it can be learned that the video frames of the video sequences compressed using the solution of the embodiment of the present application have higher image quality than the video frames of the video sequences compressed using only the second neural network.

[0307] In this experiment, for example, the offset and slope of the first fitting equation are generated, the offset and slope of the second fitting equation are generated, and the offset and slope of the first fitting equation and the offset and slope of the second fitting equation are continuously updated. Table 2 shows the experimental data. [Table 2] Table 2

[0308] See Table 2. For two groups of video sequences with different resolutions, it can be learned that the video frames of the video sequences compressed using the solution of the embodiment of the present application have higher image quality than the video frames of the video sequences compressed using only the second neural network.

[0309] An embodiment of the present application further provides a video encoding / decoding system. Figure 13 is a diagram of a system architecture of a video encoding / decoding system according to an embodiment of the present application. Figure 13 is a schematic block diagram of an example of a video encoding / decoding system 10. A video encoder 20 (or, for short, encoder 20) and a video decoder 30 (or, for short, decoder 30) of the video encoding / decoding system 10 represent devices that can be used to perform various techniques according to various examples described herein.

[0310] 13, video encoding / decoding system 10 includes a source device 12. Source device 12 is configured to provide encoded picture data 21, such as encoded pictures, to a destination device 14 that is configured to decode the encoded picture data 21.

[0311] The source device 12 includes an encoder 20 , and may optionally include a picture source 16 , a pre-processor (or pre-processing unit) 18 , such as a picture pre-processor, and a communication interface (or communication unit) 22 .

[0312] Picture source 16 may include any type of picture capture device configured to capture pictures in the real world and / or any type of picture generation device, e.g., a computer graphics processor configured to generate computer animated pictures, or any type of device configured to obtain and / or provide real-world pictures, computer-generated pictures (e.g., on-screen content, virtual reality (VR) pictures, and / or combinations thereof (e.g., augmented reality (AR) pictures). Picture source may be any type of memory or storage device that stores any of the aforementioned pictures.

[0313] To distinguish the processing performed by the pre-processor (or pre-processing unit) 18 , the picture (or picture data 17 ) is sometimes referred to as the original picture (or original picture data) 17 .

[0314] The pre-processor 18 is configured to receive the (original) picture data 17 and pre-process the picture data 17 to obtain a pre-processed picture (or pre-processed picture data) 19. For example, the pre-processing performed by the pre-processor 18 may include cropping, color format conversion (e.g., RGB to YCbCr conversion), color correction, or noise removal. It can be understood that the pre-processing unit 18 may be an optical component.

[0315] A video encoder (or encoder) 20 is configured to receive pre-processed picture data 19 and to provide encoded picture data 21 .

[0316] Communications interface 22 of source device 12 may be configured to receive encoded picture data 21 and transmit the encoded picture data 21 (or other processed version) via communications channel 13 to another device, such as destination device 14 or other device, for storage or direct reconstruction.

[0317] The destination device 14 includes a decoder 30, and may optionally include a communications interface (or communications unit) 28, a post-processor (or post-processing unit) 32, and a display device .

[0318] Communications interface 28 of destination device 14 is configured to receive encoded picture data 21 (or other processed version) from, for example, source device 12 or from any other device, such as a storage device. For example, a storage device is a device that stores encoded picture data and provides encoded picture data 21 to a decoder 30.

[0319] Communications interface 22 and communications interface 28 may be configured to transmit or receive encoded picture data (or encoded data) 21 over a direct communications link between source device 12 and destination device 14, such as a wired or wireless connection, or over a wired network, a wireless network, or any combination of wired and wireless networks, any type of private network, any type of public network, or any combination of private and public networks.

[0320] For example, the communications interface 22 may be configured to process the encoded picture data 21 using any type of transmission encoding or processing, for example to encapsulate the encoded picture data 21 in a suitable format, e.g., into packets, and / or for transmission over a communications link or network.

[0321] The communications interface 28 corresponds to the communications interface 22 and is configured, for example, to receive transmitted data and process the transmitted data through any type of corresponding transmission decoding or processing and / or decapsulation to obtain the encoded picture data 21.

[0322] Both communication interface 22 and communication interface 28 may be configured as unidirectional communication interfaces, as indicated by the arrow of communication channel 13 pointing from source device 12 to destination device 14 in FIG. 13, or as bidirectional communication interfaces, and may be configured to send and receive messages, for example, to establish a connection, to acknowledge and exchange a communication link and / or any other information related to a data transmission, for example a coded picture data transmission.

[0323] The video decoder (or decoder) 30 is configured to receive the encoded picture data 21 and provide decoded picture data (or decoded picture data) 31. The decoded picture data, also sometimes called reconstructed picture data, reconstructed frames of video frames, or by other names, is the picture data obtained after decompression has been performed on the encoded picture data 21.

[0324] The post-processor 32 is configured to perform post-processing on the decoded picture data 31, such as a decoded picture, to obtain post-processed picture data, such as a post-processed picture 33. The post-processing performed by the post-processor 32 may include, for example, color format conversion (e.g., YCbCr to RGB conversion), color correction, cropping, or resampling, or any other processing to generate the decoded picture data 31 for display, for example, by a display device 34.

[0325] A display device 34 is configured to receive the post-processed picture data 33 for displaying the picture to a user, viewer, etc. The display device 34 may be or include any type of display, such as an integrated or external display screen or display, configured to display the reconstructed picture. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any type of other display screen.

[0326] The video encoding / decoding system 10 may also include a training engine 25. The training engine 25 is configured to train neural networks, i.e., the first neural network, the second neural network, the third neural network, and the fourth neural network shown in the embodiment of the method, in the encoder 20 or the decoder 30. The training data may be stored in a database (not shown), and the training engine 25 performs training based on the training data to obtain the neural networks. It should be noted that in this embodiment of the present application, the source of the training data is not limited. For example, the training data may be obtained from the cloud or other locations to train the neural networks.

[0327] The neural network obtained by training by the training engine 25 can be applied to, for example, the video encoding / decoding system 10 and the video encoding / decoding system 40 applied to the source device 12 (e.g., the encoder 20) or the destination device 14 (e.g., the decoder 30) shown in Fig. 13. The training engine 25 performs training on the cloud to obtain the neural network, and then the video encoding / decoding system 10 can download the neural network from the cloud and use the neural network.

[0328] 13 illustrates source device 12 and destination device 14 as separate devices, embodiments of the devices may instead include both source device 12 and destination device 14, or may include functionality of both source device 12 and destination device 14, i.e., both source device 12 or corresponding functionality and destination device 14 or corresponding functionality. In these embodiments, source device 12 or corresponding functionality and destination device 14 or corresponding functionality may be implemented using the same hardware and / or software, using separate hardware and / or software, or any combination thereof.

[0329] Based on the description, the presence and (exact) division of different units or functions of the source device 12 and / or the destination device 14 shown in Fig. 13 may vary according to the actual device and application, as will be apparent to those skilled in the art.

[0330] FIG. 14 is a diagram of another system architecture of a video encoding / decoding system according to an embodiment of the present application. The following is described with reference to FIG. 13. The encoder 20 (e.g., video encoder 20), the decoder 30 (e.g., video decoder 30), or both the encoder and the decoder, can be implemented using, for example, the processing circuitry shown in FIG. 14, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, hardware, processors dedicated to video encoding, or any combination thereof. The encoder 20 can be implemented using a processing circuitry 46 to include various modules described with reference to the encoder 20 of FIG. 14 and / or other decoder systems or subsystems described herein. The decoder 30 can be implemented using a processing circuitry 46 to include various modules described with reference to the decoder 30 of FIG. 15 and / or other decoder systems or subsystems described herein. The processing circuitry 46 may be configured to perform various operations, such as: When some techniques are implemented using software, as shown in Figure 16, an apparatus may store instructions for the software in a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors to perform the techniques of the present application. As shown in Figure 14, either the video encoder 20 or the video decoder 30 may be integrated into a single device as part of a combined encoder / decoder (CODEC).

[0331] The source device 12 and the destination device 14 may include any of a variety of devices, including any type of handheld or fixed device, such as a notebook or laptop computer, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or a content delivery server), a broadcast receiving device, a broadcast sending device, and may or may not use any type of operating system. In some cases, the source device 12 and the destination device 14 may be equipped with wireless communication components. Thus, the source device 12 and the destination device 14 may be wireless communication devices.

[0332] In some cases, the video encoding / decoding system 10 shown in FIG. 13 is merely an example. The techniques provided herein are also applicable to video encoding settings (e.g., video encoding or video decoding) that do not necessarily include data communication between an encoding device and a decoding device. There are also examples of obtaining data from a local memory and transmitting it over a network. A video encoding device may encode data and store the data in memory, and / or a video decoding device may read data from memory and decode data. In some examples, encoding and decoding are performed by devices that do not communicate with each other but encode data to memory and / or read data from memory and decode data.

[0333] 14 is a diagram of an example video encoding / decoding system 40 including video encoder 20 and / or video decoder 30, according to an example embodiment. Video encoding / decoding system 40 may include an image device 41, video encoder 20, video decoder 30 (and / or a video encoder / decoder implemented using processing circuitry 46), an antenna 42, one or more processors 43, one or more memories 44, and / or a display device 45.

[0334] 14, imaging device 41, antenna 42, processing circuitry 46, video encoder 20, video decoder 30, processor 43, memory 44, and / or display device 45 may be in communication with each other. In different examples, video encoding / decoding system 40 may include only video encoder 20 or only video decoder 30.

[0335] In some examples, the antenna 42 may be configured to transmit or receive an encoded bitstream of video data. Additionally, in some examples, the display device 45 may be configured to present the video data. The processing circuitry 46 may include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, and the like. The video encoding / decoding system 40 may also include an optional processor 43. Similarly, the optional processor 43 may include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, and the like. Additionally, the memory 44 may be any type of memory, such as a volatile memory (e.g., a static random access memory (SRAM) or a dynamic random access memory (DRAM) or a non-volatile memory (e.g., a flash memory). In a non-limiting example, the memory 44 may be implemented by a cache memory. In another example, the processing circuitry 46 may include a memory (e.g., a cache) configured to implement a picture buffer.

[0336] In some examples, video encoder 20 implemented using logic circuitry may include a picture buffer (e.g., implemented by processing circuitry 46 or memory 44) and a graphics processing unit (e.g., implemented by processing circuitry 46). The graphics processing unit may be communicatively coupled to the picture buffer. The graphics processing unit includes video encoder 20 implemented using processing circuitry 46 and may implement various modules described with reference to video decoder 20 shown in FIG. 14 and / or other encoder systems or subsystems described herein. The logic circuitry may be configured to perform various operations herein.

[0337] In some examples, video decoder 30 may also be implemented using processing circuitry 46 to implement various modules described with reference to video decoder 30 shown in FIG. 14 and / or other decoder systems or subsystems described herein. In some examples, video decoder 30 implemented using logic circuitry may include a picture buffer (e.g., implemented by processing circuitry 46 or memory 44) and a graphics processing unit (e.g., implemented by processing circuitry 46). The graphics processing unit may be communicatively coupled to the picture buffer. The graphics processing unit may include video decoder 30 implemented by processing circuitry 46.

[0338] In some examples, the antenna 42 may be configured to receive an encoded bitstream of video data. As previously mentioned, the encoded bitstream may include data, indices, index values, mode selection data, etc., such as data related to coding partitions (e.g., transform coefficients or quantized transform coefficients, optional indices (described), and / or data defining coding partitions) related to the encoding of the video frames. The video encoding / decoding system 40 may further include a video decoder 30 coupled to the antenna 42 and configured to decode the encoded bitstream. The display device 45 is configured to present the video frames.

[0339] It should be understood that, for the examples described with reference to the video encoder 20 in this embodiment of the present application, the video decoder 30 may be configured to perform the inverse process. With respect to the signaling syntax elements, the video decoder 30 may be configured to receive and parse these syntax elements and decode the associated video data accordingly. In some examples, the video encoder 20 may entropy code the syntax elements to obtain an encoded video bitstream. In an example, the video decoder 30 may parse the syntax elements and decode the associated video data accordingly.

[0340] It should be noted that the encoding / decoding process described in this application exists in most video encoders / decoders, such as H.263, H.264, MPEG-2, MPEG-4, VP8, VP9, ​​and AI-based end-to-end picture encoding.

[0341] 15 is a schematic diagram of a video coding device 400 according to an embodiment of the present disclosure. The video coding device 400 is suitable for implementing the disclosed embodiments described herein. In an embodiment, the video encoding device 400 may be a video decoder, such as the video decoder 30 of FIG. 14, or an encoder, such as the video encoder 20 of FIG. 14.

[0342] The video coding device 400 includes an ingress port 410 (or input port 410) and a receiver unit (Rx) 420 configured to receive data, a processor, logic unit, or central processing unit (CPU) 430 configured to process the data, a transmitter unit (Tx) 440 and an egress port (or output port 450) configured to transmit the data, and a memory 460 configured to store the data. For example, the processing unit 430 may be a neural network processing unit 430. The video coding device 400 may also include optical-to-electrical (OE) and electrical-to-optical (EO) components coupled to the ingress port 410, the receiver unit 420, the transmitter unit 440, and the egress port 450 and used as an exit or entrance for optical or electrical signals.

[0343] The processing unit 430 is implemented using hardware and software. The processing unit 430 may be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processing unit 430 communicates with the ingress port 410, the receiver unit 420, the transmitter unit 440, the egress port 450, and the memory 460. The processing unit 430 includes a coding module 470 (e.g., a coding module 470 based on a neural network NN). The coding module 470 implements the above-mentioned embodiments. For example, the coding module 470 executes, processes, prepares, or provides various encoding operations. Thus, the use of the coding module 470 provides significant improvements in the functionality of the video coding device 400 and affects the switching of the video coding device 400 to different states. Alternatively, the coding module 470 is implemented as instructions stored in the memory 460 and executed by the processing unit 430.

[0344] Memory 460 may include one or more disks, tape drives, and solid state drives and may be used as overflow data storage, configured to store programs as they are executed, and to store instructions and data that are read as the programs are executed. Memory 460 may be non-volatile and / or non-volatile, and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random access memory (SRAM).

[0345] 16 is a simplified block diagram of an apparatus 500 according to an example embodiment, which may be used as any one or both of source device 12 and destination device 14 of FIG.

[0346] The processor 502 of the device 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or devices that can manipulate or process information, whether now existing or developed in the future. Although the disclosed implementations can be implemented using a single processor, such as the processor 502 shown in the figure, using multiple processors can be faster and more efficient.

[0347] In implementation, the memory 504 in the device 500 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as the memory 504. The memory 504 may include code and data 506 that is accessed by the processor 502 using a bus 512. The memory 504 may further include an operating system 508 and application programs 510. The application programs 510 include at least one program that enables the processor 502 to perform the methods described herein. For example, the application programs 510 may include applications 1-N, which further include a video coding application that performs the methods described herein.

[0348] The device 500 may also include one or more output devices, such as a display 518. In a further example, the display 518 may be a touch-sensitive display that combines a display with a touch-sensitive element operative to sense touch input. The display 518 may be coupled to the processor 502 using the bus 512.

[0349] Although the bus 512 of the device 500 is shown here as a single bus, the bus 512 may include multiple buses. Also, the secondary memory may be directly coupled to other components in the device 500 or accessed over a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Thus, the device 500 may have a variety of configurations.

Claims

1. 1. A method for compressing video frames, comprising: determining a target neural network from a plurality of neural networks according to a network selection policy, the plurality of neural networks including a first neural network and a second neural network; performing compression encoding on a current video frame using the target neural network to obtain compression information corresponding to the current video frame; Including, If the compression information is obtained using the first neural network, the compression information includes first compression information of a first feature of the current video frame, and a reference frame of the current video frame is used in the compression process of the first feature of the current video frame; or When the compressed information is obtained using the second neural network, the compressed information includes second compressed information of a second feature of the current video frame, and a reference frame of the current video frame is used in the generation process of the second feature of the current video frame.

2. The first neural network includes a coding network and an entropy coding layer, and the coding network is used to obtain the first feature of the current video frame from the current video frame, and the entropy coding layer is used to perform entropy coding on the first feature of the current video frame based on the reference frame of the current video frame, and output the first compressed information; and / or 2. The method of claim 1 , wherein the second neural network includes a convolutional network and an entropy coding layer, the convolutional network includes a plurality of convolutional layers and an activated ReLU layer, the convolutional network is used to obtain a residual of the current video frame based on the reference frame of the current video frame, and the entropy coding layer is used to perform entropy coding on the residual of the current video frame to output the second compressed information, and the second feature is the residual.

3. The method of claim 1 or 2, wherein the network selection policy relates to one or more of: position information of the current video frame; or an amount of data carried in the current video frame.

4. The step of determining a target neural network from the plurality of neural networks in accordance with a network selection policy includes: selecting the target neural network from the plurality of neural networks based on the position information of the current video frame in a current video sequence, the position information indicating that the current video frame is an Xth frame in the current video sequence; or The step of determining a target neural network from the plurality of neural networks in accordance with a network selection policy includes:

4. The method of claim 3, comprising: selecting the target neural network from the plurality of neural networks based on attributes of the current video frame, the attributes of the current video frame indicating an amount of data carried in the current video frame, the attributes of the current video frame including any one or combination of entropy, contrast, and saturation of the current video frame.

5. The method comprises: generating and transmitting instruction information corresponding to the compressed information, the instruction information indicating that the compressed information is to be obtained using the target neural network, the target neural network being either the first neural network or the second neural network; The method of any one of claims 1 to 4, further comprising:

6. When the target neural network is the first neural network, performing compression encoding on a current video frame using the target neural network to obtain compression information corresponding to the current video frame includes: obtaining the first feature of the current video frame from the current video frame using the encoding network; predicting, using the entropy coding layer, the first feature of the current video frame based on the reference frame of the current video frame to generate predicted features of the current video frame, the predicted features of the current video frame being a prediction result of the first feature of the current video frame; generating, using the entropy coding layer, a probability distribution for the first feature of the current video frame based on the predictive feature of the current video frame; performing entropy coding on the first feature of the current video frame using the entropy coding layer based on a probability distribution of the first feature of the current video frame to obtain the first compressed information; The method of claim 2 , comprising:

7. 1. A method for compressing video frames, comprising: performing compression encoding on a current video frame using a first neural network to obtain first compression information of a first feature of the current video frame, wherein a reference frame of the current video frame is used in the compression process of the first feature of the current video frame; generating a first video frame using the first neural network, the first video frame being a reconstructed frame of the current video frame; performing compression encoding on the current video frame using a second neural network to obtain second compression information of a second feature of the current video frame, wherein the reference frame of the current video frame is used in the generation process of the second feature of the current video frame; generating a second video frame using the second neural network, the second video frame being a reconstructed frame of the current video frame; determining compression information corresponding to the current video frame based on the first compression information, the first video frame, the second compression information, and the second video frame, where the determined compression information is obtained using the first neural network and the determined compression information is the first compression information, or the determined compression information is obtained using the second neural network and the determined compression information is the second compression information; The method includes:

8. The first neural network includes a coding network and an entropy coding layer, and the coding network is used to obtain the first feature of the current video frame from the current video frame, and the entropy coding layer is used to perform entropy coding on the first feature of the current video frame to output the first compressed information; and / or 8. The method of claim 7, wherein the second neural network includes a convolutional network and an entropy coding layer, the convolutional network includes a plurality of convolutional layers and an activated ReLU layer, the convolutional network is used to obtain a residual of the current video frame based on the reference frame of the current video frame, and the entropy coding layer is used to perform entropy coding on the residual of the current video frame, and output the second compressed information.

9. 1. A method for compressing video frames, comprising: performing compression encoding on a third video frame using a first neural network to obtain first compression information corresponding to the third video frame, the first compression information including compression information of a first feature of the third video frame, and a reference frame of the third video frame being used for compression processing of the first feature of the third video frame; performing compression encoding on a fourth video frame using a second neural network to obtain second compression information corresponding to the fourth video frame, the second compression information including compression information of a second feature of the fourth video frame, a reference frame of the fourth video frame being used in the generation process of the second feature of the fourth video frame, and the third video frame and the fourth video frame being different video frames of a same video sequence; The method includes:

10. The first neural network includes a coding network and an entropy coding layer, and the coding network is used to obtain the first feature of the current video frame from the current video frame, and the entropy coding layer is used to perform entropy coding on the first feature of the current video frame to output the first compressed information; and / or 10. The method of claim 9, wherein the second neural network includes a convolutional network and an entropy coding layer, the convolutional network includes a plurality of convolutional layers and an activated ReLU layer, the convolutional network is used to obtain a residual of the current video frame based on a reference frame of the current video frame, and the entropy coding layer is used to perform entropy coding on the residual of the current video frame, and output the second compressed information.

11. 1. A method for decompressing a video frame, comprising: obtaining compression information for a current video frame; selecting a target neural network corresponding to the current video frame from a plurality of neural networks, the plurality of neural networks including a third neural network and a fourth neural network; performing a decompression using the target neural network based on the compression information to obtain a reconstructed frame of the current video frame; If the target neural network is the third neural network, the compressed information includes first compressed information of a first feature of the current video frame, and a reference frame of the current video frame is used in a process of decompressing the first compressed information to obtain the first feature of the current video frame, and the first feature of the current video frame is used in a process of generating the reconstructed frame of the current video frame; or if the target neural network is the fourth neural network, the compression information includes second compression information of a second feature of the current video frame, the second compression information is used by a decoder to perform decompression to obtain the second feature of the current video frame, and a reference frame of the current video frame and the second feature of the current video frame are used in a process of generating the reconstructed frame of the current video frame; The method includes:

12. the third neural network includes an entropy decoding layer and a decoding network, and using the entropy decoding layer to perform entropy decoding on the first compressed information of the current video frame based on the reference frame of the current video frame, and using the decoding network to generate the reconstructed frame of the current video frame based on the first feature of the current video frame; and / or 12. The method of claim 11, wherein the fourth neural network includes an entropy decoding layer and a convolutional network, the entropy decoding layer is used to perform entropy decoding on the second compressed information, and the convolutional network is used to perform a process of generating the reconstructed frame of the current video frame based on the reference frame of the current video frame and the second features of the current video frame.

13. The method comprises: obtaining indication information corresponding to the compressed information; The step of selecting a target neural network corresponding to the current video frame from a plurality of neural networks includes:

13. The method of claim 11 or 12, comprising determining the target neural network from the plurality of neural networks based on the indication information.

14. 1. A method for decompressing a video frame, comprising: using a third neural network to decompress first compressed information of a third video frame to obtain a reconstructed frame of the third video frame, the first compressed information including compressed information of a first feature of the third video frame, a reference frame of the third video frame being used in the decompression process of the first compressed information to obtain the first feature of the third video frame, and the first feature of the third video frame being used in the generation process of the reconstructed frame of the third video frame; using a fourth neural network to decompress second compressed information of a fourth video frame to obtain a reconstructed frame of the fourth video frame, the second compressed information including compressed information of a second feature of the fourth video frame, the second compressed information being used by a decoder to perform decompression to obtain the second feature of the fourth video frame, and a reference frame of the fourth video frame and the second feature of the fourth video frame being used in a process of generating the reconstructed frame of the fourth video frame; The method includes:

15. the third neural network includes an entropy decoding layer and a decoding network, and the entropy decoding layer is used to perform entropy decoding on the first compressed information of the current video frame based on the reference frame of the current video frame, and the decoding network is used to generate the reconstructed frame of the current video frame based on the first feature of the current video frame; and / or 15. The method of claim 14, wherein the fourth neural network includes an entropy decoding layer and a convolutional network, the entropy decoding layer is used to perform entropy decoding on the second compressed information, and the convolutional network is used to perform a process of generating the reconstructed frame of the current video frame based on the reference frame of the current video frame and the second features of the current video frame.

16. An encoder comprising a processing circuit and configured to perform the method of any one of claims 1 to 10.

17. A decoder comprising processing circuitry and configured to perform the method of any one of claims 11 to 15.

18. A computer program comprising program code, the program code being adapted to perform the method according to any one of claims 1 to 15 when executed by a computer or processor.

19. 1. An encoder comprising: one or more processors; a non-transitory computer readable storage medium coupled to said processor and storing program instructions for execution by said processor, said program instructions, when executed by said processor, enabling said encoder to perform a method according to any one of claims 1 to 10; An encoder that includes

20. A decoder comprising: one or more processors; a non-transitory computer readable storage medium coupled to said processor and storing program instructions for execution by said processor, said program instructions, when executed by said processor, enabling said decoder to perform a method according to any one of claims 11 to 15; A decoder including:

21. A non-transitory computer readable storage medium comprising a program code which, when executed by a computer device, performs the method according to any one of claims 1 to 15.

Citation Information

Patent Citations

  • Method and equipment for data compression

    JP1990303220A

  • Method and apparatus for video encoding and decoding based on a neural network implementation of CABAC

    JP2021520087A

  • Method and apparatus for video encoding and decoding based on neural network implementation of cabac

    US20220264095A1

  • Method and apparatus for encoding and decoding data utilizing data compression and neural networks

    US5134396A

  • Coding device, decoding device, coding method, and decoding method

    WO2018199051A1