Video coding method and apparatus

Abstract

The application provides a video coding method and device, which makes the selection of key frames more flexible, and improves the coding efficiency and system performance. The method comprises: performing time domain downsampling on an image sequence of an original video at an encoding end to obtain N key frames, wherein N is an integer greater than or equal to 2; the encoding end encodes only the coding information of the N key frames and the time domain position information of the N key frames into a code stream, and the time domain position information of the N key frames is used to indicate the positions of the N key frames in the image sequence. Correspondingly, the decoding end decodes the reconstructed images of the N key frames and the time domain position information of the N key frames from the code stream; and the decoding end performs time domain upsampling on the time domain position information of the N key frames and the reconstructed images of the N key frames to obtain a reconstructed image sequence of the original video.

Description

Technical Field

[0001] This application relates to the field of video processing, and more particularly to a video encoding and decoding method and apparatus. Background Technology

[0002] During video storage and transmission, it is usually necessary to compress and encode the video image sequence to reduce storage capacity and transmission bandwidth. For example, to reduce storage space and computing capacity, the image sequence of the original video needs to be downsampled in the temporal domain. Representative and high-quality low frame rate images are selected as the objects of processing and analysis. The encoding end can compress these low frame rate images to obtain a compressed bitstream, and the decoding end can process the compressed bitstream sequentially to obtain the reconstructed image. Then, the reconstructed image is upsampled in the temporal domain to obtain the reconstructed video of the original video.

[0003] Current video encoding and decoding schemes employ uniform sampling when downsampling the image sequence of the original video in the temporal domain. For example, every K frames skipped, one frame is retained (K is an integer greater than or equal to 1), and the encoded end encodes the retained image as a low frame rate image. While this method has low complexity, it is prone to losing keyframes containing important information, resulting in low encoding and decoding accuracy, inflexibility, and low efficiency. Summary of the Invention

[0004] This application provides a video encoding and decoding method and apparatus, which effectively improves the flexibility of the encoding end in sampling the image sequence of the original video, which is beneficial to improving the accuracy of encoding and decoding, thereby improving the encoding and decoding efficiency.

[0005] In a first aspect, a video encoding method is provided, comprising: performing temporal downsampling on an image sequence of an original video to obtain N keyframes, where N ≥ 2 and is an integer, the image sequence including X image frames, where X is an integer greater than N; encoding only the encoding information of the N keyframes and the temporal position information of the N keyframes into the bitstream, wherein the temporal position information of the N keyframes is used to indicate the position of the N keyframes in the image sequence.

[0006] In this embodiment, the encoding end first acquires keyframes, which contain important image information, such as scenes or objects. This effectively improves encoding efficiency. Furthermore, in addition to the encoding information of the keyframes, the bitstream also contains the temporal location information of those keyframes. Therefore, the final bitstream contains more key image information, effectively improving the performance of the encoding end.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, performing temporal downsampling on the image sequence of the original video to obtain N key frames includes: determining N key frames whose probability values ​​are greater than or equal to a preset threshold based on the probability value of each image in the image sequence as a key frame.

[0008] In this embodiment, the encoding end selects keyframes by obtaining the probability of each frame in the image sequence of the original video as a keyframe. This improves the flexibility of keyframe selection and effectively avoids the omission of keyframes containing key image information by using probability, thereby effectively improving encoding efficiency.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, before determining the N frames of images with probability values ​​greater than or equal to a preset threshold as the N key frames, the method further includes: inputting the image sequence into a pre-trained first neural network model, the first neural network model being used to determine the probability value of each frame in the image sequence as a key frame; and obtaining the probability value of each frame in the image sequence as a key frame from the first neural network model.

[0010] In this embodiment, the encoding end uses a neural network to obtain the probability of the above image being a keyframe, thereby obtaining the keyframe and effectively improving the efficiency and accuracy of keyframe acquisition.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, performing temporal downsampling on the image sequence of the original video to obtain N keyframes includes: grouping the image sequence into N groups of images, each of the N groups of images including at least one image frame; inputting the N groups of images into a pre-trained first neural network model, the first neural network model being used to determine the probability value of each image frame in the image sequence as a keyframe; and determining the image frame with the highest probability value in each of the N groups of images as a keyframe based on the probability value of each image frame in the N groups of images.

[0012] In this embodiment of the application, before the encoding end obtains the probability of the above image as a key frame through the neural network, it can first group the above image sequence, that is, divide the above image sequence into several shorter groups of image sequences. Then the encoding end passes the image groups through the first neural network model in sequence to obtain the probability of each frame in each group of images as a key frame. That is, the number of image frames processed by the encoding end each time is reduced, which effectively reduces the performance requirements of the encoding end.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the above image sequence is grouped to obtain N groups of images, including: determining N-1 scene switching frames based on the probability value of each frame in the above image sequence as a scene switching frame; and grouping the above image sequence according to the above N-1 scene switching frames to obtain the above N groups of images.

[0014] In this embodiment, the encoding end groups the image sequence according to different scenes in the above image sequence, which effectively reduces the omission of key frames containing key scene information, thereby effectively improving encoding efficiency.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, before determining N-1 scene switching frames based on the probability value of each frame in the image sequence as a scene switching frame, the method further includes: inputting the image sequence into a pre-trained second neural network model, the second neural network model being used to determine the probability value of each frame in the image sequence as a scene switching frame; and obtaining the probability value of each frame in the image sequence as a scene switching frame from the second neural network model.

[0016] In this embodiment, the encoding end uses a neural network model to obtain the probability of the above image as a scene switching frame, which effectively avoids the omission of key frames containing important scenes, thereby effectively improving the efficiency and accuracy of grouping.

[0017] In conjunction with the first aspect, in some implementations of the first aspect, performing temporal downsampling on the image sequence of the original video to obtain N key frames includes: obtaining the N key frames from the image sequence based on motion detection of adjacent images in the image sequence.

[0018] In this embodiment of the application, the encoding end analyzes the motion changes between images in the above image sequence to determine the changes in various information in the image, such as the displacement of objects, effectively avoiding the omission of key frames containing objects with violent motion.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, the encoding information of the aforementioned N keyframes and the temporal location information of the aforementioned N keyframes are encoded into the bitstream, including: encoding the encoding information of the aforementioned N keyframes and the temporal location information of the aforementioned N keyframes into the following positions in the aforementioned bitstream: sequence header, image header, slice header, or supplementary enhancement information.

[0020] In this embodiment, the encoding end also incorporates the location information of keyframes into the bitstream, effectively increasing the amount of key information contained in the bitstream and thus improving encoding performance.

[0021] In conjunction with the first aspect, in certain implementations of the first aspect, the temporal position information of the aforementioned N keyframes includes any one of the following: the image sequence count (POC) of the aforementioned N keyframes, the position index of the aforementioned N keyframes, or the time interval between the aforementioned N keyframes and a preset image; wherein, the POC of the aforementioned N keyframes is used to indicate the playback sequence number of the aforementioned N keyframes in the aforementioned image sequence; the position index of the aforementioned N keyframes is used to indicate the temporal position of the aforementioned N keyframes in the aforementioned image sequence; and the time interval is the time interval between each frame of the aforementioned N keyframes and the aforementioned preset image.

[0022] Secondly, a video decoding method is provided, comprising: decoding a reconstructed image of N keyframes and temporal position information of the N keyframes from a bitstream, wherein the temporal position information of the N keyframes is used to indicate the position of the N keyframes in the image sequence of the original video, and N≥2 and is an integer; performing temporal upsampling on the temporal position information of the N keyframes and the reconstructed image of the N keyframes to obtain a reconstructed image sequence of the original video, wherein the reconstructed image sequence includes X image frames, where X is an integer greater than N.

[0023] In the embodiments of this application, in addition to using the reconstructed images of keyframes, the temporal location information of keyframes is also used during the temporal upsampling process of the decoding end, which effectively improves the reconstruction of the image sequence of the original video and effectively improves the performance of the decoding end.

[0024] In conjunction with the second aspect, in some implementations of the second aspect, the above-mentioned decoding of the reconstructed image of N keyframes and the temporal location information of the N keyframes from the bitstream includes: parsing the temporal location information of the N keyframes from the bitstream; and decoding the reconstructed image of the N keyframes from the bitstream based on the temporal location information of the N keyframes.

[0025] In this embodiment, the decoding end can decode the reconstructed image of the key frame based on the position information of the key frame in the bitstream, which effectively improves the decoding efficiency.

[0026] In conjunction with the second aspect, in some implementations of the second aspect, parsing the temporal location information of the aforementioned N keyframes from the aforementioned bitstream includes: parsing the temporal location information of the aforementioned N keyframes from the sequence header, image header, slice header, or supplementary enhancement information in the aforementioned bitstream.

[0027] In conjunction with the second aspect, in some implementations of the second aspect, the temporal position information of the aforementioned N keyframes includes any of the following: the image sequence count (POC) of the aforementioned N keyframes, the position index of the aforementioned N keyframes, or the time interval between the aforementioned N keyframes and a preset image; wherein, the POC of the aforementioned N keyframes is used to indicate the playback sequence number of the aforementioned N keyframes in the aforementioned image sequence; the position index of the aforementioned N keyframes is used to indicate the temporal position of the aforementioned N keyframes in the aforementioned image sequence; and the time interval is the time interval between each frame of the aforementioned N keyframes and the aforementioned preset image.

[0028] In conjunction with the second aspect, in some implementations of the second aspect, temporal upsampling is performed on the temporal location information of the aforementioned N keyframes and the reconstructed images of the aforementioned N keyframes to obtain the reconstructed image sequence of the aforementioned original video, including: interpolating the reconstructed images of the aforementioned N keyframes and the temporal location information of the aforementioned N keyframes to obtain K frame images, wherein K≤M and are integers, the aforementioned M frame images are the images in the aforementioned image sequence excluding the aforementioned N keyframes, and M≥1 and are integers; and obtaining the reconstructed image sequence of the aforementioned original video based on the reconstructed images of the aforementioned N keyframes and the aforementioned K frame images.

[0029] In conjunction with the second aspect, in some implementations of the second aspect, interpolating the reconstructed images of the aforementioned N keyframes and the temporal location information of the N keyframes to obtain K-frame images includes: inputting the reconstructed images of the aforementioned N keyframes and the temporal location information of the aforementioned N keyframes into a pre-trained third neural network model to obtain the aforementioned K-frame images, wherein the aforementioned third neural network model is used to obtain reconstructed images of images in the original video.

[0030] In this embodiment, the decoding end uses a neural network model to interpolate and obtain the reconstructed image, which effectively improves the interpolation efficiency and thus effectively improves the performance of the decoding end.

[0031] Thirdly, a video encoding apparatus is provided, comprising: an acquisition module and an encoding module; wherein the acquisition module is configured to: perform temporal downsampling on an image sequence of an original video to acquire N keyframes, where N ≥ 2 and is an integer, and the image sequence includes X image frames, where X is an integer greater than N. The encoding module is configured to: encode only the encoding information of the N keyframes and the temporal position information of the N keyframes into the bitstream, wherein the temporal position information of the N keyframes is used to indicate the position of the N keyframes in the image sequence.

[0032] In conjunction with the third aspect, in some implementations of the third aspect, the acquisition module is used to: determine the N frames of images whose probability values ​​are greater than or equal to a preset threshold as keyframes based on the probability values ​​of each frame in the image sequence.

[0033] In conjunction with the third aspect, in some implementations of the third aspect, the acquisition module is specifically used to: before determining the N frames of images with probability values ​​greater than or equal to a preset threshold as the N key frames, input the image sequence into a pre-trained first neural network model, the first neural network model being used to determine the probability value of each frame in the image sequence as a key frame; and obtain the probability value of each frame in the image sequence as a key frame from the first neural network model.

[0034] In conjunction with the third aspect, in some implementations of the third aspect, the acquisition module is specifically used to: group the image sequence to obtain N groups of images, each of the N groups of images including at least one frame; input the N groups of images into a pre-trained first neural network model, the first neural network model being used to determine the probability value of each frame in the image sequence as a keyframe; and, based on the probability value of each frame in the N groups of images as a keyframe, determine the frame with the highest probability value in each of the N groups of images as the keyframe.

[0035] In conjunction with the third aspect, in some implementations of the third aspect, the acquisition module is specifically used to: determine N-1 scene switching frames based on the probability value of each frame in the above image sequence as a scene switching frame; and group the above image sequence according to the above N-1 scene switching frames to obtain the above N groups of images.

[0036] In conjunction with the third aspect, in some implementations of the third aspect, the aforementioned acquisition module is specifically used to: before determining N-1 scene switching frames based on the probability value of each frame in the aforementioned image sequence as a scene switching frame, input the aforementioned image sequence into a pre-trained second neural network model, wherein the aforementioned second neural network model is used to determine the probability value of each frame in the image sequence as a scene switching frame; and obtain the probability value of each frame in the aforementioned image sequence as a scene switching frame from the aforementioned second neural network model.

[0037] In conjunction with the third aspect, in some implementations of the third aspect, the above-mentioned acquisition module is specifically used to: acquire the above-mentioned N key frames from the above-mentioned image sequence based on motion detection of adjacent images in the above-mentioned image sequence.

[0038] In conjunction with the third aspect, in some implementations of the third aspect, the above-mentioned encoding module is specifically used to: encode the encoding information of the above-mentioned N key frames and the temporal location information of the above-mentioned N key frames into the following positions in the above-mentioned bitstream: sequence header, image header, slice header or supplementary enhancement information.

[0039] In conjunction with the third aspect, in certain implementations of the third aspect, the temporal position information of the aforementioned N keyframes includes any of the following: the image sequence count (POC) of the aforementioned N keyframes, the position index of the aforementioned N keyframes, or the time interval between the aforementioned N keyframes and a preset image; wherein, the POC of the aforementioned N keyframes is used to indicate the playback sequence number of the aforementioned N keyframes in the aforementioned image sequence; the position index of the aforementioned N keyframes is used to indicate the temporal position of the aforementioned N keyframes in the aforementioned image sequence; and the aforementioned time interval is the time interval between each frame of the aforementioned N keyframes and the aforementioned preset image.

[0040] Fourthly, a video decoding apparatus is provided, comprising: a decoding module and a processing module; wherein the decoding module is configured to: decode reconstructed images of N keyframes and temporal position information of the N keyframes from a bitstream, the temporal position information of the N keyframes indicating the positions of the N keyframes in an image sequence of the original video, where N ≥ 2 and is an integer. The processing module is configured to: perform temporal upsampling on the temporal position information of the N keyframes and the reconstructed images of the N keyframes to obtain a reconstructed image sequence of the original video, the reconstructed image sequence comprising X image frames, where X is an integer greater than N.

[0041] In conjunction with the fourth aspect, in one implementation of the fourth aspect, the decoding module is used to: parse the temporal location information of the N key frames from the bitstream; and decode the reconstructed image of the N key frames from the bitstream based on the temporal location information of the N key frames.

[0042] In conjunction with the fourth aspect, in one implementation of the fourth aspect, the above-mentioned decoding module is specifically used to: parse the temporal location information of the above-mentioned N key frames from the sequence header, image header, slice header or supplementary enhancement information in the above-mentioned bitstream.

[0043] In conjunction with the fourth aspect, in one implementation of the fourth aspect, the temporal position information of the aforementioned N keyframes includes any of the following: the image sequence count (POC) of the aforementioned N keyframes, the position index of the aforementioned N keyframes, or the time interval between the aforementioned N keyframes and a preset image; wherein, the POC of the aforementioned N keyframes is used to indicate the playback sequence number of the aforementioned N keyframes in the aforementioned image sequence; the position index of the aforementioned N keyframes is used to indicate the temporal position of the aforementioned N keyframes in the aforementioned image sequence; and the aforementioned time interval is the time interval between each frame of the aforementioned N keyframes and the aforementioned preset image.

[0044] In conjunction with the fourth aspect, in one implementation of the fourth aspect, the above processing module is used to: interpolate the reconstructed images of the above N keyframes and the temporal location information of the above N keyframes to obtain K frames of images, where K ≤ M and is an integer, and the above M frames of images are the images in the above image sequence excluding the above N keyframes, where M ≥ 1 and is an integer; and obtain the reconstructed image sequence of the above original video based on the reconstructed images of the above N keyframes and the above K frames of images.

[0045] In conjunction with the fourth aspect, in one implementation of the fourth aspect, the above processing module is specifically used to: input the reconstructed images of the above N key frames and the temporal location information of the above N key frames into the pre-trained third neural network model to obtain the above K frame images, and the above third neural network model is used to obtain the reconstructed images of the images in the original video.

[0046] Fifthly, another video encoding / decoding apparatus is provided, including a processor coupled to a memory for executing instructions in the memory to implement the methods in any possible implementation of any of the foregoing aspects. Optionally, the apparatus further includes a memory. Optionally, the apparatus further includes a communication interface, to which the processor is coupled.

[0047] In one implementation, the video codec is a server. When the video codec is a server, the communication interface can be a transceiver, or an input / output interface.

[0048] In another implementation, the video codec is a chip configured in a server. When the video codec is a chip configured in a server, the communication interface can be an input / output interface.

[0049] In a sixth aspect, a processor is provided, comprising: an input circuit, an output circuit, and a processing circuit. The processing circuit is configured to receive signals through the input circuit and transmit signals through the output circuit, causing the processor to execute the method in any of the possible implementations of the first and second aspects described above.

[0050] In specific implementation, the processor can be a chip, the input circuit can be an input pin, the output circuit can be an output pin, and the processing circuit can be a transistor, gate circuit, flip-flop, and various logic circuits. The input signal received by the input circuit can be received and input by, for example, but not limited to, a receiver, and the signal output by the output circuit can be output to, for example, but not limited to, a transmitter and transmitted by the transmitter. Furthermore, the input circuit and the output circuit can be the same circuit, which is used as the input circuit and the output circuit at different times. This application does not limit the specific implementation of the processor and various circuits.

[0051] In a seventh aspect, a processing apparatus is provided, including a processor and a memory. The processor is configured to read instructions stored in the memory and to receive signals via a receiver and transmit signals via a transmitter to execute the method in any of the possible implementations of the first aspect described above.

[0052] Optionally, there may be one or more processors and one or more memories.

[0053] Alternatively, the memory can be integrated with the processor, or the memory can be set up separately from the processor.

[0054] In specific implementation, the memory can be a non-transitory memory, such as read-only memory (ROM), which can be integrated on the same chip as the processor or set on different chips. The embodiments of this application do not limit the type of memory or the way the memory and processor are set.

[0055] It should be understood that the relevant data interaction process, such as sending indication information, can be the process of outputting indication information from the processor, and receiving capability information can be the process of the processor receiving input capability information. Specifically, the processed output data can be output to the transmitter, and the input data received by the processor can come from the receiver. Here, the transmitter and receiver can be collectively referred to as a transceiver.

[0056] The processing device in the seventh aspect above can be a chip. The processor can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. The memory can be integrated into the processor or located outside the processor and exist independently.

[0057] Eighthly, a computer program product is provided, comprising: a computer program (also referred to as code or instructions) that, when run, causes a computer to perform the method in any of the possible implementations of the first and second aspects described above.

[0058] A ninth aspect provides a computer-readable storage medium storing a computer program (also referred to as code or instructions) that, when executed on a computer, causes the computer to perform the methods of any possible implementation of the first and second aspects described above. Attached Figure Description

[0059] Figure 1 This is a schematic flowchart of a typical video encoding and decoding method.

[0060] Figure 2 This is a schematic flowchart of the video encoding and decoding method provided in the embodiments of this application;

[0061] Figure 3 This is a schematic diagram of the first neural network model structure provided in the embodiments of this application;

[0062] Figure 4 This is a schematic diagram illustrating a keyframe acquisition process provided in an embodiment of this application;

[0063] Figure 5 This is a schematic diagram of another keyframe acquisition process provided in an embodiment of this application;

[0064] Figure 6 This is a schematic diagram of another keyframe acquisition process provided in an embodiment of this application;

[0065] Figure 7 This is a schematic block diagram of the video encoding apparatus provided in the embodiments of this application;

[0066] Figure 8 This is a schematic block diagram of another video decoding device provided in the embodiments of this application;

[0067] Figure 9 This is a schematic block diagram of another video encoding / decoding device provided in the embodiments of this application. Detailed Implementation

[0068] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0069] The technical solutions involved in the embodiments of this application can be applied not only to existing video coding standards (such as H.264, high efficiency video coding (HEVC) and other standards), but also to future video coding standards.

[0070] The terminology used in the implementation section of this application is only for explaining specific embodiments of this application and is not intended to limit this application. A brief introduction to some concepts that may be involved in the embodiments of this application is given below.

[0071] 1. Video encoding and decoding

[0072] Video encoding and decoding generally refers to the processing of image sequences that form a video or video file. In the field of video encoding and decoding, the terms "picture," "frame," or "image" can be used synonymously. The combination of encoding and decoding is also called encoding and decoding (encoding and decoding).

[0073] 2. Time-domain downsampling and time-domain upsampling

[0074] Time-domain sampling refers to the following: for a continuous signal f(t) with frequency band F, a series of discrete sampled values ​​f(t1), f(t1±Δt), ..., f(t1±nΔt) can be used to represent it. As long as the time interval Δt of these sampling points is ≤1 / (2F), the original signal f(t) can be completely recovered from each sampled value.

[0075] The process of representing the signal f(t) using a series of discrete sampled values ​​f(t), f(t1±Δt), ..., f(t1±nΔt) is called time-domain downsampling. The process of recovering the original signal f(t) from the sampled values ​​f(t1), f(t1±Δt), ..., f(t1±nΔt) is called time-domain upsampling.

[0076] It should be understood that the aforementioned time-domain downsampling is essentially the decimation of a signal, a process of transforming a continuous signal into a discrete one. The aforementioned time-domain upsampling is the inverse process of time-domain downsampling, a process of transforming a discrete signal into a continuous one. Time-domain upsampling can also be called time-domain upsampling, time-domain interpolation, or other names; this application does not limit the specific names used in its embodiments.

[0077] The typical video encoding and decoding process is as follows: Figure 1 As shown, the encoding end encodes the image sequence of the original video to obtain encoded information. For example, the encoding end can process the image sequence of the original video through compression, thereby reducing the amount of data required to store and / or transmit the image sequence of the video. The encoding end can transmit the above encoded information to the decoding end in the form of a bitstream. Correspondingly, the decoding end receives the bitstream from the encoding end, performs the inverse operation relative to the operation at the encoding end on the bitstream, and thus reconstructs the image sequence of the original video.

[0078] It should be understood that the above video encoding process mainly includes intra-prediction, inter-prediction, transform, quantization, entropy encoding, and in-loop filtering. The encoding end can divide the image into blocks, perform intra-prediction and inter-prediction on each block, and after obtaining the residuals, perform transform quantization, and finally perform entropy encoding to output the bitstream. Here, a block is an M×N array of pixels, and the pixel value at each pixel position is known. M and N are both positive integers; M can be equal to N or not equal to N, and this application embodiment does not limit this.

[0079] In one possible implementation, intra-frame prediction refers to the encoder using the pixel values ​​of pixels in the reconstructed region of the current image to predict the pixel values ​​of pixels within the current block. Since objects in neighboring frames have some correlation, inter-frame prediction involves the encoder searching for the position of the current block in a neighboring frame, determining the difference between the two, finding a matching reference block for the current block in the current image, and using the pixel values ​​of the reference block as the predicted pixel values ​​for the current block. This process can be called motion estimation (ME). Furthermore, the encoder needs to transmit a motion vector (MV) pointing to the reference block, as well as indication information of the image containing the reference block. The ME process requires the encoder to try multiple reference blocks in the reference image for the current block. The encoder can determine which reference block(s) to use for prediction using rate-distortion optimization (RDO) or other methods. After obtaining prediction information using intra-frame or inter-frame prediction, the encoder subtracts the corresponding prediction information from the pixel value of the current pixel in the current block to obtain residual information. Then, the encoder transforms the residual information using methods such as discrete cosine transformation (DCT) and then uses quantization entropy encoding to obtain the bitstream.

[0080] The video decoding process can be understood as the inverse of the video encoding process performed by the encoding end. For example, corresponding to the encoding process described above, the decoding end can use entropy decoding, inverse quantization, and inverse transform to obtain residual information, determining whether the current block uses intra-frame prediction or inter-frame prediction. If the current block uses intra-frame coding, the decoding end can use the pixel values ​​of pixels in the surrounding reconstructed regions within the current image to construct prediction information according to the intra-frame prediction method used. If the current block uses inter-frame coding, the decoding end can parse out motion information and use the parsed motion information to determine a reference block in the reconstructed image, using the pixel values ​​of pixels within the reference block as prediction information; this process is called motion compensation (MC). The decoding end uses the prediction information plus the residual information and performs filtering to obtain the reconstructed information.

[0081] It should be understood that the "current block" mentioned above refers to the block currently being processed. For example, in the encoding process, the current block refers to the block currently being encoded; correspondingly, in the decoding process, the current block refers to the block being decoded. The "reference block" mentioned above is the block that provides a reference signal for the current block. During the search process, the encoder can traverse multiple reference blocks to find the optimal reference block. The block that provides prediction for the current block is called the prediction block. For example, after traversing multiple reference blocks, the encoder finds the optimal reference block, which will provide prediction for the current block. The pixel values, sample values, or sample signals within the prediction block are called prediction signals.

[0082] Current video encoding and decoding schemes employ uniform sampling when downsampling the image sequence of the original video in the temporal domain. For example, every K frames skipped, one frame is retained (K is an integer greater than or equal to 1), and the encoded end encodes the retained image as a low frame rate image. While this method has low complexity, it is prone to losing keyframes containing important information, resulting in low encoding and decoding accuracy, inflexibility, and low efficiency.

[0083] In view of this, this application provides a video encoding and decoding method and apparatus. The encoding end encodes key frames and their temporal location information together into the bitstream and transmits it to the decoding end. The decoding end can reconstruct the reconstructed image sequence of the original video based on the temporal location information of the key frames, making the selection of key frames more flexible and improving both encoding efficiency and system performance.

[0084] Before introducing the methods and apparatus provided in the embodiments of this application, the following points should be made first.

[0085] First, in the embodiments shown below, the terms and English abbreviations, such as keyframe and temporal location information, are merely exemplary examples given for ease of description and should not constitute any limitation on this application. This application does not preclude the possibility of defining other terms that can achieve the same or similar functions in existing or future agreements.

[0086] Second, in the embodiments shown below, the terms "first," "second," and various numerical designations are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. For example, "first neural network model," "second neural network model," etc., distinguish different neural networks.

[0087] Third, "at least one" means one or more, while "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can mean: a, or b, or c, or a and b, or a and c, or b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0088] To make the objectives and technical solutions of this application clearer and more intuitive, the video encoding and decoding method and apparatus provided in this application will be described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0089] Figure 2 This is a schematic flowchart of the video encoding method 200 in an embodiment of this application. Figure 2 As shown, the method 200 may include the following steps:

[0090] S201. The encoding end performs temporal downsampling on the image sequence of the original video to obtain N key frames, where N ≥ 2 and is an integer. The above image sequence includes X image frames, where X is an integer greater than N.

[0091] The N keyframes can be understood as a subset of frames that embody the main information of the original video. For example, the N keyframes could be the frames containing key actions in the movement or change of an object.

[0092] S202. The encoding end only encodes the encoding information of the above N key frames and the temporal position information of the above N key frames into the bitstream. The temporal position information of the above N key frames is used to indicate the position of the above N key frames in the above image sequence.

[0093] S203. The encoding end sends the above-mentioned bitstream to the decoding end; correspondingly, the decoding end receives the bitstream.

[0094] S204. The decoding end decodes the reconstructed image of N key frames and the temporal location information of the N key frames from the above bitstream.

[0095] S205. The decoding end performs temporal upsampling on the temporal location information of the above N key frames and the reconstructed image of the above N key frames to obtain the reconstructed image sequence of the above original video. The reconstructed image sequence includes X image frames, where X is an integer greater than N.

[0096] In this embodiment, the encoding end can process the image sequence of the original video, obtaining not only N keyframes of the image sequence but also the temporal location information of the N keyframes. It should be understood that the temporal location information here can be used to represent the specific position of the N keyframes in the image sequence of the original video, for example, representing the specific order or playback sequence of the N keyframes in the image sequence of the original video.

[0097] The encoding end encodes the obtained N keyframes and can transmit only the encoded information and temporal location information of the N keyframes to the decoding end via the bitstream. It should be understood that the encoding method of the encoding end can be implemented using existing encoding standards, such as H.263, H.264, H.265, H.266, MPEG-2, MPEG-4, VP8, or VP9, ​​or other encoding standards. This application embodiment does not limit the specific encoding method used.

[0098] After receiving the bitstream from the encoding end, the decoding end parses the bitstream to obtain the reconstructed images of N keyframes and the temporal location information of the N keyframes. Based on this, it performs temporal upsampling to obtain the reconstructed image sequence of the original video. It should be understood that the decoding standard used by the decoding end corresponds to the encoding standard selected by the encoding end.

[0099] For example, if the encoding end uses H.264 intra-frame coding to encode a block, i.e., the current block, the decoding end needs to decode the encoding information of the current block, such as the encoding mode, prediction mode, and quantized transform coefficients, from the bitstream. Accordingly, in addition to performing inverse quantization and inverse transform on the quantized transform coefficients to obtain the reconstructed residual signal, the decoding end also needs to perform a prediction operation based on the prediction mode to obtain the prediction signal, and finally add the two together to obtain the reconstructed image block at the original resolution.

[0100] Compared to existing technologies, the video encoding method of this application encodes keyframes and their temporal location information together into the bitstream at the encoding end and transmits it to the decoding end. The decoding end can reconstruct the reconstructed image sequence of the original video based on the temporal location information of the keyframes. This approach makes the selection of keyframes more flexible and improves both encoding efficiency and system performance.

[0101] In this application embodiment, the encoding end can obtain the above-mentioned N keyframes in various different ways, and this application embodiment does not limit this. The process of the encoding end obtaining the above-mentioned N keyframes will be described in detail below.

[0102] In the first possible design, the encoder can determine the N keyframes based on the probability value of each image in the above image sequence as a keyframe.

[0103] As an optional embodiment, the encoding end performs temporal downsampling on the image sequence of the original video to obtain N key frames, including: the encoding end determines the N key frames as the probability value of each image in the image sequence being a key frame.

[0104] For example, the encoder can directly select the N frames with the highest probability values ​​as keyframes. That is, the encoder can sort the probability values ​​of each frame in the image sequence of the original video as keyframes from largest to smallest, and then select the N frames with the highest probability values ​​as the aforementioned N keyframes.

[0105] In this embodiment, the encoding end flexibly obtains N key frames based on the probability value of each image frame in the image sequence as a key frame. This helps to avoid missing key frames containing important information in the image sequence of the original video and improves encoding efficiency.

[0106] The probability value of each frame in the above image sequence as a keyframe can be obtained in the following manner according to the embodiments of this application.

[0107] As an optional embodiment, the encoding end can input the above image sequence into a pre-trained first neural network model, which is used to determine the probability value of each frame in the image sequence as a keyframe. The encoding end can obtain the probability value of each frame in the above image sequence as a keyframe from the first neural network model.

[0108] For example, the encoding end can input the pixel values ​​of each image sequence in the original video sequence into a pre-trained first neural network model to obtain the probability distribution P of the image sequence of the original video. i Among them, the probability distribution P i It is a two-dimensional vector. If the image sequence of the original video contains X frames, then P i This includes X element values ​​(i, j), each element value (i, j) representing the probability j of an image with temporal location information i being a keyframe. Here, i is an integer greater than or equal to 1 and less than or equal to X. The encoding end obtains the probability distribution P of the X frames. i Then, a probability value P can be selected from the X frames of images. i J frames of images exceeding a preset threshold are used to randomly select N frames as the aforementioned N keyframes. Alternatively, the encoding end can select N frames with higher probability values ​​from the X frames as the aforementioned N keyframes. In this implementation, the obtained N keyframes are more accurate and contain more key information.

[0109] Figure 3The structure of the first neural network model obtained through pre-training is shown below, as follows: Figure 3 As shown, the first neural network model mainly includes N convolutional layers, one adaptive average pooling layer, and one normalized exponential layer. The convolutional layers perform inner product operations on pixels to extract features and generate feature maps, which are then input into the adaptive average pooling layer. The adaptive average pooling layer transforms the feature map into a multi-dimensional vector, which is then input into the normalized exponential layer (softmax). The normalized exponential layer obtains the probability distribution P of the intra-frame prediction mode based on the multi-dimensional vector. i .

[0110] In a second possible design, the encoding end can divide the above image sequence into N groups, and select one frame from each of the N groups to determine the N keyframes. Specifically, the encoding end can randomly select one frame from each group, or it can select one frame from each group according to certain rules; this application does not limit this.

[0111] As an optional embodiment, the encoding end performs temporal downsampling on the image sequence of the original video to obtain N keyframes, including: the encoding end groups the image sequence into N groups of images, each of the N groups of images including at least one image frame. The encoding end inputs the N groups of images into a pre-trained first neural network model, which is used to determine the probability value of each image frame in the N groups of images as a keyframe. Based on the probability value of each image frame in the N groups of images as a keyframe, the encoding end determines the image frame with the highest probability value in each of the N groups of images as the keyframe.

[0112] It should be understood that the encoding end can divide the above image sequence into N groups of images on an equal basis, that is, each group of images has the same number of frames. Alternatively, the encoding end can divide the above image sequence into N groups according to certain rules, in which case the number of frames in each group may be equal or unequal. This application embodiment does not limit this. Then, the encoding end can input the above N groups of images into the pre-trained first neural network model, and take the frame with the highest probability value in each of the above N groups of images as the keyframe, thereby obtaining the above N keyframes.

[0113] It should be understood that, in the embodiments of this application, the method by which the encoding end groups the above-mentioned image sequence can be determined with reference to the performance of the encoding end. For example, if the encoding end has superior performance and can process a long image sequence at once, then when grouping the above-mentioned image sequence, the encoding end can appropriately increase the number of frames in each group of images, thereby improving encoding efficiency. Alternatively, if the encoding end has poor performance and processing a long image sequence is time-consuming and laborious, then when grouping the above-mentioned image sequence, the encoding end can appropriately reduce the number of frames in each group of images. Although the number of groups increases, the processing time for each group of images is shorter, the requirements for the performance of the encoding end are not high, and the encoding efficiency is effectively improved.

[0114] As an optional embodiment, the encoding end groups the above image sequence to obtain N groups of images, including: the encoding end determines N-1 scene switching frames based on the probability value of each frame in the above image sequence as a scene switching frame; the encoding end groups the above image sequence according to the above N-1 scene switching frames to obtain the above N groups of images.

[0115] It should be understood that scene transition frames can be used to identify different scenes in an image sequence. Scene transition frames may also be called scene transition points or other names, which are not limited in the embodiments of this application.

[0116] For example, assuming the above image sequence contains iconic scenes, such as indoor and outdoor scenes, the encoding end can select scene switching frames from the images containing indoor and outdoor scenes respectively, and group the image sequence according to the scene switching frames so that each group of images contains different scenes.

[0117] In this embodiment of the application, the encoding end can group the image sequence based on the scene switching frame. In other words, the encoding end can divide the above image sequence into N groups of images containing different scenes, which effectively avoids the omission of images containing key scenes, thereby effectively improving the encoding and decoding efficiency.

[0118] As an optional embodiment, before determining N-1 scene switching frames based on the probability value of each image frame in the above image sequence as a scene switching frame, the method further includes: the encoding end inputting the above image sequence into a pre-trained second neural network model, which is used to determine the probability value of each image frame in the image sequence as a scene switching frame. The encoding end obtains the probability value of each image frame in the above image sequence as a scene switching frame from the second neural network model.

[0119] It should be understood that the second neural network model here can be trained in advance with a large number of scenes, enabling it to effectively identify and judge the scenes in the images during use, thereby outputting the probability value of each frame in the image sequence as a scene switching frame. Therefore, the encoding end effectively identifies scenes in the image sequence through the second neural network model, that is, it identifies images containing different scenes, obtains images containing key information, and improves video encoding efficiency.

[0120] In a third possible design, the encoder can perform temporal downsampling on the image sequence based on motion detection of adjacent images to obtain the N keyframes. Optionally, the encoder can use mean absolute deviation (MAD) to measure the motion complexity of the original video image sequence, use the mean of MAD between adjacent images to measure the overall change in motion of several frames in the original video sequence, and use the standard deviation of MAD to measure the motion fluctuation between the current image and previous images. The encoder can then perform frame extraction or frame retention operations on the image series of the original video based on the overall change in motion of the image sequence and the motion fluctuation between images, ultimately obtaining the keyframes.

[0121] For example, the encoder can group five adjacent frames in the original video sequence, keeping the first frame as the reference frame. The encoder then compares the next four frames with their respective preceding frames. If the mean of the four MAD values ​​of the next four frames relative to their respective preceding frames is less than threshold 'a' and the standard deviation is less than 'b', it indicates that the motion of the adjacent five frames is particularly smooth. The encoder then removes the middle three frames and outputs the temporal position information of the remaining images. Otherwise, the encoder continues to determine if the mean of the three MAD values ​​of the next three frames relative to their respective preceding frames is less than threshold 'c' and the standard deviation is less than 'd', indicating that the motion of the next four frames is relatively smooth. The encoder then removes the middle two frames of the next four frames and outputs the temporal position information of the remaining images. Otherwise, it indicates that the image sequence is moving violently or even has a scene jump, and the encoder does not remove any frames and outputs the temporal position information of the remaining images.

[0122] As an optional embodiment, the temporal position information of the aforementioned N keyframes includes any of the following: the POC of the aforementioned N keyframes, the position index of the aforementioned N keyframes, or the time interval between the aforementioned N keyframes and the preset image. Specifically, the POC of the N keyframes indicates the playback sequence number of the N keyframes in the aforementioned image sequence; the position index of the N keyframes indicates the temporal position of the N keyframes in the aforementioned image sequence; and the time interval is the time interval between each frame of the N keyframes and the preset image.

[0123] In this embodiment of the application, the encoding end obtains the above-mentioned N key frames according to different temporal location information identifiers, which can be achieved in the following three possible ways.

[0124] In the first possible implementation, the temporal location information of the aforementioned N keyframes is the POC. Figure 4 This is a schematic diagram illustrating a keyframe acquisition process provided in an embodiment of this application, as shown below. Figure 4 As shown, the image sequence of the original video is initially identified using POCs. The encoding end performs temporal downsampling on the image sequence of the original video to obtain N key frames and their temporal position information. Here, the temporal position information of the N key frames can be represented by the temporal position position_n[i] corresponding to the image, where position_n[i] represents the POC of the (i+1)th key image, and 0 ≤ i < N. Figure 4 As shown, the temporal positions of the N keyframes obtained are position_n[0]=0, position_n[1]=2,…,position_n[N-1]=X-1.

[0125] In the second possible implementation, the temporal location information of the aforementioned N keyframes is the time interval between the N keyframes and the preset image. Figure 5 This is a schematic diagram illustrating another keyframe acquisition process provided in an embodiment of this application, as shown below. Figure 5 As shown, the image sequence of the original video is initially identified using POC. The encoding end performs temporal downsampling on the image sequence of the original video to obtain N key frames and the temporal position information of the N key frames. Here, the temporal position information of the N key frames can be represented by the time interval time_n[i] between the key frame and the preset image. This time_n[i] represents the time interval between the i-th key frame and the first image in the image sequence of the original video, where 0≤i<N. Figure 5 As shown, the time-domain positions of the final N keyframes are time_n[0] = 0, time_n[1] = 3, ..., time_n[N-1] = X-1. It should be understood that the preset image here can be the first frame image in the image sequence of the original video, or the last frame image in the image sequence of the original video, or any frame image in the image sequence of the original video. This application embodiment does not limit this.

[0126] In the third possible implementation, the temporal location information of the aforementioned N keyframes is a location index. Figure 6 A schematic diagram illustrating another keyframe acquisition process provided in this application embodiment, as shown below. Figure 6As shown, the image sequence of the original video is initially identified by POC. The encoding end performs temporal downsampling on the image sequence of the original video to obtain N key frames and the temporal position information of the N key frames. Here, the temporal position information of the N key frames can be represented by the key frame position index index_I[i]. This index_I[i] represents the position of the i-th key frame in the original video image sequence, where 0≤i<N.

[0127] Optionally, the N keyframes can be extracted from the original video sequence, or the N keyframes can be generated by fusion and are not in the original video's image sequence, such as... Figure 6 As shown, when i takes values ​​from 0 to 2, the temporal location information indicated by indexI[i] is: 0, 1.5, 3. Here, when the temporal location information indicated by indexI[i] is 0 or 3, the corresponding keyframe is in the original video sequence. When the temporal location information indicated by indexI[i] is 1.5, the keyframe is not in the original video sequence, and this frame is generated by fusing the preceding and following frames.

[0128] It should be understood that, while obtaining the temporal location information of the N keyframes, the temporal location information of the remaining M frames can also be obtained, or the temporal location information of the remaining M frames can be deduced from the temporal location information of the N keyframes.

[0129] As an optional embodiment, the encoding end encodes the encoded information and temporal location information of the aforementioned N keyframes into the bitstream, including: the encoding end encodes the encoded information and temporal location information of the aforementioned N keyframes into the following positions in the bitstream: sequence header, image header, slice header, or supplementary enhancement information. Correspondingly, the decoding end parses the temporal location information of the N keyframes from the bitstream, including: the decoding end parses the temporal location information of the N keyframes from the sequence header, image header, slice header, or supplementary enhancement information in the bitstream.

[0130] In the first possible implementation, Table 1 is the syntax table required for decoding. As shown in Table 1, the left side is the syntax, and u(1) in the descriptor on the right side is the specific parsing method. The encoding end can encode the encoding information of the N keyframes and the temporal position information position_n[i] of the N keyframes into the sequence header, image header, slice header, or supplementary enhancement information in the bitstream, and send the bitstream to the decoding end. Correspondingly, the decoding end can parse the temporal position information position_n[i] of the N keyframes from the sequence header, image header, slice header, or supplementary enhancement information in the bitstream according to the syntax table, and decode the reconstructed image of the N keyframes from the bitstream according to the temporal position information.

[0131] In Figure 4 In the example shown, the decoding end can determine the temporal position information of N key frames according to position_n[i], where i is the index of the key frame, 0 ≤ i < N. According to the syntax table shown in Table 1, the decoding end can parse and obtain the temporal position information position_n[i] indicated when i takes values from 0 to X - 1 as: 0, 2, 5, …, X - 1. The decoding end determines the positions of N key frames in the image sequence of the above original video, specifically corresponding to the images with POCs of 0, 2, 5, …, X - 1.

[0132] Table 1

[0133]

[0134] In the second possible implementation, Table 2 is the syntax table required for decoding. As shown in Table 2, on the left is the syntax, and u(2) in the descriptor on the right is the specific parsing method. The encoding end can incorporate the encoding information of N key frames and the temporal position information time_n[i] of the above N key frames into the code stream and send the code stream to the decoding end. Correspondingly, the decoding end can parse and obtain the temporal position information time_n[i] from the sequence header, picture header, slice header or supplementary enhancement information in the above code stream according to the above syntax table, and decode the reconstructed images of N key frames from the above code stream according to this temporal position information.

[0135] In Figure 5 In the example shown, the decoding end can determine the temporal position information of N key frames according to time_n[i], where i is the index of the key frame, 0 ≤ i < N. The decoding end parses and obtains the temporal position information time_n[i] indicated when i takes values from 0 to X - 1 as: 0, 3, …, X - 1. Thus, it knows the positions of N key frames in the image sequence of the above original video, specifically corresponding to several frames of images with POCs of 0, 3, …, X - 1, specifically as shown in the above Figure 5 shown.

[0136] Table 2

[0137]

[0138] In the third possible implementation, Table 3 shows the syntax table required for decoding. As shown in Table 3, the left side is the syntax, and u(3) in the descriptor on the right side is the specific parsing method. The encoding end can encode the encoding information of N keyframes and the temporal position information index_I[i] of the above N keyframes into the bitstream and send the bitstream to the decoding end. Correspondingly, the decoding end can parse the temporal position information Index_I[i] from the sequence header, image header, slice header or supplementary enhancement information in the bitstream according to the syntax table, and decode the reconstructed image of N keyframes from the bitstream according to the temporal position information.

[0139] exist Figure 6 In the example shown, the decoder can determine the temporal location information of N keyframes based on index_I[i]. The decoder parses the temporal location information indexI[i], which indicates the value of i from 0 to X-1, as: 0, 1.5, 3, ..., X-1. Therefore, the decoder knows the position of the N keyframes in the image sequence of the original video, specifically corresponding to the frames with POC values ​​of 0, 1.5, 3, ..., X-1, as detailed above. Figure 6 As shown.

[0140] Table 3

[0141]

[0142] It should be understood that the encoding end can also encode the encoded information of the N keyframes, their temporal position information, and the temporal position information of the remaining M frames into the bitstream. Correspondingly, the decoding end can not only parse the temporal position information of the N keyframes, but also parse the temporal position information of the remaining M frames. Alternatively, based on the temporal position information of the N keyframes, the decoding end can deduce the temporal position information of the remaining M frames. For example, for a video image sequence where the POC is a consecutive integer, the decoding end can directly interpolate the POC of the N keyframes to obtain the POC number of the M frames. Furthermore, the decoding end can use methods specified in existing video decoding standards, such as H.264, H.265, or H.266. Specifically, the parsing and decoding operations of the decoding end are determined by the encoding scheme selected by the encoding end.

[0143] As an optional embodiment, in S205 above, the decoding end performs temporal upsampling on the temporal position information of the above N key frames and the reconstructed images of the N key frames to obtain the reconstructed image sequence of the above original video, including: the decoding end interpolates the reconstructed images of the N key frames and the temporal position information of the N key frames, and outputs K frames of images. Here, K ≤ M and is an integer, and the above M frames of images are the images in the above image sequence except for the N key frames, where M ≥ 1 and is an integer. The decoding end obtains the reconstructed image sequence of the above original video according to the reconstructed images of the above N key frames and the above K frames of images.

[0144] When the encoding method adopted by the encoding end corresponding to the decoding end is lossy compression (also known as irreversible encoding), the decoding end can interpolate the reconstructed images of the N key frames and the temporal position information of the N key frames, and output K frames of images, where K < M and is an integer.

[0145] It should be understood that the above lossy compression means that when the decoding end reconstructs (i.e., decodes) the compressed data, the reconstructed data is different from the original data, but it does not affect people's misunderstanding of the information expressed by the original data. That is to say, the decoded image and the original image are different, with a certain degree of distortion, but the visual effect of this degree of distortion is generally acceptable. Lossy compression has a wide range of applications, such as video conferencing, videophone, video broadcasting, video surveillance, etc.

[0146] When the encoding method adopted by the encoding end corresponding to the decoding end is lossless compression (also known as reversible encoding). The decoding end can interpolate the reconstructed images of the N key frames and the temporal position information of the N key frames, and output K frames of images, where K = M and is an integer.

[0147] It should be understood that the above lossless compression means that when the decoding end reconstructs (i.e., decodes) the compressed data, the reconstructed data is exactly the same as the original data. That is to say, the decoded image and the original image are strictly the same, and the compression is completely recoverable or unbiased, without distortion. Lossless compression is used in situations where it is required that the reconstructed signal is exactly the same as the original signal, such as the compression of disk files.

[0148] As an optional embodiment, the decoding end interpolates the reconstructed images of the above N key frames and the temporal position information of the N key frames to obtain K frames of images, including: the decoding end inputs the reconstructed images of the above N key frames, the temporal position information of the N key frames, and the temporal position information of the above K frames of images into a pre-trained third neural network model to obtain the above K frames of images, and the third neural network model is used to obtain the reconstructed images of the images in the original video.

[0149] Combined with the above Figure 4In the example shown, the decoding end can input the reconstructed image information and temporal position information position_n[i] of N key images into the third neural network. The neural network generates new video images based on the input information, that is, the reconstructed video sequence of the original video sequence.

[0150] In combination with the above Figure 5 In the example shown, the decoding end can input the reconstructed image information and temporal location information time_n[i] of N key images into the third neural network. The neural network generates a new video image sequence based on the input information, that is, the reconstructed video image sequence of the original video image sequence.

[0151] In combination with the above Figure 6 In the example shown, the decoding end can input the reconstructed image information and temporal location information indexI[i] of N key images into the third neural network. The neural network generates new video images based on the input information, that is, the reconstructed video sequence of the original video sequence.

[0152] For example, the decoding end can input the reconstructed images of at least two decoded keyframes and their temporal location information into the third neural network model. Feature extraction can be performed through the encoder and decoder in the network. The extracted features are then input into four sub-networks to obtain four one-dimensional convolution kernels for each pixel. These four convolution kernels are then convolved with the reconstructed images to obtain the pixel value of each pixel. Finally, the entire image is output, which is located between the temporal locations of the two keyframes.

[0153] It should be understood that the decoding end can also input the reconstructed image information of N key frames, the temporal location information of N key frames, and the temporal location information of the remaining M frames into the third neural network mentioned above.

[0154] In the above embodiments, before using the first neural network and the third neural network, the networks can be trained on an NVIDIA GTX 1080Ti graphics processing unit (GPU) using PyTorch software. It should be understood that the first neural network is a temporal downsampling network, and the third neural network is a temporal upsampling network. In this embodiment, the temporal upsampling network and the temporal downsampling network can be jointly trained, and the images in the UCID image library and the DIV2K dataset can be compressed using VTM reference software. First, the input and target data of the first neural network are extracted. The input of the first neural network is the pixel values ​​of the image sequence of the original video and the corresponding temporal location information. The output of the first neural network is the probability distribution of images as key images in the image sequence of the original video. The N key images with the highest probabilities and their corresponding temporal location information are selected as the input of the third neural network at the decoding end. Finally, the third neural network at the decoding end outputs the reconstructed pixel values ​​of the images in the image sequence of the original video excluding the N key frames. Specifically, the loss function shown in the following formula can be used to train the neural network:

[0155]

[0156] Where the number of frames in the original video image sequence is X, N is the number of frames in the key image, and XN is the number of frames discarded by the first neural network, i.e., the number of frames that the third neural network at the decoding end needs to recover, I i This represents the i-th frame of image discarded by the first neural network at the encoding end. This represents the reconstructed image of the i-th frame recovered by the third neural network at the decoding end. L2Loss indicates the mean squared error (MSE) function. This represents the mean square error between the frames discarded by the first neural network at the encoding end and the reconstructed image recovered by the third neural network at the decoding end.

[0157] The above text combines Figures 2 to 6 The video encoding and decoding method of the embodiments of this application is described in detail below. Figures 7 to 9 This application describes in detail the video encoding and decoding apparatus according to embodiments of the present application.

[0158] Figure 7 The present application shows a video encoding device 700 provided in an embodiment of the present application. The device 700 includes: an acquisition module 701 and an encoding module 702.

[0159] The acquisition module 701 is used to perform temporal downsampling on the image sequence of the original video to acquire N keyframes, where N ≥ 2 and is an integer. The image sequence includes X image frames, where X is an integer greater than N. The encoding module 702 is used to encode only the encoding information and temporal position information of the N keyframes into the bitstream. The temporal position information of the N keyframes is used to indicate the position of the N keyframes in the image sequence.

[0160] Optionally, the acquisition module 701 is used to determine N frames of key frames whose probability values ​​are greater than or equal to a preset threshold based on the probability value of each frame in the above image sequence as a key frame.

[0161] Optionally, the acquisition module 701 is specifically used to input the image sequence into a pre-trained first neural network model before determining the N frames of images with a probability value greater than or equal to a preset threshold as the N key frames. The first neural network model is used to determine the probability value of each frame of the image sequence as a key frame; and to acquire the probability value of each frame of the image sequence as a key frame from the first neural network model.

[0162] Optionally, the acquisition module 701 is specifically used to group the above image sequence to obtain N groups of images, each of the N groups of images including at least one frame; input the above N groups of images into a pre-trained first neural network model, the first neural network model being used to determine the probability value of each frame in the image sequence as a keyframe; and, based on the probability value of each frame in the above N groups of images as a keyframe, determine the frame with the highest probability value in each of the above N groups of images as the keyframe.

[0163] Optionally, the acquisition module 701 is specifically used to determine N-1 scene switching frames based on the probability value of each image in the above image sequence as a scene switching frame; and to group the above image sequence according to the above N-1 scene switching frames to obtain the above N groups of images.

[0164] Optionally, the acquisition module 701 is specifically used to input the image sequence into a pre-trained second neural network model before determining N-1 scene switching frames based on the probability value of each frame in the image sequence as a scene switching frame; the second neural network model is used to determine the probability value of each frame in the image sequence as a scene switching frame; and to acquire the probability value of each frame in the image sequence as a scene switching frame from the second neural network model.

[0165] Optionally, the acquisition module 701 is specifically used to acquire the N keyframes from the image sequence based on motion detection of adjacent images in the image sequence.

[0166] Optionally, the encoding module 702 is specifically used to encode the encoding information of the above-mentioned N keyframes and the temporal location information of the above-mentioned N keyframes into the following positions in the above-mentioned bitstream: sequence header, image header, slice header or supplementary enhancement information.

[0167] Optionally, the temporal position information of the aforementioned N keyframes includes any of the following: the image sequence count (POC) of the aforementioned N keyframes, the position index of the aforementioned N keyframes, or the time interval between the aforementioned N keyframes and the preset image; wherein, the POC of the aforementioned N keyframes is used to indicate the playback sequence number of the aforementioned N keyframes in the aforementioned image sequence; the position index of the aforementioned N keyframes is used to indicate the temporal position of the aforementioned N keyframes in the aforementioned image sequence; and the aforementioned time interval is the time interval between each frame of the aforementioned N keyframes and the aforementioned preset image.

[0168] It should be understood that the device 700 here is embodied in the form of a functional module. The term "module" here can refer to application-specific integrated circuits (ASICs), electronic circuits, processors (e.g., shared processors, proprietary processors, or group processors, etc.) and memories for executing one or more software or firmware programs, integrated logic circuits, and / or other suitable components supporting the described functions. In an alternative example, those skilled in the art will understand that the device 700 may specifically be the encoding end in the above embodiments, or the functions of the encoding end in the above embodiments may be integrated into the device 700. The device 700 may be used to execute the various processes and / or steps corresponding to the encoding end in the above method embodiments; to avoid repetition, these will not be described further here.

[0169] The aforementioned device 700 has the function of implementing the corresponding steps executed by the encoding end in the aforementioned method; the aforementioned function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned function.

[0170] In the embodiments of this application, Figure 7 The device 700 in the text can also be a chip or a chip system, such as a system on chip (SoC).

[0171] Figure 8 The present application shows a video decoding device 800, which includes a decoding module 801 and a processing module 802.

[0172] The decoding module 801 is used to decode the reconstructed images of N keyframes and the temporal position information of the N keyframes from the bitstream. The temporal position information of the N keyframes indicates the position of the N keyframes in the image sequence of the original video, where N ≥ 2 and is an integer. The processing module 802 is used to perform temporal upsampling on the temporal position information of the N keyframes and the reconstructed images of the N keyframes to obtain a reconstructed image sequence of the original video. The reconstructed image sequence includes X image frames, where X is an integer greater than N.

[0173] Optionally, the decoding module 801 is used to parse the temporal location information of the N key frames from the bitstream; and to decode the reconstructed image of the N key frames from the bitstream based on the temporal location information of the N key frames.

[0174] Optionally, the decoding module 801 is used to parse the temporal location information of the N key frames from the sequence header, image header, slice header or supplementary enhancement information in the bitstream.

[0175] Optionally, the temporal position information of the aforementioned N keyframes includes any of the following: the image sequence count (POC) of the aforementioned N keyframes, the position index of the aforementioned N keyframes, or the time interval between the aforementioned N keyframes and the preset image; wherein, the POC of the aforementioned N keyframes is used to indicate the playback sequence number of the aforementioned N keyframes in the aforementioned image sequence; the position index of the aforementioned N keyframes is used to indicate the temporal position of the aforementioned N keyframes in the aforementioned image sequence; and the aforementioned time interval is the time interval between each frame of the aforementioned N keyframes and the aforementioned preset image.

[0176] Optionally, the processing module 802 is used to interpolate the reconstructed images of the N keyframes and the temporal location information of the N keyframes to obtain K frames, where K ≤ M and is an integer, and the M frames are the images in the image sequence excluding the N keyframes, where M ≥ 1 and is an integer; and to obtain the reconstructed image sequence of the original video based on the reconstructed images of the N keyframes and the K frames.

[0177] Optionally, the processing module 802 is used to input the reconstructed images of the N key frames and the temporal location information of the N key frames into a pre-trained third neural network model to obtain the K-frame images. The third neural network model is used to obtain the reconstructed images of the images in the original video.

[0178] It should be understood that the device 800 here is embodied in the form of a functional module. The term "module" here can refer to application-specific integrated circuits (ASICs), electronic circuits, processors (e.g., shared processors, proprietary processors, or group processors, etc.) and memories for executing one or more software or firmware programs, integrated logic circuits, and / or other suitable components supporting the described functions. In an alternative example, those skilled in the art will understand that the device 800 may specifically be the decoding end in the above embodiments, or the functions of the decoding end in the above embodiments may be integrated into the device 800. The device 800 may be used to execute the various processes and / or steps corresponding to the decoding end in the above method embodiments; to avoid repetition, these will not be described further here.

[0179] The aforementioned device 800 has the function of implementing the corresponding steps performed by the decoding end in the above method; the above functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0180] In the embodiments of this application, Figure 8 The device 800 in the text can also be a chip or a chip system, such as a system on chip (SoC).

[0181] Figure 9 Another video encoding / decoding apparatus 900 provided in this application embodiment is illustrated. The apparatus 900 includes a processor 901, a memory 902, a communication interface 903, and a bus 904. The memory 902 stores instructions, and the processor 901 executes the instructions stored in the memory 902. The processor 901, memory 902, and communication interface 903 are interconnected via the bus 904.

[0182] In one implementation, the device 900 may specifically be the encoding end in the above embodiments, wherein the processor 901 is used to perform downsampling on the image sequence of the original video to obtain N key frames, where N≥2 and is an integer, and the image sequence includes X image frames, where X is an integer greater than N; and to encode the encoding information of the N key frames and the temporal position information of the N key frames into the bitstream, wherein the temporal position information of the N key frames is used to indicate the position of the N key frames in the image sequence.

[0183] In another implementation, the device 900 can be specifically the decoding end in the above embodiment, wherein the processor 901 is used to decode the reconstructed image of N key frames and the temporal position information of the N key frames from the bitstream. The temporal position information of the N key frames is used to indicate the position of the N key frames in the image sequence of the original video, where N≥2 and is an integer. The processor 901 is used to perform temporal upsampling on the temporal position information of the N key frames and the reconstructed image of the N key frames to obtain the reconstructed image sequence of the original video. The reconstructed image sequence includes X image frames, where X is an integer greater than N.

[0184] It should be understood that the device 900 may specifically be the encoding end or decoding end in the above embodiments, or the functions of the encoding end or decoding end in the above embodiments may be integrated into the device 900. The device 900 may be used to execute the various steps and / or processes corresponding to the encoding end or decoding end in the above method embodiments.

[0185] Optionally, the memory 902 may include read-only memory and random access memory, and provide instructions and data to the processor 901. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information. The processor 901 can be used to execute instructions stored in the memory, and when the processor executes the instructions, the processor 901 can perform the various steps and / or processes corresponding to the encoding or decoding end in the above method embodiments.

[0186] It should be understood that in the embodiments of this application, the processor can be a Central Processing Unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The steps of the method disclosed in the embodiments of this application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor executes the instructions in the memory, combining with its hardware to complete the steps of the above method. To avoid repetition, detailed descriptions are not provided here.

[0187] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0188] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0189] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0190] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0191] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0192] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video encoding method, characterized in that, include: Temporal downsampling is performed on the image sequence of the original video to obtain N keyframes, where N ≥ 2 and is an integer. The image sequence includes X image frames, where X is an integer greater than N. The encoding information of the N keyframes and the temporal position information of the N keyframes are encoded into the bitstream. The temporal position information of the N keyframes is used to indicate the position of the N keyframes in the image sequence. The step of performing temporal downsampling on the image sequence of the original video to obtain N keyframes includes: The image sequence is input into a pre-trained first neural network model, which is used to determine the probability value of each frame in the image sequence as a keyframe. The probability value of each frame in the image sequence as a key frame is obtained from the first neural network model, and N frames with the probability value greater than or equal to a preset threshold are determined as the N key frames. or, Based on the probability value of each image in the image sequence as a scene switching frame, N-1 scene switching frames are determined; The image sequence is grouped according to the N-1 scene switching frames to obtain N groups of images, and each group of images includes at least one frame of images; The N sets of images are respectively input into a pre-trained first neural network model, which is used to determine the probability value of each frame in the image sequence as a keyframe. Based on the probability value of each frame in the N groups of images as a keyframe, the frame with the highest probability value in each of the N groups of images is determined as the N keyframes.

2. The method of claim 1, wherein, Before determining N-1 scene switching frames based on the probability value of each image in the image sequence as a scene switching frame, the method further includes: The image sequence is input into a pre-trained second neural network model, which is used to determine the probability value of each frame in the image sequence as a scene switching frame. The probability value of each frame in the image sequence as a scene switching frame is obtained from the second neural network model.

3. The method according to claim 1 or 2, characterized in that, The step of encoding the coding information and temporal location information of the N keyframes into the bitstream includes: The encoding information and temporal location information of the N keyframes are incorporated into the following locations in the bitstream: sequence header, image header, slice header, or supplementary enhancement information.

4. The method of claim 3, wherein, The temporal location information of the N keyframes includes any of the following: The image sequence count (POC) of the N keyframes, the position index of the N keyframes, or the time interval between the N keyframes and the preset image. Wherein, the POC of the N keyframes is used to indicate the playback sequence number of the N keyframes in the image sequence; the position index of the N keyframes is used to indicate the temporal position of the N keyframes in the image sequence; the time interval is the time interval between each frame of the N keyframes and the preset image.

5. A method of video decoding, the method comprising: include: The reconstructed images of N keyframes and the temporal position information of the N keyframes are decoded from the bitstream. The temporal position information of the N keyframes is used to indicate the position of the N keyframes in the image sequence of the original video, where N≥2 and is an integer. Temporal upsampling is performed on the temporal location information of the N keyframes and the reconstructed images of the N keyframes to obtain a reconstructed image sequence of the original video. The reconstructed image sequence includes X image frames, where X is an integer greater than N. The N keyframes are determined by inputting the image sequence into a pre-trained first neural network model to determine the probability value of each frame as a keyframe. After obtaining the probability value of each frame from the first neural network model, the N keyframes are selected from the N images whose probability values ​​are greater than or equal to a preset threshold. or, The N keyframes are determined by identifying N-1 scene switching frames based on the probability value of each image in the image sequence as a scene switching frame, then grouping the image sequence into N groups of images based on these N-1 scene switching frames, and then inputting each of the N groups of images into a pre-trained first neural network model to determine the probability value of each frame as a keyframe, and finally selecting the image with the highest probability value in each group of images as the N keyframes.

6. The method of claim 5, wherein, The process of decoding the reconstructed image of N keyframes from the bitstream and the temporal location information of the N keyframes includes: The temporal location information of the N keyframes is parsed from the bitstream; Based on the temporal location information of the N keyframes, the reconstructed images of the N keyframes are decoded from the bitstream.

7. The method according to claim 6, characterized in that, The step of parsing the temporal location information of the N keyframes from the bitstream includes: The temporal location information of the N keyframes is obtained by parsing the sequence header, image header, slice header, or supplementary enhancement information in the bitstream.

8. The method according to any one of claims 5 to 7, characterized in that, The temporal location information of the N keyframes includes any of the following: The image sequence count (POC) of the N keyframes, the position index of the N keyframes, or the time interval between the N keyframes and the preset image. Wherein, the POC of the N keyframes is used to indicate the playback sequence number of the N keyframes in the image sequence; the position index of the N keyframes is used to indicate the temporal position of the N keyframes in the image sequence; the time interval is the time interval between each frame of the N keyframes and the preset image.

9. The method according to any one of claims 5 to 8, characterized in that, Temporal upsampling is performed on the temporal location information of the N keyframes and the reconstructed images of the N keyframes to obtain the reconstructed image sequence of the original video, including: Interpolate the reconstructed images of the N keyframes and the temporal location information of the N keyframes to obtain K frames, where K ≤ M and is an integer, and the M frames are the images in the image sequence excluding the N keyframes, where M ≥ 1 and is an integer. Based on the reconstructed images of the N keyframes and the K-frame images, the reconstructed image sequence of the original video is obtained.

10. The method of claim 9, wherein, The step of interpolating the reconstructed images of the N keyframes and the temporal location information of the N keyframes to obtain K-frame images includes: The reconstructed images of the N keyframes and the temporal location information of the N keyframes are input into a pre-trained third neural network model to obtain the K-frame images. The third neural network model is used to obtain the reconstructed image sequence of the image sequence in the original video.

11. A video encoding apparatus, comprising: include: The acquisition module is used to perform temporal downsampling on the image sequence of the original video to acquire N key frames, where N≥2 and is an integer. The image sequence includes X image frames, where X is an integer greater than N. The encoding module is used to encode the encoding information of the N key frames and the temporal position information of the N key frames into the bitstream, wherein the temporal position information of the N key frames is used to indicate the position of the N key frames in the image sequence. The acquisition module is used for: The image sequence is input into a pre-trained first neural network model, which is used to determine the probability value of each frame in the image sequence as a keyframe. The probability value of each frame in the image sequence as a key frame is obtained from the first neural network model, and N frames with the probability value greater than or equal to a preset threshold are determined as the N key frames. or, Based on the probability value of each frame in the image sequence as a scene switching frame, N-1 scene switching frames are determined; the image sequence is grouped according to the N-1 scene switching frames to obtain N groups of images, and each group of images includes at least one frame. The N sets of images are respectively input into a pre-trained first neural network model, which is used to determine the probability value of each frame in the image sequence as a keyframe. Based on the probability value of each frame in the N groups of images as a keyframe, the frame with the highest probability value in each group of N images is determined as the N keyframes.

12. The apparatus of claim 11, wherein, The acquisition module is specifically used for: Before determining N-1 scene switching frames based on the probability value of each frame in the image sequence as a scene switching frame, the image sequence is input into a pre-trained second neural network model, which is used to determine the probability value of each frame in the image sequence as a scene switching frame. The probability value of each frame in the image sequence as a scene switching frame is obtained from the second neural network model.

13. The apparatus of claim 11 or 12, wherein, The encoding module is specifically used for: The encoding information and temporal location information of the N keyframes are incorporated into the following locations in the bitstream: sequence header, image header, slice header, or supplementary enhancement information.

14. The apparatus of claim 13, wherein, The temporal location information of the N keyframes includes any of the following: The image sequence count (POC) of the N keyframes, the position index of the N keyframes, or the time interval between the N keyframes and the preset image. Wherein, the POC of the N keyframes is used to indicate the playback sequence number of the N keyframes in the image sequence; the position index of the N keyframes is used to indicate the temporal position of the N keyframes in the image sequence; the time interval is the time interval between each frame of the N keyframes and the preset image.

15. An apparatus for video decoding, the apparatus comprising: include: The decoding module is used to decode the reconstructed images of N key frames and the temporal position information of the N key frames from the bitstream. The temporal position information of the N key frames is used to indicate the position of the N key frames in the image sequence of the original video, where N≥2 and is an integer. The processing module is used to perform temporal upsampling on the temporal location information of the N key frames and the reconstructed images of the N key frames to obtain a reconstructed image sequence of the original video, wherein the reconstructed image sequence includes X image frames, where X is an integer greater than N; The N keyframes are determined by inputting the image sequence into a pre-trained first neural network model to determine the probability value of each frame as a keyframe. After obtaining the probability value of each frame from the first neural network model, the N keyframes are selected from the N images whose probability values ​​are greater than or equal to a preset threshold. or, The N keyframes are determined by identifying N-1 scene switching frames based on the probability value of each image in the image sequence as a scene switching frame, then grouping the image sequence into N groups of images based on these N-1 scene switching frames, and then inputting each of the N groups of images into a pre-trained first neural network model to determine the probability value of each frame as a keyframe, and finally selecting the image with the highest probability value in each group of images as the N keyframes.

16. The apparatus of claim 15, wherein, The decoding module is used for: The temporal location information of the N keyframes is parsed from the bitstream; Based on the temporal location information of the N keyframes, the reconstructed images of the N keyframes are decoded from the bitstream.

17. The apparatus of claim 16, wherein, The decoding module is specifically used for: The temporal location information of the N keyframes is obtained by parsing the sequence header, image header, slice header, or supplementary enhancement information in the bitstream.

18. The apparatus of any one of claims 15-17, wherein, The temporal location information of the N keyframes includes any of the following: The image sequence count (POC) of the N keyframes, the position index of the N keyframes, or the time interval between the N keyframes and the preset image. Wherein, the POC of the N keyframes is used to indicate the playback sequence number of the N keyframes in the image sequence; the position index of the N keyframes is used to indicate the temporal position of the N keyframes in the image sequence; the time interval is the time interval between each frame of the N keyframes and the preset image.

19. The apparatus of any one of claims 15-18, wherein, The processing module is used for: Interpolate the reconstructed images of the N keyframes and the temporal location information of the N keyframes to obtain K frames, where K ≤ M and is an integer. The M frames are the images in the image sequence excluding the N keyframes, and M ≥ 1 and is an integer. Based on the reconstructed images of the N keyframes and the K-frame images, the reconstructed image sequence of the original video is obtained.

20. The apparatus of claim 19, wherein, The processing module is specifically used for: The reconstructed images of the N keyframes and the temporal location information of the N keyframes are input into a pre-trained third neural network model to obtain the K-frame images. The third neural network model is used to obtain the reconstructed images of the images in the original video.

21. An apparatus for video coding, the apparatus comprising: include: A processor coupled to a memory for storing a computer program, which, when invoked by the processor, causes the apparatus to perform the method as described in any one of claims 1 to 10.

22. A computer-readable storage medium, characterized in that, Used to store a computer program, the computer program including instructions for implementing the method as described in any one of claims 1 to 10.

Images