Coding and decoding method, codec, chip, code stream and storage medium

Abstract

The invention provides a coding and decoding method, a coding and decoding device, a chip, a code stream and a storage medium. Real and accurate eye movement information can be reconstructed at a decoding end. The decoding method comprises the following steps: decoding a supplementary enhancement information SEI message corresponding to a current image to obtain pupil position information contained in the current image; and based on the position information of the pupils, determining motion information of eyes contained in the current image.

Description

Technical Field

[0001] This application relates to the field of video encoding and decoding, and more particularly to an encoding and decoding method, codec, chip, bitstream, and storage medium. Background Technology

[0002] Supplemental enhancement information (SEI) is an important coding tool that provides additional information to improve the efficiency and quality of video coding.

[0003] In generative face video coding frameworks, the SEI (Eye Matrix Information) can carry matrix information about the eyes to represent their opening and closing states and degrees. However, current methods are insufficient to represent realistic and accurate eye movement information. Summary of the Invention

[0004] This application provides an encoding / decoding method, an encoding / decoding device, a chip, a bitstream, and a storage medium, which enables the decoding end to reconstruct realistic and accurate eye movement information.

[0005] In a first aspect, a decoding method is provided, which is applied to a decoder. The decoding method includes: decoding the supplementary enhancement information (SEI) message corresponding to the current image to obtain the position information of the pupil contained in the current image; and determining the motion information of the eye contained in the current image based on the position information of the pupil.

[0006] Secondly, an encoding method is provided, which is applied to an encoder. The encoding method includes: determining the position information of the pupil contained in the current image; encoding the parameters in the supplementary enhancement information (SEI) message corresponding to the current image based on the pupil position information, wherein the pupil position information is used to determine the eye motion information contained in the current image.

[0007] Thirdly, a decoder is provided, comprising: a decoding unit configured to decode the supplementary enhancement information (SEI) message corresponding to the current image to obtain the position information of the pupil contained in the current image; and a determining unit configured to determine the motion information of the eye contained in the current image based on the position information of the pupil.

[0008] Fourthly, a decoder is provided, comprising: a memory for storing a computer program; and a processor for executing the method of the first aspect when running the computer program.

[0009] Fifthly, an encoder is provided, comprising: a determining unit configured to determine the position information of a pupil contained in a current image; and an encoding unit configured to encode parameters in a supplementary enhancement information (SEI) message corresponding to the current image based on the pupil position information, wherein the pupil position information is used to determine the eye motion information contained in the current image.

[0010] In a sixth aspect, an encoder is provided, comprising: a memory for storing a computer program; and a processor for executing the method of the second aspect when running the computer program.

[0011] In a seventh aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program that, when executed, implements the method as described in the first or second aspect.

[0012] Eighthly, a computer program product is provided, including a computer program that, when executed, implements the method as described in the first or second aspect.

[0013] A ninth aspect provides a non-volatile computer-readable storage medium for storing bit streams, the bit streams being generated by an encoding method using an encoder, or the bit streams being decoded by a decoding method using a decoder, wherein the decoding method is the method described in the first aspect and the encoding method is the method described in the second aspect.

[0014] A tenth aspect provides a computer-readable storage medium storing a computer program that, when executed, implements the method as described in the first aspect or the second aspect.

[0015] In an eleventh aspect, a bitstream is provided, including a bitstream generated according to the method described in the second aspect.

[0016] In a twelfth aspect, a decoding chip is provided, the chip including a memory and a processor, the processor being configured to call and run a program or instructions from the memory to implement some or all of the steps described in the first aspect above.

[0017] In a thirteenth aspect, an encoding chip is provided, the chip including a memory and a processor, the processor being used to retrieve and execute a program or instructions from the memory to implement some or all of the steps described in the second aspect above.

[0018] Some embodiments of this application add pupil position information to the SEI message. Since the pupil position information can accurately reflect the eye movement information, compared with the information on the opening and closing state and degree of the eye in related technologies, some embodiments of this application enable the decoding end to generate real and accurate eye movement information. Attached Figure Description

[0019] Figure 1 Here are some structural example diagrams of video encoders that can be applied to some embodiments of this application;

[0020] Figure 2 Here are some structural example diagrams of video decoders that can be applied to some embodiments of this application;

[0021] Figure 3 A structural block diagram of a generative face video codec provided for some embodiments of this application;

[0022] Figure 4 A schematic diagram of a first coordinate system provided for some embodiments of this application;

[0023] Figure 5 A flowchart illustrating the decoding method provided in some embodiments of this application;

[0024] Figure 6 A flowchart illustrating the encoding method provided in some embodiments of this application;

[0025] Figure 7 This is a schematic diagram of the decoder structure provided in one embodiment of this application;

[0026] Figure 8 A schematic diagram of the decoder provided in another embodiment of this application;

[0027] Figure 9 This is a schematic diagram of the encoder structure provided in one embodiment of this application;

[0028] Figure 10 This is a schematic diagram of the encoder provided in another embodiment of this application. Detailed Implementation

[0029] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0030] Figure 1 This is a schematic block diagram of a video encoder involved in some embodiments of this application.

[0031] It should be understood that the video encoder 100 can be used for lossy compression of images, or for lossless compression of images. The lossless compression can be visually lossless compression or mathematically lossless compression.

[0032] This video encoder 100 can be applied to image data in luminance / chrominance (YCbCr, YUV) format. For example, the YUV ratio can be 4:2:0, 4:2:2, or 4:4:4, where Y represents luminance (Luma), Cb (U) represents blue chrominance, Cr (V) represents red chrominance, and U and V represent chrominance (Chroma) used to describe color and saturation. For example, in color format, 4:2:0 means that there are 4 luminance components and 2 chrominance components (YYYYCbCr) per 4 pixels; 4:2:2 means that there are 4 luminance components and 4 chrominance components (YYYYCbCrCbCr) per 4 pixels; and 4:4:4 means full pixel display (YYYYCbCrCbCrCbCrCbCr).

[0033] For example, the video encoder 100 reads video data and, for each image in the video data, divides the image into several coding tree units (CTUs). In some examples, CTUs may be called "tree blocks," "largest coding units" (LCUs), or "coding tree blocks" (CTBs). Each CTU can be associated with a pixel block of equal size within the image. Each pixel can correspond to one luminance (luma) sample and two chrominance (chroma) samples. Therefore, each CTU can be associated with one luminance sample block and two chrominance sample blocks. The size of a CTU is, for example, 128×128, 64×64, 32×32, etc. A CTU can be further divided into several coding units (CUs) for encoding. CUs can be rectangular or square blocks. CUs can correspond to prediction units (PUs) and transform units (TUs).

[0034] In some embodiments, the current block may be referred to as the current coding unit (CU). A prediction block may also be referred to as a prediction image block or an image prediction block, and a reconstructed image block may also be referred to as a reconstruction block or an image reconstruction block. Due to the need for parallel processing, an image can be divided into slices. Slices within the same image can be processed in parallel, meaning there is no data dependency between them. A "frame" is a commonly used term, generally understood to mean that one frame is an image. In this document, "frame" can also be replaced with "image" or "slice," etc.

[0035] The following is combined Figure 1 The video encoder 100 in some embodiments of this application will be described.

[0036] In some embodiments, the video encoder 100 may include a prediction module 110. The prediction module 110 includes an inter-frame prediction module 111 and an intra-frame prediction module 112. Because there is a strong correlation between adjacent pixels in an image of a video, intra-frame prediction is used in video encoding and decoding techniques to eliminate spatial redundancy between adjacent pixels. Because there is a strong similarity between adjacent images in a video, inter-frame prediction is used in video encoding and decoding techniques to eliminate temporal redundancy between adjacent images, thereby improving encoding efficiency.

[0037] The inter-frame prediction module 111 can be used for inter-frame prediction, which can include motion estimation and motion compensation. It can reference image information from different images. Inter-frame prediction uses motion information to find reference blocks in the reference images and generates prediction blocks based on these reference blocks to eliminate temporal redundancy. The motion information includes a list of reference images, the reference image index, and motion vectors. Motion vectors can be integer-pixel or fractional-pixel. If the motion vector is fractional-pixel, interpolation filtering needs to be used in the reference image to create the required fractional-pixel blocks. Here, the integer-pixel or fractional-pixel blocks in the reference image found based on the motion vectors are called reference blocks. Some techniques use reference blocks as prediction blocks, while others process the reference blocks further to generate prediction blocks. Processing the reference blocks further to generate prediction blocks can also be understood as using the reference blocks as prediction blocks and then processing them to generate new prediction blocks.

[0038] The intra-frame prediction module 112 refers only to information from the same image to predict pixel information within the current image block, thereby eliminating spatial redundancy.

[0039] Intra-frame prediction has multiple prediction modes. Taking the international digital video coding standards H-series as an example, the H.264 / AVC standard has 8 angular prediction modes and 1 non-angular prediction mode, while H.265 / HEVC extends this to 33 angular prediction modes and 2 non-angular prediction modes. High Efficiency Video Coding (HEVC) uses Planar, DC, and 33 angular modes, for a total of 35 prediction modes. Versatile Video Coding (VVC) uses Planar, DC, and 65 angular modes, for a total of 67 prediction modes.

[0040] It should be noted that with the increase in angle modes, intra-frame prediction will be more accurate and better meet the needs of the development of high-definition and ultra-high-definition digital video.

[0041] In some embodiments, the video encoder 100 may include a residual module 120. The residual module 120 may generate a residual block of the CU based on pixel blocks of the CU and prediction blocks of the CU. For example, the residual module 120 may generate a residual block of the CU such that each sample in the residual block has a value equal to the difference between the sample in the pixel block of the CU and the corresponding sample in the prediction block of the CU. The residual block can also be understood as residual information.

[0042] In some embodiments, the video encoder 100 may include a transformation module 125. The transformation module 125 can transform the residual information generated by the residual module 120 to obtain transformation coefficients.

[0043] In some embodiments, the video encoder 100 may include a quantization module 130. The quantization module 130 quantizes transform coefficients. The quantization module 130 quantizes the transform coefficients associated with the CU based on a quantization parameter (QP) value associated with the CU. The video encoder 100 can adjust the degree of quantization applied to the transform coefficients associated with the CU by adjusting the QP value associated with the CU.

[0044] In some embodiments, the video encoder 100 may include an inverse quantization module 135. The inverse quantization module 135 can apply inverse quantization to the quantized transform coefficients to obtain the transform coefficients.

[0045] In some embodiments, the video encoder 100 may include an inverse transform module 140. The inverse transform module 140 applies an inverse transform to the transform coefficients, which can be used to reconstruct the residual block.

[0046] In some embodiments, the video encoder 100 may include a reconstruction module 150. The reconstruction module 150 may add samples of the reconstructed residual block to corresponding samples of one or more prediction blocks generated by the prediction module 110 to generate reconstructed image blocks associated with the CU. By reconstructing each sample block of the CU in this manner, the video encoder 100 can reconstruct the pixel blocks of the CU.

[0047] In some embodiments, the video encoder 100 may include a loop filter module 160. The loop filter module 160 is used to process the pixels after inverse transform and inverse quantization to compensate for distortion information and provide a better reference for subsequent encoded pixels. For example, it may perform deblocking filtering operations to reduce the block effect of pixel blocks associated with the CU.

[0048] In some embodiments, the loop filtering module 160 includes a deblocking filtering module. The deblocking filtering module is used to remove block artifacts. Block artifacts refer to the discontinuities at the boundaries of coded blocks in an image. The deblocking filtering module can smooth the block boundaries to effectively reduce or remove block artifacts.

[0049] In some embodiments, the loop filtering module 160 includes a sample adaptive offset (SAO) module. The SAO module can be used to remove ringing effects. For strong edges in an image, due to quantization distortion of the high-frequency AC coefficients, a ripple effect will be generated around the edges after decoding; this distortion can be called ringing. The SAO module can suppress ringing effects from the pixel domain perspective.

[0050] In some embodiments, the loop filtering module 160 includes an adaptive loop filter (ALF) module, which is used to remove ringing effects. ALF technology can include luma ALF, chroma ALF, and inter-component ALF. ALF can be based on the Wiener filtering principle, using the original image information and reconstructed image information to establish the Wiener-Hough equation and solve for a series of filter coefficients with the minimum mean square error to remove ringing effects.

[0051] In some embodiments, the video encoder 100 may include a decoded image buffer 170. The decoded image buffer 170 may store reconstructed pixel blocks. The inter-frame prediction module 111 may use a reference image containing the reconstructed pixel blocks to perform inter-frame prediction on PUs of other images. In addition, the intra-frame prediction module 112 may use the reconstructed pixel blocks in the decoded image buffer 170 to perform intra-frame prediction on other PUs in the same image as the CU.

[0052] In some embodiments, the video encoder 100 may include an entropy coding module 180. The entropy coding module 180 may receive quantized transform coefficients from the quantization module 130. The entropy coding module 180 may perform one or more entropy coding operations on the quantized transform coefficients to produce entropy-coded data.

[0053] Figure 2 This is a schematic block diagram of a video decoder involved in some embodiments of this application.

[0054] Video decoder 200 can receive a bitstream. In some embodiments, video decoder 200 may include an entropy decoding module 210. Entropy decoding module 210 can parse the bitstream to extract syntax elements from it. As part of parsing the bitstream, entropy decoding module 210 can parse the entropy-encoded syntax elements in the bitstream. Prediction module 220, inverse quantization module 230, inverse transform module 235, reconstruction module 240, and loop filtering module 250 can decode video data based on the syntax elements extracted from the bitstream, i.e., generate decoded video data.

[0055] In some embodiments, the video decoder 200 may include a prediction module 220. The prediction module 220 includes an intra-frame prediction module 222 and an inter-frame prediction module 221.

[0056] Intra-prediction module 222 can perform intra-prediction to generate prediction blocks for the PU. Intra-prediction module 222 can use an intra-prediction mode to generate prediction blocks for the PU based on pixel blocks of spatially adjacent PUs. Intra-prediction module 222 can also determine the intra-prediction mode of the PU based on one or more syntax elements parsed from the bitstream.

[0057] Inter-frame prediction module 221 can construct a first reference image list (list 0) and a second reference image list (list 1) based on the syntax elements parsed from the bitstream. Furthermore, if the PU uses inter-frame prediction coding, entropy decoding module 210 can parse the motion information of the PU. Inter-frame prediction module 221 can determine one or more reference blocks of the PU based on the motion information of the PU. Inter-frame prediction module 221 can generate prediction blocks for the PU based on one or more reference blocks of the PU.

[0058] In some embodiments, the video decoder 200 may include an inverse quantization module 230. The inverse quantization module 230 applies inverse quantization to the quantized transform coefficients to obtain the transform coefficients. The inverse quantization module 230 may use the QP value associated with the CU of the TU to determine the degree of quantization.

[0059] In some embodiments, the video decoder 200 may include an inverse transform module 235. The inverse transform module 235 inversely transforms the transform coefficients associated with the TU. After inverse transforming the transform coefficients, the inverse transform module 235 can generate a residual block associated with the TU.

[0060] In some embodiments, the video decoder 200 may include a reconstruction module 240. The reconstruction module 240 uses a residual block associated with the TU of the CU and a prediction block of the PU of the CU to reconstruct pixel blocks of the CU. For example, the reconstruction module 240 may add samples of the residual block to the corresponding samples of the prediction block to reconstruct pixel blocks of the CU, thereby obtaining reconstructed image blocks.

[0061] In some embodiments, the video decoder 200 may include a loop filter module 250. The loop filter module 250 may perform deblocking filtering operations to reduce the block artifacts of pixel blocks associated with the CU.

[0062] In some embodiments, the video decoder 200 may include a decoded image buffer 260. The video decoder 200 may store the reconstructed image of the CU in the decoded image buffer 260. The video decoder 200 may use the reconstructed image in the decoded image buffer 260 as a reference image for subsequent prediction, or transmit the reconstructed image to a display device for presentation.

[0063] The basic process of video encoding and decoding is as follows: At the encoding end, an image is divided into blocks. For the current block, the prediction module 110 uses intra-frame prediction or inter-frame prediction to generate a prediction block for the current block. The residual module 120 can calculate a residual block based on the prediction block and the original block of the current block, that is, the difference between the prediction block and the original block of the current block. This residual block can also be called residual information. Through the transformation process of the transformation module 125 and the quantization process of the quantization module 130, the residual block can remove information that is not sensitive to the human eye, thereby eliminating visual redundancy. In some embodiments, the residual block before transformation by the transformation module 125 can be called a temporal residual block, and the temporal residual block after quantization by the quantization module 130 can be called a frequency residual block or a frequency domain residual block. The entropy coding module 180 receives the quantized transformation coefficients output by the quantization module 130 and can perform entropy coding on the quantized transformation coefficients to output a bitstream. For example, the entropy coding module 180 can eliminate character redundancy based on the target context model and the probability information of the binary bitstream.

[0064] At the decoding end, the entropy decoding module 210 parses the bitstream to obtain the prediction information and quantization coefficient matrix of the current block. The prediction module 220 uses the prediction information to generate a prediction block for the current block using intra-frame prediction or inter-frame prediction. The inverse quantization module 230 uses the quantization coefficient matrix obtained from the bitstream to inverse quantize the quantization coefficient matrix to obtain transform coefficients; the inverse transform module 235 performs an inverse transform on the transform coefficients to obtain a residual block. The reconstruction module 240 adds the prediction block and the residual block to obtain a reconstructed block. The reconstructed blocks form a reconstructed image. The loop filtering module 250 performs loop filtering on the reconstructed image based on the image or based on the blocks to obtain a decoded image. The encoding end also requires similar operations to the decoding end to obtain a decoded image. This decoded image can also be called a reconstructed image, which can be used as a reference image for inter-frame prediction of subsequent images.

[0065] It should be noted that the block partitioning information determined at the encoding end, as well as mode information or parameter information such as prediction, transform, quantization, entropy coding, and loop filtering, are carried in the bitstream when necessary. The decoding end determines the same block partitioning information, prediction, transform, quantization, entropy coding, and loop filtering mode information or parameter information as the encoding end by parsing the bitstream and analyzing existing information, thereby ensuring that the decoded image obtained by the encoding end is the same as the decoded image obtained by the decoding end.

[0066] It is understandable that the "inverse transformation" of the transform coefficients at the decoding end can also be referred to as "transformation" in the standard text. In some embodiments of this application, "transformation" and "inverse transformation" correspond to two opposite processes. For example, "transformation" converts the spatial domain values ​​to frequency domain coefficients, while "inverse transformation" converts the frequency domain coefficients back to spatial domain values. If the standard only specifies decoding, then "transformation" in the standard text refers to the decoding part, specifically the "inverse transformation" in this document. The "inverse transformation" of the transform coefficients at the decoding end can also be referred to as "transformation" in the standard text.

[0067] The above describes the basic flow of a video codec under a block-based hybrid coding framework. With the development of technology, some modules or steps of this framework or flow may be optimized. This application is applicable to the basic flow of the video codec under this block-based hybrid coding framework, but is not limited to this framework and flow.

[0068] The preceding text has described in detail the encoding and decoding frameworks provided by some embodiments of this application. The following section introduces the encoding and decoding technologies involved in some embodiments of this application.

[0069] In today's digital age, the generation and transmission of video content has become a crucial means of information exchange. With the rapid development of social media, online education, and remote work, the demand for high-quality video coding technologies is growing daily. However, while existing video coding standards (such as H.265 / HEVC and H.266 / VVC) perform excellently in high-bitrate environments, they often struggle to achieve satisfactory video quality and compression efficiency at extremely low bitrates. The importance of low-bitrate video coding is becoming increasingly apparent, especially in scenarios such as mobile devices, the Internet of Things (IoT), and emergency communications.

[0070] Against this backdrop, research and standardization of generative facial video coding (GFVC) are particularly urgent. GFVC technology can leverage compact representations of facial priors and the strong reasoning capabilities of deep generative models to achieve high-quality facial video communication in ultra-low bandwidth scenarios.

[0071] Generative models, such as generative adversarial networks (GANs) and variational autoencoders (VAEs), offer new possibilities for generating high-quality face videos at extremely low bit rates. These models can learn facial features from limited data and generate realistic video frames, thereby significantly reducing the required bit rate while maintaining video quality.

[0072] Driven by AI-generated content (AIGC), the joint video experts team (JVET) of ISO / IEC SC 29 and ITU-T SG16 began to work on the standardization of generative face video coding in extremely low bit rate environments, especially the discussion of the SEI proposal.

[0073] SEI, as an important coding tool, can provide additional information to improve the efficiency and quality of video coding. The experts at the meeting agreed to include the SEI proposal for generative face video in the technologies under consideration (TuC) of future VSEI (such as ITU-T H.274|ISO / IEC 23002-7), which provides an application prospect for generative face video coding in extremely low bitrate environments.

[0074] Furthermore, JVET-AH0110 proposed a scalable technique that adds more detail to the original GFVC technique. JVET-AH0127 designed two SEI schemes for this technique: 1) introducing a gating flag to control whether to send additional signals for enhancement layer features, thus expanding the current GFV SEI information; 2) introducing a new SEI information, called generative face video enhancement (GFVE) SEI information, to represent the features of the enhancement layer. After discussion among the experts present, the latter was ultimately adopted, introducing a new SEI information called GFVE SEI to enhance the quality of GFVC videos.

[0075] The encoding and decoding process of generative face video coding is described below.

[0076] Figure 3 A block diagram of a generative face video codec is shown. The generative face video codec system consists of two parts. The first part, 310, is responsible for encoding base images, which carry the basic texture information of the face. The encoding of the base images can employ standard encoding and decoding techniques, such as H.266 / VVC. Base images can also be referred to as keyframes. See [link to documentation]. Figure 3 The encoder 312 can encode the base image and obtain a bitstream. The decoder 314 can decode the bitstream to obtain the basic texture information of the face and reconstruct the face image.

[0077] Part 2, 320, involves the extraction and transmission of motion information from subsequent images, and facial image reconstruction based on this information. This process can be implemented using analysis model 322, which captures motion details. Generative model 328 can reconstruct the face image using basic texture information (as described above) and motion information. See also... Figure 3 The encoder can extract facial motion information using analysis model 322 and perform feature encoding on the facial motion information using feature encoding module 324 to obtain a bitstream. Feature decoding module 326 decodes the bitstream to obtain facial motion information, and generation model 328 processes the facial motion information and basic texture information to reconstruct the face image. The generation model can also be called a synthesis model or fusion model, etc.

[0078] Compared to VVC coding, a related technology, generative face video coding can provide higher quality face image reconstruction at extremely low bit rates.

[0079] Currently, JVET has tested five related algorithms for the above generative face video coding structure diagram, which are described below.

[0080] The FOMM takes a base image S and a subsequent image D as input, which can also be called the driving image. An unsupervised keypoint detector extracts sparse 2D keypoints of the face and a first-order motion representation consisting of local affine transformations relative to an abstract reference frame R. A dense motion network uses this motion representation to generate dense optical flow and occlusion maps from D to S. The generator uses the base image and the output of the dense motion network to render subsequent images.

[0081] The FV2V input is a base image S and a subsequent image D, which can also be called the driving image. The appearance feature extractor outputs appearance features from the base image, and the canonical keypoint detector outputs 3D canonical keypoints. For each image, an estimator network estimates keypoint perturbations caused by head pose and expression, thereby obtaining the head rotation / translation matrix and sparse 3D keypoints. The motion field estimation network uses the 3D keypoints from the base image and the subsequent image to obtain dense optical flow from D to S. The generator uses the appearance features from the base image and the output of the dense motion network to render the subsequent image.

[0082] The CFTE takes a base image S and a subsequent image D as input, which can also be called the driving image. Each image is processed by a compact feature extractor to obtain a compact feature matrix. The dense motion estimation network uses the difference between the compact feature matrices of the base image and the subsequent image, along with features from the base image, to generate a dense optical flow and occlusion map from D to S. The generator uses the base image and the output of the dense motion network to render the subsequent image.

[0083] The DAC algorithm is similar to the FOMM algorithm, but it uses only a set of 2D keypoints as a compact representation without any Jacobian matrix, in order to reduce the bit consumption of encoding GFVC parameters without affecting the reconstruction quality.

[0084] The HDAC algorithm is an extension of the DAC algorithm, designed to overcome the limitations of GFVC, such as limited bitrate coverage and unstable reconstruction quality (e.g., long-term dependencies and background motion). HDAC enhances the DAC bitstream by using a very low bitrate version of the video auxiliary stream obtained from a relevant technical video codec (e.g., HEVC or VVC). The DAC output and the auxiliary decoded video are then fed into a fusion module to reconstruct a high-quality face video, where the auxiliary video provides sufficient information to regularize the animation and compensate for motion estimation errors in the DAC output.

[0085] Table 1 shows the face representation methods corresponding to the five algorithms mentioned above.

[0086] Table 1

[0087] Face representation methods Corresponding Algorithm 2D key points + affine transformation matrix FOMM 3D key points + head rotation / translation matrix FV2V Compact feature matrix CFTE 2D Key Points DAC 2D key points + VVC video frames HDAC

[0088] The SEI information for generative face videos is described below. This SEI information can include SEI information specific to the generative face video itself, as well as SEI information for enhancements to the generative face video. These two types of information will be described separately below.

[0089] SEI information for generative face videos

[0090] Table 2 SEI Information Scope

[0091] SEI Information Duration range … … Generative face video SEI information related images

[0092] Table 3 SEI Information Syntax

[0093]

[0094]

[0095]

[0096]

[0097] The semantics of the syntax elements involved in the SEI messages in Table 3 are explained below.

[0098] The GFV SEI message carries facial parameters and instructs a facial parameter transformation network and a face image generator neural network. The facial parameter transformation network, denoted as TranslatorNN(), can be used to convert various facial parameter formats of the signal in the SEI message into a specific facial parameter format supported by the decoding system. The face image generator neural network, denoted as GenerativeNN(), can be used to generate an output image using the facial parameters converted to the specific format and the previously decoded output image.

[0099] `gfv_id` contains an identifier that can be used to recognize facial features and specifies a neural network that can be used with `TranslatorNN()`. The value of `gfv_id` ranges from 0 to 2. 32 -2 (inclusive). The value of gfv_id is 256 to 511 (inclusive) and 2. 31 ~2 32 -2 (inclusive) is reserved for future use by ITU-T|ISO / IEC. In some embodiments of this application, when the decoder receives a gfv_id in the range of 256 to 511 (inclusive) or 2... 31 ~2 32 When a GFV SEI message with a value of -2 (inclusive) is received, the SEI message will be ignored.

[0100] gfv_cnt specifies the GFV SEI message instance count value for the gfv_id value within the image unit. The value of gfv_cnt ranges from 0 to 65535 (inclusive).

[0101] When gfv_base_pic_flag equals 1, it indicates that the currently decoded output image is the base image. When gfv_base_pic_flag equals 0, it indicates that the currently decoded output image is not the base image. The value of gfv_base_pic_flag is 1 when the GFV SEI message is the first GFV SEI message in the current CLVS with a specific gfv_id value.

[0102] When `gfv_nn_present_flag` equals 1, it indicates that the SEI message contains or indicates a neural network that can be used as a TranslatorNN(). When `gfv_nn_present_flag` equals 0, it indicates that the SEI message does not contain or indicate a neural network that can be used as a TranslatorNN(). If the SEI message does not contain this flag information, it is inferred that the value of this flag information is 0.

[0103] When gfv_nn_base_flag equals 1, it indicates that the specified TranslatorNN() is a neural network post-filter (NNPF). When gfv_nn_base_flag equals 0, it indicates that the specified TranslatorNN() is an update relative to the base NNPF.

[0104] When `gfv_nn_mode_idc` equals 0, it indicates that the neural network information is contained in the Neural-network post-filter characteristics (NNPFC) SEI message, and the neural network information uses the ISO / IEC 15938-17 bitstream format. When `gfv_nn_mode_idc` equals 1, it indicates that the neural network information is identified by the URI indicated by `nnpfc_uri`, and its format is identified by the tag URI `nnpfc_tag_uri`.

[0105] gfv_nn_alignment_zero_bit_a and gfv_nn_alignment_zero_bit_b should be equal to 0.

[0106] gfv_nn_tag_uri contains a tag URI whose syntax and semantics are defined by IETF RFC 4151, used to identify a neural network format and related information as a base NNPF or an updated version of the base NNPF.

[0107] gfv_uri contains a URI whose syntax and semantics conform to IETF Internet Standard 66, used to identify a neural network used as the base NNPF, or a newer neural network relative to the base NNPF.

[0108] `gfv_nn_payload_byte[i]` contains the i-th byte of a bitstream conforming to the ISO / IEC 15938-17 standard. All the contents of `nnpfc_payload_byte[i]` should combine to form a complete bitstream conforming to a video codec standard (such as ISO / IEC 15938-17). `gfv_drive_pic_fusion_flag`: When it exists and is equal to 1, it indicates that the currently decoded image (corresponding to a driving image that may be used for fusion) can be input to `GenerativeNN()`; when it is equal to 0, it indicates that the currently decoded image should not be input to `GenerativeNN()`.

[0109] A value of 1 for `gfv_chroma_key_info_present_flag` indicates that the syntax elements `gfv_chroma_key_value_present_flag[c]` and `gfv_chroma_key_thr_present_flag[i]` exist. Additionally, `gfv_chroma_key_value[c]` and `gfv_chroma_key_thr_value[i]` may exist. A value of 0 for `gfv_chroma_key_info_present_flag` indicates that the syntax elements `gfv_chroma_key_value_present_flag[c]`, `gfv_chroma_key_thr_present_flag[i]`, `gfv_chroma_key_value[c]`, and `gfv_chroma_key_thr_value[i]` do not exist.

[0110] A value of 1 for `gfv_chroma_key_value_present_flag[c]` indicates that the syntax element `gfv_chroma_key_value[c]` exists. A value of 0 for `gfv_chroma_key_present_flag[c]` indicates that the syntax element `gfv_chroma_key_value[c]` does not exist.

[0111] gfv_chroma_key_value[c] specifies the chroma key value corresponding to the c-th color component.

[0112] When gfv_chroma_key_thr_value[i] exists, the i-th chroma key threshold is specified. When gfv_chroma_key_thr_value[i] does not exist, the value of gfv_chroma_key_thr_value[i] is inferred as follows: if i = 0, then gfv_chroma_key_thr_value[0] is set to 48; otherwise, if i = 1, gfv_chroma_key_thr_value[1] is set to 75.

[0113] When `gfv_drive_pic_fusion_flag` exists, a value of 1 indicates that the currently decoded image can be input into `GenerativeNN()`, and this image corresponds to a driving image that may be used for fusion. A value of 0 indicates that the currently decoded image should not be input into `GenerativeNN()`.

[0114] A value of 1 for `gfv_low_confence_face_parameter_flag` indicates a low confidence level in the calculated face parameters. A value of 0 for `gfv_low_confence_face_parameter_flag` indicates that no confidence information for face parameters has been specified.

[0115] When gfv_coordinate_present_flag equals 1, it indicates that the coordinate information of the key point exists. When gfv_coordinate_present_flag equals 0, it indicates that the coordinate information of the key point does not exist. If gfv_matrix_type_idx[i] is 0 or 1, then the value of gfv_coordinate_present_flag is 1.

[0116] A value of 1 for `gfv_kps_pred_flag` indicates the presence of the syntax elements `gfv_coordinate_dx_abs[i]`, `gfv_coordinate_dy_abs[i]`, and `gfv_coordinate_dz_abs[i]`, and may also include the syntax elements `gfv_coordinate_dx_sign_flag[i]`, `gfv_coordinate_dy_sign_flag[i]`, and `gfv_coordinate_dz_sign_flag[i]`. A value of 0 for `gfv_kps_pred_flag` indicates the presence of the syntax elements `gfv_coordinate_x_abs[i]`, `gfv_coordinate_y_abs[i]`, and `gfv_coordinate_z_abs[i]`, and may also include the syntax elements `gfv_coordinate_x_sign_flag[i]`, `gfv_coordinate_y_sign_flag[i]`, and `gfv_coordinate_z_sign_flag[i]`.

[0117] When gfv_coordinate_present_flag is 1, gfv_base_pic_flag is 0, and gfv_kps_pred_flag is 1, there should be a previous GFV SEI message in the current CLVS with the same gfv_id as the current GFV SEI message and gfv_base_pic_flag is 1, and its decoding order is the same.

[0118] Increasing `gfv_coordinate_precision_factor_minus1` by 1 indicates the precision of the keypoint coordinates represented in the SEI message. The value of `gfv_coordinate_precision_factor_minus1` ranges from 0 to 31 (inclusive). When `gfv_coordinate_present_flag = 1`, `gfv_base_pic_flag = 0`, and `gfv_kps_pred_flag = 1`, the value of `gfv_coordinate_precision_factor_minus1` should be inferred to be the same as the `gfv_coordinate_precision_factor_minus1` of a previous GFV SEI message with the same `gfv_id` and `gfv_base_pic_flag` equal to 1.

[0119] Increasing `gfv_num_kps_minus1` by 1 indicates the number of keypoints. The value of `gfv_num_kps_minus1` ranges from 0 to 2. 10 -1 (inclusive). When gfv_coordinate_present_flag is 1, gfv_base_pic_flag is 0, and gfv_kps_pred_flag is 1, the value of gfv_num_kps_minus1 should be inferred to be the same as the gfv_num_kps_minus1 of a previous GFV SEI message with the same gfv_id and gfv_base_pic_flag equal to 1.

[0120] A value of 1 for `gfv_coordinate_z_present_flag` indicates the presence of z-axis coordinate information for keypoints. A value of 0 for `gfv_coordinate_z_present_flag` indicates the absence of z-axis coordinate information for keypoints. When `gfv_coordinate_present_flag` is 1, `gfv_base_pic_flag` is 0, and `gfv_kps_pred_flag` is 1, the value of `coordinate_z_present_flag` should be inferred to be the same as the `coordinate_z_present_flag` of a previous GFVSEI message with the same `gfv_id` and `gfv_base_pic_flag` equal to 1.

[0121] Increasing 1 in `gfv_coordinate_z_max_value_minus1` indicates the maximum absolute value of the z-axis coordinate of the keypoint. The value range of `gfv_coordinate_z_max_value_minus1` is 0 to 2.16 -1 (inclusive). When gfv_coordinate_present_flag equals 1, gfv_base_pic_flag equals 0, and gfv_kps_pred_flag equals 1, the value of gfv_coordinate_z_max_value_minus1 should be inferred to be the same as the gfv_coordinate_z_max_value_minus1 of a previous GFV SEI message with the same gfv_id and gfv_base_pic_flag equal to 1.

[0122] gfv_coordinate_x_abs[i] is used to calculate the x-axis coordinate of the i-th key point.

[0123] `gfv_coordinate_x_sign_flag[i]` specifies the sign of the x-axis coordinate of the i-th keypoint. When `gfv_coordinate_x_sign_flag[i]` does not exist, it is inferred to be equal to 0.

[0124] gfv_coordinate_y_abs[i] is used to calculate the y-axis coordinate of the i-th key point.

[0125] `gfv_coordinate_y_sign_flag[i]` specifies the sign of the y-axis coordinate of the i-th keypoint. When `gfv_coordinate_y_sign_flag[i]` does not exist, it is inferred to be equal to 0.

[0126] gfv_coordinate_z_abs[i] is used to calculate the z-axis coordinate of the i-th key point.

[0127] `gfv_coordinate_z_sign_flag[i]` specifies the sign of the z-axis coordinate of the i-th keypoint. When `gfv_coordinate_z_sign_flag[i]` does not exist, it is inferred to be equal to 0.

[0128] gfv_coordinate_dx_abs[i] specifies a difference used to calculate the x-axis coordinate of the i-th keypoint.

[0129] `gfv_coordinate_dx_sign_flag[i]` specifies the sign of the x-axis coordinate difference of the i-th keypoint. When `gfv_coordinate_dx_sign_flag[i]` does not exist, it is inferred to be equal to 0.

[0130] gfv_coordinate_dy_abs[i] specifies a difference used to calculate the y-axis coordinate of the i-th keypoint.

[0131] `gfv_coordinate_dy_sign_flag[i]` specifies the sign of the y-axis coordinate difference of the i-th keypoint. When `gfv_coordinate_yd_sign_flag[i]` does not exist, it is inferred to be equal to 0.

[0132] gfv_coordinate_dz_abs[i] specifies a difference used to calculate the z-axis coordinate of the i-th key point.

[0133] `gfv_coordinate_dz_sign_flag[i]` specifies the sign of the z-axis coordinate difference of the i-th keypoint. When `gfv_coordinate_dz_sign_flag[i]` does not exist, it is inferred to be equal to 0.

[0134] A value of 1 for `gfv_matrix_present_flag` indicates the presence of matrix parameters. A value of 0 for `gfv_matrix_present_flag` indicates the absence of matrix parameters. When `gfv_coordinate_present_flag` is 0, `gfv_matrix_present_flag` is 1.

[0135] A value of 1 for `gfv_matrix_pred_flag` indicates that the syntax elements `gfv_matrix_element_int[i][j][k][m]` and `gfv_matrix_element_dec[i][j][k][m]` exist, and the syntax element `gfv_matrix_element_sign_flag[i][j][k][m]` may exist. A value of 0 for `gfv_matrix_pred_flag` indicates that the syntax elements `gfv_matrix_delta_element_int[i][j][k][m]` and `gfv_matrix_delta_element_dec[i][j][k][m]` exist, and the syntax element `gfv_matrix_delta_element_sign_flag[i][j][k][m]` may exist. When `gfv_matrix_pred_flag` does not exist, it is inferred to be 0.

[0136] When gfv_matrix_present_flag is 1, gfv_base_pic_flag is 0, and gfv_matrix_pred_flag is 1, there should be a GFV SEI message in the current CLVS with the same gfv_id as the current GFV SEI message and gfv_base_pic_flag is 1, and the decoding order is earlier.

[0137] Increasing `gfv_matrix_element_precision_factor_minus1` by 1 indicates the precision of the matrix elements that emit signals in the SEI message. The value of `gfv_matrix_element_precision_factor_minus1` ranges from 0 to 31 (inclusive). When `gfv_matrix_present_flag` is 1, `gfv_base_pic_flag` is 0, and `gfv_matrix_pred_flag` is 1, the value of `gfv_matrix_element_precision_factor_minus1` should be inferred to be the same as the `gfv_matrix_element_precision_factor_minus1` of a previous GFV SEI message with the same `gfv_id` and `gfv_base_pic_flag` equal to 1.

[0138] Increasing `gfv_num_matrix_types_minus1` by 1 indicates the number of matrix types of signals emitted in the SEI message. The value of `gfv_num_matrix_types_minus1` ranges from 0 to 2. 6 -1 (including 2) 6 -1). Regarding bitstream consistency requirements, when `gfv_matrix_pred_flag` equals 1 and `gfv_base_pic_flag` equals 0, the value of `gfv_num_matrix_types_minus1` should be the same as the value of `gfv_num_matrix_types_minus1` in the current CLVS decoding order for GFV SEI messages with the same `gfv_id` and `gfv_base_pic_flag` equal to 1. When `gfv_matrix_present_flag` is 1, `gfv_base_pic_flag` is 0, and `gfv_matrix_pred_flag` is 1, the value of `gfv_num_matrix_types_minus1` should be inferred to be the same as the value of `gfv_num_matrix_types_minus1` in a previous GFV SEI message with the same `gfv_id` and `gfv_base_pic_flag` equal to 1.

[0139] `gfv_matrix_type_idx[i]` represents the index of the i-th matrix type specified in Table 4. The value range of `gfv_matrix_type_idx[i]` is 0 to 63 (inclusive). The value range of `gfv_matrix_type_idx[i]` is 0 to 31 (inclusive). In some embodiments of this application, the decoder allows `gfv_matrix_type_idx[i]` to appear in the bitstream if it is greater than 31. If `gfv_matrix_type_idx[i]` is greater than 31, the decoder ignores all information of the i-th matrix type.

[0140] Table 4

[0141]

[0142] When gfv_num_matrices_equal_to_num_kps_flag[i] equals 1, it means that the number of matrices of the i-th matrix type is equal to gfv_num_kps_minus1+1. When gfv_num_matrices_equal_to_num_kps_flag[i] equals 0, it means that the number of matrices is not equal to gfv_num_kps_minus1+1.

[0143] gfv_num_matrices_info[i] calculates the number of matrices of the i-th matrix type.

[0144] Adding 1 to gfv_matrix_width_minus1[i] indicates the width of the matrix of the i-th matrix type.

[0145] Adding 1 to gfv_matrix_height_minus1[i] indicates the height of the matrix of the i-th matrix type.

[0146] When gfv_matrix_for_3D_space_flag[i] equals 1, it indicates that the i-th matrix type is a matrix defined in three-dimensional space. When gfv_matrix_for_3D_space_flag[i] equals 0, it indicates that the i-th matrix is ​​defined in two-dimensional space.

[0147] The increment of `gfv_num_matrices_minus1[i]` by 1 indicates the number of matrices of the i-th matrix type. The value of `gfv_num_matrices_minus1[i]` ranges from 0 to 2. 10 -1 (inclusive).

[0148] gfv_matrix_element_int[i][j][k][m] indicates the integer part of the matrix element in the k-th row and m-th column of the j-th matrix of the i-th matrix type.

[0149] gfv_matrix_element_dec[i][j][k][m] indicates the fractional part of the matrix element in the k-th row and m-th column of the j-th matrix of the i-th matrix type.

[0150] `gfv_matrix_element_sign_flag[i][j][k][m]` indicates the sign of the matrix element in the k-th row and m-th column of the j-th matrix of the i-th matrix type. If `gfv_matrix_element_sign_flag[i][j][k][m]` does not exist, it is inferred to be equal to 0.

[0151] gfv_matrix_delta_element_int[i][j][k][m] indicates the integer part of the difference between the matrix elements in the k-th row and m-th column of the j-th matrix of the i-th matrix type.

[0152] gfv_matrix_delta_element_dec[i][j][k][m] indicates the fractional part of the difference between the matrix elements in the k-th row and m-th column of the j-th matrix of the i-th matrix type.

[0153] `gfv_matrix_delta_element_sign_flag[i][j][k][m]` indicates the sign of the difference between the matrix elements in the k-th row and m-th column of the j-th matrix of the i-th matrix type. If `gfv_matrix_delta_element_sign_flag[i][j][k][m]` does not exist, it is inferred to be equal to 0.

[0154] Generative Face Video Enhancement SEI Information

[0155] Table 5 SEI Information Scope

[0156] SEI Information Duration range … … Generative face video enhancement SEI information related images

[0157] Table 6 SEI Information Syntax

[0158]

[0159] The semantics of the syntax elements involved in the SEI messages in Table 6 are explained below.

[0160] The GFVE SEI message indicates the enhancement of facial parameters and specifies an enhancement network, denoted as EnhancerNN(), which can be used to enhance the visual quality of the face image generated by the GFV SEI message.

[0161] `gfve_id` contains an identifier that can be used to recognize GFVE SEI messages and specifies the neural network that can be used as EnhancerNN(). The value of `gfve_id` ranges from 0 to 2. 32 -2 (inclusive). The value of gfve_id is 256 to 511 (inclusive) and 2. 31 ~2 32 -2 (inclusive) is reserved for future use by ITU-T|ISO / IEC. In some embodiments of this application, the decoder receives a gfve_id ranging from 256 to 511 (inclusive) or 2. 31 ~2 32 When a GFVE SEI message with a value of -2 (inclusive) is received, the SEI message is ignored.

[0162] `gfve_gfv_id` and `gfve_gfv_cnt` specify the `gfv_id` and `gfv_cnt` of the associated GFV SEI message. The associated GFV SEI message is the GFV SEI message in the same image unit as the GFVE SEI message whose `gfv_id` is equal to `gfve_gfv_id` and `gfv_cnt` is equal to `gfve_gfv_cnt`. The GFVE message is used to enhance the image generated with the associated GFV SEI message.

[0163] A value of 1 for `gfve_nn_present_flag` indicates that the neural network that can be used as EnhancerNN() is included or indicated by the SEI message. A value of 0 for `gfve_nn_present_flag` indicates that the neural network that can be used as EnhancerNN() is not included or indicated by the SEI message.

[0164] gfve_nn_base_flag, gfve_nn_mode_idc, gfve_nn_alignment_zero_bit_a, gfve_nn_tag_uri, gfve_nn_uri, gfve_nn_alignment_zero_bit_b, and gfve_nn_payload_byte[i] specify a neural network that can be used as EnhancerNN().

[0165] `gfve_matrix_element_precision_factor` represents the quantization factor of the matrix element represented in the SEI message. The value range of `gfve_matrix_element_precision_factor` is 0 to 32 (inclusive).

[0166] Increasing `gfv_num_matrices_minus1` by 1 specifies the number of matrices that are signaled in the SEI message. The value of `gfv_num_matrices_minus1` ranges from 0 to 2. 10 -1 (inclusive).

[0167] The increment of `gfve_matrix_height_minus1[i]` by 1 indicates the height of the i-th matrix. The value of `gfve_matrix_height_minus1[i]` ranges from 0 to 2. 10 -1 (inclusive).

[0168] The increment of `gfve_matrix_width_minus1[i]` by 1 indicates the width of the i-th matrix. The value of `gfve_matrix_width_minus1[i]` ranges from 0 to 2. 10 -1 (inclusive).

[0169] `gfve_matrix_element[i][j][k]` is used to calculate the value of the element at position (k,j) in the i-th matrix. The value of `gfve_matrix_element[i][j][k]` ranges from 0 to 2. 32 -2 (inclusive).

[0170] gfve_matrix_element_sign_flag[i][j][k] represents the sign of the matrix element at position (k,j) of the i-th matrix.

[0171] As shown in Table 4, eye movement information can currently be represented by gfv_matrix_type_idx[i]. When i = 3, gfv_matrix_type_idx[i] represents the eye matrix, used to represent the opening and closing state and degree of the eyes. However, this representation is insufficient to represent real and accurate eye movements; that is, the decoder cannot reconstruct real and accurate eye movement information using the eye matrix information.

[0172] To improve the realism of generative face video coding, some embodiments of this application have added corresponding coding content. Figure 3For example, in the generative face video encoding and decoding process, standard encoding and decoding technologies, such as H.266 / VVC, are used for the base image. For subsequent images, they are first fed into an analysis model to extract corresponding image features and eye movement information. Then, feature encoding is performed on the corresponding image features and eye movement information. At the decoding end, feature decoding is performed on the received corresponding image features and eye movement information. Finally, the image features, eye movement information, and base image are fed into the generative model to generate an output sequence with realistic and relatively accurate eye movements compared to the input sequence.

[0173] The aforementioned eye movement information may include pupil position information, which reflects eye movement information.

[0174] Some embodiments of this application provide an encoding method, including: determining the position information of a pupil contained in a current image; encoding parameters in a Supplemental Enhancement Information (SEI) message corresponding to the current image based on the pupil position information, wherein the pupil position information is used to determine the eye motion information contained in the current image.

[0175] Furthermore, some embodiments of this application also provide a decoding method, including: decoding the SEI message corresponding to the current image to obtain the position information of the pupil contained in the current image; and determining the motion information of the eye contained in the current image based on the position information of the pupil.

[0176] Some embodiments of this application add pupil position information to the SEI message. Since the pupil position information can accurately reflect the eye movement information, compared with the information on the opening and closing state and degree of the eye in related technologies, some embodiments of this application enable the decoding end to generate real and accurate eye movement information.

[0177] In video encoding, SEI (Eye Movement Information) is a technique used to add additional information to the video bitstream. This additional information can include metadata related to the video content, such as timestamps, scene information, copyright information, color space information, etc. The use of SEI can provide more functionality and optimize video quality; therefore, some embodiments of this application may choose to add eye movement information via SEI to improve video quality.

[0178] Regarding the encoding of pupil information, some embodiments of this application provide four schemes: no pupil information encoding, encoding only the pupil information of one eye (left or right eye), independently encoding the pupil information of both eyes, and using residual encoding of one eye to the other eye (left eye to right eye, or right eye to left eye). Specifically, when both eyes are absent in the face video, or the encoder deems it unnecessary to transmit pupil information in this frame, only one identifier is encoded. When one eye is absent, or the encoder deems it necessary to transmit only one eye, only the pupil information of one eye is encoded. When both eyes need to be transmitted, there are two schemes: one is to encode both eyes independently, and the other is to use residual encoding of the left eye to the right eye (or right eye to left eye) to further improve encoding efficiency. It should be noted that when the encoder deems it unnecessary to transmit the pupil information of a certain eye, the decoder can adopt a default value mechanism, such as using the position information of the left (or right) eye pupil in the reference image as the position information of the left (or right) eye pupil in the current image.

[0179] When transmitting pupil position information from the encoder to the decoder, there are 11 possible transmission schemes. These 11 transmission schemes can be divided into two main categories: predictive transmission schemes and non-predictive transmission schemes. Prediction refers to using information from a reference image to predict the information of the current image, while non-predictive transmission refers to not using information from a reference image to predict the information of the current image. Table 7 shows the detailed information of these 11 transmission schemes.

[0180] Table 7

[0181]

[0182]

[0183] Scheme B1 is the same as scheme B6, which means that it does not transmit the pupil position information of the current image.

[0184] Scheme B2 uses a predictive approach, transmitting only the position information of the left pupil. The encoder can use the position information of the left pupil in the reference image to predict the position information of the left pupil in the current image, obtaining residual information 1. The encoder can encode residual information 1, and the decoder can decode residual information 1 to determine the position information of the left pupil.

[0185] Scheme B3 uses a predictive approach, transmitting only the position information of the right pupil. The encoder can use the position information of the right pupil in the reference image to predict the position information of the right pupil in the current image, obtaining residual information 2. The encoder can encode residual information 2, and the decoder can decode residual information 2 to determine the position information of the right pupil.

[0186] Scheme B4 employs a predictive approach to transmit the position information of both the left and right pupils. It also uses information from the left pupil in the current image to predict the position of the right pupil. For example, the encoder can use information from the reference image to predict the current image's information, obtaining residual information 1 and residual information 2. Furthermore, the encoder can use residual information 1 to predict residual information 2, obtaining residual information 3. The encoder can encode residual information 1 and residual information 3, and the decoder can decode them to determine the position information of both the left and right pupils.

[0187] The B5 scheme employs a predictive approach to transmit the position information of both the left and right pupils. It also uses information from the right pupil in the current image to predict the position of the left pupil. For example, the encoder can use information from a reference image to predict the current image's information, obtaining residual information 1 and residual information 2. Furthermore, the encoder can use residual information 2 to predict residual information 1, obtaining residual information 4. The encoder can encode residual information 2 and residual information 4, and the decoder can decode them to determine the position information of both the left and right pupils.

[0188] The B7 scheme uses a non-predictive method to transmit the position information of the left pupil. This left pupil position information is independent; that is, the position information of the left pupil in each frame of the image is independent. The encoding end can encode the position information of the left pupil, and the decoding end can decode it to obtain the position information of the left pupil.

[0189] The B8 scheme uses a non-predictive method to transmit the position information of the right pupil. This right pupil position information is independent; that is, the position information of the right pupil in each frame of the image is independent. The encoding end can encode the position information of the right pupil, and the decoding end can decode it to obtain the position information of the right pupil.

[0190] The B9 scheme transmits the position information of the left and right pupils, and uses the position information of the left pupil to predict the position information of the right pupil. The encoder can use the position information of the left pupil to predict the position information of the right pupil, obtaining residual information 5, and then encodes the position information of the left pupil and residual information 5. The decoder can decode the position information of the left pupil and residual information 5 to determine the position information of the left and right pupils.

[0191] The B10 scheme transmits the position information of the left and right pupils, and uses the position information of the right pupil to predict the position information of the left pupil. The encoder can use the position information of the right pupil to predict the position information of the left pupil, obtaining residual information 6, and then encodes the position information of the right pupil and residual information 6. The decoder can decode the position information of the right pupil and residual information 6 to determine the position information of the left and right pupils.

[0192] The B11 scheme transmits the position information of the left and right pupils, which are encoded independently. The encoding end can encode the position information of the left and right pupils, and the decoding end can decode the position information of the left and right pupils to obtain their respective position information.

[0193] The quantification of pupil information can be determined by a quantization parameter, and a skipping mechanism can be used to avoid the additional overhead caused by repeated transmission.

[0194] In the specific encoding process, the first frame of the image needs to encode the specific quantization parameters or the index of the quantization parameters. For subsequent images, if the quantization parameters have not changed, they do not need to be transmitted. If the quantization parameters have changed, the difference between the new quantization parameters and the quantization parameters of the reference image, or the index of the new quantization parameters, can be transmitted.

[0195] In some embodiments of this application, the pupil position information can refer to the pupil coordinate information. In face videos, the pupil's positional movement relative to the entire eye region is relatively small, and encoding the absolute position of the pupil in each frame is very resource-intensive. To improve the efficiency of pupil encoding, this application can specify the midpoint of the line connecting the two corners of the eyes (left and right corners) as the origin to establish an image coordinate system. Furthermore, considering that eye motion reconstruction will employ a neural network approach, some embodiments of this application propose a coordinate system normalization scheme. A square eye image is cropped using n times the horizontal distance L between the two corners of the eyes as the cropping length. Then, the pupil coordinates on the eye image are normalized to [-1, 1], where n = {1.2, 1.4, 1.6}. Finally, the pupil coordinate information in each frame is encoded, or residual encoding is performed using information from a reference image.

[0196] For ease of description, the coordinate system mentioned above will be referred to as the first coordinate system. The origin of the first coordinate system is the midpoint of the line connecting the two corners of the eyes, and the unit length of the first coordinate system is m times the horizontal distance between the two corners of the eyes. To present as complete or multiple eye images as possible, m is greater than 0.5. For example, m can take values ​​of 0.6, 0.7, or 0.8. The horizontal distance between the two corners of the eyes can refer to the distance in the horizontal direction between the two corners of the eyes, or the projected length of the distance between the two corners of the eyes in the horizontal direction.

[0197] The following is combined Figure 4 The first coordinate system will be described in detail.

[0198] In some embodiments of this application, a square region is selected as the center point, and the horizontal distance L between the two corners of the eyes is n times the side length to establish an image coordinate system. This center point is the origin of the coordinate system, the horizontal axis to the right is the positive direction of the x-axis, the vertical axis downwards is the positive direction of the y-axis, and all coordinates are normalized to [-1, 1]. The possible values ​​of n are {1.2, 1.4, 1.6}, and n = 2m.

[0199] n > 1 is used to crop out the eye image as much as possible. A larger value for n results in a larger cropping area, encompassing more of the eye region. However, this also leads to a smaller dynamic range of the normalized pupil position, affecting the accuracy of subsequent quantization. Therefore, to include as much of the eye image as possible while avoiding an excessively small dynamic range of the normalized pupil position, in some embodiments of this application, the value of n can be {1.2, 1.4, 1.6}.

[0200] The following describes some quantization schemes in some embodiments of this application. It should be noted that the following quantization schemes are only examples, and the quantization schemes in some embodiments of this application can be any scheme capable of quantization.

[0201] For a normalized Cartesian coordinate system, a unified quantization scheme is used for the x-axis and y-axis. For example, a quantization step size of 1 / q can be selected, where q ranges from 1 to 32.

[0202] The following section will first provide detailed examples of the decoding methods of some embodiments of this application.

[0203] Figure 5 This is a flowchart illustrating a decoding method provided in some embodiments of this application. Figure 5 The method can be applied to the decoder.

[0204] See Figure 5 In step S510, the SEI message corresponding to the current image is decoded to obtain the position information of the pupil contained in the current image.

[0205] The current image can be any frame from a video. For example, the current image can be a base image or a driving image. The base image can be the first frame of the video, and the driving image can be any other image in the video besides the first frame; the driving image can also be called a subsequent image. Of course, the base image can also be any other image in the video besides the first frame. In some embodiments, the base image can also be called a reference image.

[0206] SEI messages can be either GFV SEI messages or GFVE SEI messages. The following explanation will primarily use GFVE SEI messages as an example.

[0207] An image can correspond to one SEI message. Each SEI message can be transmitted independently, or multiple SEI messages can be transmitted together, such as multiple SEI messages transmitted through a single carrier. If multiple SEI messages are transmitted together, then each of the multiple SEI messages can correspond to a different index.

[0208] See also Figure 5 In step S520, based on the pupil position information, the eye motion information contained in the current image is determined. For example, a neural network model can be used to reconstruct the eye motion information.

[0209] Some embodiments of this application can generate eye position information based on pupil position information, making the generated eye position information more accurate.

[0210] Following step S520, a generative model can be used to reconstruct the face image based on eye movement information, resulting in a more realistic and accurate reconstructed face image. This generative model can... Figure 3 The generative model in the generative face video coding framework of this application, some embodiments of which can utilize generative face video coding technology to generate face images.

[0211] In some implementations, if the current image is the base image, the decoder can obtain the image features of the base image by parsing the bitstream, and reconstruct the face image based on these features to obtain the base image. If the current image is the driving image, the decoder can obtain the image features and pupil position information of the driving image by parsing the bitstream, and reconstruct the face image based on the image features, eye movement information, and the base image to obtain subsequent images. For example, the decoder can input image features, eye movement information, and the base image into the generative model to generate a face image.

[0212] The above-mentioned generative model can be a generative model in a generative face video coding framework.

[0213] The SEI message may include one or more parameters that can be used by the decoder to determine the pupil's position information. In some embodiments of this application, the term "parameter" may also be replaced with the term "syntax element." The parameters included in the SEI message are described in detail below.

[0214] SEI messages may include one or more of the following parameters: first parameter, second parameter, third parameter, fourth parameter, fifth parameter, sixth parameter, seventh parameter, and eighth parameter. These parameters are described below.

[0215] The first parameter represents the first position information. The first position information is the position of the left eye pupil in the current image.

[0216] The second parameter represents the second location information. The second location information is the location of the right eye pupil in the current image.

[0217] The first and second position information can be absolute position information. Taking coordinate information as an example, the first position information can be the coordinates of the left pupil, and the second position information can be the coordinates of the right pupil. The decoder decodes the first parameter to obtain the position information of the left pupil. The decoder decodes the second parameter to obtain the position information of the right pupil.

[0218] The third parameter represents the difference between the first position information and the third position information. This difference can also be called the residual. Hereafter, this difference will also be referred to as the first difference. The third position information is the position information of the left pupil in the reference image. Alternatively, the third parameter can be used to represent the residual information obtained by predicting the position information of the left pupil in the current image using the position information of the left pupil in the reference image. By carrying the difference information in the SEI message, the encoding efficiency of the pupil position information can be improved.

[0219] The first and third location information are location information for the pupil of the same eye in the same face. For example, if the image includes face a and face b, the first and third location information can be the location information for the left pupil of face a, or the location information for the left pupil of face b.

[0220] Taking position information as coordinate information as an example, the difference between the first position information and the third position information can refer to the difference between the coordinates of the left pupil in the current image and the coordinates of the left pupil in the reference image.

[0221] The fourth parameter represents the difference between the second positional information and the fourth positional information. This difference can also be called the residual. Hereafter, this difference will also be referred to as the second difference. The fourth positional information is the position of the right pupil in the reference image. Alternatively, the fourth parameter can be used to represent the residual information obtained by predicting the position of the right pupil in the current image using the position information of the right pupil in the reference image. By carrying the difference information in the SEI message, the encoding efficiency of the pupil position information can be improved.

[0222] The second and fourth location information are location information for the pupil of the same eye in the same face. For example, if the image includes face a and face b, the second and fourth location information could be the location information for the right pupil of face a, or the location information for the right pupil of face b.

[0223] Taking position information as coordinate information as an example, the difference between the second position information and the fourth position information can refer to the difference between the coordinates of the right pupil in the current image and the coordinates of the right pupil in the reference image.

[0224] The fifth parameter represents the difference between the first position information and the second position information. Alternatively, the fifth parameter can be used to represent the residual information obtained by predicting the position of the left pupil using the position information of the right pupil. By carrying the difference information in the SEI message, the encoding efficiency of the pupil position information can be improved.

[0225] Taking location information as coordinate information as an example, the difference between the first location information and the second location information can refer to the difference between the coordinate values ​​of the left pupil and the right pupil.

[0226] The sixth parameter represents the difference between the second position information and the first position information. Alternatively, the sixth parameter can be used to represent the residual information obtained by predicting the position of the right pupil using the position information of the left pupil. By carrying the difference information in the SEI message, the encoding efficiency of the pupil position information can be improved.

[0227] Taking location information as coordinate information as an example, the difference between the second location information and the first location information can refer to the difference between the coordinate values ​​of the right pupil and the left pupil.

[0228] The seventh parameter represents the difference between the first difference and the second difference. By performing difference processing on the first and second differences, the encoding efficiency of pupil position information can be further improved.

[0229] Taking location information as coordinate information as an example, the first difference is the difference in the coordinates of the left pupil, the second difference is the difference in the coordinates of the right pupil, and the seventh parameter can be used to represent the residual information obtained by predicting the difference in the coordinates of the left pupil using the difference in the coordinates of the right pupil.

[0230] The eighth parameter represents the difference between the second difference and the first difference. By performing difference processing on the second and first differences, the encoding efficiency of pupil position information can be further improved.

[0231] Taking location information as coordinate information as an example, the first difference is the difference in the coordinates of the left pupil, the second difference is the difference in the coordinates of the right pupil, and the eighth parameter can be used to represent the residual information obtained by predicting the difference in the coordinates of the right pupil using the difference in the coordinates of the left pupil.

[0232] For example, an SEI message may include a first parameter. Or, an SEI message may include a second parameter. Or, an SEI message may include both a first and a second parameter. Or, an SEI message may include a third parameter. Or, an SEI message may include a fourth parameter. Or, an SEI message may include both a third and a fourth parameter. Or, an SEI message may include a fifth parameter. Or, an SEI message may include a sixth parameter. Or, an SEI message may include both a fifth and a sixth parameter. Or, an SEI message may include a seventh parameter. Or, an SEI message may include an eighth parameter. Or, an SEI message may include both a seventh and an eighth parameter.

[0233] The reference image mentioned above can be the first frame of the video, or another image in the video. For example, a reference image can be set every p frames. Alternatively, the reference image can be the frame preceding the current image.

[0234] The SEI message also includes first identification information, which indicates the parameters contained in the SEI message. The parameters contained in the SEI message refer to the parameters that may exist within the SEI message. The first identification information indicates which parameters the decoding end needs to decode, which helps the decoding end obtain correct parameter information and improves the decoding efficiency.

[0235] The first identifier can be represented by gfve_pupil_pred_flag. Of course, the first identifier can also be represented by any other letters and / or numbers.

[0236] The first identifier information can be used to indicate whether a predictive method is used to represent the pupil position information. If the first identifier information takes the first value, it indicates that a predictive method is used to represent the pupil position information. If the first identifier information takes the second value, it indicates that a non-predictive method is used to represent the pupil position information. Prediction means using the pupil position information of the reference image to predict the pupil position information of the current image, while no prediction means not using the pupil position information of the reference image to predict the pupil position information of the current image.

[0237] The first value can be 1, and the second value can be 0; or, the first value can be 0, and the second value can be 1. The following explanation uses the example of the first value being 1 and the second value being 0.

[0238] Whether or not a predictive approach is used to represent pupil position information determines which parameters are carried in the SEI message.

[0239] If the first identifier information is set to the first value, then the SEI message contains one or more of the following parameters: the third parameter, the fourth parameter, the seventh parameter, and the eighth parameter. In other words, the SEI message may contain the third parameter, the fourth parameter, the seventh parameter, and the eighth parameter.

[0240] For example, the SEI message includes at least one of the third and fourth parameters. If the current image includes only the left pupil, or if the encoder determines that only the position information of the left pupil needs to be transmitted, then the SEI message includes the third parameter but not the fourth parameter. If the current image includes only the right pupil, or if the encoder determines that only the position information of the right pupil needs to be transmitted, then the SEI message includes the fourth parameter but not the third parameter. If the current image includes both the left and right pupils, or if the encoder determines that the position information of both the left and right pupils needs to be transmitted, then the SEI message includes both the third and fourth parameters.

[0241] For example, the SEI message may include at least one of the seventh and eighth parameters. If the current image contains only the left pupil, or if the encoder determines that only the position information of the left pupil needs to be transmitted, then the SEI message includes the seventh parameter but not the eighth parameter. If the current image contains only the right pupil, or if the encoder determines that only the position information of the right pupil needs to be transmitted, then the SEI message includes the eighth parameter but not the seventh parameter. If the current image contains both the left and right pupils, or if the encoder determines that the position information of both the left and right pupils needs to be transmitted, then the SEI message includes both the seventh and eighth parameters.

[0242] If the first identifier is a second value, then the SEI message contains one or more of the following parameters: the first parameter, the second parameter, the fifth parameter, and the sixth parameter. In other words, the SEI message may contain the first parameter, the second parameter, or the fifth parameter and the sixth parameter.

[0243] For example, the SEI message includes at least one of the first parameter and the second parameter. If the current image includes only the left pupil, or if the encoder determines that only the position information of the left pupil needs to be transmitted, then the SEI message includes the first parameter but not the second parameter. If the current image includes only the right pupil, or if the encoder determines that only the position information of the right pupil needs to be transmitted, then the SEI message includes the second parameter but not the first parameter. If the current image includes both the left and right pupils, or if the encoder determines that the position information of both the left and right pupils needs to be transmitted, then the SEI message includes both the first and second parameters.

[0244] For example, the SEI message includes at least one of the fifth and sixth parameters. If the current image only includes the left pupil, or the encoder determines that only the position information of the left pupil needs to be transmitted, then the SEI message includes the fifth parameter but not the sixth parameter. If the current image only includes the right pupil, or the encoder determines that only the position information of the right pupil needs to be transmitted, then the SEI message includes the sixth parameter but not the fifth parameter. If the current image includes both the left and right pupils, or the encoder determines that the position information of both the left and right pupils needs to be transmitted, then the SEI message includes both the fifth and sixth parameters.

[0245] The SEI message also includes a ninth or tenth parameter.

[0246] The ninth parameter represents the bit quantization depth of the current image. Bit quantization depth refers to the number of bits required to represent location information. For example, if the bit quantization depth is 3, it means that 3 bits are needed to represent the location information.

[0247] If the SEI message contains at least one of the first position information and the second position information, the ninth parameter can be used to represent the bit quantization depth of the position information; if the SEI message contains difference information, the ninth parameter can be used to represent the bit quantization depth of the difference.

[0248] To reduce transmission overhead, the ninth parameter may be omitted from the SEI message if the bit quantization depth of the current image is the same as that of the reference image. If the ninth parameter is not present in the SEI message, it is assumed that the bit quantization depth of the current image is the same as that of the reference image.

[0249] The tenth parameter represents the change in bit quantization depth of the current image relative to the bit quantization depth of the reference image. The tenth parameter can also be understood as representing the bit quantization depth of the current image using a prediction method. This change can also be referred to as residual information.

[0250] Assuming the current image has a bit quantization depth of 1 and the reference image has a bit quantization depth of 2, the tenth parameter can be used to represent the difference between bit quantization depth 1 and bit quantization depth 2.

[0251] By using the tenth parameter to indicate the change value of the bit quantization depth, the bit quantization depth of each frame can be adjusted with fewer bits, allowing the bit quantization depth of each frame to change, which helps to improve the flexibility of the video encoding and decoding system.

[0252] The initial value of the bit quantization depth can be set to 0. If the bit quantization depth of the first frame is 3, then the change value of the tenth parameter corresponding to the first frame is 3. Assuming the reference image is the first frame, if the bit quantization depth of the second frame is 3, then the change value of the tenth parameter corresponding to the second frame is 0, as 0 only requires 1 bit. If the bit quantization depth of the third frame is 2, then the change value of the tenth parameter corresponding to the third frame is 1.

[0253] The SEI message includes first identification information, which can be used to indicate whether the SEI message includes a ninth parameter or a tenth parameter. For example, if the value of the first identification information is the first value, then the SEI message includes a tenth parameter, or the SEI message may contain a tenth parameter; if the value of the first identification information is the second value, then the SEI message includes a ninth parameter, or the SEI message may contain a ninth parameter.

[0254] As mentioned above, the first identifier value indicates that the pupil position information is represented using a predicted method, and the tenth parameter also uses a predicted method to represent the bit quantization depth. Therefore, the SEI message can include the tenth parameter. Alternatively, the first identifier value indicates that the pupil position information is represented using a non-predictive method, and the ninth parameter also uses a non-predictive method to represent the bit quantization depth. Therefore, the SEI message can include the ninth parameter.

[0255] The SEI message may also include second identification information, which indicates whether the SEI message includes the ninth parameter. If the value of the second identification information is the seventh value, it means that the SEI message includes the ninth parameter; if the value of the second identification information is the eighth value, it means that the SEI message may not include the ninth parameter. The seventh value is 1 and the eighth value is 0; or the seventh value is 0 and the eighth value is 1.

[0256] The ninth parameter can be represented by `gfve_pupil_coordinate_precision_factor_minus1`, and the tenth parameter can be represented by `gfve_pupil_coordinate_delta_precision_factor`. Of course, these parameters can also be represented using any other letters and / or numbers.

[0257] If the SEI message does not include the second identification information, it is inferred that the value of the second identification information is the eighth value.

[0258] If the SEI message does not include the ninth parameter, then the bit quantization depth of the current image is the same as that of the reference image.

[0259] The second identifier can be represented by `gfve_pupil_update_parameter_flag`. Of course, the second identifier can also be represented by any other letters and / or numbers.

[0260] The SEI message also includes third identification information, which indicates which pupil location information is included in the SEI message. For example, the third identification information can be used to indicate the location information of a pupil that is not present in the SEI message, or the pupil corresponding to the location information included in the SEI message. The pupil corresponding to the location information included in the SEI message includes at least one of the left and right pupils.

[0261] The third identifier value is the third value (e.g., 0), indicating that the SEI message does not contain pupil position information; the third identifier value is the fourth value (e.g., 1), indicating that the SEI message contains the position information of the left pupil; the third identifier value is the fifth value (e.g., 2), indicating that the SEI message contains the position information of the right pupil; the third identifier value is the sixth value (e.g., 3), indicating that the SEI message contains the position information of both the left and right pupils.

[0262] The third identifier can be represented by `gfve_pupil_coordinate_present_idx`. Of course, the third identifier can also be represented using any other letters and / or numbers.

[0263] The pupil position information mentioned above can refer to the pupil's coordinate information. The pupil's coordinate information can include at least one of the x-coordinate and y-coordinate. The x-coordinate information can include at least one of the absolute value of the x-coordinate and the sign of the x-coordinate. The y-coordinate information can include at least one of the absolute value of the y-coordinate and the sign of the y-coordinate. The sign can refer to the positive or negative sign of the coordinate value.

[0264] The information about the difference mentioned above may include at least one of the absolute value of the difference and the sign of the difference.

[0265] The following section uses location information as coordinate information as an example to explain in detail the various parameters included in the SEI message.

[0266] The first parameter mentioned above may include one or more of the following parameters: parameter 1, parameter 2, parameter 3, and parameter 4. Parameter 1 is used to represent the absolute value of the x-coordinate of the left pupil, parameter 2 is used to represent the sign of the x-coordinate of the left pupil, parameter 3 is used to represent the absolute value of the y-coordinate of the left pupil, and parameter 4 is used to represent the sign of the y-coordinate of the left pupil.

[0267] For example, the first parameter includes parameter 1 and parameter 2. Or, the first parameter includes parameter 3 and parameter 4. Or, the first parameter includes parameter 1, parameter 2, parameter 3, and parameter 4.

[0268] If the first parameter does not include parameters 2 and 4, then parameters 2 and 4 are inferred to be 0.

[0269] Parameter 1 can be represented by `gfve_pupil_left_eye_x_coordinate_abs`, parameter 2 by `gfve_pupil_left_eye_x_coordinate_sign_flag`, parameter 3 by `gfve_pupil_left_eye_y_coordinate_abs`, and parameter 4 by `gfve_pupil_left_eye_y_coordinate_sign_flag`. Of course, these parameters can also be represented using any other letters and / or numbers.

[0270] The second parameter mentioned above may include one or more of the following parameters: parameter 5, parameter 6, parameter 7, and parameter 8. Parameter 5 is used to represent the absolute value of the x-coordinate of the right pupil, parameter 6 is used to represent the sign of the x-coordinate of the right pupil, parameter 7 is used to represent the absolute value of the y-coordinate of the right pupil, and parameter 8 is used to represent the sign of the y-coordinate of the right pupil.

[0271] For example, the second parameter includes parameters 5 and 6. Or, the second parameter includes parameters 7 and 8. Or, the second parameter includes parameters 5, 6, 7, and 8.

[0272] If parameters 6 and 8 are not included in the second parameter, then parameters 6 and 8 are inferred to be 0.

[0273] Parameter 5 can be represented by `gfve_pupil_right_eye_x_coordinate_abs`, parameter 6 by `gfve_pupil_right_eye_x_coordinate_sign_flag`, parameter 7 by `gfve_pupil_right_eye_y_coordinate_abs`, and parameter 8 by `gfve_pupil_right_eye_y_coordinate_sign_flag`. Of course, these parameters can also be represented by any other letters and / or numbers.

[0274] The third parameter mentioned above may include one or more of the following parameters: parameter 9, parameter 10, parameter 11, and parameter 12. Parameter 9 is used to represent the absolute value of the difference (denoted as difference 1) between the x-coordinate of the left pupil in the current image and the x-coordinate of the left pupil in the reference image; parameter 10 is used to represent the sign of difference 1; parameter 11 is used to represent the absolute value of the difference (denoted as difference 2) between the y-coordinate of the left pupil in the current image and the y-coordinate of the left pupil in the reference image; and parameter 12 is used to represent the sign of difference 2.

[0275] For example, the third parameter includes parameters 9 and 10. Another example is that the third parameter includes parameters 11 and 12. Yet another example is that the third parameter includes parameters 9, 10, 11, and 12.

[0276] If the third parameter does not include parameters 10 and 12, then parameters 10 and 12 are inferred to be 0.

[0277] Parameter 9 can be represented by `gfve_pupil_left_eye_dx_coordinate_abs`, parameter 10 by `gfve_pupil_left_eye_dx_coordinate_sign_flag`, parameter 11 by `gfve_pupil_left_eye_dy_coordinate_abs`, and parameter 12 by `gfve_pupil_left_eye_dy_coordinate_sign_flag`. Of course, these parameters can also be represented by any other letters and / or numbers.

[0278] The fourth parameter mentioned above may include one or more of the following parameters: parameter 13, parameter 14, parameter 15, and parameter 16. Parameter 13 is used to represent the absolute value of the difference (denoted as difference 3) between the x-coordinate of the right pupil in the current image and the x-coordinate of the right pupil in the reference image; parameter 14 is used to represent the sign of difference 3; parameter 15 is used to represent the absolute value of the difference (denoted as difference 4) between the y-coordinate of the right pupil in the current image and the y-coordinate of the right pupil in the reference image; and parameter 16 is used to represent the sign of difference 4.

[0279] For example, the fourth parameter includes parameters 13 and 14. Or, the fourth parameter includes parameters 15 and 16. Or, the fourth parameter includes parameters 13, 14, 15, and 16.

[0280] If the fourth parameter does not include parameters 14 and 16, then parameters 14 and 16 are inferred to be 0.

[0281] Parameter 13 can be represented by `gfve_pupil_right_eye_dx_coordinate_abs`, parameter 14 by `gfve_pupil_right_eye_dx_coordinate_sign_flag`, parameter 15 by `gfve_pupil_right_eye_dy_coordinate_abs`, and parameter 16 by `gfve_pupil_right_eye_dy_coordinate_sign_flag`. Of course, these parameters can also be represented by any other letters and / or numbers.

[0282] The fifth parameter mentioned above may include one or more of the following parameters: parameter 17, parameter 18, parameter 19, and parameter 20. Parameter 17 is used to represent the absolute value of the difference (denoted as difference 5) between the x-coordinate of the left pupil and the x-coordinate of the right pupil in the current image; parameter 18 is used to represent the sign of difference 5; parameter 19 is used to represent the absolute value of the difference (denoted as difference 6) between the y-coordinate of the left pupil and the y-coordinate of the right pupil in the current image; and parameter 20 is used to represent the sign of difference 6.

[0283] For example, the fifth parameter includes parameters 17 and 18. Another example is that the fifth parameter includes parameters 19 and 20. Yet another example is that the fifth parameter includes parameters 17, 18, 19, and 20.

[0284] If the fifth parameter does not include parameters 18 and 20, then parameters 18 and 20 are inferred to be 0.

[0285] The sixth parameter mentioned above may include one or more of the following parameters: parameter 21, parameter 22, parameter 23, and parameter 24. Parameter 21 is used to represent the absolute value of the difference (denoted as difference 7) between the x-coordinate of the right pupil and the x-coordinate of the left pupil in the current image; parameter 22 is used to represent the sign of difference 7; parameter 23 is used to represent the absolute value of the difference (denoted as difference 8) between the y-coordinate of the right pupil and the y-coordinate of the left pupil in the current image; and parameter 24 is used to represent the sign of difference 8.

[0286] For example, the sixth parameter includes parameters 21 and 22. Another example is that the sixth parameter includes parameters 23 and 24. Yet another example is that the sixth parameter includes parameters 21, 22, 23, and 24.

[0287] If the sixth parameter does not include parameters 22 and 24, then parameters 22 and 24 are inferred to be 0.

[0288] Parameter 21 can be represented by `gfve_pupil_pre_residual_right_eye_x_coordinate_abs`, parameter 22 by `gfve_pupil_pre_residual_right_eye_x_coordinate_sign_flag`, parameter 23 by `gfve_pupil_pre_residual_right_eye_y_coordinate_abs`, and parameter 24 by `gfve_pupil_pre_residual_right_eye_y_coordinate_sign_flag`. Of course, these parameters can also be represented by any other letters and / or numbers.

[0289] The seventh parameter mentioned above may include one or more of the following parameters: parameter 25, parameter 26, parameter 27, and parameter 28. Parameter 25 is used to represent the absolute value of the difference between difference 1 and difference 3, parameter 26 is used to represent the sign of the difference between difference 1 and difference 3, parameter 27 is used to represent the absolute value of the difference between difference 2 and difference 4, and parameter 28 is used to represent the sign of the difference between difference 2 and difference 4.

[0290] For example, the seventh parameter includes parameters 25 and 26. Another example is that the seventh parameter includes parameters 27 and 28. Yet another example is that the seventh parameter includes parameters 25, 26, 27, and 28.

[0291] If parameters 26 and 28 are not included in the seventh parameter, then parameters 26 and 28 are inferred to be 0.

[0292] The eighth parameter mentioned above may include one or more of the following parameters: parameter 29, parameter 30, parameter 31, and parameter 32. Parameter 29 is used to represent the absolute value of the difference between difference 3 and difference 1, parameter 30 is used to represent the sign of the difference between difference 3 and difference 1, parameter 31 is used to represent the absolute value of the difference between difference 4 and difference 2, and parameter 32 is used to represent the sign of the difference between difference 4 and difference 2.

[0293] For example, the eighth parameter includes parameters 29 and 30. Another example is that the eighth parameter includes parameters 31 and 32. Yet another example is that the eighth parameter includes parameters 29, 30, 31, and 32.

[0294] If parameters 30 and 32 are not included in the eighth parameter, then parameters 30 and 32 are inferred to be 0.

[0295] Parameter 29 can be represented by `gfve_pupil_pre_residual_right_eye_dx_coordinate_abs`, parameter 30 by `gfve_pupil_pre_residual_right_eye_dx_coordinate_sign_flag`, parameter 31 by `gfve_pupil_pre_residual_right_eye_dy_coordinate_abs`, and parameter 32 by `gfve_pupil_pre_residual_right_eye_dy_coordinate_sign_flag`. Of course, these parameters can also be represented by any other letters and / or numbers.

[0296] The above text combined Figure 5 This document describes in detail some of the decoding methods provided in embodiments of this application. The following section, in conjunction with... Figure 6 This application provides a detailed description of some of the encoding methods provided in its embodiments.

[0297] Figure 6 This is a flowchart illustrating the encoding method provided in some embodiments of this application. Figure 6 The method can be applied to encoders.

[0298] See Figure 6 In step S610, the position information of the pupil contained in the current image is determined.

[0299] Step S610 may include: extracting pupil position information using an analysis model. For example, the current image can be input into the analysis model, the image features of the current image and the pupil position information can be extracted by the analysis model, and the image features and pupil position information can be encoded into the bitstream.

[0300] The above analysis model can be used as an analysis model in a generative face video coding framework.

[0301] In step S620, based on the pupil position information, the parameters in the SEI message corresponding to the current image are encoded. The pupil position information is used to determine the eye motion information contained in the current image.

[0302] After step S620, the SEI message can be encoded into the bitstream.

[0303] The current image can be any frame from a video. For example, the current image can be a base image or a driving image. The base image can be the first frame of the video, and the driving image can be any other image in the video besides the first frame; the driving image can also be called a subsequent image. Of course, the base image can also be any other image in the video besides the first frame. In some embodiments, the base image can also be called a reference image.

[0304] In some implementations, if the current image is the base image, the encoder can extract image features from the base image (such as basic facial texture information) and encode these features into the bitstream. If the current image is the driving image, the encoder can extract image features and pupil position information from the driving image and encode these features and pupil position information into the bitstream. For example, the encoder can input the driving image into the analysis model to obtain image features and pupil position information from the driving image.

[0305] SEI messages can be either GFV SEI messages or GFVE SEI messages. The following explanation will primarily use GFVE SEI messages as an example.

[0306] An image can correspond to one SEI message. Each SEI message can be transmitted independently, or multiple SEI messages can be transmitted together, such as multiple SEI messages transmitted through a single carrier. If multiple SEI messages are transmitted together, then each of the multiple SEI messages can correspond to a different index.

[0307] Some embodiments of this application can generate eye position information based on pupil position information, making the generated eye position information more accurate.

[0308] The SEI message may include one or more parameters that can be used by the decoder to determine the pupil's position information. In some embodiments of this application, the term "parameter" may also be replaced with the term "syntax element." The parameters included in the SEI message are described in detail below.

[0309] SEI messages may include one or more of the following parameters: first parameter, second parameter, third parameter, fourth parameter, fifth parameter, sixth parameter, seventh parameter, and eighth parameter. These parameters are described below.

[0310] The first parameter represents the first position information. The first position information is the position of the left eye pupil in the current image.

[0311] The second parameter represents the second location information. The second location information is the location of the right eye pupil in the current image.

[0312] The first and second position information can be absolute position information. Taking coordinate information as an example, the first position information can be the coordinates of the left pupil, and the second position information can be the coordinates of the right pupil. The encoder can encode the first parameter based on the position information of the left pupil. The encoder can encode the second parameter based on the position information of the right pupil.

[0313] The third parameter represents the difference between the first position information and the third position information. This difference can also be called the residual. Hereafter, this difference will also be referred to as the first difference. The third position information is the position information of the left pupil in the reference image. Alternatively, the third parameter can be used to represent the residual information obtained by predicting the position information of the left pupil in the current image using the position information of the left pupil in the reference image. By carrying the difference information in the SEI message, the encoding efficiency of the pupil position information can be improved.

[0314] The first and third location information are location information for the pupil of the same eye in the same face. For example, if the image includes face a and face b, the first and third location information can be the location information for the left pupil of face a, or the location information for the left pupil of face b.

[0315] Taking position information as coordinate information as an example, the difference between the first position information and the third position information can refer to the difference between the coordinates of the left pupil in the current image and the coordinates of the left pupil in the reference image.

[0316] The fourth parameter represents the difference between the second positional information and the fourth positional information. This difference can also be called the residual. Hereafter, this difference will also be referred to as the second difference. The fourth positional information is the position of the right pupil in the reference image. Alternatively, the fourth parameter can be used to represent the residual information obtained by predicting the position of the right pupil in the current image using the position information of the right pupil in the reference image. By carrying the difference information in the SEI message, the encoding efficiency of the pupil position information can be improved.

[0317] The second and fourth location information are location information for the pupil of the same eye in the same face. For example, if the image includes face a and face b, the second and fourth location information could be the location information for the right pupil of face a, or the location information for the right pupil of face b.

[0318] Taking position information as coordinate information as an example, the difference between the second position information and the fourth position information can refer to the difference between the coordinates of the right pupil in the current image and the coordinates of the right pupil in the reference image.

[0319] The fifth parameter represents the difference between the first position information and the second position information. Alternatively, the fifth parameter can be used to represent the residual information obtained by predicting the position of the left pupil using the position information of the right pupil. By carrying the difference information in the SEI message, the encoding efficiency of the pupil position information can be improved.

[0320] Taking location information as coordinate information as an example, the difference between the first location information and the second location information can refer to the difference between the coordinate values ​​of the left pupil and the right pupil.

[0321] The sixth parameter represents the difference between the second position information and the first position information. Alternatively, the sixth parameter can be used to represent the residual information obtained by predicting the position of the right pupil using the position information of the left pupil. By carrying the difference information in the SEI message, the encoding efficiency of the pupil position information can be improved.

[0322] Taking location information as coordinate information as an example, the difference between the second location information and the first location information can refer to the difference between the coordinate values ​​of the right pupil and the left pupil.

[0323] The seventh parameter represents the difference between the first difference and the second difference. By performing difference processing on the first and second differences, the encoding efficiency of pupil position information can be further improved.

[0324] Taking location information as coordinate information as an example, the first difference is the difference in the coordinates of the left pupil, the second difference is the difference in the coordinates of the right pupil, and the seventh parameter can be used to represent the residual information obtained by predicting the difference in the coordinates of the left pupil using the difference in the coordinates of the right pupil.

[0325] The eighth parameter represents the difference between the second difference and the first difference. By performing difference processing on the second and first differences, the encoding efficiency of pupil position information can be further improved.

[0326] Taking location information as coordinate information as an example, the first difference is the difference in the coordinates of the left pupil, the second difference is the difference in the coordinates of the right pupil, and the eighth parameter can be used to represent the residual information obtained by predicting the difference in the coordinates of the right pupil using the difference in the coordinates of the left pupil.

[0327] For example, an SEI message may include a first parameter. Or, an SEI message may include a second parameter. Or, an SEI message may include both a first and a second parameter. Or, an SEI message may include a third parameter. Or, an SEI message may include a fourth parameter. Or, an SEI message may include both a third and a fourth parameter. Or, an SEI message may include a fifth parameter. Or, an SEI message may include a sixth parameter. Or, an SEI message may include both a fifth and a sixth parameter. Or, an SEI message may include a seventh parameter. Or, an SEI message may include an eighth parameter. Or, an SEI message may include both a seventh and an eighth parameter.

[0328] The reference image mentioned above can be the first frame of the video, or another image in the video. For example, a reference image can be set every p frames. Alternatively, the reference image can be the frame preceding the current image.

[0329] The SEI message also includes first identification information, which indicates the parameters contained in the SEI message. The parameters contained in the SEI message refer to the parameters that may exist within the SEI message. The first identification information indicates which parameters the decoding end needs to decode, which helps the decoding end obtain correct parameter information and improves the decoding efficiency.

[0330] The first identifier can be represented by gfve_pupil_pred_flag. Of course, the first identifier can also be represented by any other letters and / or numbers.

[0331] The first identifier information can be used to indicate whether a predictive method is used to represent the pupil position information. If the first identifier information takes the first value, it indicates that a predictive method is used to represent the pupil position information. If the first identifier information takes the second value, it indicates that a non-predictive method is used to represent the pupil position information. Prediction means using the pupil position information of the reference image to predict the pupil position information of the current image, while no prediction means not using the pupil position information of the reference image to predict the pupil position information of the current image.

[0332] The first value can be 1, and the second value can be 0; or, the first value can be 0, and the second value can be 1. The following explanation uses the example of the first value being 1 and the second value being 0.

[0333] Whether or not a predictive approach is used to represent pupil position information determines which parameters are carried in the SEI message.

[0334] If the first identifier information is set to the first value, then the SEI message contains one or more of the following parameters: the third parameter, the fourth parameter, the seventh parameter, and the eighth parameter. In other words, the SEI message may contain the third parameter, the fourth parameter, the seventh parameter, and the eighth parameter.

[0335] For example, the SEI message includes at least one of the third and fourth parameters. If the current image includes only the left pupil, or if the encoder determines that only the position information of the left pupil needs to be transmitted, then the SEI message includes the third parameter but not the fourth parameter. If the current image includes only the right pupil, or if the encoder determines that only the position information of the right pupil needs to be transmitted, then the SEI message includes the fourth parameter but not the third parameter. If the current image includes both the left and right pupils, or if the encoder determines that the position information of both the left and right pupils needs to be transmitted, then the SEI message includes both the third and fourth parameters.

[0336] For example, the SEI message may include at least one of the seventh and eighth parameters. If the current image contains only the left pupil, or if the encoder determines that only the position information of the left pupil needs to be transmitted, then the SEI message includes the seventh parameter but not the eighth parameter. If the current image contains only the right pupil, or if the encoder determines that only the position information of the right pupil needs to be transmitted, then the SEI message includes the eighth parameter but not the seventh parameter. If the current image contains both the left and right pupils, or if the encoder determines that the position information of both the left and right pupils needs to be transmitted, then the SEI message includes both the seventh and eighth parameters.

[0337] If the first identifier is a second value, then the SEI message contains one or more of the following parameters: the first parameter, the second parameter, the fifth parameter, and the sixth parameter. In other words, the SEI message may contain the first parameter, the second parameter, or the fifth parameter and the sixth parameter.

[0338] For example, the SEI message includes at least one of the first parameter and the second parameter. If the current image includes only the left pupil, or if the encoder determines that only the position information of the left pupil needs to be transmitted, then the SEI message includes the first parameter but not the second parameter. If the current image includes only the right pupil, or if the encoder determines that only the position information of the right pupil needs to be transmitted, then the SEI message includes the second parameter but not the first parameter. If the current image includes both the left and right pupils, or if the encoder determines that the position information of both the left and right pupils needs to be transmitted, then the SEI message includes both the first and second parameters.

[0339] For example, the SEI message includes at least one of the fifth and sixth parameters. If the current image only includes the left pupil, or the encoder determines that only the position information of the left pupil needs to be transmitted, then the SEI message includes the fifth parameter but not the sixth parameter. If the current image only includes the right pupil, or the encoder determines that only the position information of the right pupil needs to be transmitted, then the SEI message includes the sixth parameter but not the fifth parameter. If the current image includes both the left and right pupils, or the encoder determines that the position information of both the left and right pupils needs to be transmitted, then the SEI message includes both the fifth and sixth parameters.

[0340] The SEI message also includes a ninth or tenth parameter.

[0341] The ninth parameter represents the bit quantization depth of the current image. Bit quantization depth refers to the number of bits required to represent location information. For example, if the bit quantization depth is 3, it means that 3 bits are needed to represent the location information.

[0342] If the SEI message contains at least one of the first position information and the second position information, the ninth parameter can be used to represent the bit quantization depth of the position information; if the SEI message contains difference information, the ninth parameter can be used to represent the bit quantization depth of the difference.

[0343] To reduce transmission overhead, the ninth parameter may be omitted from the SEI message if the bit quantization depth of the current image is the same as that of the reference image. If the ninth parameter is not present in the SEI message, it is assumed that the bit quantization depth of the current image is the same as that of the reference image.

[0344] The tenth parameter represents the change in bit quantization depth of the current image relative to the bit quantization depth of the reference image. The tenth parameter can also be understood as representing the bit quantization depth of the current image using a prediction method. This change can also be referred to as residual information.

[0345] Assuming the current image has a bit quantization depth of 1 and the reference image has a bit quantization depth of 2, the tenth parameter can be used to represent the difference between bit quantization depth 1 and bit quantization depth 2.

[0346] By using the tenth parameter to indicate the change value of the bit quantization depth, the bit quantization depth of each frame can be adjusted with fewer bits, allowing the bit quantization depth of each frame to change, which helps to improve the flexibility of the video encoding and decoding system.

[0347] The initial value of the bit quantization depth can be set to 0. If the bit quantization depth of the first frame is 3, then the change value of the tenth parameter corresponding to the first frame is 3. Assuming the reference image is the first frame, if the bit quantization depth of the second frame is 3, then the change value of the tenth parameter corresponding to the second frame is 0, as 0 only requires 1 bit. If the bit quantization depth of the third frame is 2, then the change value of the tenth parameter corresponding to the third frame is 1.

[0348] The SEI message includes first identification information, which can be used to indicate whether the SEI message includes a ninth parameter or a tenth parameter. For example, if the value of the first identification information is the first value, then the SEI message includes a tenth parameter, or the SEI message may contain a tenth parameter; if the value of the first identification information is the second value, then the SEI message includes a ninth parameter, or the SEI message may contain a ninth parameter.

[0349] As mentioned above, the first identifier value indicates that the pupil position information is represented using a predicted method, and the tenth parameter also uses a predicted method to represent the bit quantization depth. Therefore, the SEI message can include the tenth parameter. Alternatively, the first identifier value indicates that the pupil position information is represented using a non-predictive method, and the ninth parameter also uses a non-predictive method to represent the bit quantization depth. Therefore, the SEI message can include the ninth parameter.

[0350] The SEI message may also include second identification information, which indicates whether the SEI message includes the ninth parameter. If the value of the second identification information is the seventh value, it means that the SEI message includes the ninth parameter; if the value of the second identification information is the eighth value, it means that the SEI message may not include the ninth parameter. The seventh value is 1 and the eighth value is 0; or the seventh value is 0 and the eighth value is 1.

[0351] The ninth parameter can be represented by `gfve_pupil_coordinate_precision_factor_minus1`, and the tenth parameter can be represented by `gfve_pupil_coordinate_delta_precision_factor`. Of course, these parameters can also be represented using any other letters and / or numbers.

[0352] If the SEI message does not include the second identification information, it is inferred that the value of the second identification information is the eighth value.

[0353] If the SEI message does not include the ninth parameter, then the bit quantization depth of the current image is the same as that of the reference image.

[0354] The second identifier can be represented by `gfve_pupil_update_parameter_flag`. Of course, the second identifier can also be represented by any other letters and / or numbers.

[0355] The SEI message also includes third identification information, which indicates which pupil location information is included in the SEI message. For example, the third identification information can be used to indicate the location information of a pupil that is not present in the SEI message, or the pupil corresponding to the location information included in the SEI message. The pupil corresponding to the location information included in the SEI message includes at least one of the left and right pupils.

[0356] The third identifier value is the third value (e.g., 0), indicating that the SEI message does not contain pupil position information; the third identifier value is the fourth value (e.g., 1), indicating that the SEI message contains the position information of the left pupil; the third identifier value is the fifth value (e.g., 2), indicating that the SEI message contains the position information of the right pupil; the third identifier value is the sixth value (e.g., 3), indicating that the SEI message contains the position information of both the left and right pupils.

[0357] The third identifier can be represented by `gfve_pupil_coordinate_present_idx`. Of course, the third identifier can also be represented using any other letters and / or numbers.

[0358] The pupil position information mentioned above can refer to the pupil's coordinate information. The pupil's coordinate information can include at least one of the x-coordinate and y-coordinate. The x-coordinate information can include at least one of the absolute value of the x-coordinate and the sign of the x-coordinate. The y-coordinate information can include at least one of the absolute value of the y-coordinate and the sign of the y-coordinate. The sign can refer to the positive or negative sign of the coordinate value.

[0359] The information about the difference mentioned above may include at least one of the absolute value of the difference and the sign of the difference.

[0360] The following section uses location information as coordinate information as an example to explain in detail the various parameters included in the SEI message.

[0361] The first parameter mentioned above may include one or more of the following parameters: parameter 1, parameter 2, parameter 3, and parameter 4. Parameter 1 is used to represent the absolute value of the x-coordinate of the left pupil, parameter 2 is used to represent the sign of the x-coordinate of the left pupil, parameter 3 is used to represent the absolute value of the y-coordinate of the left pupil, and parameter 4 is used to represent the sign of the y-coordinate of the left pupil.

[0362] For example, the first parameter includes parameter 1 and parameter 2. Or, the first parameter includes parameter 3 and parameter 4. Or, the first parameter includes parameter 1, parameter 2, parameter 3, and parameter 4.

[0363] If the first parameter does not include parameters 2 and 4, then parameters 2 and 4 are inferred to be 0.

[0364] Parameter 1 can be represented by `gfve_pupil_left_eye_x_coordinate_abs`, parameter 2 by `gfve_pupil_left_eye_x_coordinate_sign_flag`, parameter 3 by `gfve_pupil_left_eye_y_coordinate_abs`, and parameter 4 by `gfve_pupil_left_eye_y_coordinate_sign_flag`. Of course, these parameters can also be represented using any other letters and / or numbers.

[0365] The second parameter mentioned above may include one or more of the following parameters: parameter 5, parameter 6, parameter 7, and parameter 8. Parameter 5 is used to represent the absolute value of the x-coordinate of the right pupil, parameter 6 is used to represent the sign of the x-coordinate of the right pupil, parameter 7 is used to represent the absolute value of the y-coordinate of the right pupil, and parameter 8 is used to represent the sign of the y-coordinate of the right pupil.

[0366] For example, the second parameter includes parameters 5 and 6. Or, the second parameter includes parameters 7 and 8. Or, the second parameter includes parameters 5, 6, 7, and 8.

[0367] If parameters 6 and 8 are not included in the second parameter, then parameters 6 and 8 are inferred to be 0.

[0368] Parameter 5 can be represented by `gfve_pupil_right_eye_x_coordinate_abs`, parameter 6 by `gfve_pupil_right_eye_x_coordinate_sign_flag`, parameter 7 by `gfve_pupil_right_eye_y_coordinate_abs`, and parameter 8 by `gfve_pupil_right_eye_y_coordinate_sign_flag`. Of course, these parameters can also be represented by any other letters and / or numbers.

[0369] The third parameter mentioned above may include one or more of the following parameters: parameter 9, parameter 10, parameter 11, and parameter 12. Parameter 9 is used to represent the absolute value of the difference (denoted as difference 1) between the x-coordinate of the left pupil in the current image and the x-coordinate of the left pupil in the reference image; parameter 10 is used to represent the sign of difference 1; parameter 11 is used to represent the absolute value of the difference (denoted as difference 2) between the y-coordinate of the left pupil in the current image and the y-coordinate of the left pupil in the reference image; and parameter 12 is used to represent the sign of difference 2.

[0370] For example, the third parameter includes parameters 9 and 10. Another example is that the third parameter includes parameters 11 and 12. Yet another example is that the third parameter includes parameters 9, 10, 11, and 12.

[0371] If the third parameter does not include parameters 10 and 12, then parameters 10 and 12 are inferred to be 0.

[0372] Parameter 9 can be represented by `gfve_pupil_left_eye_dx_coordinate_abs`, parameter 10 by `gfve_pupil_left_eye_dx_coordinate_sign_flag`, parameter 11 by `gfve_pupil_left_eye_dy_coordinate_abs`, and parameter 12 by `gfve_pupil_left_eye_dy_coordinate_sign_flag`. Of course, these parameters can also be represented by any other letters and / or numbers.

[0373] The fourth parameter mentioned above may include one or more of the following parameters: parameter 13, parameter 14, parameter 15, and parameter 16. Parameter 13 is used to represent the absolute value of the difference (denoted as difference 3) between the x-coordinate of the right pupil in the current image and the x-coordinate of the right pupil in the reference image; parameter 14 is used to represent the sign of difference 3; parameter 15 is used to represent the absolute value of the difference (denoted as difference 4) between the y-coordinate of the right pupil in the current image and the y-coordinate of the right pupil in the reference image; and parameter 16 is used to represent the sign of difference 4.

[0374] For example, the fourth parameter includes parameters 13 and 14. Another example is that the fourth parameter includes parameters 15 and 16. Yet another example is that the fourth parameter includes parameters 13, 14, 15, and 16.

[0375] If the fourth parameter does not include parameters 14 and 16, then parameters 14 and 16 are inferred to be 0.

[0376] Parameter 13 can be represented by `gfve_pupil_right_eye_dx_coordinate_abs`, parameter 14 by `gfve_pupil_right_eye_dx_coordinate_sign_flag`, parameter 15 by `gfve_pupil_right_eye_dy_coordinate_abs`, and parameter 16 by `gfve_pupil_right_eye_dy_coordinate_sign_flag`. Of course, these parameters can also be represented by any other letters and / or numbers.

[0377] The fifth parameter mentioned above may include one or more of the following parameters: parameter 17, parameter 18, parameter 19, and parameter 20. Parameter 17 is used to represent the absolute value of the difference (denoted as difference 5) between the x-coordinate of the left pupil and the x-coordinate of the right pupil in the current image; parameter 18 is used to represent the sign of difference 5; parameter 19 is used to represent the absolute value of the difference (denoted as difference 6) between the y-coordinate of the left pupil and the y-coordinate of the right pupil in the current image; and parameter 20 is used to represent the sign of difference 6.

[0378] For example, the fifth parameter includes parameters 17 and 18. Another example is that the fifth parameter includes parameters 19 and 20. Yet another example is that the fifth parameter includes parameters 17, 18, 19, and 20.

[0379] If the fifth parameter does not include parameters 18 and 20, then parameters 18 and 20 are inferred to be 0.

[0380] The sixth parameter mentioned above may include one or more of the following parameters: parameter 21, parameter 22, parameter 23, and parameter 24. Parameter 21 is used to represent the absolute value of the difference (denoted as difference 7) between the x-coordinate of the right pupil and the x-coordinate of the left pupil in the current image; parameter 22 is used to represent the sign of difference 7; parameter 23 is used to represent the absolute value of the difference (denoted as difference 8) between the y-coordinate of the right pupil and the y-coordinate of the left pupil in the current image; and parameter 24 is used to represent the sign of difference 8.

[0381] For example, the sixth parameter includes parameters 21 and 22. Another example is that the sixth parameter includes parameters 23 and 24. Yet another example is that the sixth parameter includes parameters 21, 22, 23, and 24.

[0382] If the sixth parameter does not include parameters 22 and 24, then parameters 22 and 24 are inferred to be 0.

[0383] Parameter 21 can be represented by `gfve_pupil_pre_residual_right_eye_x_coordinate_abs`, parameter 22 by `gfve_pupil_pre_residual_right_eye_x_coordinate_sign_flag`, parameter 23 by `gfve_pupil_pre_residual_right_eye_y_coordinate_abs`, and parameter 24 by `gfve_pupil_pre_residual_right_eye_y_coordinate_sign_flag`. Of course, these parameters can also be represented by any other letters and / or numbers.

[0384] The seventh parameter mentioned above may include one or more of the following parameters: parameter 25, parameter 26, parameter 27, and parameter 28. Parameter 25 is used to represent the absolute value of the difference between difference 1 and difference 3, parameter 26 is used to represent the sign of the difference between difference 1 and difference 3, parameter 27 is used to represent the absolute value of the difference between difference 2 and difference 4, and parameter 28 is used to represent the sign of the difference between difference 2 and difference 4.

[0385] For example, the seventh parameter includes parameters 25 and 26. Another example is that the seventh parameter includes parameters 27 and 28. Yet another example is that the seventh parameter includes parameters 25, 26, 27, and 28.

[0386] If parameters 26 and 28 are not included in the seventh parameter, then parameters 26 and 28 are inferred to be 0.

[0387] The eighth parameter mentioned above may include one or more of the following parameters: parameter 29, parameter 30, parameter 31, and parameter 32. Parameter 29 is used to represent the absolute value of the difference between difference 3 and difference 1, parameter 30 is used to represent the sign of the difference between difference 3 and difference 1, parameter 31 is used to represent the absolute value of the difference between difference 4 and difference 2, and parameter 32 is used to represent the sign of the difference between difference 4 and difference 2.

[0388] For example, the eighth parameter includes parameters 29 and 30. Another example is that the eighth parameter includes parameters 31 and 32. Yet another example is that the eighth parameter includes parameters 29, 30, 31, and 32.

[0389] If parameters 30 and 32 are not included in the eighth parameter, then parameters 30 and 32 are inferred to be 0.

[0390] Parameter 29 can be represented by `gfve_pupil_pre_residual_right_eye_dx_coordinate_abs`, parameter 30 by `gfve_pupil_pre_residual_right_eye_dx_coordinate_sign_flag`, parameter 31 by `gfve_pupil_pre_residual_right_eye_dy_coordinate_abs`, and parameter 32 by `gfve_pupil_pre_residual_right_eye_dy_coordinate_sign_flag`. Of course, these parameters can also be represented by any other letters and / or numbers.

[0391] To facilitate understanding, the SEI messages in some embodiments of this application will be described in detail below with reference to three examples.

[0392] Example 1

[0393] Table 8

[0394]

[0395]

[0396] The following explains the grammatical elements involved in Table 8.

[0397] `gfve_pupil_coordinate_present_idx` represents the index of which pupil information is sent. When `gfve_pupil_coordinate_present_idx` equals 0, it means that pupil position information does not exist; when `gfve_pupil_coordinate_present_idx` equals 1, it means that only the left eye pupil position information is included; when `gfve_pupil_coordinate_present_idx` equals 2, it means that only the right eye pupil position information is included; when `gfve_pupil_coordinate_present_idx` equals 3, it means that both the left and right eye pupil position information are included.

[0398] When `gfve_pupil_pred_flag` equals 1, it indicates that the syntax elements `gfve_pupil_left_eye_dx_coordinate_abs`, `gfve_pupil_left_eye_dx_coordinate_sign_flag`, `gfve_pupil_left_eye_dy_coordinate_abs`, and `gfve_pupil_left_eye_dy_coordinate_sign_flag` may exist, as well as `gfve_pupil_right_eye_dx_coordinate_abs`, `gfve_pupil_right_eye_dx_coordinate_sign_flag`, and `gfve_pupil_`. right_eye_dy_coordinate_abs, gfve_pupil_right_eye_dy_coordinate_sign_flag; or there may be syntax elements gfve_pupil_pre_residual_right_eye_dx_coordinate_abs, gfve_pupil_pre_r esidual_right_eye_dx_coordinate_sign_flag, gfve_pupil_pre_residual_right_eye_dy_coordinate_abs, gfve_pupil_pre_residual_right_eye_dy_coordinate_sign_flag. When gfve_pupil_pred_flag equals 0, it indicates that the syntax elements gfve_pupil_left_eye_x_coordinate_abs, gfve_pupil_left_eye_x_coordinate_sign_flag, gfve_pupil_left_eye_y_coordinate_abs, and gfve_pupil_left_eye_y_coordinate_sign_flag may exist, as may gfve_pupil_right_eye_x_coordinate_abs, gfve_pupil_right_eye_x_coordinate_sign_flag, gfve_pupil_right_eye_y_coordinate_abs, and gfve_pupil_right_eye_y_coordinate_sign_flag.When gfve_pupil_pred_flag is 1, the corresponding difference value refers to the difference between the pupil position and the pupil position in the reference frame (such as the previous frame).

[0399] When `gfve_pupil_update_parameter_flag` equals 1, it indicates the existence of the syntax element `gfve_pupil_coordinate_precision_factor_minus1`; when `gfve_pupil_update_parameter_flag` equals 0, it indicates the possible non-existence of the syntax element `gfve_pupil_coordinate_precision_factor_minus1`. If `gfve_pupil_update_parameter_flag` does not exist, its value is inferred to be 0.

[0400] gfve_pupil_coordinate_precision_factor_minus1 plus 1 represents the syntax elements gfve_pupil_left_eye_dx_coordinate_abs, gfve_pupil_left_eye_dy_coordinate_abs, gfve_pupil_right_eye_dx_coordinate_abs, gfve_pupil_right_eye_dy_coordinate_abs, gfve_pupil_pre_residual_rig ht_eye_dx_coordinate_abs, gfve_pupil_pre_residual_right_eye_dy_coordinate_abs, gfve_pupil_left_eye_x_coordinate_abs, gfv Bit quantization depth of e_pupil_left_eye_y_coordinate_abs, gfve_pupil_right_eye_x_coordinate_abs, gfve_pupil_right_eye_y_coordinate_abs. The value of gfve_pupil_coordinate_precision_factor_minus1 ranges from 0 to 31 (inclusive). When gfve_pupil_coordinate_precision_factor_minus1 is not equal to 0, gfve_pupil_update_parameter_flag = 0, and gfve_pupil_pred_flag = 1, the value of gfve_pupil_coordinate_precision_factor_minus1 should be inferred to be the same as the gfve_pupil_coordinate_precision_factor_minus1 of a previous SEI message with the same gfve_id and gfve_pupil_update_parameter_flag = 1.

[0401] gfve_pupil_left_eye_dx_coordinate_abs represents the absolute value of the difference in the x-coordinate of the left eye.

[0402] `gfve_pupil_left_eye_dx_coordinate_sign_flag` represents the sign of the difference in the left eye's x-coordinate. If `gfve_pupil_left_eye_dx_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0403] gfve_pupil_left_eye_dy_coordinate_abs represents the absolute value of the difference in the y-coordinate of the left eye.

[0404] `gfve_pupil_left_eye_dy_coordinate_sign_flag` represents the sign of the difference in the y-coordinates of the left eye. If `gfve_pupil_left_eye_dy_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0405] gfve_pupil_right_eye_dx_coordinate_abs represents the absolute value of the difference in the x-coordinate of the right eye.

[0406] `gfve_pupil_right_eye_dx_coordinate_sign_flag` represents the sign of the difference in the x-coordinate of the right eye. If `gfve_pupil_right_eye_dx_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0407] gfve_pupil_right_eye_dy_coordinate_abs represents the absolute value of the difference in the y-coordinate of the right eye.

[0408] `gfve_pupil_right_eye_dy_coordinate_sign_flag` represents the sign of the difference in the y-coordinate of the right eye. If `gfve_pupil_right_eye_dy_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0409] gfve_pupil_pre_residual_right_eye_dx_coordinate_abs represents the absolute value of the residual used to predict the x-coordinate difference of the right eye using the x-coordinate difference of the left eye.

[0410] `gfve_pupil_pre_residual_right_eye_dx_coordinate_sign_flag` represents the sign of the residual used to predict the x-coordinate difference of the right eye using the x-coordinate difference of the left eye. If `gfve_pupil_pre_residual_right_eye_dx_coordinate_sign_flag` does not exist, it is inferred to be equal to 0.

[0411] gfve_pupil_pre_residual_right_eye_dy_coordinate_abs represents the absolute value of the residual used to predict the right eye's y-coordinate difference using the left eye's y-coordinate difference.

[0412] `gfve_pupil_pre_residual_right_eye_dy_coordinate_sign_flag` represents the sign of the residual used to predict the right eye's y-coordinate difference using the left eye's y-coordinate difference. If `gfve_pupil_pre_residual_right_eye_dy_coordinate_sign_flag` does not exist, it is inferred to be equal to 0.

[0413] gfve_pupil_left_eye_x_coordinate_abs represents the absolute value of the left eye's x-coordinate.

[0414] `gfve_pupil_left_eye_x_coordinate_sign_flag` represents the sign of the left eye's x-coordinate. If `gfve_pupil_left_eye_x_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0415] gfve_pupil_left_eye_y_coordinate_abs represents the absolute value of the y-coordinate of the left eye.

[0416] `gfve_pupil_left_eye_y_coordinate_sign_flag` represents the sign of the y-coordinate of the left eye. If `gfve_pupil_left_eye_y_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0417] gfve_pupil_right_eye_x_coordinate_abs represents the absolute value of the right eye's x-coordinate.

[0418] `gfve_pupil_right_eye_x_coordinate_sign_flag` represents the sign of the right eye's x-coordinate. If `gfve_pupil_right_eye_x_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0419] gfve_pupil_right_eye_y_coordinate_abs represents the absolute value of the y-coordinate of the right eye.

[0420] `gfve_pupil_right_eye_y_coordinate_sign_flag` represents the sign of the right eye's y-coordinate. If `gfve_pupil_right_eye_y_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0421] Example 2

[0422] Compared to Example 1, Example 2 introduces a scheme to predict the position of the right eye pupil using the position information of the left eye pupil.

[0423] Table 9

[0424]

[0425]

[0426]

[0427] The grammatical elements mentioned in Table 9 are explained below.

[0428] `gfve_pupil_coordinate_present_idx` represents the index of which pupil information is sent. When `gfve_pupil_coordinate_present_idx` equals 0, it means that pupil position information does not exist; when `gfve_pupil_coordinate_present_idx` equals 1, it means that only the left eye pupil position information is included; when `gfve_pupil_coordinate_present_idx` equals 2, it means that only the right eye pupil position information is included; when `gfve_pupil_coordinate_present_idx` equals 3, it means that both the left and right eye pupil position information are included.

[0429] When `gfve_pupil_pred_flag` equals 1, it indicates that the syntax elements `gfve_pupil_left_eye_dx_coordinate_abs`, `gfve_pupil_left_eye_dx_coordinate_sign_flag`, `gfve_pupil_left_eye_dy_coordinate_abs`, and `gfve_pupil_left_eye_dy_coordinate_sign_flag` may exist, as well as `gfve_pupil_right_eye_dx_coordinate_abs`, `gfve_pupil_right_eye_dx_coordinate_sign_flag`, and `gfve_pupil_`. right_eye_dy_coordinate_abs, gfve_pupil_right_eye_dy_coordinate_sign_flag, or there may be syntax elements gfve_pupil_pre_residual_right_eye_dx_coordinate_abs, gfve_pupil_pre_r esidual_right_eye_dx_coordinate_sign_flag, gfve_pupil_pre_residual_right_eye_dy_coordinate_abs, gfve_pupil_pre_residual_right_eye_dy_coordinate_sign_flag.When `gfve_pupil_pred_flag` equals 0, it indicates that the syntax elements `gfve_pupil_left_eye_x_coordinate_abs`, `gfve_pupil_left_eye_x_coordinate_sign_flag`, `gfve_pupil_left_eye_y_coordinate_abs`, and `gfve_pupil_left_eye_y_coordinate_sign_flag` may exist, as well as `gfve_pupil_right_eye_x_coordinate_abs`, `gfve_pupil_right_eye_x_coordinate_sign_flag`, and `gfve_pupil_`. `right_eye_y_coordinate_abs`, `gfve_pupil_right_eye_y_coordinate_sign_flag`; or possibly the syntax elements `gfve_pupil_pre_residual_right_eye_x_coordinate_abs`, `gfve_pupil_pre_residual_right_eye_x_coordinate_sign_flag`, `gfve_pupil_pre_residual_right_eye_y_coordinate_abs`, `gfve_pupil_pre_residual_right_eye_y_coordinate_sign_flag`. When `gfve_pupil_pred_flag` is 1, the corresponding difference refers to the difference between the pupil position and the pupil position in the reference frame (such as the previous frame).

[0430] When `gfve_pupil_update_parameter_flag` equals 1, it indicates the existence of the syntax element `gfve_pupil_coordinate_precision_factor_minus1`. When `gfve_pupil_update_parameter_flag` equals 0, it indicates that the syntax element `gfve_pupil_coordinate_precision_factor_minus1` may not exist. If `gfve_pupil_update_parameter_flag` does not exist, it is inferred to be 0.

[0431] gfve_pupil_coordinate_precision_factor_minus1 plus 1 represents the syntax elements gfve_pupil_left_eye_dx_coordinate_abs, gfve_pupil_left_eye_dy_coordinate_abs, gfve_pupil_right_eye_dx_coordinate_abs, gfve_pupil_right_eye_dy_coordinate_abs, gfve_pupil_pre_residual_rig ht_eye_dx_coordinate_abs, gfve_pupil_pre_residual_right_eye_dy_coordinate_abs, gfve_pupil_left_eye_x_coordinate_abs, gfv Bit quantization depth of e_pupil_left_eye_y_coordinate_abs, gfve_pupil_right_eye_x_coordinate_abs, gfve_pupil_right_eye_y_coordinate_abs. When gfve_pupil_update_parameter_flag = 0 and gfve_pupil_pred_flag = 1, the value of gfve_pupil_coordinate_precision_factor_minus1 should be inferred to be the same as the gfve_pupil_coordinate_precision_factor_minus1 of a previous SEI message with the same gfve_id and gfve_pupil_update_parameter_flag = 1.

[0432] gfve_pupil_left_eye_dx_coordinate_abs represents the absolute value of the difference in the x-coordinate of the left eye.

[0433] `gfve_pupil_left_eye_dx_coordinate_sign_flag` represents the sign of the difference in the left eye's x-coordinate. If `gfve_pupil_left_eye_dx_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0434] gfve_pupil_left_eye_dy_coordinate_abs represents the absolute value of the difference in the y-coordinate of the left eye.

[0435] `gfve_pupil_left_eye_dy_coordinate_sign_flag` represents the sign of the difference in the y-coordinates of the left eye. If `gfve_pupil_left_eye_dy_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0436] gfve_pupil_right_eye_dx_coordinate_abs represents the absolute value of the difference in the x-coordinate of the right eye.

[0437] `gfve_pupil_right_eye_dx_coordinate_sign_flag` represents the sign of the difference in the x-coordinate of the right eye. If `gfve_pupil_right_eye_dx_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0438] gfve_pupil_right_eye_dy_coordinate_abs represents the absolute value of the difference in the y-coordinate of the right eye.

[0439] `gfve_pupil_right_eye_dy_coordinate_sign_flag` represents the sign of the difference in the y-coordinate of the right eye. If `gfve_pupil_right_eye_dy_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0440] gfve_pupil_pre_residual_right_eye_dx_coordinate_abs represents the absolute value of the residual used to predict the x-coordinate difference of the right eye using the x-coordinate difference of the left eye.

[0441] `gfve_pupil_pre_residual_right_eye_dx_coordinate_sign_flag` represents the sign of the residual used to predict the x-coordinate difference of the right eye using the x-coordinate difference of the left eye. If `gfve_pupil_pre_residual_right_eye_dx_coordinate_sign_flag` does not exist, it is inferred to be equal to 0.

[0442] gfve_pupil_pre_residual_right_eye_dy_coordinate_abs represents the absolute value of the residual used to predict the right eye's y-coordinate difference using the left eye's y-coordinate difference.

[0443] `gfve_pupil_pre_residual_right_eye_dy_coordinate_sign_flag` represents the sign of the residual used to predict the right eye's y-coordinate difference using the left eye's y-coordinate difference. If `gfve_pupil_pre_residual_right_eye_dy_coordinate_sign_flag` does not exist, it is inferred to be equal to 0.

[0444] gfve_pupil_left_eye_x_coordinate_abs represents the absolute value of the left eye's x-coordinate.

[0445] `gfve_pupil_left_eye_x_coordinate_sign_flag` represents the sign of the left eye's x-coordinate. If `gfve_pupil_left_eye_x_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0446] gfve_pupil_left_eye_y_coordinate_abs represents the absolute value of the y-coordinate of the left eye.

[0447] `gfve_pupil_left_eye_y_coordinate_sign_flag` represents the sign of the y-coordinate of the left eye. If `gfve_pupil_left_eye_y_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0448] gfve_pupil_right_eye_x_coordinate_abs represents the absolute value of the right eye's x-coordinate.

[0449] `gfve_pupil_right_eye_x_coordinate_sign_flag` represents the sign of the right eye's x-coordinate. If `gfve_pupil_right_eye_x_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0450] gfve_pupil_right_eye_y_coordinate_abs represents the absolute value of the y-coordinate of the right eye.

[0451] `gfve_pupil_right_eye_y_coordinate_sign_flag` represents the sign of the right eye's y-coordinate. If `gfve_pupil_right_eye_y_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0452] gfve_pupil_pre_residual_right_eye_x_coordinate_abs represents the absolute value of the residual used to predict the right eye's x-coordinate using the left eye's x-coordinate.

[0453] `gfve_pupil_pre_residual_right_eye_x_coordinate_sign_flag` represents the sign of the residual used to predict the right eye's x-coordinate using the left eye's x-coordinate. If `gfve_pupil_pre_residual_right_eye_x_coordinate_sign_flag` does not exist, it is inferred to be equal to 0.

[0454] gfve_pupil_pre_residual_right_eye_y_coordinate_abs represents the absolute value of the residual used to predict the right eye's y-coordinate using the left eye's y-coordinate.

[0455] `gfve_pupil_pre_residual_right_eye_y_coordinate_sign_flag` represents the sign of the residual used to predict the right-eye y-coordinate using the left-eye y-coordinate. If `gfve_pupil_pre_residual_right_eye_y_coordinate_sign_flag` does not exist, it is inferred to be equal to 0.

[0456] Example 3

[0457] The schemes in Example 1 and Example 2 use the value of the bit quantization depth, while the scheme in Example 3 introduces a variation in the bit quantization depth.

[0458] Table 10

[0459]

[0460]

[0461] The following explains the grammatical elements involved in Table 10.

[0462] `gfve_pupil_coordinate_present_idx` represents the index of which pupil information is sent. When `gfve_pupil_coordinate_present_idx` equals 0, it means that pupil position information does not exist; when `gfve_pupil_coordinate_present_idx` equals 1, it means that only the left eye pupil position information is included; when `gfve_pupil_coordinate_present_idx` equals 2, it means that only the right eye pupil position information is included; when `gfve_pupil_coordinate_present_idx` equals 3, it means that both the left and right eye pupil position information are included.

[0463] When `gfve_pupil_pred_flag` equals 1, it indicates the presence of the syntax element `gfve_pupil_coordinate_delta_precision_factor`. This may include the syntax elements `gfve_pupil_left_eye_dx_coordinate_abs`, `gfve_pupil_left_eye_dx_coordinate_sign_flag`, `gfve_pupil_left_eye_dy_coordinate_abs`, and `gfve_pupil_left_eye_dy_coordinate_sign_flag`. It may also include the syntax elements `gfve_pupil_right_eye_dx_coordinate_abs` and `gfve_pupil_right_eye_dx_coordinate_sign_flag`. nate_sign_flag, gfve_pupil_right_eye_dy_coordinate_abs, gfve_pupil_right_eye_dy_coordinate_sign_flag; or there may be syntax elements gfve_pupil_pre_residual_right_eye_dx_coordinate_abs, gfv e_pupil_pre_residual_right_eye_dx_coordinate_sign_flag, gfve_pupil_pre_residual_right_eye_dy_coordinate_abs, gfve_pupil_pre_residual_right_eye_dy_coordinate_sign_flag.When gfve_pupil_pred_flag equals 0, it indicates the existence of the syntax element gfve_pupil_coordinate_precision_factor_minus1. This may include the syntax elements gfve_pupil_left_eye_x_coordinate_abs, gfve_pupil_left_eye_x_coordinate_sign_flag, gfve_pupil_left_eye_y_coordinate_abs, and gfve_pupil_left_eye_y_coordinate_sign_flag. It may also include the syntax elements gfve_pupil_right_eye_x_coordinate_abs, gfve_pupil_right_eye_x_coordinate_sign_flag, gfve_pupil_right_eye_y_coordinate_abs, and gfve_pupil_right_eye_y_coordinate_sign_flag. When gfve_pupil_pred_flag is 1, the corresponding difference refers to the difference between the pupil position and the pupil position in the reference frame (such as the previous frame).

[0464] `gfve_pupil_coordinate_delta_precision_factor` represents the change in bit quantization depth of the syntax elements `gfve_pupil_left_eye_dx_coordinate_abs`, `gfve_pupil_left_eye_dy_coordinate_abs`, `gfve_pupil_right_eye_dx_coordinate_abs`, `gfve_pupil_right_eye_dy_coordinate_abs`, `gfve_pupil_pre_residual_right_eye_dx_coordinate_abs`, and `gfve_pupil_pre_residual_right_eye_dy_coordinate_abs` relative to the reference frame (such as the previous frame). The default initial value of the bit quantization depth is 0, meaning that the bit quantization depth of the first frame is `gfve_pupil_coordinate_delta_precision_factor + 0`.

[0465] The increment of 1 in gfve_pupil_coordinate_precision_factor_minus1 indicates the bit quantization depth of the syntax elements gfve_pupil_left_eye_x_coordinate_abs, gfve_pupil_left_eye_y_coordinate_abs, gfve_pupil_right_eye_x_coordinate_abs, and gfve_pupil_right_eye_y_coordinate_abs.

[0466] gfve_pupil_left_eye_dx_coordinate_abs represents the absolute value of the difference in the x-coordinate of the left eye.

[0467] `gfve_pupil_left_eye_dx_coordinate_sign_flag` represents the sign of the difference in the left eye's x-coordinate. If `gfve_pupil_left_eye_dx_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0468] gfve_pupil_left_eye_dy_coordinate_abs represents the absolute value of the difference in the y-coordinate of the left eye.

[0469] `gfve_pupil_left_eye_dy_coordinate_sign_flag` represents the sign of the difference in the y-coordinates of the left eye. If `gfve_pupil_left_eye_dy_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0470] gfve_pupil_right_eye_dx_coordinate_abs represents the absolute value of the difference in the x-coordinate of the right eye.

[0471] `gfve_pupil_right_eye_dx_coordinate_sign_flag` represents the sign of the difference in the x-coordinate of the right eye. If `gfve_pupil_right_eye_dx_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0472] gfve_pupil_right_eye_dy_coordinate_abs represents the absolute value of the difference in the y-coordinate of the right eye.

[0473] `gfve_pupil_right_eye_dy_coordinate_sign_flag` represents the sign of the difference in the y-coordinate of the right eye. If `gfve_pupil_right_eye_dy_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0474] gfve_pupil_pre_residual_right_eye_dx_coordinate_abs represents the absolute value of the residual used to predict the x-coordinate difference of the right eye using the x-coordinate difference of the left eye.

[0475] `gfve_pupil_pre_residual_right_eye_dx_coordinate_sign_flag` represents the sign of the residual used to predict the x-coordinate difference of the right eye using the x-coordinate difference of the left eye. If `gfve_pupil_pre_residual_right_eye_dx_coordinate_sign_flag` does not exist, it is inferred to be equal to 0.

[0476] gfve_pupil_pre_residual_right_eye_dy_coordinate_abs represents the absolute value of the residual used to predict the right eye's y-coordinate difference using the left eye's y-coordinate difference.

[0477] `gfve_pupil_pre_residual_right_eye_dy_coordinate_sign_flag` represents the sign of the residual used to predict the right eye's y-coordinate difference using the left eye's y-coordinate difference. If `gfve_pupil_pre_residual_right_eye_dy_coordinate_sign_flag` does not exist, it is inferred to be equal to 0.

[0478] gfve_pupil_left_eye_x_coordinate_abs represents the absolute value of the left eye's x-coordinate.

[0479] `gfve_pupil_left_eye_x_coordinate_sign_flag` represents the sign of the left eye's x-coordinate. If `gfve_pupil_left_eye_x_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0480] gfve_pupil_left_eye_y_coordinate_abs represents the absolute value of the y-coordinate of the left eye.

[0481] `gfve_pupil_left_eye_y_coordinate_sign_flag` represents the sign of the y-coordinate of the left eye. If `gfve_pupil_left_eye_y_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0482] gfve_pupil_right_eye_x_coordinate_abs represents the absolute value of the right eye's x-coordinate.

[0483] `gfve_pupil_right_eye_x_coordinate_sign_flag` represents the sign of the right eye's x-coordinate. If `gfve_pupil_right_eye_x_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0484] gfve_pupil_right_eye_y_coordinate_abs represents the absolute value of the y-coordinate of the right eye.

[0485] `gfve_pupil_right_eye_y_coordinate_sign_flag` represents the sign of the right eye's y-coordinate. If `gfve_pupil_right_eye_y_coordinate_sign_flag` does not exist, it is assumed to be equal to 0.

[0486] The above text combined Figures 1 to 6 The method embodiments of this application are described in detail below, in conjunction with... Figures 7 to 10 The present application provides a detailed description of the apparatus embodiments. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be found in the preceding method embodiments.

[0487] Figure 7 This is a schematic diagram of the decoder structure provided in one embodiment of this application. Figure 7 As shown, the decoder 700 includes a decoding unit 710 and a determining unit 720. The decoding unit 710 is configured to decode the SEI message corresponding to the current image to obtain the pupil position information contained in the current image; the determining unit 720 is configured to determine the eye motion information contained in the current image based on the pupil position information.

[0488] In some implementations, the SEI message includes one or more of the following parameters: a first parameter representing first location information; a second parameter representing second location information; a third parameter representing the difference between the first location information and the third location information; a fourth parameter representing the difference between the second location information and the fourth location information; a fifth parameter representing the difference between the first location information and the second location information; a sixth parameter representing the difference between the second location information and the first location information; a seventh parameter representing the difference between the first difference and the second difference; and an eighth parameter representing the difference between the second difference and the first difference.

[0489] Wherein, the first position information is the position information of the left pupil in the current image, the second position information is the position information of the right pupil in the current image, the third position information is the position information of the left pupil in the reference image, the fourth position information is the position information of the right pupil in the reference image, the first difference is the difference between the position information of the left pupil in the current image and the position information of the left pupil in the reference image, and the second difference is the difference between the position information of the right pupil in the current image and the position information of the right pupil in the reference image.

[0490] In some implementations, the SEI message also includes first identification information, which is used to indicate the parameters contained in the SEI message.

[0491] In some implementations, the first identifier information is a first value, and the parameters included in the SEI message include one or more of the following parameters: the third parameter, the fourth parameter, the seventh parameter, and the eighth parameter; or the first identifier information is a second value, and the parameters included in the SEI message include one or more of the following parameters: the first parameter, the second parameter, the fifth parameter, and the sixth parameter.

[0492] In some implementations, the SEI message also includes a ninth parameter or a tenth parameter, wherein the ninth parameter is used to represent the bit quantization depth corresponding to the current image, and the tenth parameter is used to represent the change in bit quantization depth corresponding to the current image relative to the bit quantization depth corresponding to the reference image.

[0493] In some implementations, if the value of the first identifier information is a first value, then the SEI message includes the tenth parameter; if the value of the first identifier information is a second value, then the SEI message includes the ninth parameter.

[0494] In some implementations, the SEI message also includes second identification information, which is used to indicate whether the SEI message includes the ninth parameter.

[0495] In some implementations, if the second identification information indicates that the SEI message does not include the ninth parameter, then the bit quantization depth corresponding to the current image is the same as the bit quantization depth corresponding to the reference image.

[0496] In some implementations, the SEI message also includes third identification information, which is used to indicate that there is no pupil location information in the SEI message, or the pupil corresponding to the location information included in the SEI message.

[0497] In some implementations, the third identifier information is a third value, and the SEI message does not contain pupil position information; the third identifier information is a fourth value, and the SEI message only contains the position information of the left pupil; the third identifier information is a fifth value, and the SEI message only contains the position information of the right pupil; the third identifier information is a sixth value, and the SEI message contains the position information of both the left and right pupils.

[0498] In some implementations, the location information includes the coordinate information of the pupil in a first coordinate system, the origin of which is the midpoint of the line connecting the two corners of the eye, and the unit length of the first coordinate system is m times the horizontal distance between the two corners of the eye, where m is greater than 0.5.

[0499] In some implementations, m can take any of the following values: 0.6, 0.7, 0.8.

[0500] In some implementations, the coordinate information includes at least one of the x-coordinate information and the y-coordinate information.

[0501] In some implementations, the x-coordinate information includes at least one of the absolute value of the x-coordinate value and the sign of the x-coordinate value; the y-coordinate information includes at least one of the absolute value of the y-coordinate value and the sign of the y-coordinate value.

[0502] In some implementations, the information of the difference includes at least one of the absolute value of the difference and the sign of the difference.

[0503] In some implementations, the decoder further includes a reconstruction unit configured to reconstruct a face image using a generative model based on the motion information of the eyes.

[0504] In some implementations, the SEI message is a GFVE SEI message.

[0505] Understandably, in some embodiments of this application, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and can also be a module or a non-modular component. Furthermore, the components in this embodiment can be integrated into a single processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The integrated unit described above can be implemented in hardware or as a software functional module.

[0506] If the integrated unit is implemented as a software functional module and not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0507] Therefore, some embodiments of this application provide a computer-readable storage medium applied to a decoder 700, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned decoding method.

[0508] Based on the composition of the decoder 700 and the computer-readable storage medium described above, see [link to documentation]. Figure 8 This illustrates a schematic diagram of the specific hardware structure of the decoder provided in some embodiments of this application. For example... Figure 8 As shown, the decoder 800 may include a communication interface 810, a memory 820, and a processor 830; the various components are coupled together via a bus system 840. It is understood that the bus system 840 is used to implement communication between these components. In addition to a data bus, the bus system 840 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 8 The general designated all buses as Bus System 840. Among them,

[0509] The communication interface 810 is used for receiving and sending signals during the process of sending and receiving information with other external network elements.

[0510] The memory 820 is used to store computer programs.

[0511] Processor 830, when running the computer program, performs the following:

[0512] Decode the SEI message corresponding to the current image to obtain the position information of the pupil contained in the current image;

[0513] Based on the pupil position information, the eye motion information contained in the current image is determined.

[0514] It is understood that the memory 820 in some embodiments of this application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 820 of the systems and methods described in this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0515] The processor 830 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 830 or by instructions in software form. The processor 830 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in some embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in some embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 820. Processor 830 reads the information in memory 820 and, in conjunction with its hardware, completes the steps of the above method.

[0516] It is understood that the embodiments described in this application can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), DSP devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in this application, or combinations thereof. For software implementation, the technology described in this application can be implemented through modules (e.g., procedures, functions, etc.) that perform the functions described in this application. Software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0517] In some embodiments, as another example, the processor 830 is also configured to execute the decoding method described in the foregoing embodiments when running the computer program.

[0518] In some embodiments of this application, a decoding chip is also provided, which includes a processor and a memory. The processor is configured to call and run a computer program from the memory, enabling a device equipped with the decoding chip to perform any of the decoding methods described above.

[0519] Figure 9 This is a schematic diagram of the encoder provided in one embodiment of this application. Figure 9 As shown, the encoder 900 includes a determining unit 910 and an encoding unit 920. The determining unit 910 is configured to determine the position information of the pupil contained in the current image; the encoding unit 920 is configured to encode the parameters in the supplementary enhancement information (SEI) message corresponding to the current image based on the pupil position information, wherein the pupil position information is used to determine the eye motion information contained in the current image.

[0520] In some implementations, the SEI message includes one or more of the following parameters: a first parameter representing first location information; a second parameter representing second location information; a third parameter representing the difference between the first location information and the third location information; a fourth parameter representing the difference between the second location information and the fourth location information; a fifth parameter representing the difference between the first location information and the second location information; a sixth parameter representing the difference between the second location information and the first location information; a seventh parameter representing the difference between the first difference and the second difference; and an eighth parameter representing the difference between the second difference and the first difference.

[0521] Wherein, the first position information is the position information of the left pupil in the current image, the second position information is the position information of the right pupil in the current image, the third position information is the position information of the left pupil in the reference image, the fourth position information is the position information of the right pupil in the reference image, the first difference is the difference between the position information of the left pupil in the current image and the position information of the left pupil in the reference image, and the second difference is the difference between the position information of the right pupil in the current image and the position information of the right pupil in the reference image.

[0522] In some implementations, the SEI message also includes first identification information, which is used to indicate the parameters contained in the SEI message.

[0523] In some implementations, the first identifier information is a first value, and the parameters included in the SEI message include one or more of the following parameters: the third parameter, the fourth parameter, the seventh parameter, and the eighth parameter; or the first identifier information is a second value, and the parameters included in the SEI message include one or more of the following parameters: the first parameter, the second parameter, the fifth parameter, and the sixth parameter.

[0524] In some implementations, the SEI message also includes a ninth parameter or a tenth parameter, wherein the ninth parameter is used to represent the bit quantization depth corresponding to the current image, and the tenth parameter is used to represent the change in bit quantization depth corresponding to the current image relative to the bit quantization depth corresponding to the reference image.

[0525] In some implementations, if the value of the first identifier information is a first value, then the SEI message includes the tenth parameter; if the value of the first identifier information is a second value, then the SEI message includes the ninth parameter.

[0526] In some implementations, the SEI message also includes second identification information, which is used to indicate whether the SEI message includes the ninth parameter.

[0527] In some implementations, if the second identification information indicates that the SEI message does not include the ninth parameter, then the bit quantization depth corresponding to the current image is the same as the bit quantization depth corresponding to the reference image.

[0528] In some implementations, the SEI message also includes third identification information, which is used to indicate that the location information of the pupil is not present in the SEI message, or that the location information included in the SEI message corresponds to the pupil.

[0529] In some implementations, the third identifier information is a third value, and the SEI message does not contain pupil position information; the third identifier information is a fourth value, and the SEI message only contains the position information of the left pupil; the third identifier information is a fifth value, and the SEI message only contains the position information of the right pupil; the third identifier information is a sixth value, and the SEI message contains the position information of both the left and right pupils.

[0530] In some implementations, the location information includes the coordinate information of the pupil in a first coordinate system, the origin of which is the midpoint of the line connecting the two corners of the eye, and the unit length of the first coordinate system is m times the horizontal distance between the two corners of the eye, where m is greater than 0.5.

[0531] In some implementations, m can take any of the following values: 0.6, 0.7, 0.8.

[0532] In some implementations, the coordinate information includes at least one of the x-coordinate information and the y-coordinate information.

[0533] In some implementations, the x-coordinate information includes at least one of the absolute value of the x-coordinate and the sign of the x-coordinate; the y-coordinate information includes at least one of the absolute value of the y-coordinate and the sign of the y-coordinate.

[0534] In some implementations, the information of the difference includes at least one of the absolute value of the difference and the sign of the difference.

[0535] In some implementations, the SEI message is a GFVE SEI message.

[0536] Understandably, in some embodiments of this application, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and can also be a module or a non-modular component. Furthermore, the components in this embodiment can be integrated into a single processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The integrated unit described above can be implemented in hardware or as a software functional module.

[0537] If the integrated unit is implemented as a software functional module and not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0538] Therefore, some embodiments of this application provide a computer-readable storage medium applied to an encoder 900, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the encoding method in the foregoing embodiments.

[0539] Based on the composition of the encoder 900 and the computer-readable storage medium described above, see [link to documentation]. Figure 10 This illustrates a schematic diagram of the specific hardware structure of the encoder provided in some embodiments of this application. For example... Figure 10 As shown, encoder 1000 may include: communication interface 1010, memory 1020, and processor 1030; the various components are coupled together via bus system 1040. It is understood that bus system 1040 is used to implement communication between these components. In addition to a data bus, bus system 1040 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 10 The various buses are all labeled as Bus System 1040. Among them,

[0540] The communication interface 1010 is used for receiving and sending signals during the process of sending and receiving information with other external network elements.

[0541] Memory 1020 is used to store computer programs.

[0542] Processor 1030, when running the computer program, performs the following:

[0543] Determine the positional information of the pupils contained in the current image;

[0544] Based on the pupil position information, the parameters in the SEI message corresponding to the current image are encoded, and the pupil position information is used to determine the eye motion information contained in the current image.

[0545] It is understood that the memory 1020 in some embodiments of this application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 1020 of the system and method described in this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0546] The processor 1030 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 1030 or by instructions in software form. The processor 1030 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in some embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in some embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 1020. Processor 1030 reads the information in memory 1020 and completes the steps of the above method in conjunction with its hardware.

[0547] It is understood that the embodiments described in this application can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), DSP devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in this application, or combinations thereof. For software implementation, the technology described in this application can be implemented through modules (e.g., procedures, functions, etc.) that perform the functions described in this application. Software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0548] In some embodiments, as another example, the processor 1030 is also configured to execute the encoding method described in the foregoing embodiments when running the computer program.

[0549] In some embodiments of this application, an encoding chip is also provided, which includes a processor and a memory, the processor being configured to call and run a computer program from the memory, enabling a device equipped with the encoding chip to perform any of the decoding methods described above.

[0550] It should be noted that, in this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0551] The serial numbers in some embodiments of this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0552] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0553] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0554] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0555] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A decoding method, characterized in that, Applied to the decoder, including: Decode the supplementary enhancement information (SEI) message corresponding to the current image to obtain the pupil position information contained in the current image; Based on the pupil position information, the eye motion information contained in the current image is determined.

2. The method according to claim 1, characterized in that, The SEI message includes one or more of the following parameters: The first parameter is used to represent the first position information; The second parameter is used to represent the second position information; The third parameter is used to represent the difference between the first position information and the third position information; The fourth parameter is used to represent the difference between the second position information and the fourth position information; The fifth parameter is used to represent the difference between the first position information and the second position information; The sixth parameter is used to represent the difference between the second position information and the first position information; The seventh parameter is used to represent information about the difference between the first difference and the second difference; The eighth parameter is used to represent information about the difference between the second difference and the first difference; Wherein, the first position information is the position information of the left pupil in the current image, the second position information is the position information of the right pupil in the current image, the third position information is the position information of the left pupil in the reference image, the fourth position information is the position information of the right pupil in the reference image, the first difference is the difference between the position information of the left pupil in the current image and the position information of the left pupil in the reference image, and the second difference is the difference between the position information of the right pupil in the current image and the position information of the right pupil in the reference image.

3. The method according to claim 2, characterized in that, The SEI message also includes first identification information, which is used to indicate the parameters contained in the SEI message.

4. The method according to claim 3, characterized in that, The first identifier information is a first value, and the parameters included in the SEI message include one or more of the following parameters: the third parameter, the fourth parameter, the seventh parameter, and the eighth parameter; The first identification information is a second value, and the parameters included in the SEI message include one or more of the following parameters: the first parameter, the second parameter, the fifth parameter, and the sixth parameter.

5. The method according to claim 3, characterized in that, The SEI message also includes a ninth parameter or a tenth parameter. The ninth parameter is used to represent the bit quantization depth corresponding to the current image, and the tenth parameter is used to represent the change in the bit quantization depth corresponding to the current image relative to the bit quantization depth corresponding to the reference image.

6. The method according to claim 5, characterized in that, If the value of the first identifier information is a first value, then the SEI message includes the tenth parameter; if the value of the first identifier information is a second value, then the SEI message includes the ninth parameter.

7. The method according to claim 5, characterized in that, The SEI message also includes second identification information, which is used to indicate whether the SEI message includes the ninth parameter.

8. The method according to claim 7, characterized in that, If the second identification information indicates that the SEI message does not include the ninth parameter, then the bit quantization depth corresponding to the current image is the same as the bit quantization depth corresponding to the reference image.

9. The method according to claim 2, characterized in that, The SEI message also includes third identification information, which is used to indicate that there is no pupil location information in the SEI message, or the pupil corresponding to the location information included in the SEI message.

10. The method according to claim 9, characterized in that, The value of the third identification information is a third value, and the SEI message does not contain pupil position information; The value of the third identification information is the fourth value, and the SEI message only contains the position information of the left pupil; The value of the third identification information is the fifth value, and the SEI message only contains the position information of the right pupil; The value of the third identification information is the sixth value, and the SEI message contains the position information of the left pupil and the position information of the right pupil.

11. The method according to claim 2, characterized in that, The location information includes the coordinates of the pupil in a first coordinate system, where the origin of the first coordinate system is the midpoint of the line connecting the two corners of the eye, and the unit length of the first coordinate system is m times the horizontal distance between the two corners of the eye, where m is greater than 0.

5.

12. The method according to claim 11, characterized in that, The value of m can be any of the following: 0.6, 0.7, 0.

8.

13. The method according to claim 11, characterized in that, The coordinate information includes at least one of the x-coordinate information and the y-coordinate information.

14. The method according to claim 13, characterized in that, The information of the x-coordinate includes at least one of the absolute value of the x-coordinate and the sign of the x-coordinate; The information of the y-coordinate includes at least one of the absolute value of the y-coordinate and the sign of the y-coordinate.

15. The method according to claim 2, characterized in that, The information about the difference includes at least one of the absolute value of the difference and the sign of the difference.

16. The method according to any one of claims 1-15, characterized in that, The method further includes: Based on the eye movement information, a generative model is used to reconstruct the face image.

17. The method according to any one of claims 1-15, characterized in that, The SEI message is a Generative Face Video Enhancement (GFVE) SEI message.

18. An encoding method, characterized in that, Applied to encoders, including: Determine the positional information of the pupils contained in the current image; Based on the pupil position information, the parameters in the supplementary enhancement information (SEI) message corresponding to the current image are encoded, and the pupil position information is used to determine the eye motion information contained in the current image.

19. The method according to claim 18, characterized in that, The SEI message includes one or more of the following parameters: The first parameter is used to represent the first position information; The second parameter is used to represent the second position information; The third parameter is used to represent the difference between the first position information and the third position information; The fourth parameter is used to represent the difference between the second position information and the fourth position information; The fifth parameter is used to represent the difference between the first position information and the second position information; The sixth parameter is used to represent the difference between the second position information and the first position information; The seventh parameter is used to represent information about the difference between the first difference and the second difference; The eighth parameter is used to represent information about the difference between the second difference and the first difference; Wherein, the first position information is the position information of the left pupil in the current image, the second position information is the position information of the right pupil in the current image, the third position information is the position information of the left pupil in the reference image, the fourth position information is the position information of the right pupil in the reference image, the first difference is the difference between the position information of the left pupil in the current image and the position information of the left pupil in the reference image, and the second difference is the difference between the position information of the right pupil in the current image and the position information of the right pupil in the reference image.

20. The method according to claim 19, characterized in that, The SEI message also includes first identification information, which is used to indicate the parameters contained in the SEI message.

21. The method according to claim 20, characterized in that, The first identifier information is a first value, and the parameters included in the SEI message include one or more of the following parameters: the third parameter, the fourth parameter, the seventh parameter, and the eighth parameter; The first identification information is a second value, and the parameters included in the SEI message include one or more of the following parameters: the first parameter, the second parameter, the fifth parameter, and the sixth parameter.

22. The method according to claim 20, characterized in that, The SEI message also includes a ninth parameter or a tenth parameter. The ninth parameter is used to represent the bit quantization depth corresponding to the current image, and the tenth parameter is used to represent the change in the bit quantization depth corresponding to the current image relative to the bit quantization depth corresponding to the reference image.

23. The method according to claim 22, characterized in that, If the value of the first identifier information is a first value, then the SEI message includes the tenth parameter; if the value of the first identifier information is a second value, then the SEI message includes the ninth parameter.

24. The method according to claim 22, characterized in that, The SEI message also includes second identification information, which is used to indicate whether the SEI message includes the ninth parameter.

25. The method according to claim 24, characterized in that, If the second identification information indicates that the SEI message does not include the ninth parameter, then the bit quantization depth corresponding to the current image is the same as the bit quantization depth corresponding to the reference image.

26. The method according to claim 19, characterized in that, The SEI message also includes third identification information, which is used to indicate that there is no pupil location information in the SEI message, or the pupil corresponding to the location information included in the SEI message.

27. The method according to claim 26, characterized in that, The value of the third identification information is a third value, and the SEI message does not contain pupil position information; The value of the third identification information is the fourth value, and the SEI message only contains the position information of the left pupil; The value of the third identification information is the fifth value, and the SEI message only contains the position information of the right pupil; The value of the third identification information is the sixth value, and the SEI message contains the position information of the left pupil and the position information of the right pupil.

28. The method according to claim 19, characterized in that, The location information includes the coordinates of the pupil in a first coordinate system, where the origin of the first coordinate system is the midpoint of the line connecting the two corners of the eye, and the unit length of the first coordinate system is m times the horizontal distance between the two corners of the eye, where m is greater than 0.

5.

29. The method according to claim 28, characterized in that, The value of m can be any of the following: 0.6, 0.7, 0.

8.

30. The method according to claim 28, characterized in that, The coordinate information includes at least one of the x-coordinate information and the y-coordinate information.

31. The method according to claim 30, characterized in that, The information of the x-coordinate includes at least one of the absolute value of the x-coordinate and the sign of the x-coordinate; The information of the y-coordinate includes at least one of the absolute value of the y-coordinate and the sign of the y-coordinate.

32. The method according to claim 19, characterized in that, The information about the difference includes at least one of the absolute value of the difference and the sign of the difference.

33. The method according to any one of claims 18-32, characterized in that, The SEI message is a Generative Face Video Enhancement (GFVE) SEI message.

34. A decoder, characterized in that, include: Memory, used to store computer programs; A processor, configured to perform the method as described in any one of claims 1-17 when running the computer program.

35. An encoder, characterized in that, include: Memory, used to store computer programs; A processor, configured to perform the method as described in any one of claims 18-33 when running the computer program.

36. A decoding chip, characterized in that, It includes a processor and a memory, the processor being configured to invoke a program or instructions from the memory to cause the method as described in any one of claims 1-17 to be executed.

37. An encoding chip, characterized in that, It includes a processor and a memory, the processor being configured to invoke a program or instructions from the memory to cause the method as described in any one of claims 18-33 to be executed.

38. A non-volatile computer-readable storage medium for storing bit streams, characterized in that, The bitstream is generated by using an encoder's encoding method, or the bitstream is decoded by using a decoder's decoding method, wherein the decoding method is the method as described in any one of claims 1-17, and the encoding method is the method as described in any one of claims 18-33.

39. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the method as described in any one of claims 1-17 or 18-33.

40. A bitstream, characterized in that, The bitstream includes the bitstream generated by the method as described in any one of claims 18-33.

Images