Video coding method, video decoding method and device

Abstract

The embodiment of the invention provides a video coding method and device and a video decoding method and device, and relates to the technical field of video coding and decoding. Comprises: determining whether a current video frame is a basic image; under the condition that the current video frame is not the basic image, performing feature extraction on the current video frame to obtain an image feature corresponding to the current video frame; the image feature corresponding to the current video frame comprises pupil position information of the current video frame; and encoding the image features corresponding to the current video frame to obtain encoded data of the current video frame. Some embodiments of the invention are used for improving the accuracy of representing eye movement in generative face video coding.

Description

TECHNICAL FIELD

[0001] Some embodiments of the present application relate to the technical field of video coding. More specifically, it relates to a video encoding method and a video decoding method and apparatus. BACKGROUND

[0002] In today's digital age, the generation and transmission of video content have become an important way of information exchange. With the rapid development of social media, online education and remote work, the demand for high-quality video coding technology is growing. However, although existing video coding standards such as High Efficiency Video Coding (H.265 / HEVC) and Versatile Video Coding (H.266 / VVC) perform well in high bit rate environments, in very low bit rate environments, these standards often have difficulty achieving satisfactory video quality and compression efficiency. In particular, in scenarios such as mobile devices, Internet of Things and emergency communications, the importance of low bit rate video coding is increasingly highlighted.

[0003] In the context of the increasing importance of low bit rate video coding, the research and standardization of generative face video coding are particularly urgent. Generative models such as Generative Adversarial Network (GAN) and Variational AutoEncoder (VAE) provide new possibilities for generating high-quality face videos at very low bit rates. These models can learn facial features from limited data and generate realistic video frames, thereby significantly reducing the required bit rate while maintaining video quality. Currently, when describing the movement of the eyes, generative face video coding only represents the eye opening and closing state and degree through a matrix. However, simply representing the eye opening and closing state and degree is not sufficient to represent accurate eye movement. SUMMARY

[0004] Exemplary embodiments of the present application provide a video encoding method, a video decoding method and apparatus for improving the accuracy of representing eye movement in generative face video coding.

[0005] Some embodiments of the present application provide technical solutions as follows:

[0006] In a first aspect, some embodiments of the present application provide a video encoding method, comprising:

[0007] determining whether the current video frame is a base image;

[0008] In a case where the current video frame is not a base image, feature extraction is performed on the current video frame to obtain image features corresponding to the current video frame; the image features corresponding to the current video frame include pupil position information of the current video frame;

[0009] The image features corresponding to the current video frame are encoded to obtain encoding data of the current video frame.

[0010] In a second aspect, some embodiments of the present application provide an image decoding method, comprising:

[0011] Obtaining encoding data of the current video frame;

[0012] Decoding the encoding data of the current video frame to obtain image features corresponding to the current video frame; the image features corresponding to the current video frame include pupil position information of the current video frame;

[0013] Reconstructing the current video according to the image features corresponding to the current video frame to obtain a reconstructed current video frame.

[0014] In a third aspect, some embodiments of the present application provide an image encoding device, comprising:

[0015] A memory configured to store a computer program;

[0016] A processor configured to, when the computer program is invoked, cause the video encoding device to implement the image encoding method of the first aspect.

[0017] In a fourth aspect, some embodiments of the present application provide an image decoding device, comprising:

[0018] A memory configured to store a computer program;

[0019] A processor configured to, when the computer program is invoked, cause the video decoding device to implement the image encoding method of the first aspect.

[0020] In a fifth aspect, some embodiments of the present application provide a computer readable storage medium, the computer readable storage medium storing a computer program, when the computer program is executed by a computing device, causing the computing device to implement the method of the first aspect or the second aspect.

[0021] In a sixth aspect, some embodiments of the present application provide a computer program product, when the computer program product is run on a computer, causing the computer to implement the method of the first aspect or the second aspect.

[0022] According to the technical solution, the video encoding method provided by the embodiment of the present application first determines whether the current video frame is a base image, and in the case that the current video frame is not a base image, feature extraction is performed on the current video frame to obtain image features corresponding to the current video frame, and then the image features corresponding to the current video frame are encoded to obtain the encoding data of the current video frame. Since the image features corresponding to the current video frame obtained by the feature extraction of the current video frame in the embodiment of the present application include the pupil position information of the current video frame, the decoding end can improve the accuracy of representing eye movement, thereby more accurately reconstructing the current video frame. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate some embodiments of the present application or the implementation manner in the related art, the drawings needed to be used in the embodiment or related art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art according to these drawings.

[0024] Figure 1 The structural block diagram of the generative face video coding system in some embodiments of the present application is shown;

[0025] Figure 2 The coordinate system diagram representing the pupil position information in some embodiments of the present application is shown;

[0026] Figure 3 The coordinate system diagram representing the pupil position information in some embodiments of the present application is shown;

[0027] Figure 4 The coordinate system diagram representing the pupil position information in some embodiments of the present application is shown;

[0028] Figure 5 The coordinate system diagram representing the pupil position information in some embodiments of the present application is shown;

[0029] Figure 6 The coordinate system diagram representing the pupil position information in some embodiments of the present application is shown;

[0030] Figure 7 The coordinate system diagram representing the pupil position information in some embodiments of the present application is shown;

[0031] Figure 8 The coordinate system diagram representing the pupil position information in some embodiments of the present application is shown;

[0032] Figure 9 The coordinate system diagram representing the pupil position information in some embodiments of the present application is shown;

[0033] Figure 10 A step flowchart of a video encoding method provided by some embodiments of the present application is shown.

[0034] Figure 11 A step flowchart of a video decoding method provided by some embodiments of the present application is shown. DETAILED DESCRIPTION

[0035] For the purpose of making the purpose and embodiments of the present application clearer, the following will clearly and completely describe the exemplary embodiments of the present application with reference to the accompanying drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only a part of the embodiments of the present application, but not all of the embodiments of the present application.

[0036] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the embodiments described next, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0037] The terms “comprising” and “having” and any variations thereof are intended to cover, but not exclusively, inclusion, for example, a product or device comprising a series of components does not have to be limited to all the components listed clearly, but can include other components that are not listed clearly or inherent to these products or devices.

[0038] The description of “some implementations”, “some embodiments” and the like in the specification indicates that the described implementations or embodiments can include a specific feature, structure or property, but can not necessarily include this specific feature, structure or property in each embodiment. In addition, such phrases do not necessarily refer to the same implementation. In addition, when a specific feature, structure or property is described in connection with an embodiment, it is considered that it is within the knowledge of those skilled in the art to realize such a feature, structure or property in connection with other implementations (whether described herein or not).

[0039] Under the promotion of artificial intelligence generated content (AIGC), the joint video expert group (JVET) of ISO / IEC SC 29 and ITU-T SG 16 is committed to researching the standardization work of realizing generative face video coding in an extremely low bit rate environment.

[0040] Referring to Figure 1 As shown, Figure 1 is a structural block diagram of a generative face video coding and decoding system. As Figure 1 shown, the generative face video coding and decoding system mainly consists of two parts. The first part is a basic image coding and decoding module 11, and the second part is a generative face video coding and decoding module 12.

[0041] The base image coding module 11 includes a base image encoder 111 and a base image decoder 112. The base image encoder 111 is responsible for encoding the base image carrying the facial base texture information to generate the code stream corresponding to the base image. The base image decoder 112 is responsible for decoding the code stream corresponding to the base image to obtain the reconstructed base image. The base image encoder 111 and the base image decoder 112 respectively use standard coding and decoding technologies for coding and decoding. For example: H.265 / HEVC, H.266 / VVC, etc.

[0042] The generative facial video coding module 12 includes an analysis model 121, a feature encoding module 122, a feature decoding module 123, and a generation model 124. The analysis model 121 is used to extract the motion information of the subsequent image of the base image to obtain the motion information of the subsequent image. The feature encoding module 122 is used to encode the motion information extracted by the analysis model 121 to generate the code stream corresponding to the subsequent image. The feature decoding module 123 is used to decode the code stream corresponding to the subsequent image to obtain the motion information of the subsequent image. The generation model 124 is used to reconstruct the subsequent image according to the motion information and the facial base texture information in the base image.

[0043] Compared with the coding and decoding basis of H.265 / HEVC, H.266 / VVC, etc., the generative facial video coding technology can provide higher quality facial image reconstruction at a very low bit rate.

[0044] The following describes the algorithm for implementing the analysis model and the generation model in the generative facial video coding module.

[0045] The facial video coding algorithm of the above generative facial video coding module implemented by the current JVET test is shown in Table 1 as follows:

[0046] Table 1

[0047] Respective algorithm Face representation FOMM 2D key points + affine transformation matrix FV2V 3D key points + head rotation / translation matrix CFTE Compact feature matrix

[0048] The FOMM input is a base image and a subsequent image. The unsupervised key point detector extracts sparse 2D key points of the face and a first-order motion representation composed of local affine transformations relative to an abstract reference frame. The dense motion network generates a dense optical flow and an occlusion map from the subsequent image to the base image using the above motion representation. The generator uses the base image and the output of the dense motion network to render the subsequent image.

[0049] The FV2V input is the base image and the subsequent image. The appearance feature extractor outputs the base image appearance features, and the canonical keypoint detector outputs the 3D canonical keypoints. For each image, the estimator network estimates the keypoint perturbation caused by the head pose and expression, resulting in the head rotation / translation matrix and the sparse 3D keypoints. The motion field estimation network uses the 3D keypoints of the base image and the 3D keypoints of the subsequent image to obtain the dense optical flow from D to S. The generator uses the base image appearance features and the output of the dense motion network to render the subsequent image.

[0050] The CFTE input is the base image and the subsequent image. Each image goes through the compact feature extractor to obtain the compact feature matrix, and the dense motion estimation network uses the compact feature matrix difference between the base image and the subsequent image and the base image features to generate the dense optical flow and the occlusion map from the subsequent image to the base image. The generator uses the base image and the output of the dense motion network to render the subsequent image.

[0051] At the latest JVET meeting, experts focused on the standardization of generative face video coding, especially the proposal content of Supplemental Enhancement Information (SEI). SEI, as an important coding tool, can provide additional information to improve the efficiency and quality of video coding. The experts agreed to add the SEI proposal content of generative face video to the proposal (Technologies under Consideration, TuC) of future Versatile Supplemental Enhancement Information (VSEI) - ITU-T H.274 | ISO / IEC 23002-7, which shows the forward-looking thinking of future video coding technology development, especially the potential and application prospect of generative face video coding in the environment of very low bit rate. The following describes the SEI information of generative face video.

[0052] Referring to Table 2, the range of the SEI information of the generative face video is as follows:

[0053] Table 2 SEI information range

[0054] SEI information Continuity range … … Generative face video SEI information related image

[0055] That is, the range of the SEI information of the generative face video only works for the video frames related to the SEI information, not the video frames in the entire video.

[0056] The syntax elements in the SEI information of the generative face video are shown in Table 3 as follows:

[0057] Table 3 Syntax elements in SEI information of generated face videos

[0058]

[0059]

[0060]

[0061]

[0062]

[0063] The following Figure 3 The semantics of each syntax element shown is explained below:

[0064] "gfv_id": contains an identification number used to identify facial feature information, which may be used to indicate a neural network as GenerativeNN(). The value of gfv_id is between 0 and 2. 32 -2. The values ​​are 256 to 511, and 2 31 to 2 32 The gfv_id value of -2 is reserved for future use by ITU-T|ISO / IEC. When the value of "gfv_id" is 256 to 511, and 231 to 232-2, the decoder shall ignore the SEI message.

[0065] "gfv_base_pic_flag": When the value of "gfv_base_pic_flag" is 1, it indicates that the currently decoded image is a base image. When the value of "gfv_base_pic_flag" is 0, it indicates that the currently decoded image is not a base image. The constraint on the value of "gfv_base_pic_flag" is: when the SEI information of the generative face video is the first GFVSEI message in the current layer-wise coded video sequence (CLVS) with a specific "gfv_id" value, the value of "gfv_base_pic_flag" is 1.

[0066] "gfv_nn_present_flag" is equal to 1, the SEI message contains or indicates a neural network that can be used as a TranslatorNN(). When the value of "gfv_nn_present_flag" is 0, the SEI message does not contain or indicate a neural network that can be used as a TranslatorNN(). If this syntax element is not present, the value of "gfv_nn_present_flag" is inferred to be 0.

[0067] "gfv_nn_base_flag" is equal to 1, the indicated TranslatorNN() is a base neural network post-filter (NNPF). When the value of "gfv_nn_base_flag" is 1, the indicated TranslatorNN() is an update relative to the base NNPF.

[0068] "gfv_nn_mode_idc" is equal to 0, the neural network information is contained in the neural network post-filter characteristics (NNPFC) SEI message and the neural network information is in the ISO / IEC 15938-17 bitstream format. When the value of "gfv_nn_mode_idc" is 1, the neural network information is identified by the URI indicated by "nnpfc_uri", which is in the format identified by "nnpfc_tag_uri".

[0069] "gfv_nn_reserved_zero_bit_a" is equal to 0.

[0070] "gfv_nn_tag_uri" contains a tag URI, whose syntax and semantics are specified by IETF RFC 4151, for identifying the neural network format and related information that is a base NNPF or an update relative to the base NNPF.

[0071] "gfv_nn_uri" contains a URI, whose syntax and semantics are in accordance with IETF Internet Standard 66, for identifying the neural network that is used as a base NNPF or a neural network that is an update relative to the base NNPF.

[0072] “gfv_nn_payload_byte[i]”: contains the i-th byte of the bitstream conforming to ISO / IEC 15938-17. All nnpfc_payload_byte[i] contents should be combined to be the complete bitstream conforming to ISO / IEC 15938-17.

[0073] “gfv_drive_pic_fusion_flag”: when “gfv_drive_pic_fusion_flag” is present and has a value of 1, it indicates that the current decoded picture (corresponding to one of the drive pictures that can be used for fusion) should be input into GenerativeNN(). When the value of “gfv_drive_pic_fusion_flag” is 0, it indicates that the current decoded picture should not be input into GenerativeNN().

[0074] “gfv_coordinate_present_flag”: when the value of “gfv_coordinate_present_flag” is 1, it indicates that the coordinate information of the key points is present. When the value of “gfv_coordinate_present_flag” is 0, it indicates that the coordinate information of the key points is not present. If “gfv_matrix_type_idx[i]” is 0 or 1, the value of “gfv_coordinate_present_flag” is required to be 1.

[0075] “gfv_coordinate_precision_factor_minus1”: represents the length (in bits) of “gfv_coordinate_x_abs[i]”, “gfv_coordinate_y_abs[i]”, and “gfv_coordinate_z_abs[i]”, specifically: the value of “gfv_coordinate_precision_factor_minus1” plus 1.

[0076] “gfv_num_kps_minus1”: the value of “gfv_num_kps_minus1” plus 1 represents the number of key points. The value of “gfv_num_kps_minus1” should be in the range of 0 to 2 10 -1.

[0077] "gfv_kp_pred_flag": When the value of "gfv_kp_pred_flag" is 1, it indicates that the syntax element "gfv_coordinate_dx_abs[i]", the syntax element "gfv_coordinate_dy_abs[i]" and the syntax element "gfv_coordinate_dz_abs[i]" exist, and the syntax element "gfv_coordinate_dx_sign_flag[i]", the syntax element "gfv_coordinate_dy_sign_flag[i]" and the syntax element "gfv_coordinate_dz_sign_flag[i]" can exist. When the value of "gfv_kp_pred_flag" is 0, it indicates that "gfv_coordinate_x_abs[i]", "gfv_coordinate_y_abs[i]" and "gfv_coordinate_z_abs[i]" exist, and the syntax element "gfv_coordinate_x_sign_flag[i]", the syntax element "gfv_coordinate_y_sign_flag[i]" and the syntax element "gfv_coordinate_z_sign_flag[i]" can exist. When the value of "gfv_kp_pred_flag" is 1, the absolute difference value for the reference frame (base image) is the absolute difference value of the i-th key point and the i-1-th key point; the absolute difference value for the non-reference frame (subsequent image) is the absolute difference value of the i-th key point and the i-th key point of the reference frame (base image).

[0078] "gfv_coordinate_z_present_flag": indicates whether the z-axis coordinate information of the key point exists, the value of "gfv_coordinate_z_present_flag" is 1, indicating that the z-axis coordinate information of the key point exists; the value of "gfv_coordinate_z_present_flag" is 0, indicating that the z-axis coordinate information of the key point does not exist.

[0079] "gfv_coordinate_x_abs[i]": indicates the normalized absolute value of the x-axis coordinate of the i-th key point.

[0080] "gfv_coordinate_x_sign_flag[i]": indicates the sign of the x-axis coordinate of the i-th key point. If "gfv_coordinate_x_sign_flag[i]" does not exist, it is inferred that the value of "gfv_coordinate_x_sign_flag[i]" is 0.

[0081] “gfv_coordinate_y_abs[i]”: indicates the normalized absolute value of the y-axis coordinate of the i-th key point.

[0082] “gfv_coordinate_y_sign_flag[i]”: indicates the sign of the y-axis coordinate of the i-th key point. If “gfv_coordinate_y_sign_flag[i]” is not present, the value of “gfv_coordinate_y_sign_flag[i]” is inferred to be 0.

[0083] “gfv_coordinate_z_abs[i]”: indicates the normalized absolute value of the z-axis coordinate of the i-th key point.

[0084] “gfv_coordinate_z_sign_flag[i]”: indicates the sign of the z-axis coordinate of the i-th key point. If “gfv_coordinate_z_sign_flag[i]” is not present, the value of “gfv_coordinate_z_sign_flag[i]” is inferred to be 0.

[0085] “gfv_coordinate_dx_abs[i]”: indicates the absolute difference value of the x-axis coordinate of the i-th key point.

[0086] “gfv_coordinate_dx_sign_flag[i]”: indicates the sign of the x-axis coordinate difference value of the i-th key point. If “gfv_coordinate_dx_sign_flag[i]” is not present, the value of “gfv_coordinate_dx_sign_flag[i]” is inferred to be 0.

[0087] “gfv_coordinate_dy_abs[i]”: indicates the absolute difference value of the y-axis coordinate of the i-th key point.

[0088] “gfv_coordinate_dy_sign_flag[i]”: indicates the sign of the y-axis coordinate difference value of the i-th key point. If “gfv_coordinate_dy_sign_flag[i]” is not present, the value of “gfv_coordinate_dy_sign_flag[i]” is inferred to be 0.

[0089] “gfv_coordinate_dz_abs[i]”: indicates the absolute difference value of the z-axis coordinate of the i-th key point.

[0090] “gfv_coordinate_dz_sign_flag[i]”: indicates the sign of the z-axis coordinate difference of the i-th key point. If “gfv_coordinate_dz_sign_flag[i]” is not present, its value is inferred to be 0.

[0091] “gfv_matrix_present_flag”: when the value of “gfv_matrix_present_flag” is 1, it indicates that there are matrix parameters in the SEI message. When the value of “gfv_matrix_present_flag” is 0, it indicates that there are no matrix parameters in the SEI message.

[0092] “gfv_matrix_element_precision_factor_minus1”: indicates the number of bits of the decimal part of the matrix element, i.e. the length (in bits) of “gfv_matrix_element_dec[i][j][k][m]”. Specifically, the value of “gfv_matrix_element_precision_factor_minus1” plus 1.

[0093] “gfv_num_matrix_types_minus1”: the value of “gfv_num_matrix_types_minus1” plus 1 indicates the number of matrix types of the signal in the SEI message. The value of “gfv_num_matrix_types_minus1” shall be in the range of 0 to 2 6 -1.

[0094] "gfv_matrix_pred_flag": When the value of "gfv_matrix_pred_flag" is 1, it indicates that the syntax elements "gfv_matrix_element_int[i][j][k][m]", "gfv_matrix_element_dec[i][j][k][m]" and "gfv_matrix_element_sign_flag[i][j][k][m]" can be present. When the value of "gfv_matrix_pred_flag" is 0, it indicates that the syntax elements "gfv_matrix_delta_element_int[i][j][k][m]", "gfv_matrix_delta_element_dec[i][j][k][m]" are present and the syntax elements "gfv_matrix_delta_element_sign_flag[i][j][k][m]" can be present. If this flag is not present, the value of "gfv_matrix_pred_flag" is inferred to be 0. When the value of "gfv_matrix_pred_flag" is 1, the difference is relative to the reference frame (base image) matrix element value.

[0095] "gfv_matrix_type_idx[i]": Indicates the index of the i-th matrix type, as indicated in Table 4. Table 4 details the different "gfv_matrix_type_idx[i]" values and their corresponding matrix types, such as affine transform matrix, covariance matrix, lip shape matrix, etc.

[0096] Table 4 Semantics of gfv_matrix_type_idx

[0097]

[0098] "gfv_num_matrices_equal_to_num_kps_flag[i]": When the value is equal to 1, it indicates that the number of matrices of the i-th matrix type is equal to the value of "gfv_num_kps_minus1" plus 1. When the value of "gfv_num_matrices_equal_to_num_kps_flag[i]" is 0, it indicates that the number of matrices of the i-th matrix type is not equal to the value of "gfv_num_kps_minus1" plus 1.

[0099] "gfv_num_matrices_info[i]": Provides information to derive the number of matrices of the i-th matrix type.

[0100] "gfv_matrix_width_minus1[ i ]": The value of "gfv_matrix_width_minus1[ i ]" plus 1 represents the matrix width of the i-th matrix type.

[0101] "gfv_matrix_height_minus1[ i ]": The value of "gfv_matrix_height_minus1[ i ]" plus 1 represents the matrix height of the i-th matrix type.

[0102] "gfv_matrix_for_3D_space_flag[ i ]": When the value of "gfv_matrix_for_3D_space_flag[ i ]" is 1, it indicates that the i-th matrix type is a matrix defined in three-dimensional space. When the value of "gfv_matrix_for_3D_space_flag[ i ]" is 0, it indicates that the i-th matrix type is a matrix defined in two-dimensional space.

[0103] "gfv_matrix_element_int[ i ][ j ][ k ][ m ]": Indicates the integer part of the matrix element of the k-th row and m-th column of the j-th matrix of the i-th matrix type.

[0104] "gfv_matrix_element_dec[ i ][ j ][ k ][ m ]": Indicates the decimal part of the matrix element of the k-th row and m-th column of the j-th matrix of the i-th matrix type.

[0105] "gfv_matrix_element_sign_flag[ i ][ j ][ k ][ m ]": Indicates the sign of the matrix element of the k-th row and m-th column of the j-th matrix of the i-th matrix type. If "gfv_matrix_element_sign_flag[ i ][ j ][ k ][ m ]" does not exist, it is inferred that the value of "gfv_matrix_element_sign_flag[ i ][ j ][ k ][ m ]" is 0.

[0106] "gfv_matrix_delta_element_int[ i ][ j ][ k ][ m ]": Indicates the integer part of the difference value of the matrix element of the k-th row and m-th column of the j-th matrix of the i-th matrix type.

[0107] "gfv_matrix_delta_element_dec[ i ][ j ][ k ][ m ]": Indicates the decimal part of the difference value of the matrix element of the k-th row and m-th column of the j-th matrix of the i-th matrix type.

[0108] "gfv_matrix_delta_element_sign_flag[i][j][k][m]": indicates the sign of the difference of the matrix element of the k-th row and m-th column of the j-th matrix of the i-th matrix type. If "gfv_matrix_delta_element_sign_flag[i][j][k][m]" is not present, its value is inferred to be 0.

[0109] The descriptor of each syntax element shown in Table 3 represents the entropy decoding algorithm of the corresponding syntax element, and the correspondence between the descriptor and the entropy decoding algorithm is shown in Table 5 as follows:

[0110] Table 5: Entropy decoding algorithm represented by descriptor

[0111] In addition, when the parameter in the descriptor () is n, it indicates that the corresponding syntax element is fixed-length coded; when the parameter in the descriptor () is v, it indicates that the corresponding element is variable-length coded.

[0112] As shown in Table 4 above, the eye matrix represented by "gfv_matrix_type_idx[3]" in the related art is used to represent the eye opening state and degree. However, the eye matrix alone is not sufficient to represent the real and accurate eye movement. In order to solve the above problem, some embodiments of the present application add the pupil position to the eye movement information, so as to more realistically and accurately represent the eye movement. The implementation scheme of adding the pupil position to the eye movement information is described in detail below.

[0113] The coordinate system for representing the pupil position is described below:

[0114] Referring to FIG. 1, Figure 2 In some embodiments, as shown in FIG. 1, the position of the pupil can be represented in a rectangular coordinate system with the left corner of the eye as the origin o and the horizontal axis and the vertical axis as the x and y axes. Figure 2 In the embodiment shown in FIG. 1, the left corner of the eye is taken as the origin, and the horizontal axis and the vertical axis are taken as the x and y axes. Therefore, the manner of determining the position of the pupil can be as follows: a perpendicular line is drawn from the center of the pupil to the y axis, and the length of the perpendicular line from the center of the pupil to the y axis is determined as the x-axis coordinate value x1 of the pupil; a perpendicular line is drawn from the center of the pupil to the x axis, and the length of the perpendicular line from the center of the pupil to the x axis is determined as the y-axis coordinate value y1 of the pupil, thereby determining the coordinate value (x1, y1) of the pupil.

[0115] Referring to FIG. 2, Figure 3 In some embodiments, as shown in FIG. 2, the position of the pupil can be represented in a rectangular coordinate system with the right corner of the eye as the origin o and the horizontal axis and the vertical axis as the x and y axes. Figure 3In the shown embodiment, the right eye corner is taken as the origin, the horizontal axis and the vertical axis are taken as the x and y axes, and thus the manner of determining the position of the pupil can be: a perpendicular line is drawn from the pupil center to the y axis, and the length of the perpendicular line from the pupil center to the y axis is determined as the x axis coordinate value x1 of the pupil, a perpendicular line is drawn from the pupil center to the x axis, and the length of the perpendicular line from the pupil center to the x axis is determined as the y axis coordinate value y1 of the pupil, thereby determining the coordinate value (x1, y1) of the pupil.

[0116] Referring to Figure 4 As shown, in some embodiments, the position of the pupil can be represented in a rectangular coordinate system with the left eye corner as the origin o, the line connecting the left eye corner and the right eye corner as the x axis, and the perpendicular line of the vertical x axis passing through the origin o as the y axis. Figure 4 In the shown embodiment, the left eye corner is taken as the origin, the line connecting the left eye corner and the right eye corner is taken as the x axis, and the perpendicular line of the vertical x axis is taken as the y axis, and thus the manner of determining the position of the pupil can be: a perpendicular line is drawn from the pupil center to the y axis, and the length of the perpendicular line from the pupil center to the y axis is determined as the x axis coordinate value x1 of the pupil, a perpendicular line is drawn from the pupil center to the x axis, and the length of the perpendicular line from the pupil center to the x axis is determined as the y axis coordinate value y1 of the pupil, thereby determining the coordinate value (x1, y1) of the pupil.

[0117] Referring to Figure 5 As shown, in some embodiments, the position of the pupil can be represented in a rectangular coordinate system with the right eye corner as the origin o, the line connecting the left eye corner and the right eye corner as the x axis, and the perpendicular line of the vertical x axis passing through the origin o as the y axis. Figure 5 In the shown embodiment, the left eye corner is taken as the origin, the line connecting the left eye corner and the right eye corner is taken as the x axis, and the perpendicular line of the vertical x axis is taken as the y axis, and thus the manner of determining the position of the pupil can be: a perpendicular line is drawn from the pupil center to the y axis, and the length of the perpendicular line from the pupil center to the y axis is determined as the x axis coordinate value x1 of the pupil, a perpendicular line is drawn from the pupil center to the x axis, and the length of the perpendicular line from the pupil center to the x axis is determined as the y axis coordinate value y1 of the pupil, thereby determining the coordinate value (x1, y1) of the pupil.

[0118] Referring to Figure 6 As shown, in some embodiments, the position of the pupil can be represented in a polar coordinate system with the left eye corner as the pole o and the horizontal axis as the polar axis ox. Figure 6 In the shown embodiment, the left eye corner is taken as the pole o and the horizontal axis as the polar axis ox, and thus the manner of determining the position of the pupil can be: the distance from the pupil center to the left eye corner is determined as the polar radius r of the pupil, and the angle between the polar radius and the polar axis ox is determined as the polar angle θ of the pupil, thereby determining the coordinate value (r, θ) of the pupil.

[0119] Referring to Figure 7 As shown, in some embodiments, the position of the pupil can be represented in a polar coordinate system with the right eye corner as the pole o and the horizontal axis as the polar axis ox. Figure 7In the embodiment shown, the right corner of the eye is taken as the pole o and the horizontal axis is the polar axis ox. Therefore, the method for determining the position of the pupil can be: determining the distance from the center of the pupil to the right corner of the eye as the polar diameter r of the pupil, and determining the angle between the polar diameter and the polar axis ox as the polar angle θ of the pupil, thereby determining the coordinate value (r, θ) of the pupil.

[0120] Reference Figure 8 As shown, in some embodiments, the position of the pupil can be represented by a polar coordinate system with the left corner of the eye as the pole o and the line connecting the left corner of the eye and the right corner of the eye as the polar axis ox. Figure 8 In the embodiment shown, the left corner of the eye is the pole o, and the line connecting the left corner of the eye and the right corner of the eye is the polar axis ox. Therefore, the method for determining the position of the pupil can be: determining the distance from the center of the pupil to the left corner of the eye as the polar diameter r of the pupil, and determining the angle between the polar diameter and the polar axis ox as the polar angle θ of the pupil, thereby determining the coordinate value (r, θ) of the pupil.

[0121] Reference Figure 9 As shown, in some embodiments, the position of the pupil can be represented by a polar coordinate system with the right eye corner as the pole o and the line connecting the right eye corner and the left eye corner as the polar axis ox. Figure 9 In the embodiment shown, the right corner of the eye is taken as the pole o, and the line connecting the right corner of the eye and the left corner of the eye is the polar axis ox. Therefore, the method for determining the position of the pupil can be: determining the distance from the center of the pupil to the right corner of the eye as the polar diameter r of the pupil, and determining the angle between the polar diameter and the polar axis ox as the polar angle θ of the pupil, thereby determining the coordinate value (r, θ) of the pupil.

[0122] The following describes the quantization scheme for quantifying pupil position information:

[0123] In some embodiments, when using Figure 2 The coordinate system shown (the left corner of the eye is the origin o, the horizontal axis and the vertical axis are the rectangular coordinate system of the x and y axes) or Figure 3 When the coordinate system shown (the right corner of the eye is the origin o, the horizontal axis and the vertical axis are the x-axis and the y-axis are the rectangular coordinate system), the x-axis can be normalized by the projection distance of the line connecting the left corner of the eye and the right corner of the eye to the x-axis, and the quantization step size of the x-axis is {1 / 32, 1 / 16, 1 / 8}; the y-axis can be normalized by the distance between the highest point and the lowest point of both eyes, and the quantization step size of the y-axis is {1 / 16, 1 / 8}.

[0124] In some embodiments, when using Figure 4 The coordinate system shown (left corner of the eye is the origin o, the line connecting the left and right corners of the eye is x, and the perpendicular line of the x-axis passing through the origin o is the rectangular coordinate system of the y-axis) or Figure 5In the illustrated coordinate system (the right eye corner as the origin o, the line connecting the left eye corner and the right eye corner as the x-axis, and the perpendicular line of the x-axis passing through the origin o as the y-axis), the x-axis can be normalized by the distance between the left eye corner and the right eye corner, and the x-axis can be selected with quantization steps of {1 / 32, 1 / 16, 1 / 8}; and the y-axis can be normalized by the maximum distance of the projection of the left eye and the right eye in the y-axis direction, and the y-axis can be selected with quantization steps of {1 / 16, 1 / 8}.

[0125] In some embodiments, when the polar coordinate system is adopted Figure 2 In the illustrated coordinate system (the left eye corner as the origin o, the horizontal axis and the vertical axis as the x-axis and the y-axis), or Figure 3 In the illustrated coordinate system (the right eye corner as the origin o, the horizontal axis and the vertical axis as the x-axis and the y-axis), or Figure 4 In the illustrated coordinate system (the left eye corner as the origin o, the line connecting the left eye corner and the right eye corner as the x-axis, and the perpendicular line of the x-axis passing through the origin o as the y-axis), or Figure 5 In the illustrated coordinate system (the right eye corner as the origin o, the line connecting the left eye corner and the right eye corner as the x-axis, and the perpendicular line of the x-axis passing through the origin o as the y-axis), or

[0126] In some embodiments, when the polar coordinate system is adopted Figure 6 In the illustrated coordinate system (the left eye corner as the origin o, the horizontal axis as the polar axis), or Figure 7 In the illustrated coordinate system (the right eye corner as the origin o, the horizontal axis as the polar axis), or Figure 8 In the illustrated coordinate system (the right eye corner as the origin o, the line connecting the left eye corner and the right eye corner as the polar axis), or Figure 9 In the illustrated coordinate system (the right eye corner as the origin o, the line connecting the left eye corner and the right eye corner as the polar axis), or

[0127] In addition, when the pupil position information is encoded, the first frame of picture needs to encode a specific quantization parameter or an index of the quantization parameter, and if the quantization parameter of the subsequent image does not change, the new quantization parameter can not be transmitted, and if the quantization parameter changes, the difference between the new quantization parameter and the first frame quantization parameter or the index of the new quantization parameter can be transmitted.

[0128] In the process of generating a face video code, the pupil is relatively fixed in each frame of video position, and the absolute position coding of each frame of image is resource-consuming. In order to accurately reproduce the pupil position at the decoding end and not to occupy too much code stream in the coding process, the application specifies a reference point, which is the left or right corner of the eye where the pupil is located. After the reference point is determined, the relative coordinate information of the pupil position and the reference point is determined. Finally, the relative coordinate information of each frame is coded, or the relative coordinate residual information between each frame and the previous frame is coded.

[0129] Regarding the pupil position content, there are two schemes of one set of pupil position data and two sets of pupil position data. Specifically, one set of pupil positions can be shared by two eyes, in which case the same reference point is selected for both eyes. For example, the left eye selects the left corner and the right eye selects the left corner, or the left eye selects the right corner and the right eye selects the right corner. Alternatively, the pupil position can be transmitted separately for each eye, in which case the reference point for the left and right eyes is selected arbitrarily.

[0130] In combination with the above-mentioned coordinate system for representing the pupil position and the quantization scheme for quantizing the pupil position information, the application embodiment provides a scheme for encoding the pupil position information as shown in Table 6:

[0131] Table 6 Pupil position information encoding scheme

[0132]

[0133]

[0134] Table 6 shows all the schemes provided by selecting different coordinate systems to represent the pupil position and different quantization schemes for pupil position information. A unique coordinate system is determined for different eye movements, and a unique quantization parameter is determined for different transmission conditions and different coordinate system selections.

[0135] In combination with the transmission of pupil position information, which can transmit coordinate information or coordinate residual information, and the pupil coordinates of both eyes or the pupil coordinates of two eyes are not shared, the application embodiment provides 8x6x2x2 (8 coordinate system selections x 6 quantization parameter selections x coordinate information\coordinate residual information transmissions x two eye pupil coordinate sharing\two eye pupil coordinate not sharing) = 192 different pupil position encoding schemes.

[0136] In some embodiments, the pupil position information can be represented by SEI.

[0137] In video coding, SEI technology is a technology for adding additional information to the video code stream. These additional information can include metadata related to video content, such as timestamp, scene information, copyright information, color space information, etc. The use of SEI can provide more functions and optimize video quality.

[0138] It should be noted that in some embodiments, the pupil position information can also be represented by other ways that can add content to the eyes, and the embodiments of the present application do not limit this.

[0139] The following describes the use of syntax elements when the pupil position information is represented by SEI. Referring to Table 7, the syntax elements used when the pupil position information is represented by SEI include:

[0140] Table 7 represents the SEI syntax elements of the pupil position information

[0141]

[0142]

[0143]

[0144]

[0145]

[0146] "gfv_pupil_coordinate_present_flag": "gfv_pupil_coordinate_present_flag" is used to indicate whether the pupil position information exists. When the value of "gfv_pupil_coordinate_present_flag" is 1, it indicates that the current decoded image contains pupil position information; when the value of "gfv_pupil_coordinate_present_flag" is 0, it indicates that the current decoded image does not contain pupil position information. If the syntax element "gfv_pupil_coordinate_present_flag" does not exist, it is inferred that the value of "gfv_pupil_coordinate_present_flag" is 0.

[0147] "gfv_pupil_both_eyes_flag": is used to indicate whether the pupil position information is common for both eyes. When the value of "gfv_pupil_both_eyes_flag" is 1, it means that the pupil position information is common for both eyes, and the pupil position parameters in the SEI message apply to the pupil position of both eyes; when the value of "gfv_pupil_both_eyes_flag" is 0, it means that the pupil position information is for the left eye and the right eye respectively, and the SEI message can contain two sets of pupil position information.

[0148] "gfv_pupil_coordinate_system_idx": is used to indicate the index of the coordinate system used by the pupil position information. The index is used to select the reference coordinate system of the pupil position coordinates from the predefined coordinate system list. The value of "gfv_pupil_coordinate_system_idx" and the corresponding relationship of the coordinate system can be shown in Table 8 as follows:

[0149] Table 8 Corresponding relationship of index value and coordinate system

[0150]

[0151]

[0152] "gfv_pupil_pred_flag": indicates whether the binocular pupil position information is represented by normalized absolute difference or normalized coordinate value. Specifically, when the value of "gfv_pupil_pred_flag" is 1, it means that the binocular pupil position information is represented by normalized absolute difference, there are syntax elements "gfv_pupil_dx_coordinate_abs" and "gfv_pupil_dy_coordinate_abs", and there can be syntax elements "gfv_pupil_dx_coordinate_sign_flag" and "gfv_pupil_dy_coordinate_sign_flag". When the value of "gfv_pupil_pred_flag" is 0, it means that the binocular pupil position information is represented by normalized coordinate value, there are syntax elements "gfv_pupil_x_coordinate" and "gfv_pupil_y_coordinate". When the value of "gfv_pupil_pred_flag" is 1, the absolute difference is the absolute difference between the pupil position of this frame and the pupil position of the reference frame (base image).

[0153] "gfv_pupil_x_coordinate_precision_factor_minus1": specifies the length (in bits) of the syntax element "gfv_pupil_x_coordinate_abs" or "gfv_pupil_x_coordinate_abs". In particular, the value of "gfv_pupil_x_coordinate_precision_factor_minus1" plus 1.

[0154] "gfv_pupil_y_coordinate_precision_factor_minus1": specifies the length (in bits) of the syntax element "gfv_pupil_y_coordinate_abs" or "gfv_pupil_y_coordinate_abs". In particular, the value of "gfv_pupil_y_coordinate_precision_factor_minus1" plus 1.

[0155] "gfv_pupil_dx_coordinate_abs": specifies the normalized absolute difference of the x-axis coordinate when the pupil position information is shared for both eyes.

[0156] "gfv_pupil_dx_coordinate_sign_flag": specifies the sign of the normalized difference of the x-axis coordinate when the pupil position information is shared for both eyes. If the syntax element "gfv_pupil_dx_coordinate_sign_flag" is not present, the value of "gfv_pupil_dx_coordinate_sign_flag" is inferred to be 0.

[0157] "gfv_pupil_dy_coordinate_abs": specifies the normalized absolute difference of the y-axis coordinate when the pupil position information is shared for both eyes.

[0158] "gfv_pupil_dy_coordinate_sign_flag": specifies the sign of the normalized difference of the y-axis coordinate when the pupil position information is shared for both eyes. If the syntax element "gfv_pupil_dy_coordinate_sign_flag" is not present, the value of "gfv_pupil_dy_coordinate_sign_flag" is inferred to be 0.

[0159] "gfv_pupil_x_coordinate": indicates the x-axis normalized coordinate value when the pupil position information is shared for both eyes.

[0160] "gfv_pupil_y_coordinate": indicates the y-axis normalized coordinate value of the pupil position information when the pupil position information is shared by both eyes.

[0161] "gfv_pupil_polar_radius_precision_factor_minus1": specifies the length (bits) of the syntax element "gfv_pupil_polar_delta_radius_abs" or "gfv_pupil_polar_radius". Specifically, the value of "gfv_pupil_polar_radius_precision_factor_minus1" plus 1.

[0162] "gfv_pupil_polar_angle_precision_factor_minus1": specifies the length (bits) of the syntax element "gfv_pupil_polar_delta_angle_abs" or "gfv_pupil_polar_angle". Specifically, the value of "gfv_pupil_polar_angle_precision_factor_minus1" plus 1.

[0163] "gfv_pupil_polar_delta_radius_abs": specifies the normalized absolute difference value of the polar radius when the pupil position information is shared by both eyes.

[0164] "gfv_pupil_polar_delta_radius_sign_flag": specifies the sign of the normalized difference value of the polar radius when the pupil position information is shared by both eyes. If the syntax element "gfv_pupil_polar_delta_radius_sign_flag" does not exist, the value of "gfv_pupil_polar_delta_radius_sign_flag" is inferred to be 0.

[0165] "gfv_pupil_polar_delta_angle_abs": specifies the normalized absolute difference value of the polar angle when the pupil position information is shared by both eyes.

[0166] "gfv_pupil_polar_delta_angle_sign_flag": specifies the sign of the normalized difference of the polar angle of the pupil position information when shared by both eyes. If the syntax element "gfv_pupil_polar_delta_angle_sign_flag" is not present, the value of "gfv_pupil_polar_delta_angle_sign_flag" is inferred to be 0.

[0167] "gfv_pupil_polar_radius": indicates the normalized polar radius value of the pupil position information when shared by both eyes.

[0168] "gfv_pupil_polar_angle": indicates the normalized polar angle value of the pupil position information when shared by both eyes.

[0169] "gfv_pupil_left_eye_present_flag": indicates whether the left eye pupil position information is present. If the syntax element "gfv_pupil_left_eye_present_flag" is not present, the value of "gfv_pupil_left_eye_present_flag" is inferred to be 0.

[0170] "gfv_pupil_right_eye_present_flag": indicates whether the right eye pupil position information is present. If the syntax element "gfv_pupil_right_eye_present_flag" is not present, the value of "gfv_pupil_right_eye_present_flag" is inferred to be 0.

[0171] "gfv_pupil_left_eye_coordinate_system_idx": indicates the index of the coordinate system used for the left eye pupil position information. The index is used to select the reference coordinate system of the pupil position from the list of pre-defined coordinate systems. The coordinate system represented by the index can refer to Table 7.

[0172] "gfv_pupil_left_eye_pred_flag": indicates whether the left eye pupil position information is represented by normalized absolute difference value or normalized coordinate value. Specifically, when the value of "gfv_pupil_left_eye_pred_flag" is 1, it indicates that the left eye pupil position information is represented by normalized absolute difference value, the syntax elements "gfv_pupil_left_eye_dx_coordinate_abs" and "gfv_pupil_left_eye_dy_coordinate_abs" exist, and the syntax elements "gfv_pupil_left_eye_dx_coordinate_sign_flag" and "gfv_pupil_left_eye_dy_coordinate_sign_flag" can exist. When the value of "gfv_pupil_left_eye_pred_flag" is 0, it indicates that the left eye pupil position information is represented by normalized coordinate value, the syntax elements "gfv_pupil_left_eye_x_coordinate" and "gfv_pupil_left_eye_y_coordinate" exist. When the value of "gfv_pupil_left_eye_pred_flag" is 1, the absolute difference value is the absolute difference value between the pupil position of this frame and the pupil position of the reference frame (base image).

[0173] "gfv_pupil_left_eye_x_coordinate_precision_factor_minus1": indicates the length (bit) of the syntax element "gfv_pupil_left_eye_dx_coordinate_abs" or "gfv_pupil_left_eye_x_coordinate_abs". Specifically, the value of "gfv_pupil_left_eye_x_coordinate_precision_factor_minus1" plus 1.

[0174] "gfv_pupil_left_eye_y_coordinate_precision_factor_minus1": indicates the length (bit) of the syntax element "gfv_pupil_left_eye_dy_coordinate_abs" or "gfv_pupil_left_eye_y_coordinate_abs". Specifically, the value of "gfv_pupil_left_eye_y_coordinate_precision_factor_minus1" plus 1.

[0175] "gfv_pupil_left_eye_dx_coordinate_abs": specifies the normalized absolute difference value of the x-axis coordinate of the left eye pupil position information.

[0176] "gfv_pupil_left_eye_dx_coordinate_sign_flag": specifies the normalized difference value sign of the x-axis coordinate of the left eye pupil position information. If the syntax element "gfv_pupil_left_eye_dx_coordinate_sign_flag" is not present, the value 0 of "gfv_pupil_left_eye_dx_coordinate_sign_flag" is inferred.

[0177] "gfv_pupil_left_eye_dy_coordinate_abs": specifies the normalized absolute difference value of the y-axis coordinate of the left eye pupil position information.

[0178] "gfv_pupil_left_eye_dy_coordinate_sign_flag": specifies the normalized difference value sign of the y-axis coordinate of the left eye pupil position information. If the syntax element "gfv_pupil_left_eye_dy_coordinate_sign_flag" is not present, it is inferred to be equal to 0.

[0179] "gfv_pupil_left_eye_x_coordinate": indicates the x-axis normalized coordinate value of the left eye pupil position information.

[0180] "gfv_pupil_left_eye_y_coordinate": indicates the y-axis normalized coordinate value of the left eye pupil position information.

[0181] "gfv_pupil_left_eye_polar_radius_precision_factor_minus1": specifies the length (bits) of the syntax element "gfv_pupil_left_eye_polar_delta_angle_abs" or "gfv_pupil_left_eye_polar_angle". Specifically, the value of "gfv_pupil_left_eye_polar_radius_precision_factor_minus1" plus 1.

[0182] "gfv_pupil_left_eye_polar_delta_radius_sign_flag": specifies the sign of the normalized difference value of the polar radius of the left eye pupil position information. If the syntax element "gfv_pupil_left_eye_polar_delta_radius_sign_flag" is not present, the value 0 of "gfv_pupil_left_eye_polar_delta_radius_sign_flag" is inferred.

[0183] "gfv_pupil_left_eye_polar_delta_radius_abs": specifies the normalized absolute difference value of the polar radius of the left eye pupil position information.

[0184] "gfv_pupil_left_eye_polar_delta_radius_sign_flag": specifies the sign of the normalized difference value of the polar radius of the left eye pupil position information. If the syntax element "gfv_pupil_left_eye_polar_delta_radius_sign_flag" is not present, the value 0 of "gfv_pupil_left_eye_polar_delta_radius_sign_flag" is inferred.

[0185] "gfv_pupil_left_eye_polar_delta_angle_abs": specifies the normalized absolute difference value of the polar angle of the left eye pupil position information.

[0186] "gfv_pupil_left_eye_polar_delta_angle_sign_flag": specifies the sign of the normalized difference value of the polar angle of the left eye pupil position information. If the syntax element "gfv_pupil_left_eye_polar_delta_angle_sign_flag" is not present, the value 0 of "gfv_pupil_left_eye_polar_delta_angle_sign_flag" is inferred.

[0187] "gfv_pupil_left_eye_polar_radius": indicates the normalized polar radius value of the left eye pupil position information.

[0188] "gfv_pupil_left_eye_polar_angle": indicates the normalized polar angle value of the left eye pupil position information.

[0189] "gfv_pupil_right_eye_coordinate_system_idx" indicates the index of the coordinate system used for the right eye pupil position information. The index is used to select the reference coordinate system of the pupil position from the predefined list of coordinate systems. The index can refer to Table 7.

[0190] "gfv_pupil_right_eye_pred_flag" indicates whether the right eye pupil position information is represented by normalized absolute difference values or normalized coordinate values. Specifically, when the value of "gfv_pupil_right_eye_pred_flag" is 1, the right eye pupil position information is represented by normalized absolute difference values, the syntax elements "gfv_pupil_right_eye_dx_coordinate_abs" and "gfv_pupil_right_eye_dy_coordinate_abs" exist, and the syntax elements "gfv_pupil_right_eye_dx_coordinate_sign_flag" and "gfv_pupil_right_eye_dy_coordinate_sign_flag" can exist. When the value of "gfv_pupil_right_eye_pred_flag" is 0, the right eye pupil position information is represented by normalized coordinate values, the syntax elements "gfv_pupil_right_eye_x_coordinate" and "gfv_pupil_right_eye_y_coordinate" exist. When the value of "gfv_pupil_right_eye_pred_flag" is 1, the absolute difference values are the absolute differences between the pupil position of this frame and the pupil position of the reference frame (base image).

[0191] "gfv_pupil_right_eye_x_coordinate_precision_factor_minus1" indicates the length (bits) of the syntax element "gfv_pupil_left_eye_dx_coordinate_abs" or "gfv_pupil_right_eye_x_coordinate_abs". Specifically, the value of "gfv_pupil_right_eye_x_coordinate_precision_factor_minus1" plus 1.

[0192] "gfv_pupil_right_eye_y_coordinate_precision_factor_minus1": specifies the length (in bits) of the syntax element "gfv_pupil_right_eye_dy_coordinate_abs" or "gfv_pupil_right_eye_y_coordinate_abs". Specifically, the value of "gfv_pupil_right_eye_y_coordinate_precision_factor_minus1" plus 1.

[0193] "gfv_pupil_right_eye_dx_coordinate_abs": specifies the normalized absolute difference value of the x-axis coordinate of the right eye pupil position information.

[0194] "gfv_pupil_right_eye_dx_coordinate_sign_flag": specifies the normalized difference value sign of the x-axis coordinate of the right eye pupil position information. If the syntax element "gfv_pupil_right_eye_dx_coordinate_sign_flag" is not present, the value of "gfv_pupil_right_eye_dx_coordinate_sign_flag" is inferred to be 0.

[0195] "gfv_pupil_right_eye_dy_coordinate_abs": specifies the normalized absolute difference value of the y-axis coordinate of the right eye pupil position information.

[0196] "gfv_pupil_right_eye_dy_coordinate_sign_flag": specifies the normalized difference value sign of the y-axis coordinate of the right eye pupil position information. If the syntax element "gfv_pupil_right_eye_dy_coordinate_sign_flag" is not present, the value of "gfv_pupil_right_eye_dy_coordinate_sign_flag" is inferred to be 0.

[0197] "gfv_pupil_right_eye_x_coordinate": indicates the x-axis normalized coordinate value of the right eye pupil position information.

[0198] "gfv_pupil_right_eye_y_coordinate": indicates the y-axis normalized coordinate value of the right eye pupil position information.

[0199] "gfv_pupil_right_eye_polar_radius_precision_factor_minus1": specifies the length (in bits) of the syntax element "gfv_pupil_right_eye_polar_delta_radius_abs" or "gfv_pupil_right_eye_polar_angle". In particular, the value of "gfv_pupil_right_eye_polar_radius_precision_factor_minus1" plus 1.

[0200] "gfv_pupil_right_eye_polar_angle_precision_factor_minus1": specifies the length (in bits) of the syntax element "gfv_pupil_right_eye_polar_delta_angle_abs" or "gfv_pupil_right_eye_polar_angle". In particular, the value of "gfv_pupil_right_eye_polar_angle_precision_factor_minus1" plus 1.

[0201] "gfv_pupil_right_eye_polar_delta_radius_abs": specifies the normalized absolute difference value of the polar radius of the right eye pupil position information.

[0202] "gfv_pupil_right_eye_polar_delta_radius_sign_flag": specifies the sign of the normalized difference value of the polar radius of the right eye pupil position information. If the syntax element "gfv_pupil_right_eye_polar_delta_radius_sign_flag" is not present, the value of "gfv_pupil_right_eye_polar_delta_radius_sign_flag" is inferred to be 0.

[0203] "gfv_pupil_right_eye_polar_delta_angle_abs": specifies the normalized absolute difference value of the polar angle of the right eye pupil position information.

[0204] "gfv_pupil_right_eye_polar_delta_angle_sign_flag": specifies the sign of the normalized difference of the polar angle of the right eye pupil position information. If the syntax element "gfv_pupil_right_eye_polar_delta_angle_sign_flag" is not present, the value of "gfv_pupil_right_eye_polar_delta_angle_sign_flag" is inferred to be 0.

[0205] "gfv_pupil_right_eye_polar_radius": indicates the normalized polar radius value of the right eye pupil position information.

[0206] "gfv_pupil_right_eye_polar_angle": indicates the normalized polar angle value of the right eye pupil position information.

[0207] Based on the above, the embodiment of the present application also provides a video encoding method, referring to FIG. 1, the video encoding method comprises the following steps S101-S103: Figure 10

[0208] S101, determine whether the current video frame is a base image.

[0209] In the above step S101, if the current video frame is not a base image, the following step S102 is performed:

[0210] S102, performing feature extraction on the current video frame to obtain image features corresponding to the current video frame.

[0211] Among them, the image features corresponding to the current video frame include the pupil position information of the current video frame.

[0212] S103, encoding the image features corresponding to the current video frame to obtain the encoding data of the current video frame.

[0213] The video encoding method provided by the embodiment of the present application first determines whether the current video frame is a base image, and in the case that the current video frame is not a base image, performs feature extraction on the current video frame to obtain image features corresponding to the current video frame, and then encodes the image features corresponding to the current video frame to obtain the encoding data of the current video frame. Since the image features corresponding to the current video frame obtained by the embodiment of the present application performing feature extraction on the current video frame include the pupil position information of the current video frame, the decoding end can improve the accuracy of representing eye movement, thereby more accurately reconstructing the current video frame.

[0214] ​In some embodiments, the feature extraction on the current video frame to obtain the image features corresponding to the current video frame comprises:

[0215] A first rectangular coordinate system is constructed with the left corner of the eye as the origin, the horizontal axis as the x-axis, and the vertical axis as the y-axis;

[0216] The distance from the geometric center of the pupil to the y-axis is obtained to obtain the x-coordinate of the pupil in the first rectangular coordinate system;

[0217] The distance from the geometric center of the pupil to the x-axis is obtained to obtain the y-coordinate of the pupil in the first rectangular coordinate system;

[0218] The pupil position information is generated according to the x-coordinate and the y-coordinate of the pupil in the first rectangular coordinate system.

[0219] That is, the pupil position of the current video frame is represented by the coordinate system shown in Figure 2

[0220] In some embodiments, the feature extraction on the current video frame to obtain the image features corresponding to the current video frame comprises:

[0221] A second rectangular coordinate system is constructed with the right corner of the eye as the origin, the horizontal axis as the x-axis, and the vertical axis as the y-axis;

[0222] The distance from the geometric center of the pupil to the y-axis is obtained to obtain the x-coordinate of the pupil in the second rectangular coordinate system;

[0223] The distance from the geometric center of the pupil to the x-axis is obtained to obtain the y-coordinate of the pupil in the second rectangular coordinate system;

[0224] The pupil position information is generated according to the x-coordinate and the y-coordinate of the pupil in the second rectangular coordinate system.

[0225] That is, the pupil position of the current video frame is represented by the coordinate system shown in Figure 3

[0226] In some embodiments, the feature extraction on the current video frame to obtain the image features corresponding to the current video frame comprises:

[0227] A third rectangular coordinate system is constructed with the left corner of the eye as the origin, the line connecting the left corner and the right corner of the eye as the x-axis, and the axis passing through the origin and perpendicular to the x-axis as the y-axis;

[0228] The distance from the geometric center of the pupil to the y-axis is obtained to obtain the x-coordinate of the pupil in the third rectangular coordinate system;

[0229] ​​obtaining a distance from the geometric center of the pupil to the x-axis to obtain a y-coordinate of the pupil in the third rectangular coordinate system;

[0230] generating the pupil position information according to the x-coordinate and the y-coordinate of the pupil in the third rectangular coordinate system.

[0231] That is, the pupil position of the current video frame is represented by the coordinate system shown in the following figure. Figure 4

[0232] In some embodiments, the feature extraction on the current video frame to obtain the image features corresponding to the current video frame comprises:

[0233] constructing a fourth rectangular coordinate system with the right corner of the eye as the origin, the line connecting the right corner and the left corner of the eye as the x-axis, and the axis passing through the origin and perpendicular to the x-axis as the y-axis;

[0234] obtaining a distance from the geometric center of the pupil to the y-axis to obtain an x-coordinate of the pupil in the fourth rectangular coordinate system;

[0235] obtaining a distance from the geometric center of the pupil to the x-axis to obtain a y-coordinate of the pupil in the fourth rectangular coordinate system;

[0236] generating the pupil position information according to the x-coordinate and the y-coordinate of the pupil in the fourth rectangular coordinate system.

[0237] That is, the pupil position of the current video frame is represented by the coordinate system shown in the following figure. Figure 5

[0238] In some embodiments, the feature extraction on the current video frame to obtain the image features corresponding to the current video frame comprises:

[0239] constructing a first polar coordinate system with the left corner of the eye as the pole and the horizontal axis as the polar axis;

[0240] obtaining a distance from the geometric center of the pupil to the pole to obtain a polar radius of the pupil in the first polar coordinate system;

[0241] obtaining an angle between the line connecting the geometric center of the pupil and the pole and the polar axis to obtain a polar angle of the pupil in the first polar coordinate system;

[0242] generating the pupil position information according to the polar radius and the polar angle of the pupil in the first polar coordinate system.

[0243] That is, the pupil position of the current video frame is represented by the coordinate system shown in the following figure. Figure 6

[0244] ​​​In some embodiments, the feature extraction on the current video frame to obtain the image features corresponding to the current video frame comprises:

[0245] A second polar coordinate system is constructed with the right corner of the eye as a pole and a horizontal axis as a polar axis;

[0246] A distance from the geometric center of the pupil to the pole is obtained to obtain a polar radius of the pupil in the second polar coordinate system;

[0247] An angle between a line connecting the geometric center of the pupil and the pole and the polar axis is obtained to obtain a polar angle of the pupil in the second polar coordinate system;

[0248] The pupil position information is generated according to the polar radius and the polar angle of the pupil in the second polar coordinate system.

[0249] That is, the pupil position of the current video frame is represented by the polar coordinate system shown in FIG. 8. Figure 7

[0250] In some embodiments, the feature extraction on the current video frame to obtain the image features corresponding to the current video frame comprises:

[0251] A third polar coordinate system is constructed with the left corner of the eye as a pole and a line connecting the left corner and the right corner of the eye as a polar axis;

[0252] A distance from the geometric center of the pupil to the pole is obtained to obtain a polar radius of the pupil in the third polar coordinate system;

[0253] An angle between a line connecting the geometric center of the pupil and the pole and the polar axis is obtained to obtain a polar angle of the pupil in the third polar coordinate system;

[0254] The pupil position information is generated according to the polar radius and the polar angle of the pupil in the third polar coordinate system.

[0255] That is, the pupil position of the current video frame is represented by the polar coordinate system shown in FIG. 9. Figure 8

[0256] In some embodiments, the feature extraction on the current video frame to obtain the image features corresponding to the current video frame comprises:

[0257] A fourth polar coordinate system is constructed with the right corner of the eye as a pole and a line connecting the right corner and the left corner of the eye as a polar axis;

[0258] A distance from the geometric center of the pupil to the pole is obtained to obtain a polar radius of the pupil in the fourth polar coordinate system;

[0259] ​​An angle between a line connecting the geometric center of the pupil and the pole point and the polar axis is acquired to acquire a polar angle of the pupil in the fourth polar coordinate system;

[0260] According to the polar radius and the polar angle of the pupil in the fourth polar coordinate system, the pupil position information is generated.

[0261] That is, by Figure 9 The coordinate system shown represents the pupil position of the current video frame.

[0262] In some embodiments, the image features corresponding to the current video frame are encoded to obtain the encoded data of the current video frame, including:

[0263] The x-axis is normalized by the length of the projection of the eye on the x-axis, and the x-coordinate of the pupil is quantized by 1 / 32 or 1 / 16 or 1 / 8 as a quantization step.

[0264] The y-axis is normalized by the length of the projection of the eye on the y-axis, and the y-coordinate of the pupil is quantized by 1 / 16 or 1 / 8 as a quantization step.

[0265] In some embodiments, the image features corresponding to the current video frame are encoded to obtain the encoded data of the current video frame, including:

[0266] The x-axis is normalized by the length of the line connecting the left and right corners of the eye, and the x-coordinate of the pupil is quantized by 1 / 32 or 1 / 16 or 1 / 8 as a quantization step.

[0267] The y-axis is normalized by the length of the projection of the eye on the y-axis, and the y-coordinate of the pupil is quantized by 1 / 16 or 1 / 8 as a quantization step.

[0268] In some embodiments, the image features corresponding to the current video frame are encoded to obtain the encoded data of the current video frame, including:

[0269] The x-axis is normalized by the length of the line connecting the left and right corners of the eye, and the x-coordinate of the pupil is quantized by 1 / 16 or 1 / 8 as a quantization step.

[0270] The y-axis is normalized by the length of the line connecting the left and right corners of the eye, and the y-coordinate of the pupil is quantized by 1 / 16 or 1 / 8 as a quantization step.

[0271] In some embodiments, the image features corresponding to the current video frame are encoded to obtain the encoded data of the current video frame, including:

[0272] The polar axis is normalized by the length of the line connecting the left corner of the eye to the right corner of the eye, and the polar diameter of the pupil is quantized by 1 / 32 or 1 / 16 or 1 / 8 as a quantization step.

[0273] The polar angle is normalized by the angle between the line connecting the highest point of the eye and the polar point and the line connecting the lowest point of the eye and the polar point, and the polar angle of the pupil is quantized by 1 / 16 or 1 / 8 as a quantization step.

[0274] In some embodiments, the image features corresponding to the current video frame are encoded to obtain the encoded data of the current video frame, including:

[0275] The supplementary enhancement information is generated according to the pupil position information.

[0276] The supplementary enhancement information is entropy encoded to obtain the encoded data corresponding to the pupil position information.

[0277] In some embodiments, the supplementary enhancement information is generated according to the pupil position information, including:

[0278] A first identification syntax element for indicating whether the pupil position information exists or not is added in the supplementary enhancement information, and the value of the first identification syntax element is set to a first preset value.

[0279] In some embodiments, the first identification syntax element can be "gfv_pupil_coordinate_present_flag".

[0280] In some embodiments, the first preset value is 1.

[0281] In some embodiments, the supplementary enhancement information is generated according to the pupil position information, including:

[0282] A second identification syntax element for indicating whether the left eye and the right eye share the same pupil position information is added in the supplementary enhancement information, and in the case that the left eye and the right eye share the same pupil position information, the value of the second identification syntax element is set to a first preset value, and in the case that the left eye and the right eye do not share the same pupil position information, the value of the second identification syntax element is set to a second preset value.

[0283] In some embodiments, the second identification syntax element is "gfv_pupil_both_eyes_flag".

[0284] In some embodiments, the first preset value is 1, and the second preset value is 0.

[0285] In some embodiments, the generating the supplemental enhancement information according to the pupil position information comprises:

[0286] In case that the left eye and the right eye share the same pupil position information, a first coordinate system syntax element for representing a coordinate system to which the shared pupil position information belongs is added in the supplemental enhancement information, and a value of the first coordinate system syntax element is set according to the coordinate system to which the shared pupil position information belongs.

[0287] In some embodiments, the first coordinate system syntax element is “gfv_pupil_coordinate_system_idx”.

[0288] In some embodiments, setting the value of the first coordinate system syntax element according to the coordinate system to which the shared pupil position information belongs comprises:

[0289] when the coordinate system to which the shared pupil position information belongs is the coordinate system shown in Figure 2 , the value of the first coordinate system syntax element is set to 0;

[0290] when the coordinate system to which the shared pupil position information belongs is the coordinate system shown in Figure 3 , the value of the first coordinate system syntax element is set to 1;

[0291] when the coordinate system to which the shared pupil position information belongs is the coordinate system shown in Figure 4 , the value of the first coordinate system syntax element is set to 2;

[0292] when the coordinate system to which the shared pupil position information belongs is the coordinate system shown in Figure 5 , the value of the first coordinate system syntax element is set to 3;

[0293] when the coordinate system to which the shared pupil position information belongs is the coordinate system shown in Figure 6 , the value of the first coordinate system syntax element is set to 4;

[0294] when the coordinate system to which the shared pupil position information belongs is the coordinate system shown in Figure 7 , the value of the first coordinate system syntax element is set to 5;

[0295] when the coordinate system to which the shared pupil position information belongs is the coordinate system shown in Figure 8 , the value of the first coordinate system syntax element is set to 6;

[0296] when the coordinate system to which the shared pupil position information belongs is the coordinate system shown in Figure 9 , the value of the first coordinate system syntax element is set to 7.

[0297] In some embodiments, the generating the supplemental enhancement information according to the pupil position information comprises:

[0298] In the case that the left eye and the right eye share the same pupil position information and the coordinate system to which the pupil position information belongs is a rectangular coordinate system, a first depth syntax element for representing the bit depth of the x coordinate and a second depth syntax element for representing the bit depth of the y coordinate are added in the supplemental enhancement information, and the value of the first depth syntax element is set according to the bit depth of the x coordinate, and the value of the second depth syntax element is set according to the bit depth of the y coordinate.

[0299] In some embodiments, the first depth syntax element is “gfv_pupil_x_coordinate_precision_factor_minus1”, and the second depth syntax element is “gfv_pupil_y_coordinate_precision_factor_minus1”.

[0300] In some embodiments, the generating supplemental enhancement information according to the pupil position information comprises:

[0301] In the case that the left eye and the right eye share the same pupil position information, a third identification syntax element for representing the representation manner of the shared pupil position information is added in the supplemental enhancement information, and in the case that the shared pupil position information is represented by a normalized absolute difference value from the pupil position information of the base image, the value of the third identification syntax element is set to a first preset value; and in the case that the shared pupil position information is represented by a normalized value, the value of the third identification syntax element is set to a second preset value.

[0302] In some embodiments, the third identification syntax element is “gfv_pupil_pred_flag”.

[0303] In some embodiments, the generating supplemental enhancement information according to the pupil position information comprises:

[0304] In the case that the left eye and the right eye share the same pupil position information, the coordinate system to which the pupil position information belongs is a rectangular coordinate system, and the shared pupil position information is represented by a normalized absolute difference value from the pupil position information of the base image, a first position information syntax element and a second position information syntax element are added in the supplemental enhancement information, and the value of the first position information syntax element is set to a normalized absolute difference value of the x-axis coordinate of the shared pupil position information, and the value of the second position information syntax element is set to a normalized absolute difference value of the y-axis coordinate of the shared pupil position information.

[0305] In some embodiments, the first position information syntax element is "gfv_pupil_dx_coordinate_abs", and the second position information syntax element is "gfv_pupil_dy_coordinate_abs".

[0306] In some embodiments, the generating the supplemental enhancement information according to the pupil position information further comprises:

[0307] In the supplemental enhancement information, a first sign syntax element and / or a second sign syntax element are added, and a value of the first sign syntax element is set according to a sign of a normalized difference of x-axis coordinates of the common pupil position information, and a value of the second sign syntax element is set according to a sign of a normalized difference of y-axis coordinates of the common pupil position information.

[0308] In some embodiments, the first sign syntax element is "gfv_pupil_dx_coordinate_sign_flag", and the second sign syntax element is "gfv_pupil_dy_coordinate_sign_flag".

[0309] In some embodiments, the generating the supplemental enhancement information according to the pupil position information comprises:

[0310] In the case that the left eye and the right eye share the same pupil position information, and the coordinate system to which the common pupil position information belongs is a rectangular coordinate system, and the common pupil position information is represented by a normalized value, in the supplemental enhancement information, a third position information syntax element and a fourth position information syntax element are added, and a value of the third position information syntax element is set as a normalized value of an x-axis coordinate of the common pupil position information, and a value of the fourth position information syntax element is set as a normalized value of a y-axis coordinate of the common pupil position information.

[0311] In some embodiments, the third position information syntax element is "gfv_pupil_x_coordinate", and the fourth position information syntax element is "gfv_pupil_y_coordinate".

[0312] In some embodiments, the generating the supplemental enhancement information according to the pupil position information comprises:

[0313] In the case that the left eye and the right eye share the same pupil position information, and the coordinate system to which the common pupil position information belongs is a polar coordinate system, in the supplemental enhancement information, a third bit depth syntax element for representing a bit depth of a polar radius and a fourth bit depth syntax element for representing a bit depth of a polar angle are added, and a value of the third bit depth syntax element is set according to the bit depth of the polar radius, and a value of the fourth bit depth syntax element is set according to the bit depth of the polar angle.

[0314] In some embodiments, the third bit-depth syntax element is “gfv_pupil_polar_radius_precision_factor_minus”, and the fourth bit-depth syntax element is “gfv_pupil_polar_angle_precision_factor_minus1”.

[0315] In some embodiments, the generating the supplemental enhancement information according to the pupil position information comprises:

[0316] In the case that the left eye and the right eye share the same pupil position information, the coordinate system to which the shared pupil position information belongs is a polar coordinate system, and the shared pupil position information is represented by a normalized absolute difference value with the pupil position information of the base image, a fifth position information syntax element and a sixth position information syntax element are added in the supplemental enhancement information, the value of the fifth position information syntax element is set to a normalized absolute difference value of a polar radius of the shared pupil position information, and the value of the sixth position information syntax element is set to a normalized absolute difference value of a polar angle of the shared pupil position information.

[0317] In some embodiments, the fifth position information syntax element is “gfv_pupil_polar_delta_radius_abs”, and the sixth position information syntax element is “gfv_pupil_polar_delta_angle_abs”.

[0318] In some embodiments, the generating the supplemental enhancement information according to the pupil position information further comprises:

[0319] In the supplemental enhancement information, a third sign syntax element and / or a fourth sign syntax element are added, the value of the third sign syntax element is set according to a sign of a normalized difference value of a polar radius of the shared pupil position information, and the value of the fourth sign syntax element is set according to a sign of a normalized difference value of a polar angle of the shared pupil position information.

[0320] In some embodiments, the third sign syntax element is “gfv_pupil_polar_delta_radius_sign_flag”, and the fourth sign syntax element is “gfv_pupil_polar_delta_angle_sign_flag”.

[0321] In some embodiments, the generating the supplemental enhancement information according to the pupil position information comprises:

[0322] In the case that the left eye and the right eye share the same pupil position information, the coordinate system to which the pupil position information belongs is a polar coordinate system, and the shared pupil position information is represented by a normalized value, the seventh position information syntax element and the eighth position information syntax element are added in the supplemental enhancement information, the value of the seventh position information syntax element is set to the normalized value of the polar radius of the shared pupil position information, and the value of the eighth position information syntax element is set to the normalized value of the polar angle of the shared pupil position information.

[0323] In some embodiments, the seventh position information syntax element is "gfv_pupil_x_coordinate", and the eighth position information syntax element is "gfv_pupil_y_coordinate".

[0324] In some embodiments, the generating supplemental enhancement information according to the pupil position information comprises:

[0325] In the case that the left eye and the right eye do not share the same pupil position information, the fourth identification syntax element for indicating whether the left eye pupil position information exists and the fifth identification syntax element for indicating whether the right eye pupil position information exists are added in the supplemental enhancement information, and the value of the fourth identification syntax element is set according to whether the left eye pupil position information exists, and the value of the fifth identification syntax element is set according to whether the right eye pupil position information exists.

[0326] In some embodiments, the fourth identification syntax element is "gfv_pupil_left_eye_present_flag", and the fifth identification syntax element is "gfv_pupil_right_eye_present_flag".

[0327] In some embodiments, the generating supplemental enhancement information according to the pupil position information comprises:

[0328] In the case that the left eye pupil position information exists, the second coordinate system syntax element for indicating the coordinate system to which the left eye pupil position information belongs is added in the supplemental enhancement information, and the value of the second coordinate system syntax element is set according to the coordinate system constructed when the left eye pupil position information is acquired.

[0329] In some embodiments, the value of the second coordinate system syntax element is "gfv_pupil_left_eye_coordinate_system_idx".

[0330] In some embodiments, the generating supplemental enhancement information according to the pupil position information comprises:

[0331] In a case where the left eye pupil position information exists and a coordinate system to which the left eye pupil position information belongs is a rectangular coordinate system, a fifth depth syntax element for representing a bit depth of an x coordinate and a sixth depth syntax element for representing a bit depth of a y coordinate are added in the supplemental enhancement information, and a value of the fifth depth syntax element is set according to the bit depth of the x coordinate, and a value of the sixth depth syntax element is set according to the bit depth of the y coordinate.

[0332] In some embodiments, the fifth depth syntax element is “gfv_pupil_left_eye_x_coordinate_precision_factor_minus1”, and the sixth depth syntax element is “gfv_pupil_left_eye_y_coordinate_precision_factor_minus1”.

[0333] In some embodiments, the generating the supplemental enhancement information according to the pupil position information comprises:

[0334] In a case where the left eye pupil position information exists, a sixth identification syntax element for representing a representation manner of the left eye pupil position information is added in the supplemental enhancement information, and in a case where the left eye pupil position information is represented by a normalized absolute difference value of the pupil position information of the base image, a value of the sixth identification syntax element is set as a first preset value; in a case where the left eye pupil position information is represented by a normalized value, the value of the sixth identification syntax element is set as a second preset value.

[0335] In some embodiments, the sixth identification syntax element is “gfv_pupil_left_eye_pred_flag”.

[0336] In some embodiments, the generating the supplemental enhancement information according to the pupil position information comprises:

[0337] In a case where the left eye pupil position information exists, a coordinate system to which the left eye pupil position information belongs is a rectangular coordinate system, and the left eye pupil position information is represented by a normalized absolute difference value of the pupil position information of the base image, a ninth position information syntax element and a tenth position information syntax element are added in the supplemental enhancement information, and a value of the ninth position information syntax element is set as a normalized absolute difference value of an x axis coordinate of the left eye pupil position information, and a value of the tenth position information syntax element is set as a normalized absolute difference value of a y axis coordinate of the left eye pupil position information.

[0338] In some embodiments, the ninth position information syntax element is "gfv_pupil_left_eye_dx_coordinate_abs", and the tenth position information syntax element is "gfv_pupil_left_eye_dy_coordinate_abs".

[0339] In some embodiments, the generating the supplemental enhancement information according to the pupil position information further comprises:

[0340] In the supplemental enhancement information, a fifth sign syntax element and / or a sixth sign syntax element are added, and a value of the fifth sign syntax element is set according to a sign of a normalized difference value of an x-axis coordinate of the left eye pupil position information, and a value of the sixth sign syntax element is set according to a sign of a normalized difference value of a y-axis coordinate of the left eye pupil position information.

[0341] In some embodiments, the fifth sign syntax element is "gfv_pupil_left_eye_dx_coordinate_sign_flag", and the fifth sign syntax element is "gfv_pupil_left_eye_dy_coordinate_sign_flag".

[0342] In some embodiments, the generating the supplemental enhancement information according to the pupil position information comprises:

[0343] In a case where the left eye pupil position information exists, a coordinate system to which the left eye pupil position information belongs is a rectangular coordinate system, and the left eye pupil position information is represented by a normalized value, in the supplemental enhancement information, an eleventh position information syntax element and a twelfth position information syntax element are added, and a value of the eleventh position information syntax element is set as a normalized value of an x-axis coordinate of the left eye pupil position information, and a value of the twelfth position information syntax element is set as a normalized value of a y-axis coordinate of the left eye pupil position information.

[0344] In some embodiments, the eleventh position information is "gfv_pupil_left_eye_x_coordinate", and the eleventh position information is "gfv_pupil_left_eye_y_coordinate".

[0345] In some embodiments, the generating the supplemental enhancement information according to the pupil position information comprises:

[0346] In a case where the left eye pupil position information exists and a coordinate system to which the left eye pupil position information belongs is a polar coordinate system, a seventh depth syntax element for representing a bit depth of a polar radius and an eighth depth syntax element for representing a bit depth of a polar angle are added in the supplemental enhancement information, and a value of the seventh depth syntax element is set according to the bit depth of the polar radius, and a value of the eighth depth syntax element is set according to the bit depth of the polar angle.

[0347]

[0348] “gfv_pupil_left_eye_polar_radius_precision_factor_minus1”, the eighth depth syntax element “gfv_pupil_left_eye_polar_angle_precision_factor_minus1”.

[0349] In some embodiments, the generating the supplemental enhancement information according to the pupil position information comprises:

[0350] In a case where the left eye pupil position information exists, a coordinate system to which the left eye pupil position information belongs is a polar coordinate system, and the left eye pupil position information is represented by a normalized absolute difference value from the pupil position information of the base image, a thirteenth position information syntax element and a fourteenth position information syntax element are added in the supplemental enhancement information, a value of the thirteenth position information syntax element is set as a normalized absolute difference value of the polar radius of the left eye pupil position information, and a value of the fourteenth position information syntax element is set as a normalized absolute difference value of the polar angle of the left eye pupil position information.

[0351] In some embodiments, the thirteenth position information syntax element is “gfv_pupil_left_eye_polar_delta_radius_abs”, and the fourteenth position information syntax element is “gfv_pupil_left_eye_polar_delta_angle_abs.

[0352] In some embodiments, the generating the supplemental enhancement information according to the pupil position information further comprises:

[0353] A seventh sign syntax element and / or an eighth sign syntax element are added in the supplemental enhancement information, a value of the seventh sign syntax element is set according to a sign of the normalized difference value of the polar radius of the left eye pupil position information, and a value of the eighth sign syntax element is set according to a sign of the normalized difference value of the polar angle of the left eye pupil position information.

[0354] In some embodiments, the seventh syntax element is "gfv_pupil_left_eye_polar_delta_radius_sign_flag", and the eighth syntax element is "gfv_pupil_left_eye_polar_delta_angle_sign_flag".

[0355] In some embodiments, the generating the supplemental enhancement information according to the pupil position information comprises:

[0356] In the case that the left eye pupil position information exists, the coordinate system to which the left eye pupil position information belongs is a polar coordinate system, and the left eye pupil position information is represented by a normalized value, a fifteenth position information syntax element and a sixteenth position information syntax element are added in the supplemental enhancement information, the value of the fifteenth position information syntax element is set as the normalized value of the polar radius of the left eye pupil position information, and the value of the sixteenth position information syntax element is set as the normalized value of the polar angle of the left eye pupil position information.

[0357] In some embodiments, the fifteenth position information syntax element is "gfv_pupil_left_eye_polar_radius", and the sixteenth position information syntax element is "gfv_pupil_left_eye_polar_angle".

[0358] In some embodiments, the generating the supplemental enhancement information according to the pupil position information comprises:

[0359] In the case that the right eye pupil position information exists, a third coordinate system syntax element for representing the coordinate system to which the right eye pupil position information belongs is added in the supplemental enhancement information, and the value of the third coordinate system syntax element is set according to the coordinate system constructed when the right eye pupil position information is acquired.

[0360] In some embodiments, the third coordinate system syntax element is "gfv_pupil_right_eye_coordinate_system_idx".

[0361] In some embodiments, the generating the supplemental enhancement information according to the pupil position information comprises:

[0362] In the case that the right eye pupil position information exists and the coordinate system to which the right eye pupil position information belongs is a rectangular coordinate system, a ninth bit depth syntax element for representing the bit depth of the x coordinate and a tenth bit depth syntax element for representing the bit depth of the y coordinate are added in the supplemental enhancement information, the value of the ninth bit depth syntax element is set according to the bit depth of the x coordinate, and the value of the tenth bit depth syntax element is set according to the bit depth of the y coordinate.

[0363] In some embodiments, the value of the ninth bit-depth syntax element is "gfv_pupil_right_eye_x_coordinate_precision_factor_minus1", and the value of the tenth bit-depth syntax element is "gfv_pupil_right_eye_y_coordinate_precision_factor_minus1".

[0364] In some embodiments, the generating the supplemental enhancement information according to the pupil position information comprises:

[0365] In the case that the right eye pupil position information exists, a seventh identification syntax element for representing a representation manner of the right eye pupil position information is added in the supplemental enhancement information, and in the case that the right eye pupil position information is represented by a normalized absolute difference value from the pupil position information of the base image, the value of the seventh identification syntax element is set to a first preset value; in the case that the right eye pupil position information is represented by a normalized value, the value of the seventh identification syntax element is set to a second preset value.

[0366] In some embodiments, the seventh identification syntax element is "gfv_pupil_right_eye_pred_flag".

[0367] In some embodiments, the generating the supplemental enhancement information according to the pupil position information comprises:

[0368] In the case that the right eye pupil position information exists, the coordinate system to which the right eye pupil position information belongs is a rectangular coordinate system, and the right eye pupil position information is represented by a normalized absolute difference value from the pupil position information of the base image, a seventeenth position information syntax element and an eighteenth position information syntax element are added in the supplemental enhancement information, the value of the seventeenth position information syntax element is set to a normalized absolute difference value of an x-axis coordinate of the right eye pupil position information, and the value of the eighteenth position information syntax element is set to a normalized absolute difference value of a y-axis coordinate of the right eye pupil position information.

[0369] In some embodiments, the seventeenth position information syntax element is "gfv_pupil_right_eye_dx_coordinate_abs", and the eighteenth position information syntax element is "gfv_pupil_right_eye_dy_coordinate_abs".

[0370] In some embodiments, the generating the supplemental enhancement information according to the pupil position information further comprises:

[0371] The ninth and tenth sign syntax elements are added in the supplemental enhancement information, and a value of the ninth sign syntax element is set according to a sign of a normalized difference value of an x-axis coordinate of the right eye pupil position information, and a value of the tenth sign syntax element is set according to a sign of a normalized difference value of a y-axis coordinate of the right eye pupil position information.

[0372] In some embodiments, the ninth sign syntax element is “gfv_pupil_right_eye_dx_coordinate_sign_flag”, and the tenth sign syntax element is “gfv_pupil_right_eye_dy_coordinate_sign_flag”.

[0373] In some embodiments, the generating supplemental enhancement information according to the pupil position information comprises:

[0374] In a case where the right eye pupil position information exists, a coordinate system to which the right eye pupil position information belongs is a rectangular coordinate system, and the right eye pupil position information is represented by a normalized value, a nineteenth position information syntax element and a twentieth position information syntax element are added in the supplemental enhancement information, a value of the nineteenth position information syntax element is set as a normalized value of an x-axis coordinate of the right eye pupil position information, and a value of the twentieth position information syntax element is set as a normalized value of a y-axis coordinate of the right eye pupil position information.

[0375] In some embodiments, the nineteenth position information syntax element is “gfv_pupil_right_eye_x_coordinate”, and the twentieth position information syntax element is “gfv_pupil_right_eye_y_coordinate”.

[0376] In some embodiments, the generating supplemental enhancement information according to the pupil position information comprises:

[0377] In a case where the right eye pupil position information exists, and a coordinate system to which the right eye pupil position information belongs is a polar coordinate system, an eleventh bit depth syntax element for representing a bit depth of a polar radius and a twelfth bit depth syntax element for representing a bit depth of a polar angle are added in the supplemental enhancement information, a value of the eleventh bit depth syntax element is set according to the bit depth of the polar radius, and a value of the twelfth bit depth syntax element is set according to the bit depth of the polar angle.

[0378] In some embodiments, the eleventh depth syntax element is "gfv_pupil_right_eye_polar_radius_precision_factor_minus1", and the twelfth depth syntax element is "gfv_pupil_right_eye_polar_angle_precision_factor_minus1".

[0379] In some embodiments, the generating the supplemental enhancement information according to the pupil position information comprises:

[0380] In the case that the right eye pupil position information exists, the coordinate system to which the right eye pupil position information belongs is a polar coordinate system, and the pupil position information is represented by a normalized absolute difference value with the pupil position information of the base image, the twenty-first position information syntax element and the twenty-second position information syntax element are added in the supplemental enhancement information, the value of the twenty-first position information syntax element is set as the normalized absolute difference value of the polar radius of the right eye pupil position information, and the value of the twenty-second position information syntax element is set as the normalized absolute difference value of the polar angle of the right eye pupil position information.

[0381] In some embodiments, the twenty-first position information syntax element is "gfv_pupil_right_eye_polar_delta_radius_abs", and the twenty-first position information syntax element is "gfv_pupil_right_eye_polar_delta_angle_abs".

[0382] In some embodiments, the generating the supplemental enhancement information according to the pupil position information further comprises:

[0383] The eleventh sign syntax element and / or the twelfth sign syntax element are added in the supplemental enhancement information, the value of the eleventh sign syntax element is set according to the sign of the normalized difference value of the polar radius of the right eye pupil position information, and the value of the twelfth sign syntax element is set according to the sign of the normalized difference value of the polar angle of the right eye pupil position information.

[0384] In some embodiments, the eleventh sign syntax element is "gfv_pupil_right_eye_polar_delta_radius_sign_flag", and the eleventh sign syntax element is "gfv_pupil_right_eye_polar_delta_angle_sign_flag".

[0385] In some embodiments, the generating the supplemental enhancement information according to the pupil position information comprises:

[0386] In a case where the right eye pupil position information exists, a coordinate system to which the right eye pupil position information belongs is a polar coordinate system, and the right eye pupil position information is represented by a normalized value, the twenty-third position information syntax element and the twenty-fourth position information syntax element are added in the supplementary enhancement information, a value of the twenty-third position information syntax element is set as a normalized value of a polar radius of the right eye pupil position information, and a value of the twenty-fourth position information syntax element is set as a normalized value of a polar angle of the right eye pupil position information.

[0387] In some embodiments, the twenty-third position information syntax element is gfv_pupil_right_eye_polar_radius, and the twenty-fourth position information syntax element is gfv_pupil_right_eye_polar_angle.

[0388] The embodiments of the present application can also provide an image decoding method. Referring to FIG. 11, the image decoding method comprises the following steps. Figure 11

[0389] S111, obtaining encoding data of a current video frame.

[0390] S112, decoding the encoding data of the current video frame to obtain image features corresponding to the current video frame.

[0391] The image features corresponding to the current video frame comprise pupil position information of the current video frame.

[0392] S113, reconstructing the current video according to the image features corresponding to the current video frame to obtain a reconstructed current video frame.

[0393] The implementation manner of decoding the encoding data of the current video frame to obtain the image features of the current video frame in the step S112 corresponds to the implementation manner of encoding the image features of the current video frame to generate the encoding data of the current video frame. It should be understood by those skilled in the art that decoding the encoding data of the current video frame can obtain the image features comprising the pupil position information of the current video frame, for example, the second identification syntax element can be obtained by decoding to determine whether the left eye and the right eye share the same set of pupil position information. For another example, the coordinate system to which the shared pupil position information belongs can be determined by decoding the value of the first coordinate system syntax element. Without avoiding repetition, the above-mentioned implementation manners are not repeated here.

[0394] In some embodiments, the embodiments of the present application provide a video encoding device, comprising:

[0395] a memory configured to store a computer program;

[0396] ​The processor is configured to cause the video encoding apparatus to implement the video encoding method according to any one of the preceding embodiments when the computer program is invoked.

[0397] In some embodiments, the present application provides a video decoding apparatus, comprising:

[0398] The memory is configured to store the computer program.

[0399] The processor is configured to cause the video decoding apparatus to implement the video decoding method according to any one of the preceding embodiments when the computer program is invoked.

[0400] In some embodiments, the present application provides a computer readable storage medium having a computer program stored thereon, which, when executed by a computing device, causes the computing device to implement the video encoding method or the video decoding method according to any one of the preceding embodiments.

[0401] In some embodiments, the present application provides a computer program product, which, when executed on a computer, causes the computer to implement the video encoding method or the video decoding method according to any one of the preceding embodiments.

[0402] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0403] In order to facilitate explanation, the above description has been made in combination with specific embodiments. However, the above exemplary discussion is not intended to exhaust or limit the embodiments to the specific forms disclosed above. According to the above teaching, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different modified embodiments suitable for specific use considerations.

Claims

1. A method of video coding, the method comprising: The method comprises the following steps: determining whether the current video frame is a base image; in the case that the current video frame is not a base image, performing feature extraction on the current video frame to obtain image features corresponding to the current video frame; the image features corresponding to the current video frame include pupil position information of the current video frame; encoding the image features corresponding to the current video frame to obtain encoding data of the current video frame.

2. The method of claim 1, wherein, The feature extraction on the current video frame to obtain the image features corresponding to the current video frame comprises: a first rectangular coordinate system is constructed with the left corner of the eye as the origin, the horizontal axis as the x-axis and the vertical axis as the y-axis; the distance from the geometric center of the pupil to the y-axis is obtained to obtain the x-coordinate of the pupil in the first rectangular coordinate system; the distance from the geometric center of the pupil to the x-axis is obtained to obtain the y-coordinate of the pupil in the first rectangular coordinate system; the pupil position information is generated according to the x-coordinate and the y-coordinate of the pupil in the first rectangular coordinate system.

3. The method of claim 1, wherein, The feature extraction on the current video frame to obtain the image features corresponding to the current video frame comprises: a second rectangular coordinate system is constructed with the right corner of the eye as the origin, the horizontal axis as the x-axis and the vertical axis as the y-axis; the distance from the geometric center of the pupil to the y-axis is obtained to obtain the x-coordinate of the pupil in the second rectangular coordinate system; the distance from the geometric center of the pupil to the x-axis is obtained to obtain the y-coordinate of the pupil in the second rectangular coordinate system; the pupil position information is generated according to the x-coordinate and the y-coordinate of the pupil in the second rectangular coordinate system.

4. The method of claim 1, wherein, The feature extraction on the current video frame to obtain the image features corresponding to the current video frame comprises: a third rectangular coordinate system is constructed with the left corner of the eye as the origin, the line connecting the left corner and the right corner of the eye as the x-axis and the axis passing through the origin and perpendicular to the x-axis as the y-axis; the distance from the geometric center of the pupil to the y-axis is obtained to obtain the x-coordinate of the pupil in the third rectangular coordinate system; the distance from the geometric center of the pupil to the x-axis is obtained to obtain the y-coordinate of the pupil in the third rectangular coordinate system; the pupil position information is generated according to the x-coordinate and the y-coordinate of the pupil in the third rectangular coordinate system.

5. The method of claim 1, wherein, The feature extraction on the current video frame to obtain the image features corresponding to the current video frame comprises: a fourth rectangular coordinate system is constructed with the right corner of the eye as the origin, the line connecting the right corner and the left corner of the eye as the x-axis and the axis passing through the origin and perpendicular to the x-axis as the y-axis; the distance from the geometric center of the pupil to the y-axis is obtained to obtain the x-coordinate of the pupil in the fourth rectangular coordinate system; the distance from the geometric center of the pupil to the x-axis is obtained to obtain the y-coordinate of the pupil in the fourth rectangular coordinate system; the pupil position information is generated according to the x-coordinate and the y-coordinate of the pupil in the fourth rectangular coordinate system.

6. The method of claim 1, wherein, The feature extraction on the current video frame to obtain the image features corresponding to the current video frame comprises: a first polar coordinate system is constructed with the left corner of the eye as the pole and the horizontal axis as the polar axis; acquire a distance from the geometric center of the pupil to the pole point to acquire a polar radius of the pupil in the first polar coordinate system; acquire an angle between a line connecting the geometric center of the pupil and the pole point and the polar axis to acquire a polar angle of the pupil in the first polar coordinate system; generate the pupil position information according to the polar radius and the polar angle of the pupil in the first polar coordinate system.

7. The method of claim 1, wherein, The feature extraction on the current video frame to acquire the image feature corresponding to the current video frame comprises: a second polar coordinate system is constructed with a right corner of an eye as a pole point and a horizontal axis as a polar axis; a distance from the geometric center of the pupil to the pole point is acquired to acquire a polar radius of the pupil in the second polar coordinate system; an angle between a line connecting the geometric center of the pupil and the pole point and the polar axis is acquired to acquire a polar angle of the pupil in the second polar coordinate system; the pupil position information is generated according to the polar radius and the polar angle of the pupil in the second polar coordinate system.

8. The method of claim 1, wherein, The feature extraction on the current video frame to acquire the image feature corresponding to the current video frame comprises: a third polar coordinate system is constructed with a left corner of an eye as a pole point and a line connecting the left corner and the right corner of the eye as a polar axis; a distance from the geometric center of the pupil to the pole point is acquired to acquire a polar radius of the pupil in the third polar coordinate system; an angle between a line connecting the geometric center of the pupil and the pole point and the polar axis is acquired to acquire a polar angle of the pupil in the third polar coordinate system; the pupil position information is generated according to the polar radius and the polar angle of the pupil in the third polar coordinate system.

9. The method of claim 1, wherein, The feature extraction on the current video frame to acquire the image feature corresponding to the current video frame comprises: a fourth polar coordinate system is constructed with a right corner of an eye as a pole point and a line connecting the right corner and the left corner of the eye as a polar axis; a distance from the geometric center of the pupil to the pole point is acquired to acquire a polar radius of the pupil in the fourth polar coordinate system; an angle between a line connecting the geometric center of the pupil and the pole point and the polar axis is acquired to acquire a polar angle of the pupil in the fourth polar coordinate system; the pupil position information is generated according to the polar radius and the polar angle of the pupil in the fourth polar coordinate system.

10. The method of claim 2 or 3, wherein, The encoding on the image feature corresponding to the current video frame to acquire the encoding data of the current video frame comprises: a length of a projection of the eye on an x axis is used to normalize the x axis, and a length of a projection of the eye on a y axis is used to normalize the y axis. a length of a line connecting the left corner and the right corner of the eye is used to normalize the x axis, and a length of a projection of the eye on a y axis is used to normalize the y axis.

11. The method of claim 4 or 5, wherein, ​ ​ The length of the projection of the eye on the y-axis is normalized, and the y-coordinate of the pupil is quantized with a quantization step of 1 / 16 or 1 / 8.

12. The method according to any one of claims 2 to 5, characterized in that, The image features corresponding to the current video frame are encoded to obtain the encoding data of the current video frame, including: The length of the line connecting the left and right corners of the eye is normalized on the x-axis, and the x-coordinate of the pupil is quantized with a quantization step of 1 / 16 or 1 / 8. The length of the line connecting the left and right corners of the eye is normalized on the y-axis, and the y-coordinate of the pupil is quantized with a quantization step of 1 / 16 or 1 / 8.

13. The method according to any one of claims 6 to 9, characterized in that, The image features corresponding to the current video frame are encoded to obtain the encoding data of the current video frame, including: The length of the line connecting the left and right corners of the eye is normalized on the y-axis, and the y-coordinate of the pupil is quantized with a quantization step of 1 / 16 or 1 / 8. The angle between the line connecting the highest point and the pole point of the eye and the line connecting the lowest point and the pole point of the eye is normalized on the polar angle, and the polar angle of the pupil is quantized with a quantization step of 1 / 16 or 1 / 8.

14. The method of claim 1, wherein, The image features corresponding to the current video frame are encoded to obtain the encoding data of the current video frame, including: The supplementary enhancement information is generated according to the pupil position information. The supplementary enhancement information is entropy encoded to obtain the encoding data corresponding to the pupil position information.

15. The method of claim 14, wherein, The supplementary enhancement information is generated according to the pupil position information, including: A first identification syntax element is added in the supplementary enhancement information to indicate whether the pupil position information exists or not, and the value of the first identification syntax element is set to a first preset value.

16. The method of claim 14, wherein, The supplementary enhancement information is generated according to the pupil position information, including: A second identification syntax element is added in the supplementary enhancement information to indicate whether the left eye and the right eye share the same pupil position information, and in the case that the left eye and the right eye share the same pupil position information, the value of the second identification syntax element is set to a first preset value, and in the case that the left eye and the right eye do not share the same pupil position information, the value of the second identification syntax element is set to a second preset value.

17. The method of claim 14, wherein, The supplementary enhancement information is generated according to the pupil position information, including: In the case that the left eye and the right eye share the same pupil position information, a first coordinate system syntax element is added in the supplementary enhancement information to indicate the coordinate system to which the shared pupil position information belongs, and the value of the first coordinate system syntax element is set according to the coordinate system to which the shared pupil position information belongs.

18. The method of claim 14, wherein, The supplementary enhancement information is generated according to the pupil position information, including: In the case that the left eye and the right eye share the same pupil position information and the coordinate system to which the shared pupil position information belongs is a rectangular coordinate system, a first bit depth syntax element is added in the supplementary enhancement information to indicate the bit depth of the x-coordinate, and a second bit depth syntax element is added in the supplementary enhancement information to indicate the bit depth of the y-coordinate, and the value of the first bit depth syntax element is set according to the bit depth of the x-coordinate, and the value of the second bit depth syntax element is set according to the bit depth of the y-coordinate.

19. The method of claim 14, wherein, The generating the supplemental enhancement information according to the pupil position information comprises: In the case that the left eye and the right eye share the same pupil position information, and the coordinate system to which the shared pupil position information belongs is a rectangular coordinate system, and the shared pupil position information is represented by a normalized absolute difference value with respect to the pupil position information of the base image, a first position information syntax element and a second position information syntax element are added in the supplemental enhancement information, and a value of the first position information syntax element is set as a normalized absolute difference value of an x-axis coordinate of the shared pupil position information, and a value of the second position information syntax element is set as a normalized absolute difference value of a y-axis coordinate of the shared pupil position information.

20. The method of claim 14, wherein, The generating the supplemental enhancement information according to the pupil position information comprises: In the case that the left eye and the right eye share the same pupil position information, and the coordinate system to which the shared pupil position information belongs is a rectangular coordinate system, and the shared pupil position information is represented by a normalized absolute difference value with respect to the pupil position information of the base image, a first position information syntax element and a second position information syntax element are added in the supplemental enhancement information, and a value of the first position information syntax element is set as a normalized absolute difference value of an x-axis coordinate of the shared pupil position information, and a value of the second position information syntax element is set as a normalized absolute difference value of a y-axis coordinate of the shared pupil position information.

21. The method of claim 20, wherein, The generating the supplemental enhancement information according to the pupil position information further comprises: A first sign syntax element and / or a second sign syntax element are added in the supplemental enhancement information, and a value of the first sign syntax element is set according to a sign of a normalized difference value of an x-axis coordinate of the shared pupil position information, and a value of the second sign syntax element is set according to a sign of a normalized difference value of a y-axis coordinate of the shared pupil position information.

22. The method of claim 14, wherein, The generating the supplemental enhancement information according to the pupil position information comprises: In the case that the left eye and the right eye share the same pupil position information, and the coordinate system to which the shared pupil position information belongs is a rectangular coordinate system, and the shared pupil position information is represented by a normalized absolute difference value with respect to the pupil position information of the base image, a first position information syntax element and a second position information syntax element are added in the supplemental enhancement information, and a value of the first position information syntax element is set as a normalized absolute difference value of an x-axis coordinate of the shared pupil position information, and a value of the second position information syntax element is set as a normalized absolute difference value of a y-axis coordinate of the shared pupil position information.

23. The method of claim 14, wherein, The generating the supplemental enhancement information according to the pupil position information comprises: In the case that the left eye and the right eye share the same pupil position information, and the coordinate system to which the shared pupil position information belongs is a rectangular coordinate system, and the shared pupil position information is represented by a normalized absolute difference value with respect to the pupil position information of the base image, a first position information syntax element and a second position information syntax element are added in the supplemental enhancement information, and a value of the first position information syntax element is set as a normalized absolute difference value of an x-axis coordinate of the shared pupil position information, and a value of the second position information syntax element is set as a normalized absolute difference value of a y-axis coordinate of the shared pupil position information.

24. The method of claim 14, wherein, The generating the supplemental enhancement information according to the pupil position information comprises: In the case that the left eye and the right eye share the same pupil position information, and the coordinate system to which the shared pupil position information belongs is a rectangular coordinate system, and the shared pupil position information is represented by a normalized absolute difference value with respect to the pupil position information of the base image, a first position information syntax element and a second position information syntax element are added in the supplemental enhancement information, and a value of the first position information syntax element is set as a normalized absolute difference value of an x-axis coordinate of the shared pupil position information, and a value of the second position information syntax element is set as a normalized absolute difference value of a y-axis coordinate of the shared pupil position information. When the left eye and the right eye share the same pupil position information, the coordinate system to which the shared pupil position information belongs is a polar coordinate system, and the shared pupil position information is represented by the normalized absolute difference with the pupil position information of the base image, a fifth position information syntax element and a sixth position information syntax element are added to the supplemental enhancement information, and the value of the fifth position information syntax element is set to the normalized absolute difference of the polar diameter of the shared pupil position information, and the value of the sixth position information syntax element is set to the normalized absolute difference of the polar angle of the shared pupil position information.

25. The method of claim 24, wherein, The generating of supplementary enhancement information according to the pupil position information further includes: A third symbol syntax element and / or a fourth symbol syntax element is added to the supplemental enhancement information, and a value of the third symbol syntax element is set according to the sign of the normalized difference of the polar diameters of the shared pupil position information, and a value of the fourth symbol syntax element is set according to the sign of the normalized difference of the polar angles of the shared pupil position information.

26. The method of claim 14, wherein, Generating supplementary enhancement information according to the pupil position information includes: When the left eye and the right eye share the same pupil position information, the coordinate system to which the pupil position information belongs is a polar coordinate system, and the shared pupil position information is represented by a normalized value, a seventh position information syntax element and an eighth position information syntax element are added to the supplemental enhancement information, and the value of the seventh position information syntax element is set to the normalized value of the polar diameter of the shared pupil position information, and the value of the eighth position information syntax element is set to the normalized value of the polar angle of the shared pupil position information.

27. The method of claim 14, wherein, Generating supplementary enhancement information according to the pupil position information includes: In the case that the left eye and the right eye do not share the same pupil position information, a fourth identification syntax element for indicating whether the left eye pupil position information exists and a fifth identification syntax element for indicating whether the right eye pupil position information exists are added to the supplemental enhancement information, and the value of the fourth identification syntax element is set according to whether the left eye pupil position information exists, and the value of the fifth identification syntax element is set according to whether the right eye pupil position information exists.

28. The method of claim 14, wherein, Generating supplementary enhancement information according to the pupil position information includes: When the left-eye pupil position information exists, a second coordinate system syntax element for indicating the coordinate system to which the left-eye pupil position information belongs is added to the supplemental enhancement information, and the value of the second coordinate system syntax element is set according to the coordinate system constructed when obtaining the left-eye pupil position information.

29. The method of claim 14, wherein, Generating supplementary enhancement information according to the pupil position information includes: If left-eye pupil position information exists and the coordinate system to which the left-eye pupil position information belongs is a rectangular coordinate system, a fifth bit-depth syntax element for indicating the bit depth of the x-coordinate and a sixth bit-depth syntax element for indicating the bit depth of the y-coordinate are added to the supplemental enhancement information, and the value of the fifth bit-depth syntax element is set according to the bit depth of the x-coordinate, and the value of the sixth bit-depth syntax element is set according to the bit depth of the y-coordinate.

30. The method of claim 14, wherein, Generating supplementary enhancement information according to the pupil position information includes: In the case that the left eye pupil position information exists, a sixth identification syntax element for representing a left eye pupil position information representation mode is added in the supplemental enhancement information, and in the case that the left eye pupil position information is represented by a normalized absolute difference value of the pupil position information of the base image, a value of the sixth identification syntax element is set as a first preset value; in the case that the left eye pupil position information is represented by a normalized value, the value of the sixth identification syntax element is set as a second preset value.

31. The method of claim 14, wherein, The generating supplemental enhancement information according to the pupil position information comprises: In the case that the left eye pupil position information exists, a coordinate system to which the left eye pupil position information belongs is a rectangular coordinate system, and the left eye pupil position information is represented by a normalized absolute difference value of the pupil position information of the base image, a ninth position information syntax element and a tenth position information syntax element are added in the supplemental enhancement information, a value of the ninth position information syntax element is set as a normalized absolute difference value of an x-axis coordinate of the left eye pupil position information, and a value of the tenth position information syntax element is set as a normalized absolute difference value of a y-axis coordinate of the left eye pupil position information.

32. The method of claim 31, wherein, The generating supplemental enhancement information according to the pupil position information further comprises: A fifth sign syntax element and / or a sixth sign syntax element are added in the supplemental enhancement information, a value of the fifth sign syntax element is set according to a sign of a normalized difference value of an x-axis coordinate of the left eye pupil position information, and a value of the sixth sign syntax element is set according to a sign of a normalized difference value of a y-axis coordinate of the left eye pupil position information.

33. The method of claim 14, wherein, The generating supplemental enhancement information according to the pupil position information comprises: In the case that the left eye pupil position information exists, a coordinate system to which the left eye pupil position information belongs is a rectangular coordinate system, and the left eye pupil position information is represented by a normalized value, an eleventh position information syntax element and a twelfth position information syntax element are added in the supplemental enhancement information, a value of the eleventh position information syntax element is set as a normalized value of an x-axis coordinate of the left eye pupil position information, and a value of the twelfth position information syntax element is set as a normalized value of a y-axis coordinate of the left eye pupil position information.

34. The method of claim 14, wherein, The generating supplemental enhancement information according to the pupil position information comprises: In the case that the left eye pupil position information exists, and a coordinate system to which the left eye pupil position information belongs is a polar coordinate system, a seventh bit depth syntax element for representing a bit depth of a polar radius and an eighth bit depth syntax element for representing a bit depth of a polar angle are added in the supplemental enhancement information, a value of the seventh bit depth syntax element is set according to the bit depth of the polar radius, and a value of the eighth bit depth syntax element is set according to the bit depth of the polar angle.

35. The method of claim 14, wherein, The generating supplemental enhancement information according to the pupil position information comprises: In the case that the left eye pupil position information exists, the coordinate system to which the left eye pupil position information belongs is a polar coordinate system, and the left eye pupil position information is represented by a normalized absolute difference value of the pupil position information of the base image, the thirteenth position information syntax element and the fourteenth position information syntax element are added in the supplemental enhancement information, the value of the thirteenth position information syntax element is set as the normalized absolute difference value of the polar radius of the left eye pupil position information, and the value of the fourteenth position information syntax element is set as the normalized absolute difference value of the polar angle of the left eye pupil position information.

36. The method of claim 35, wherein, The generating supplemental enhancement information according to the pupil position information further includes: In the case that the left eye pupil position information exists, the coordinate system to which the left eye pupil position information belongs is a polar coordinate system, and the left eye pupil position information is represented by a normalized absolute difference value of the pupil position information of the base image, the thirteenth position information syntax element and the fourteenth position information syntax element are added in the supplemental enhancement information, the value of the thirteenth position information syntax element is set as the normalized absolute difference value of the polar radius of the left eye pupil position information, and the value of the fourteenth position information syntax element is set as the normalized absolute difference value of the polar angle of the left eye pupil position information.

37. The method of claim 14, wherein, The generating supplemental enhancement information according to the pupil position information includes: In the case that the left eye pupil position information exists, the coordinate system to which the left eye pupil position information belongs is a polar coordinate system, and the left eye pupil position information is represented by a normalized absolute difference value of the pupil position information of the base image, the thirteenth position information syntax element and the fourteenth position information syntax element are added in the supplemental enhancement information, the value of the thirteenth position information syntax element is set as the normalized absolute difference value of the polar radius of the left eye pupil position information, and the value of the fourteenth position information syntax element is set as the normalized absolute difference value of the polar angle of the left eye pupil position information.

38. The method of claim 14, wherein, The generating supplemental enhancement information according to the pupil position information includes: In the case that the right eye pupil position information exists, a third coordinate system syntax element for representing the coordinate system to which the right eye pupil position information belongs is added in the supplemental enhancement information, and the value of the third coordinate system syntax element is set according to the coordinate system constructed when the right eye pupil position information is acquired.

39. The method of claim 14, wherein, The generating supplemental enhancement information according to the pupil position information includes: In the case that the right eye pupil position information exists, and the coordinate system to which the right eye pupil position information belongs is a rectangular coordinate system, a ninth bit depth syntax element for representing the bit depth of the x coordinate and a tenth bit depth syntax element for representing the bit depth of the y coordinate are added in the supplemental enhancement information, the value of the ninth bit depth syntax element is set according to the bit depth of the x coordinate, and the value of the tenth bit depth syntax element is set according to the bit depth of the y coordinate.

40. The method of claim 14, wherein, The generating supplemental enhancement information according to the pupil position information includes: In the case that the right eye pupil position information exists, a seventh identification syntax element for representing the representation manner of the right eye pupil position information is added in the supplemental enhancement information, the value of the seventh identification syntax element is set as a first preset value in the case that the right eye pupil position information is represented by a normalized absolute difference value of the pupil position information of the base image, and the value of the seventh identification syntax element is set as a second preset value in the case that the right eye pupil position information is represented by a normalized value.

41. The method of claim 14, wherein, The generating supplemental enhancement information according to the pupil position information includes: In a case where the right eye pupil position information exists, the coordinate system to which the right eye pupil position information belongs is a rectangular coordinate system, and the right eye pupil position information is represented by a normalized absolute difference value with respect to the pupil position information of the base image, the seventeenth position information syntax element and the eighteenth position information syntax element are added in the supplemental enhancement information, a value of the seventeenth position information syntax element is set as a normalized absolute difference value of an x-axis coordinate of the right eye pupil position information, and a value of the eighteenth position information syntax element is set as a normalized absolute difference value of a y-axis coordinate of the right eye pupil position information.

42. The method of claim 41, wherein, The generating supplemental enhancement information according to the pupil position information further includes: In a case where the right eye pupil position information exists, the coordinate system to which the right eye pupil position information belongs is a rectangular coordinate system, and the right eye pupil position information is represented by a normalized absolute difference value with respect to the pupil position information of the base image, the seventeenth position information syntax element and the eighteenth position information syntax element are added in the supplemental enhancement information, a value of the seventeenth position information syntax element is set as a normalized absolute difference value of an x-axis coordinate of the right eye pupil position information, and a value of the eighteenth position information syntax element is set as a normalized absolute difference value of a y-axis coordinate of the right eye pupil position information.

43. The method of claim 14, wherein, The generating supplemental enhancement information according to the pupil position information includes: In a case where the right eye pupil position information exists, the coordinate system to which the right eye pupil position information belongs is a rectangular coordinate system, and the right eye pupil position information is represented by a normalized absolute difference value with respect to the pupil position information of the base image, the seventeenth position information syntax element and the eighteenth position information syntax element are added in the supplemental enhancement information, a value of the seventeenth position information syntax element is set as a normalized absolute difference value of an x-axis coordinate of the right eye pupil position information, and a value of the eighteenth position information syntax element is set as a normalized absolute difference value of a y-axis coordinate of the right eye pupil position information.

44. The method of claim 14, wherein, The generating supplemental enhancement information according to the pupil position information includes: In a case where the right eye pupil position information exists, the coordinate system to which the right eye pupil position information belongs is a rectangular coordinate system, and the right eye pupil position information is represented by a normalized absolute difference value with respect to the pupil position information of the base image, the seventeenth position information syntax element and the eighteenth position information syntax element are added in the supplemental enhancement information, a value of the seventeenth position information syntax element is set as a normalized absolute difference value of an x-axis coordinate of the right eye pupil position information, and a value of the eighteenth position information syntax element is set as a normalized absolute difference value of a y-axis coordinate of the right eye pupil position information.

45. The method of claim 14, wherein, The generating supplemental enhancement information according to the pupil position information includes: In a case where the right eye pupil position information exists, the coordinate system to which the right eye pupil position information belongs is a rectangular coordinate system, and the right eye pupil position information is represented by a normalized absolute difference value with respect to the pupil position information of the base image, the seventeenth position information syntax element and the eighteenth position information syntax element are added in the supplemental enhancement information, a value of the seventeenth position information syntax element is set as a normalized absolute difference value of an x-axis coordinate of the right eye pupil position information, and a value of the eighteenth position information syntax element is set as a normalized absolute difference value of a y-axis coordinate of the right eye pupil position information.

46. The method of claim 45, wherein, The generating supplemental enhancement information according to the pupil position information further includes: The eleventh and twelfth symbol syntax elements are added in the supplemental enhancement information, and a value of the eleventh symbol syntax element is set according to a sign of a normalized difference value of a polar radius of the right eye pupil position information, and a value of the twelfth symbol syntax element is set according to a sign of a normalized difference value of a polar angle of the right eye pupil position information.

47. The method of claim 14, wherein, The generating the supplemental enhancement information according to the pupil position information comprises: In a case where the right eye pupil position information exists, a coordinate system to which the right eye pupil position information belongs is a polar coordinate system, and the right eye pupil position information is represented by a normalized value, a twenty-third position information syntax element and a twenty-fourth position information syntax element are added in the supplemental enhancement information, a value of the twenty-third position information syntax element is set as a normalized value of a polar radius of the right eye pupil position information, and a value of the twenty-fourth position information syntax element is set as a normalized value of a polar angle of the right eye pupil position information.

48. An image decoding method, comprising: The method comprises: obtaining encoded data of a current video frame; decoding the encoded data of the current video frame to obtain image features corresponding to the current video frame; the image features corresponding to the current video frame comprise pupil position information of the current video frame; reconstructing the current video according to the image features corresponding to the current video frame to obtain a reconstructed current video frame.

49. An apparatus for video encoding, the apparatus comprising: The method comprises: storing computer program in the memory; when the computer program is invoked, the processor is configured to enable the video encoding device to implement the video encoding method in any one of claims 1-47.

50. An apparatus for video decoding, the apparatus comprising: The method comprises: storing computer program in the memory; when the computer program is invoked, the processor is configured to enable the video decoding device to implement the video decoding method in claim 48.

Images