Video decoding method and apparatus and video coding method and apparatus

Abstract

Some embodiments of the present application relate to the technical field of video coding and decoding. Provided are a video decoding method and apparatus and a video coding method and apparatus. The video decoding method comprises: acquiring coded data of the current block, wherein the current block is any coding unit in a video sequence; on the basis of the coded data of the current block, acquiring a residual block of the current block and a reference frame of the current frame, wherein the current frame is a video frame including the current block; acquiring camera parameters of the current frame and camera parameters of the reference frame; on the basis of the camera parameters of the current frame and the camera parameters of the reference frame, acquiring a mapping relationship between a pixel point in the current block and a pixel point in the reference frame; on the basis of the mapping relationship and the reference frame, acquiring a predicted block of the current block; and on the basis of the residual block of the current block and the predicted block of the current block, reconstructing the current block, so as to acquire a reconstructed block corresponding to the current block. Some embodiments of the present application are used for describing motion information of objects more accurately.

Description

A video decoding method, a video encoding method, and an apparatus

[0001] This disclosure claims priority to Chinese Patent Application No. 2024104875129, filed on April 22, 2024, entitled "A Video Decoding Method, Video Encoding Method and Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] Some embodiments of this application relate to the field of video encoding and decoding technology. More specifically, they relate to a video decoding method, a video encoding method, and an apparatus. Background Technology

[0003] To address the significant temporal redundancy in videos, traditional encoders perform intra-frame coding on keyframes and inter-frame coding on other video frames. During transmission, only the complete keyframe and the residuals of other video frames relative to the keyframe are transmitted, thus achieving redundancy removal. The core techniques of inter-frame coding are motion estimation and motion compensation, which involve finding the best-matching coded block for each coded block in the current frame within already encoded frames and obtaining the corresponding motion vector for each coded block.

[0004] Traditional video encoding and decoding algorithms are effective at estimating and compensating for translational motion in video sequences. However, they are not sensitive to deformations caused by camera rotation, forward movement, and backward movement. For example, affine transformations do not encompass the concept of depth; they are linear transformations between different two-dimensional coordinates. Therefore, in three-dimensional space, affine transformations can represent relatively limited motion information. However, in 3D animated videos such as cloud gaming, the motion of objects is much more diverse, encompassing scaling, rotation, perspective motion, and many other irregular movements. Consequently, affine transformations cannot fully describe the non-affine situations involving depth changes in these movements. Summary of the Invention

[0005] Exemplary embodiments of this application provide a video decoding method, a video encoding method, and an apparatus for more accurately describing the motion information of objects.

[0006] The technical solutions provided by some embodiments of this application are as follows:

[0007] In a first aspect, some embodiments of this application provide a video decoding method, including:

[0008] Obtain the encoded data of the current block; the current block is any encoded unit in the video sequence;

[0009] Based on the encoded data of the current block, obtain the residual block of the current block and the reference frame of the current frame; the current frame is a video frame containing the current block;

[0010] Obtain the camera parameters of the current frame and the camera parameters of the reference frame;

[0011] Based on the camera parameters of the current frame and the camera parameters of the reference frame, obtain the mapping relationship between the pixels in the current block and the pixels in the reference frame;

[0012] Based on the mapping relationship and the reference frame, obtain the predicted block of the current block;

[0013] Based on the residual block and the prediction block of the current block, the current block is reconstructed to obtain the reconstructed block corresponding to the current block.

[0014] Secondly, some embodiments of this application provide a video encoding method, including:

[0015] Determine the reference frame for the current frame, where the current frame is the video frame containing the current block;

[0016] Obtain the camera parameters of the current frame and the camera parameters of the reference frame;

[0017] Based on the camera parameters of the current frame and the camera parameters of the reference frame, obtain the mapping relationship between the pixels in the current block and the pixels in the reference frame;

[0018] Based on the mapping relationship between the pixels in the current block and the pixels in the reference frame, the prediction block of the current block is obtained;

[0019] Obtain the residual block of the current block based on the current block and the predicted block;

[0020] Based on the residual block of the current block, obtain the encoded data of the current block.

[0021] Thirdly, some embodiments of this application provide an image decoding apparatus, including:

[0022] Memory, configured to store computer programs;

[0023] The processor is configured to cause the video encoding apparatus to implement the image decoding method described in the first aspect when a computer program is invoked.

[0024] Fourthly, some embodiments of this application provide a video encoding apparatus, including:

[0025] Memory, configured to store computer programs;

[0026] The processor is configured to cause the video decoding device to implement the image encoding method described in the first aspect when a computer program is invoked.

[0027] Fifthly, some embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a computing device, causes the computing device to implement the method described in the first or second aspect.

[0028] Sixthly, some embodiments of this application provide a computer program product that, when run on a computer, causes the computer to implement the method described in the first or second aspect.

[0029] As can be seen from the above technical solutions, the video encoding method provided in this application, after obtaining the encoding data of the current block, first obtains the residual block of the current block and the reference frame of the current frame based on the encoding data of the current block, then obtains the camera parameters of the current frame and the camera parameters of the reference frame, and obtains the mapping relationship between the pixels in the current block and the pixels in the reference frame based on the camera parameters of the current frame and the camera parameters of the reference frame, then obtains the prediction block of the current block based on the mapping relationship and the reference frame, and reconstructs the current block based on the residual block and the prediction block of the current block to obtain the reconstructed block corresponding to the current block. Since the video encoding method provided in this application can obtain the mapping relationship between the pixels in the current block and the pixels in the reference frame based on the camera parameters of the current frame and the camera parameters of the reference frame, this application can accurately obtain the motion vector of each pixel in the current block based on the mapping relationship between the pixels in the current block and the pixels in the reference frame. Therefore, this application can more accurately describe the motion information of the object, thereby improving the quality of the reconstructed block of the current block. Attached Figure Description

[0030] To more clearly illustrate the implementation methods in some embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0031] Figure 1 shows a structural block diagram of a depth-based image rendering framework in some embodiments of this application;

[0032] Figure 2 shows a flowchart of the steps of the motion vector prediction model in some embodiments of this application;

[0033] Figure 3 shows a flowchart illustrating the process of obtaining the observation matrix and projection matrix in some embodiments of this application;

[0034] Figure 4 shows a schematic diagram of the coordinate space transformation in some embodiments of this application;

[0035] Figure 5 shows a schematic diagram of the view cone in some embodiments of this application;

[0036] Figure 6 shows a schematic diagram illustrating the basic principles of three-dimensional transformation in some embodiments of this application;

[0037] Figure 7 shows a schematic diagram of a region with uniformly varying depth in some embodiments of this application;

[0038] Figure 8 shows a schematic diagram of the control points of the current block in some embodiments of this application;

[0039] Figure 9 shows a schematic diagram of motion vector compensation in some embodiments of this application;

[0040] Figure 10 shows a flowchart of the steps of a video decoding method provided in some other embodiments of this application;

[0041] Figure 11 shows a flowchart of the steps of a video encoding method provided in some other embodiments of this application. Detailed Implementation

[0042] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0043] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0044] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0045] The use of phrases such as "some implementations" or "some embodiments" in the specification indicates that the described implementations or embodiments may include specific features, structures, or characteristics, but not every embodiment may necessarily include that specific feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same implementation. Additionally, when describing a specific feature, structure, or characteristic in connection with an embodiment, it is considered that implementing that feature, structure, or characteristic in connection with other implementations (whether explicitly described herein or not) is within the knowledge of those skilled in the art.

[0046] Some embodiments of this application relate to depth image based rendering (DIBR), and the following describes the depth image based rendering technology.

[0047] The core of depth image-based rendering technology lies in using the depth and color information of the old view to construct the three-dimensional information of the target viewpoint. By simulating the rendering process of the graphics engine through 3D mapping, new views from other viewpoints are generated. The essence of depth image-based rendering technology is the transformation of images in three-dimensional space; therefore, depth image rendering technology is also known as 3D image warping technology.

[0048] Referring to Figure 1, the depth-based image rendering framework mainly includes: a 3D transformation module 11, a view fusion module 12, and a hole filling module 13.

[0049] The 3D transformation module 11 is a core component of the depth-based image rendering framework, primarily responsible for achieving accurate conversion from the reference image to the target image. During this conversion, the 3D transformation module 11 performs a series of complex mapping transformations within 3D space. Specifically, the 3D transformation module 11 first identifies a pixel in the reference image and maps it from the reference image coordinate system to the 3D world coordinate system via the reference camera coordinate system. Subsequently, this pixel is projected again through the target camera coordinate system, ultimately mapping it precisely to the corresponding point in the target image coordinate system.

[0050] The view fusion module 12 is primarily responsible for fully utilizing the occlusion relationships of objects in multiple views to generate a more complete and accurate target view. When a camera captures objects in a 3D scene from a fixed position and perspective, objects may occlude each other. Therefore, after the 3D transformation module 11 generates virtual views corresponding to the left and right reference views, these virtual views need to be input into the view fusion module 12. This allows for the generation of a more complete and accurate target view by utilizing the different manifestations of object occlusion relationships in multiple views. The view fusion module 12 can reduce holes in the target view caused by object occlusion, thereby significantly improving the image quality of the target view and presenting users with a more realistic and detailed visual experience.

[0051] The hole-filling module 13 is primarily responsible for filling holes in the target view. After processing by the view fusion module 12, some pixels may still exist in the target view that cannot find corresponding reference information in the reference view. These pixels will appear in the target view as holes, severely affecting the overall image quality. The hole-filling module 13 can utilize the pixel information of non-empty pixels around the holes to accurately fill them, ensuring that the final output target view is complete and of high quality.

[0052] The following is a brief explanation of video coding technology.

[0053] Currently, commonly used video codec protocols are still primarily High Efficiency Video Coding (H.265 / HEVC) and Advanced Video Coding (H.264 / AVC). Among the optimal coding schemes, H.265 / HEVC can save approximately 36% of data compared to 1080p video compressed with H.264 / AVC while maintaining the same Peak Signal-to-Noise Ratio (PSNR). Furthermore, it can further increase the data saving to 50% while maintaining similar image quality.

[0054] Compared to its predecessor, H.265 / HEVC, the Versatile Video Coding (H.266 / VVC) standard largely retains the motion estimation, motion vector prediction, and motion compensation techniques from H.265 / HEVC in terms of inter-frame coding. Building upon this foundation, H.266 / VVC introduces new motion vector prediction techniques and improves upon existing ones. These improvements primarily target the candidate list of motion vectors (MVs) in Merge mode, focusing on optimizing their length, refining the construction process, and adjusting the inspection range of adjacent coding blocks. Improvements to motion vector prediction techniques include affine transformation and adaptive motion vector precision techniques.

[0055] In H.266 / VVC, the Merge mode has undergone significant changes, particularly in the construction of the MV candidate list. One of the most significant changes is the increase in the length of the MV candidate list, expanding from the original 5 candidate MVs to the current 6 candidate MVs. This is due to the addition of several new methods for checking adjacent coding blocks. During the construction of the MV candidate list, adjacent coding blocks are still given priority, followed by a rigorous check of the validity of temporal candidates. To ensure technical continuity and stability, this construction process is consistent with H.265 / HEVC. In H.266 / VVC, if the candidate list fails to reach the predetermined full length during construction, an innovative candidate padding method, namely the History Based Motion Vector Predictor (HMVP) technique, is introduced. After the candidate list is constructed, the encoder calculates the rate-distortion cost of each candidate in the list, thereby accurately selecting the best candidate MV. In addition to the Merge mode mentioned above, H.266 / VVC also introduces several new technologies, such as Multiple Motion Vector Difference (MMVD) technology. These techniques independently compute candidate motion vector prediction (MVs), using the optimal result as the representative of the current mode. Then, at the overall level, these modes are evaluated again for rate-distortion cost (RD cost). Finally, through comprehensive comparison, the encoder determines an optimal motion vector prediction merge mode and simultaneously selects the corresponding best MV candidate.

[0056] In H.265 / HEVC, motion compensation prediction technology primarily targets translational motion, which limits its ability to describe complex motions in nature. In reality, natural motion is diverse, including scaling, shrinking, rotation, and various irregular movements. When encoders encounter these complex motions, they often experience large prediction errors, leading to increased residuals in the coded blocks. To address this issue, H.266 / VVC introduces a sub-block-based affine transformation technique (Affine Mode). Affine Mode can be further divided into sub-block-based affine transformation and affine motion vector prediction (Affine AMVP), primarily targeting coding units of different sizes. The benefits of using Affine AMVP mode are not significant with small coding unit sizes; therefore, the minimum coding unit size for Affine AMVP is larger than that for Affine Merge mode. Within any Affine mode, there are two options: four-DOF and six-DOF modes. The affine motion of the block to be encoded can be represented by two models: a four-parameter model composed of motion vectors with two control points, and a six-parameter model composed of motion vectors with three control points. Since pixel-level motion compensation computation is too cumbersome, blocks are typically used as the unit to calculate their corresponding motion vectors. In H.266 / VVC, a process for establishing corresponding candidate lists is specified for the two Affine modes: an AMVP candidate list (2 candidates) is established for Affine Inter Mode, and a Merge candidate list (5 candidates) is established for Affine Merge Mode. The main function of Subblock-based Temporal Motion Vector Prediction (SbTMVP) is to obtain the motion information of the sub-blocks in the current coding unit from the corresponding sub-blocks in the same frame, serving as candidates for the Merge list. Prediction correction techniques based on optical flow fields are also used.

[0057] Bi-Directional Optical Flow (BDOF) technology evolved from BIO technology in the JEM reference model. Compared to BIO, BDOF has lower computational complexity, especially in terms of the number of multiplication operations and the size of the multipliers. The main function of BDOF is to correct the bi-directionally predicted pixel values ​​of 4x4 sub-blocks within the coding unit. For each 4x4 sub-block, motion correction (v) is calculated by minimizing the difference between the L0 and L1 predicted pixels. x ,v yThe correction results are then used to adjust the bidirectional prediction pixel values ​​in the 4x4 sub-block. BDOF is applicable to coding units in regular Merge or Inter modes under bidirectional prediction, and requires that the two reference frames and the current frame be equidistant.

[0058] In 3D animation videos such as cloud gaming videos, there are various motion modes, such as zooming in, zooming out, rotation, and perspective. However, traditional video encoding and decoding algorithms can only estimate and compensate for translational motion in video sequences, and are not sensitive to deformation caused by camera rotation, forward movement, and backward movement. Therefore, considering the characteristics of 3D animation videos and the shortcomings of traditional encoders, this application proposes a motion vector prediction model based on the idea of ​​3D transformation in some embodiments. This enables the encoder to better adapt to various motion modes during encoding, thereby improving encoding performance. Referring to Figure 2, in some embodiments, the construction process of the motion vector prediction model includes the following steps:

[0059] S21. Obtain the observation matrix and projection matrix of the virtual camera.

[0060] For 3D animated videos such as cloud gaming videos, the view matrix and projection matrix are crucial in the process of rendering 3D models into video frames and ultimately displaying them on the screen of a display device. The view matrix transforms the position information of points on the 3D model from the corresponding world space to the view space, while the projection matrix transforms the position information of points on the 3D model from the view space to the projection space. The view matrix and projection matrix are also commonly referred to as the extrinsic and intrinsic parameters of a virtual camera.

[0061] S22. Determine the transformation relationship between the coordinates of the pixels in the current frame and the coordinates in the reference frame based on the observation matrix and projection matrix of the virtual camera.

[0062] S23. Construct a motion vector prediction model based on the transformation relationship between the pixel points in the current frame and the coordinates in the reference frame.

[0063] The following section details the implementation methods for obtaining the observation matrix and projection matrix.

[0064] Referring to Figure 3, S30 (obtaining the observation matrix and projection matrix) can be divided into: S31, obtaining the extrinsic parameter information of the virtual camera, and S32, obtaining the intrinsic parameter information of the virtual camera. S32 (obtaining the intrinsic parameter information of the virtual camera) includes: S321, directly calling the interface to obtain the intrinsic parameter information of the virtual camera, and S322, estimating the intrinsic parameter information using the camera's extrinsic parameter information and video frames.

[0065] In some embodiments, obtaining the extrinsic parameters of the virtual camera includes: calling the corresponding interface to obtain the extrinsic parameters of the virtual camera.

[0066] In the Unity3D game engine, you can obtain the absolute extrinsic matrix (absolute observation matrix) that changes relative to the origin in world space by directly calling the corresponding interface. Alternatively, you can obtain the relative observation matrix (extrinsic matrix) that changes the camera pose information by calculating the changes in the camera's position information in the world space coordinate system as shown on the game engine's interactive panel.

[0067] It should be noted that if the corresponding interface is called directly to obtain the camera's extrinsic parameters, when the virtual camera only rotates, the elements in the extrinsic parameter matrix that represent the camera's position in world space will also change, making it impossible to directly determine the motion type of the virtual camera through the extrinsic parameter information.

[0068] In some embodiments, obtaining the intrinsic parameter information of a virtual camera includes: calling the corresponding interface to obtain the intrinsic parameter information of the virtual camera or estimating the intrinsic parameter information using the camera's extrinsic parameter information and video frames.

[0069] The intrinsic parameters of a virtual camera can be obtained directly from the engine by calling the corresponding interface. However, developers typically do not disclose the intrinsic parameters of virtual cameras, therefore, this information cannot be directly obtained by calling the corresponding interface. Building a motion vector prediction model requires obtaining the camera's intrinsic parameters; therefore, the intrinsic parameters of the virtual camera can be estimated using its extrinsic parameters and video frames.

[0070] The following section details how to estimate the intrinsic parameters of a virtual camera using its extrinsic information and video frames.

[0071] Currently, most 3D animation videos are generated using perspective projection technology. The perspective projection-related frustum parameters can be changed through the field of view (FOV) in the virtual camera component and the near and far clipping plane parameters in the clipping planes. The height of the near clipping plane, the height of the far clipping plane, the distance between the near and far clipping planes, and the field of view have the following relationships (1) and (2):

[0072] Where nearClipPlaneHeight is the height of the near clipping plane, Near is the distance to the near clipping plane (the distance from the virtual camera to the near clipping plane), and FOV is the field of view.

[0073] Where farClipPlaneHeight is the height of the far clipping plane, Far is the distance to the far clipping plane (the distance from the virtual camera to the far clipping plane), and FOV is the field of view.

[0074] The widths of the near and far clipping planes can be obtained from the aspect ratio of the virtual camera, and can also be configured using the virtual camera's aspect ratio. Let the aspect ratio of the virtual camera be Aspect, then:

[0075] Where nearClipPlaneWidth is the width of the near clipping plane, and farClipPlaneWidth is the width of the far clipping plane.

[0076] Combining equations (1) to (3) above, the projection matrix K of the virtual camera can be expressed as follows:

[0077] Furthermore, the above equation (4) can be simplified as follows:

[0078] It should be noted that during spatial transformations, the rotation of coordinates changes in different spaces. Specifically, world space is a left-handed coordinate system, the observation space is a right-handed coordinate system, and the projected space reverts back to a left-handed coordinate system. Therefore, when performing matrix transformations, the corresponding rotation changes between different coordinate systems must be considered.

[0079] In a video sequence where the virtual camera rotates purely (without displacement), the ORB algorithm is used to find matching feature points in two frames of pure virtual camera rotation. Constraints between frames are established based on these feature points, and finally, the values ​​of f1 and f2 are estimated using the least squares method, allowing them to be used in subsequent model derivation calculations. The specific calculation process is as follows:

[0080] Let the virtual camera projection matrix P be... Projection for:

[0081] Therefore, the virtual camera projection matrix includes four unknowns: f1, f2, c1, and c2.

[0082] In a video sequence where the virtual camera rotates purely, the observation matrix only contains a rotation matrix and not a translation matrix; therefore, the observation matrix V... rotation for:

[0083] Among them, R z R x R yThese are rotation matrices about the z-axis, x-axis, and y-axis, respectively, where z, x, and y are the rotation angles about the z-axis, x-axis, and y-axis, respectively.

[0084] Given two adjacent video frames and their corresponding observation matrices in a video sequence with pure virtual camera rotation. First, based on the ORB-based feature point matching algorithm described above, feature points in two adjacent video frames are matched. Furthermore, according to the principle of coordinate space mapping transformation based on 3D transformation, the matched feature points originate from the same point on the 3D model in world space. Therefore, we can assume the coordinates of a point on the 3D model in world space are P. world =(X,Y,Z,1) T The corresponding matching point in the adjacent video frame is The coordinates in the corresponding clipping space are Then we have:

[0085] Among them, P world The coordinates of the feature point pairs to be matched in world space. P is the observation matrix of the virtual camera. Projection The projection matrix of the virtual camera, The coordinates of the matching feature point pairs in the clipping space. These are the coordinates of the matching feature point pairs in screen space. `pixelwidth` and `pixelheight` are the width and height of the video frame, respectively.

[0086] Will It can be represented as:

[0087] Then, according to equations (9) to (12), equation (8) can be rearranged as follows:

[0088] Based on the principle that the matching feature points in two adjacent video frames in a 3D transformation are the same coordinates in world space, the following constraints can be established for f1 and f2 in equation (13):

[0089] Further utilizing the classical nonlinear least squares method, by min((AB)) 2 +(CB) 2 +(AC) 2 By doing so, the values ​​of f1 and f2 can be obtained.

[0090] In 3D animation videos, especially first-person game videos, there are a large number of video frames with pure rotational motion of the virtual camera. Therefore, the above method can be used to obtain the values ​​of f1 and f2 relatively easily.

[0091] It should be noted that when using different reference views with different camera rotation methods for calculation, it can be found that when the camera rotates only around one of the x, y, and z axes, each of the three constraint terms in equation (14) contains only one of f1 or f2, resulting in a decrease in the accuracy of the final constraint calculation. Therefore, using a virtual camera motion sequence with multi-directional composite rotation can yield more accurate f1 and f2 calculations.

[0092] In some embodiments, the intrinsic parameters of the virtual camera include the ratio of the half-angle cotangent of the camera's field of view to the camera's aspect ratio and / or the half-angle cotangent of the camera's field of view, i.e., f1 and f2. Therefore, the intrinsic parameter information of the virtual camera includes the values ​​of f1 and f2.

[0093] Therefore, based on the above camera intrinsic parameter estimation algorithm, the unknowns (f1, f2) related to the field of view (FOV) and aspect ratio in the camera's projection matrix can be obtained from different frames of pure virtual camera rotation in the video. However, the unknowns (c1, c2) related to the distances to the near and far clipping planes of the virtual camera's view frustum cannot establish similar constraints, so c1 and c2 cannot be predicted. Furthermore, the depth information in the 3D animation video is also unavailable. Therefore, the following derivation of the model will treat the depth information, c1, and c2 as unknowns.

[0094] Furthermore, the following explains how to convert the coordinates of a point in the current frame to coordinates in the reference frame.

[0095] In the process of generating 3D animation videos, points on the 3D model undergo multiple coordinate space transformations before being finally displayed on a 2D screen. Referring to Figure 4, the transformation process includes:

[0096] S41. Transform the vertex coordinates of the 3D model from model space to world space.

[0097] In a virtual 3D scene, each created model has its own independent coordinate space, called model space. Model space changes as the model moves or rotates. World space describes the absolute position within the virtual 3D scene; therefore, the first step in coordinate space transformation is to transform the vertex coordinates of the 3D model from model space to world space. This transformation is called model transformation.

[0098] S42. Transform the vertex coordinates of the 3D model from world space to view space.

[0099] In the process of generating 3D animation, the virtual camera determines the user's perspective, which is the perspective used to render the video. Therefore, the coordinates in world space need to be mapped to the view space (camera space). The process of transforming the vertex coordinates of the 3D model from world space to view space is called view transformation, and the matrix that implements view transformation is called the view matrix, which is also the extrinsic parameter matrix of the camera.

[0100] When transforming the vertex coordinates of a 3D model from world space to view space, the process follows the rule of translation followed by rotation. Therefore, when acquiring the view matrix, the entire view space is moved, and the position of the virtual camera in world space is inversely transformed, i.e., rotated first and then translated.

[0101] Let P be the coordinates of a point on a 3D model in world space. world =(X world ,Y world Z world ,1) T In the world space coordinate system, let the camera's rotation matrix be R, translation matrix be T, and observation matrix be V. Then the observation matrix can be described as: V = R·T (15)

[0102] P world After the observation transformation, the coordinates in the observation space are P. view Then we have: P view =V·P world (16)

[0103] S43. Transform the vertex coordinates of the 3D model from view space to clip space.

[0104] The primary goal of clip space is to simplify the clipping process for rendered primitives. Clip space is determined by the view frustum, which is surrounded by six planes called clipping planes. The near and far clipping planes determine the depth range visible to the camera. A projection matrix can be used to transform the coordinates of the view space to the corresponding clip space. The core function of the projection matrix is ​​to scale the x, y, and z components at different ratios and to translate the z component. This scaling design is mainly for the convenience of subsequent clipping operations.

[0105] As shown in Figure 5, all points on ray L, originating at the location of camera 50 and within the view frustum, will be rendered onto the same point P. The vertical plane containing point P is called the projection plane. Let the vertical opening angle of the view frustum be denoted by FOV, the aspect ratio of the video frame be denoted by Aspect, and the distances from the near clipping plane 51 and the far clipping plane 52 of the view frustum to the camera be denoted by Near and Far, respectively. Then, the projection matrix K can be expressed as:

[0106] After transforming the vertex coordinates of a 3D model from view space to clip space, the vertex coordinates of the 3D model are still coordinates in 3D space. Let the coordinates in clip space be... Then we have: P clip =K·P view (18)

[0107] The obtained coordinates in the clipping space are then subjected to homogeneous division to obtain normalized device coordinates (NDC). Next, the obtained u and v coordinates can be used to map the corresponding pixel coordinates of the output window.

[0108] S44. Transform the vertex coordinates of the 3D model from clip space to screen space.

[0109] In some embodiments, the vertex coordinates of the 3D model within the video cone are projected onto screen space. After transforming the vertex coordinates of the 3D model from clip space to screen space, the virtual 3D coordinates are converted into actual pixel positions on the screen, thereby presenting the realistic image of the 3D animated video.

[0110] Based on the above process of spatial transformation and coordinate mapping, when the next point P in world space... world During camera movement, the coordinates are mapped to different positions in the current frame and the reference frame, respectively. Let P be the clipping space coordinates of the previous frame. old The crop space coordinates of the next frame are P. new Then we have:

[0111] Among them, V new V old are the observation matrices corresponding to the current frame and the reference frame, respectively, and K is the projection matrix corresponding to the current frame and the reference frame, respectively.

[0112] Therefore, the coordinate transformation relationship between the current frame and the reference frame can be established according to equation (19):

[0113] Rearranging equation (20) yields:

[0114] As shown in equation (21) above, the NDC coordinates of a point in the new viewpoint projection of the camera in the same world space can be calculated from the NDC coordinate information of the previous frame, the camera motion information, and the camera intrinsic parameter matrix (projection matrix). Based on this relationship, the positions of most points on a frame can be calculated pixel by pixel to obtain the motion vector between the current frame and the reference frame.

[0115] The following explains how to establish a motion vector prediction model based on the transformation relationship between the pixel points in the current frame and the coordinates in the reference frame.

[0116] Referring to Figure 6, based on the fundamental principle of 3D transformation, the coordinates (x, y) of point P in world space sequentially pass through the observation matrix V of the first video frame. 1 After obtaining the projection matrix K and the mapping between clipping space and screen space, the pixel coordinates (u) of the point in the first video frame can be obtained. 1 v 1 The coordinates of this point pass through the observation matrix V of the second video frame in sequence. 2 After obtaining the projection matrix K and the mapping between clipping space and screen space, the pixel coordinates (u) of the point in the second video frame can be obtained. 2 v 2 Therefore, the coordinates of a point on the current video frame are first inversely mapped from screen space back to clip space, then from clip space back to view space, and then from view space back to world space. Similarly, the matching point of the same point on the current video frame is first inversely mapped from screen space back to clip space, then from clip space back to view space, and then from view space back to world space to obtain the same world space coordinates.

[0117] In some embodiments, it is assumed that the absolute observation matrices of the cameras corresponding to the current frame and the reference frame are V, respectively. i Let i ∈ {1, 2}, and the coordinates of the matching points in the current frame and the reference frame in the clipping space be respectively... The coordinates P of the point in world space corresponding to the matching point between the current frame and the reference frame. world =(X,Y,Z,1) T ,and

[0118] in, These are the elements at each position in the absolute rotation matrix of the camera relative to the initial direction. These are the values ​​of the camera's absolute coordinates in world space after being transformed by a rotation matrix.

[0119] Combining the above assumptions and equation (13), we can obtain:

[0120] From the above equation (24), we can see that A, P1, P2, P3, P4, Q1, Q2, Q3, Q4, S1, S2, S3, and S4 are all known quantities that can be obtained from the observation matrix.

[0121] Based on equation (20), establish a relationship between the coordinate values ​​X and Y in world space in equation (23):

[0122] Let the coordinates of a point in the current frame be (u 1 v 1 The coordinates of the matching point corresponding to this point in the reference frame are (u 2 v 2 Then, according to equation (17), we can obtain u. 1 v 1 u 2 v 2 The relationship is:

[0123] Given the projection matrix, observation matrix, two-dimensional coordinates of pixels in the current frame, and depth information, a series of points in two-dimensional coordinate systems mapped to a point in the same world coordinate system can be calculated using equations (26) and (27). Therefore, theoretically, by matching the transformation relationship of feature points between video frames, the transformation relationship of a certain region where a feature point is located can be estimated, thereby establishing a motion vector prediction model and determining the motion vectors of all pixels within the region.

[0124] For ease of analysis and subsequent calculations, equation (26) can be simplified as follows:

[0125] in,

[0126] From the above equation (28), we can obtain that θ, S, and J are parameters related to depth information and projection matrix; E, H, I, and M are parameters related to observation matrix and known matching point pair coordinate information.

[0127] At this point, u 2 v 2 with u 1 v 1 The relationship can be fully represented by universally relevant information. Based on the establishment of the motion relationship between the matching points of the current frame and the reference frame, Equation (28) is further transformed according to objectively available information, so that the final motion vector prediction model can be flexibly applied to various types of 3D animation videos.

[0128] For 3D animation videos such as cloud gaming videos, the observation matrix representing camera motion and the coordinates of points on the video frames can be obtained. However, the camera's projection matrix and the depth information of the 3D model in world space are usually unavailable. Therefore, in the derivation of the motion vector prediction model in this embodiment, the projection matrix and depth information are treated as unknowns, while the observation matrix and the coordinates of points in the current frame are treated as knowns. Furthermore, since f1 and f2 in the projection matrix can be estimated based on the matching point pairs in the observation matrix and the current and reference frames, f1 and f2 in this embodiment are also used as knowns in the subsequent derivation of the motion vector prediction model.

[0129] In some embodiments, this application provides a motion vector prediction model for coded blocks where all pixels have the same depth information.

[0130] When encoding video frames, the encoder first divides the video frame into blocks, creating multiple coding blocks. When the size of the coding block is small enough, it can be assumed that the motion vectors of every pixel in each coding block are identical, as are the depth information of each pixel.

[0131] The core idea of ​​motion vector prediction for coded blocks with identical pixel depth information is as follows: each video frame in the video frame sequence is divided into multiple non-overlapping coded blocks. Within a given search range, feature points will find matching points in the reference frame according to certain calculation criteria. The relative displacement between the matching point and the current point is the motion vector of the coded block.

[0132] Based on the above theories and assumptions, the quantities θ, S, and J related to depth information and projection matrix in equations (27) and (28) are set as unknowns; the quantities E, H, I, and M related to the observation matrix are considered as knowns. Therefore, u in equation (28) 2 v 2 Each of these can be considered as a linear equation in two variables. By using the coordinates of the two known matching points, the values ​​of θ, S, and J can be obtained.

[0133] However, by observing equation (26), we can conclude that u 2 The expression contains v 2 v 2 The expression contains u 2 To avoid estimating u 2 v 2 The coupling in the process requires u 2 v 2 Transformed into a form consisting only of the current point u in the current frame 1 v 1 An expression with known parameters.

[0134] Combining equations (27), (28), and (29), equation (26) can be reformulated as:

[0135] In equations (30) and (31) above, θ, S, and J are related to the depth information and the projection matrix, and are unknowns that need to be calculated. H, M, and N are known quantities that can be obtained from the observation matrix. Only E and I are related to the coordinates of the matching point, and only include the coordinates u of the current point in the current frame. 1 v 1 Therefore, the coordinates u of the matching point in the reference frame 2 v 2 The coordinates can be determined solely by the coordinates u of the current point in the current frame. 1 v 1 The prediction is derived from the observation matrices of the current frame and the reference frame, as well as f1 and f2 estimated from the current frame and the reference frame. 2 v 2 with u 1 v 1 The coordinate difference between the points is the motion vector between the matching point pairs.

[0136] From equation (23), we can obtain:

[0137] Combining equations (27), (29), and (32), the three unknowns related to depth information, θ, S, and J, can be expressed as:

[0138] Observing equations (33) to (35), it can be seen that the motion vector prediction model established at this time only has Two unknown parameters, both of which are related to depth information but not to other physical quantities.

[0139] Let P be the coordinates of a point in world space. world =[X world ,Y xorld ,z world ,1] T After observing the changes, the coordinates of that point become P. view =[x view ,y view ,z view ,1] T After projection transformation and device coordinate normalization transformation, the coordinates are respectively P clip and P NDC Then we have:

[0140] According to equation (36), we can obtain: clip w =z view(37)

[0141] Among them, z view This represents the depth information of the current point in the observation space.

[0142] As can be seen from the above, the motion vector prediction model only needs to solve... The correspondence between the current point and the matching point between two frames can be obtained using two unknown parameters. Therefore, according to equation (37), the unknown parameters obtained by this model are the observation space depth values ​​corresponding to the current matching point under different camera motion states. Under the assumption that a code has the same depth, the depth value of this coded block can be solved from a pair of matching points. Therefore, equation (30) can be written as:

[0143] The above embodiments can predict the coordinates of the corresponding pixel in the next video frame based on the coordinates of the pixel in the current video frame. Therefore, the motion vector prediction model of the coded block with the same depth information of the pixel can be expressed as Equation (38).

[0144] In some embodiments, this application provides a motion vector prediction model for a coded block with uniformly varying depth information of pixels.

[0145] Geometric planes are fundamental spatial primitives and frequently encountered elements in 3D animation. Therefore, in some cases, parts of a 3D scene with varying depths can be considered as a plane with uniformly varying depths. By establishing plane equations using the depth values ​​of each pixel within a video frame as the measure of change, depth information can be incorporated into the motion vector prediction process, thereby improving the accuracy of motion vector prediction, reducing residuals, and ultimately lowering the bitrate.

[0146] In three-dimensional space, any plane Γ can be represented by the following general parametric equation:

[0147] Where A, B, C, and D are constants, and at least one of them is non-zero. Let be the coordinates of the point, that is, any point that satisfies equation (39) lies on the plane Γ.

[0148] Referring to Figure 7, within a region 70 with uniformly varying depth, the depth of the upper left corner is known. Depth in the upper right corner Depth in the lower left corner Depth value The coefficient C is set to 1, and the origin of the coordinate system is translated to the projection of the upper left control point onto the depth plane where the camera is located. Then the depth of the current plane can be represented by the plane equation (40).

[0149] Taking the partial derivatives with respect to x and y respectively, and expressing them in terms of depth, the plane equation (40) can be written as:

[0150] Simplifying equation (41), we can obtain that, in the camera coordinate system, the depth value of each pixel can be expressed as:

[0151] Where x and y are the coordinates of the current point relative to the top-left control point of the current block. The depth is relative to the top-left control point of the current block, where w and h are the width and height of the current block, respectively.

[0152] Let λ1 = x / w, λ2 = y / h, then the above equation (42) can be expressed as follows:

[0153] Based on the premise assumptions, the derivation of this plane equation is based on a three-dimensional scene; therefore, the coordinate system is established based on camera space, which is also the viewing space. Thus, For observation space

[0154] set up: Then, according to equation (38), for any point k within the current block, we have:

[0155] According to equation (35), we can obtain:

[0156] Combining equations (43) and (44), we can obtain:

[0157] Where, α k The depth of the observation space representing the current block midpoint k β k This represents the observation space depth of the matching point corresponding to point k on the reference frame.

[0158] As shown in Figure 8, based on the observation space depths α1, α2, and α3 of the upper left, upper right, and lower left control points of the current block 81, and the observation space depths β1, β2, and β3 of the predicted points of the upper left, upper right, and lower left control points in the reference frame 82, the observation space depth α of any point in the current block 81 and the observation space depth β of the corresponding matching point in the reference frame can be obtained.

[0159] Based on the above derivation, in a regular motion sequence, with the current frame pixel coordinates P(u) known... 1 ,v 1In the case of ), the pixel coordinates P'(u) of the matching point corresponding to the current frame pixel in the reference frame can be calculated according to the following formula (48). 2 ,v 2 ):

[0160] in:

[0161] In some embodiments, the motion vector prediction model for a coded block with uniformly varying depth information can be expressed as equations (46) to (49) above. In some embodiments, the motion vector prediction model is determined by equations (46) to (49) above and the pixel coordinates P(u) in the current frame. 1 ,v 1 The pixel coordinates P'(u) of the corresponding matching point in the reference frame can be calculated. 2 ,v 2 ), thus through P(u 1 ,v 1 ) and P'(u 2 ,v 2 ) Calculate the motion vector of each pixel in the current block.

[0162] By incorporating depth information into the calculations, this motion vector prediction model is better suited to the projection transformations present in 3D animated videos.

[0163] In some embodiments, this application provides a motion vector prediction model for a coded block of a camera with pure rotational motion.

[0164] When the camera only rotates and its absolute position in world space remains unchanged, and the translation matrix T holds true in a pure rotation sequence where all data remains constant:

[0165] Among them, t x t y , This indicates the camera's absolute position in world space.

[0166] From equation (24), it can be seen that in equation (48), only the terms P2 and Q2 are related to the translation matrix, that is:

[0167] Among them, R 11 R 12 R 13 R 21 R 22 R 23 R 31 R 32 R 33t1, t2, and t3 are the elements of the absolute rotation matrix of the camera relative to the initial direction, and t1, t2, and t3 are the values ​​of the absolute coordinates of the camera in world space after the rotation matrix transformation.

[0168] From the above equation (50), we can see that:

[0169] Therefore, t1, t2, and t3 can be derived from t x t y , It is expressed as follows:

[0170] Substituting equation (53) into equation (51), and simplifying according to the characteristics of rotation matrices, we get:

[0171] From the above and equation (50), it can be seen that when the camera only rotates, the t corresponding to the current frame and the reference frame is... x t y , Since it remains unchanged, equation (48) can be simplified to the case of pure rotational motion as follows:

[0172] In equation (55), there is only one unknown, θ. Therefore, by solving for θ, the coordinates of the matching point can be calculated based on the coordinates of the current point in the current frame. In some embodiments, the vector prediction model in the pure rotation case can be expressed as equation (55).

[0173] According to the above formula (35), we know that:

[0174] Where A 1 A 2 These are the determinants of the rotation matrices for the current frame and the reference frame, respectively.

[0175] Since the determinant of the rotation matrix is ​​always 1 in pure rotational motion, therefore:

[0176] In some embodiments, θ is the ratio of the normalization factor used for the matching point pairs in the current frame and the reference frame during normalization.

[0177] According to equation (38), a relationship between the coordinate transformations of the current point and the matching point can be established for any pair of matching points between any two frames. When the camera only rotates, the observation matrix V and the projection matrix K can be expressed as:

[0178] Therefore, equation (23) can be expressed as follows in the case of pure rotation:

[0179] By establishing relationships between X, Y, and 1 in equation (59) and rearranging, we can obtain:

[0180] in:

[0181] Combining equations (60) and (61), we can solve for:

[0182] in:

[0183] From equations (62) and (63), θ can be obtained using the pixel coordinates of known points in the current frame and the extrinsic parameters of the camera in the current frame and the reference frame. Substituting θ into the expression (55) of the motion vector prediction model in the case of pure rotation, the vector prediction model does not include any depth-related information and has no unknowns. Therefore, for this special motion mode of pure rotational motion, the predicted point can be obtained based on the current point using only the camera's pose information and intrinsic parameter information, and finally the pixel-level motion vector can be obtained.

[0184] H.266 / VVC adds an affine merge mode to the conventional merge mode of H.265 / HEVC, further improving coding performance. The motion vector prediction model provided in the above embodiments can supplement the affine merge mode on this basis, adapting to more motion modes and predicting motion vectors more accurately, thereby improving overall performance while reducing bitrate to some extent.

[0185] Furthermore, when using the aforementioned motion vector prediction model for actual encoding, if encoding is performed using a motion vector prediction model built based on the uniform depth of all pixels in the current block or a motion vector prediction model built based on the uniformly varying depth of all pixels in the current block, then the mapping relationship between the pixels in the current block and the pixels in the reference frame needs to be determined using the motion vectors of the control points. However, control point motion information usually has a certain error at 1 / 16 precision, and this error will gradually accumulate in subsequent calculations to form a more serious error. Therefore, in some embodiments, when obtaining control point motion information, control point compensation techniques based on optical flow compensation can also be used to keep the calculation of the motion vector prediction model as accurate as possible.

[0186] In some embodiments, as shown in Figure 9, during the construction of the motion vector list, when obtaining the motion vectors of control points from neighboring coded blocks in the spatial domain, it is no longer simply obtaining the motion vector of the center point 91 of the 4x4 coded blocks adjacent to the control point. Instead, the distance from the center point 91 of the neighboring coded blocks to the control point is calculated, and the motion vector of the center point 91 of the neighboring coded blocks is compensated based on the distance from the center point 91 of the neighboring coded blocks to the control point. This achieves accurate calculation of the optimal construction motion vector of the control point for the current vertex by using the motion vector prediction model at a precision of 16 pixels. Compared to the motion vector of the center point 91 of the neighboring 4x4 coded blocks, the motion vector of this control point is more accurate, making the calculation of the motion vectors of each pixel in subsequent coded blocks more precise, thereby reducing the amount of residual data in the current block.

[0187] Based on the above, this application provides a video decoding method. Referring to FIG10, the video decoding method includes the following steps:

[0188] S101. Obtain the encoded data of the current block.

[0189] The current block is any coding unit in the video sequence.

[0190] In some embodiments, obtaining the encoded data of the current block includes receiving the encoded data of the current block sent by a media resource server.

[0191] S102. Based on the encoded data of the current block, obtain the residual block of the current block and the reference frame of the current frame.

[0192] The current frame is a video frame that contains the current block.

[0193] S103. Obtain the camera parameters of the current frame and the camera parameters of the reference frame.

[0194] In some embodiments, the camera parameters of the current frame and the camera parameters of the reference frame include one or more of the camera extrinsic parameters of the current frame, the camera extrinsic parameters of the reference frame, and the camera intrinsic parameters. In some embodiments, the camera parameters of the current frame and the camera parameters of the reference frame include the camera extrinsic parameters of the current frame, the camera extrinsic parameters of the reference frame, a first parameter, and a second parameter. In some embodiments, the camera intrinsic parameters include a first parameter and / or a second parameter, wherein the first parameter is the ratio of the half-angle cotangent of the camera's field of view to the aspect ratio of the camera, and the second parameter is the half-angle cotangent of the camera's field of view.

[0195] In some embodiments, the camera parameters of the current frame and the camera parameters of the reference frame can be obtained through corresponding interfaces. Specifically, the camera extrinsic parameters of the current frame and the reference frame can be obtained through corresponding interfaces, or the camera intrinsic parameters can be obtained from the engine through corresponding interfaces.

[0196] In some embodiments, the camera extrinsic parameters of the current frame and the camera extrinsic parameters of the reference frame can be obtained through corresponding interfaces, and the camera intrinsic parameters can be obtained based on the camera extrinsic parameters of the current frame and the camera extrinsic parameters of the reference frame.

[0197] In some embodiments, camera extrinsic parameters of video frames can be obtained through a corresponding interface. The video frames may include a first video frame and a second video frame. The camera intrinsic parameters are estimated based on the camera extrinsic parameters of the first video frame and the camera extrinsic parameters of the second video frame.

[0198] The estimation of the camera intrinsic parameters can be made with reference to the following steps a to d:

[0199] Step a: Obtain the first video frame and the second video frame. The first video frame is a video frame in the video sequence, and the camera pose corresponding to the first video frame only has rotational motion relative to the camera pose corresponding to the second video frame. That is, extract two video frames with pure camera rotational motion from the video sequence.

[0200] It should be noted that the first video frame and the second video frame can be adjacent video frames or non-adjacent video frames. This application embodiment does not limit this, and the only requirement is that the camera pose corresponding to the first video frame has rotational motion relative to the camera pose corresponding to the second video frame.

[0201] Step b: Determine the coordinates of the first pixel and the second pixel.

[0202] Wherein, the first pixel coordinates are the pixel coordinates of the first pixel point in the first video frame, and the second pixel coordinates are the pixel coordinates of the second pixel point in the second video frame that matches the first pixel point.

[0203] In some embodiments, matching the second pixel with the first pixel means that the second pixel is a pixel where the first pixel and the second pixel are the same point in world space.

[0204] In some embodiments, a pair of matching pixels in the first video frame and the second video frame can be obtained by using the ORB algorithm, and the pixel coordinates of the pair of pixels can be determined as the first pixel coordinates and the second pixel coordinates, respectively.

[0205] Step c: Obtain the first parameter and the second parameter based on the camera extrinsic parameters of the first video frame, the camera extrinsic parameters of the second video frame, the first pixel coordinates, and the second pixel coordinates.

[0206] In some embodiments, the camera intrinsic parameters are obtained based on the camera extrinsic parameters of the first video frame, the camera extrinsic parameters of the second video frame, the first pixel coordinates, and the second pixel coordinates.

[0207] In some embodiments, obtaining the first parameter and the second parameter based on the camera extrinsic parameters of the first video frame, the camera extrinsic parameters of the second video frame, the first pixel coordinates, and the second pixel coordinates includes the following steps c1 to c3:

[0208] Step c1: Construct the observation matrix of the first video frame and the observation matrix of the second video frame based on the camera extrinsic parameters of the first video frame and the camera extrinsic parameters of the second video frame, respectively.

[0209] Step c2: Determine the camera intrinsic parameters based on the observation matrix of the first video frame, the observation matrix of the second video frame, the first pixel coordinates, and the second pixel coordinates.

[0210] In some embodiments, the following constraint is constructed based on the observation matrix of the first video frame, the observation matrix of the second video frame, the first pixel coordinates, and the second pixel coordinates:

[0211] in, The element in the m-th row and n-th column of the observation matrix of the first video frame. Let f1 be the element in the m-th row and n-th column of the observation matrix of the second video frame, and f2 be the first parameter and the second parameter, respectively. 1 and v 1 These are the x and y coordinates of the first pixel, respectively. 2 and v 2 These are the x and y coordinates of the second pixel, respectively.

[0212] The implementation method and principle of constructing the above equation (64) based on the observation matrix of the first video frame, the observation matrix of the second video frame, the first pixel coordinates, and the second pixel coordinates can be referred to equations (1) to (14) above. To avoid redundancy, it will not be repeated here.

[0213] Step c3: Based on min((AB) 2 +(CB) 2 +(AC) 2 Determine the first parameter and the second parameter.

[0214] In some embodiments, according to min((AB)) 2 +(CB) 2 +(AC) 2 Determine the camera intrinsic parameters.

[0215] In some embodiments, (AB) is obtained. 2 +(CB) 2 +(AC) 2 Find the minimum value, and then (AB) 2 +(CB) 2 +(AC) 2 When the values ​​of f1 and f2 are minimized, the first parameter and the second parameter are determined, respectively.

[0216] In some embodiments, (AB) is obtained. 2 +(CB) 2 +(AC) 2 Find the minimum value, and then (AB) 2 +(CB) 2 +(AC) 2 The values ​​of f1 and f2 when they are at their minimum are determined as the camera intrinsic parameters.

[0217] S104. Based on the camera parameters of the current frame and the camera parameters of the reference frame, obtain the mapping relationship between the pixels in the current block and the pixels in the reference frame.

[0218] In some embodiments, a motion vector prediction model is obtained based on the camera parameters of the current frame and the camera parameters of the reference frame.

[0219] Since the prediction point corresponding to the pixel in the current block in the reference frame can be determined based on the mapping relationship between the pixel in the current block and the pixel in the reference frame, thereby determining the motion vector of the pixel in the current block, the mapping relationship between the pixel in the current block and the pixel in the reference frame can also indicate the motion vector prediction model of the current block.

[0220] In some embodiments, obtaining the mapping relationship between pixels in the current block and pixels in the reference frame based on camera parameters of the current frame and camera parameters of the reference frame includes: obtaining the control point motion vector (CPMV) of the current block; and obtaining the mapping relationship between pixels in the current frame and pixels in the reference frame based on the control point motion vector of the current block, camera parameters of the current frame, and camera parameters of the reference frame.

[0221] In some embodiments, obtaining the motion vector of the control point of the current block may include:

[0222] The index value of the motion vector prediction model is obtained based on the encoded data of the current block. The prediction model of the motion vector of the control point of the current block is obtained based on the index value of the motion vector prediction model and the candidate list of motion vector prediction models. The motion vector of the current control point is predicted based on the prediction model of the motion vector of the control point of the current block to obtain the motion vector of the control point of the current block.

[0223] In some embodiments, a spatial reference motion vector of the control point of the current block is obtained; the spatial reference motion vector is compensated according to the distance from the center of the image block corresponding to the spatial reference motion vector to the control point of the current block, so as to obtain the motion vector of the control point of the current block.

[0224] Compared to directly using the motion vector of the neighboring coded block as the motion vector of the control point of the current block, the above embodiment can compensate the motion vector of the neighboring coded block according to the distance from the center of the neighboring coded block to the control point of the current block, and use the compensated motion vector as the motion vector of the current block. Therefore, the above embodiment can obtain the motion vector of the control point of the current block more accurately, thereby reducing the amount of residual data.

[0225] In some embodiments, the mapping relationship between pixels in the current frame and pixels in the reference frame is obtained based on the motion vector of the control point of the current block, the camera parameters of the current frame, and the camera parameters of the reference frame, including the following steps 1 and 2:

[0226] Step 1: Obtain the pixel coordinates of the predicted point of the target control point of the current block based on the motion vector of the control point of the current block.

[0227] The target control point is any control point of the current block.

[0228] Step 2: Based on the pixel coordinates of the target control point, the pixel coordinates of the predicted point of the target control point, the camera parameters of the current frame, and the camera parameters of the reference frame, obtain the mapping relationship between the pixels in the current frame and the pixels in the reference frame.

[0229] In some embodiments, step 2 above (obtaining the mapping relationship between pixels in the current frame and pixels in the reference frame based on the pixel coordinates of the target control point, the pixel coordinates of the predicted point of the target control point, the camera parameters of the current frame, and the camera parameters of the reference frame) includes the following steps 21 to 22:

[0230] Step 21: Generate a first observation matrix and a second observation matrix based on the camera extrinsic parameters of the current frame and the camera extrinsic parameters of the reference frame, respectively.

[0231] Step 22: Determine the motion vector prediction model based on the first observation matrix, the second observation matrix, the first parameter, and the second parameter, that is, the mapping relationship between the pixels in the current frame and the pixels in the reference frame.

[0232] In some embodiments, the following first relation can be constructed based on the first observation matrix, the second observation matrix, the first parameter, and the second parameter:

[0233] in,

[0234] Let be the element in the m-th row and n-th column of the first observation matrix. The element in the m-th row and fourth column of the first observation matrix. The element in the m-th row and n-th column of the second observation matrix. Let f1 and f2 be the elements in the m-th row and fourth column of the second observation matrix, respectively, and u be the first parameter and the second parameter. 1 and v 1 These are the x and y coordinates of the pixel coordinates of the pixel point in the current block, respectively. 2 and v 2 These are the x and y coordinates of the pixel coordinates in the reference frame, respectively. Let be the second coordinate component of the pixel in the current block in the clipping space. The second coordinate component of the pixel in the reference frame in the clipping space.

[0235] The implementation method and principle of constructing the first relation based on the first observation matrix, the second observation matrix, the first parameter and the second parameter can be referred to the above equations (15) to (38). To avoid redundancy, it will not be repeated here.

[0236] Step 22: Substitute the pixel coordinates of the target control point and the pixel coordinates of the predicted point of the target control point into the first relational expression to obtain the second coordinate components of the pixels in the current block in the clipping space and the second coordinate components of the corresponding pixels in the reference frame in the clipping space, so as to obtain the mapping relationship between the pixels in the current frame and the pixels in the reference frame.

[0237] In some embodiments, the second coordinate component of a pixel in the current block in the clipping space and the second coordinate component of a pixel in the corresponding reference frame in the clipping space can be determined by the pixel coordinates of the target control point, the pixel coordinates of the predicted point of the target control point, and the first relational expression, thereby determining the motion vector prediction model.

[0238] Only in the above equation (67) is there and Two unknowns, therefore it can be understood as and The first linear equation in two variables can be solved by substituting the pixel coordinates of the target control point and the pixel coordinates of the predicted point of the target control point into the first relational expression. and The value of . Calculate and After setting the value, for any pixel in the current block, the pixel coordinates (u) of the pixel can be determined. 1 ,v 1 Substituting into equation (67) above, the corresponding pixel coordinates (u) in the reference frame are calculated. 2 ,v 2 ), and then according to (u 1 ,v 1 ) and (u 2 ,v 2 Calculate the motion vector of each pixel.

[0239] In some embodiments, the pixel coordinates of the target control point and the pixel coordinates of the predicted point of the target control point are substituted into the first relational expression to calculate the result. and The value. In obtaining... and After determining the value, the motion vector prediction model can be determined according to equation (67). Based on the motion vector prediction model, for any pixel coordinate (u) of any pixel in the current block... 1 ,v 1 This allows for the prediction of the corresponding pixel coordinates (u) in the reference frame. 2 ,v 2 This allows us to obtain the motion vector of the pixel.

[0240] In some embodiments, the mapping relationship between pixels in the current frame and pixels in the reference frame is obtained based on the motion vector of the control point of the current block, the camera parameters of the current frame, and the camera parameters of the reference frame, including the following steps ① and ②:

[0241] Step ①: Determine the predicted points of the upper left corner control point, the upper right corner control point, and the lower left corner control point of the current block based on the motion vector of the control points of the current block.

[0242] Step 2: Based on the depths of the upper left corner control point, the upper right corner control point, and the lower left corner control point of the current block, and the depths of the predicted points of the upper left corner control point, the upper right corner control point, and the lower left corner control point in the reference frame, construct the following second relational expression;

[0243] Wherein, α1, α2, and α3 are the depths of the top-left, top-right, and bottom-left control points of the current block, respectively; β1, β2, and β3 are the depths of the predicted points of the top-left, top-right, and bottom-left control points in the reference frame, respectively; λ1 = x / w, λ2 = y / h, where x and y are the coordinates of the pixels in the current block relative to the top-left control point of the current block; and w and h are the width and height of the current block.

[0244] It should be noted that, in step ②, the depths of the top left corner control point, the top right corner control point, the bottom left corner control point, the predicted depths of the top left corner control point, the predicted depths of the top right corner control point, and the predicted depths of the bottom left corner control point in the reference frame are all depth values ​​in the observation space.

[0245] The implementation method and principle of constructing the third relation based on the depth of the upper left corner control point, the upper right corner control point, the lower left corner control point, the predicted depth of the upper left corner control point, the predicted depth of the upper right corner control point, and the predicted depth of the lower left corner control point in the reference frame can be referred to the above equations (39) to (49). To avoid redundancy, it will not be repeated here.

[0246] Step 3: Based on the second relational expression, the camera parameters of the current frame, and the camera parameters of the reference frame, obtain the mapping relationship between the pixels in the current frame and the pixels in the reference frame.

[0247] In some embodiments, step ③ (obtaining the mapping relationship between pixels in the current frame and pixels in the reference frame based on the second relation, the camera parameters of the current frame, and the camera parameters of the reference frame) includes the following steps ③1 to ③3:

[0248] Step 31: Generate a first observation matrix and a second observation matrix based on the camera extrinsic parameters of the current frame and the camera extrinsic parameters of the reference frame, respectively.

[0249] Step ③2: Based on the first observation matrix, the second observation matrix, the first parameter, and the second parameter, construct the following third relation:

[0250] in,

[0251] Let be the element in the m-th row and n-th column of the first observation matrix. The element in the m-th row and fourth column of the first observation matrix. The element in the m-th row and n-th column of the second observation matrix. Let f1 and f2 be the elements in the m-th row and fourth column of the second observation matrix, respectively, and u be the first parameter and the second parameter. 1 and v 1 These are the x and y coordinates of the pixel coordinates of the pixel point in the current block, respectively. 2 and v 2 α and B are the x and y coordinates of the pixel in the reference frame, respectively; α is the reciprocal of the second coordinate component of the pixel in the current block in the clipping space; and B is the reciprocal of the second coordinate component of the pixel in the reference frame in the clipping space.

[0252] The implementation method and principle of constructing the third relation based on the first observation matrix, the second observation matrix, the first parameter and the second parameter can be referred to the above equations (15) to (38). To avoid redundancy, it will not be repeated here.

[0253] Step 3: Based on the second relation and the third relation, obtain the mapping relationship between the pixels in the current frame and the pixels in the reference frame.

[0254] For any pixel in the current block, you can first set the pixel coordinates (u) of the pixel. 1 ,v 1 Substitute these into equation (68) to calculate α and β, and then use (u) 1 ,v 1 Substituting α and β into equation (70) above, we can calculate (u). 2 ,v 2 ), and then according to (u 1 ,v 1 ) and (u 2 ,v 2 Calculate the motion vector of the pixel.

[0255] In some embodiments, equations (68) and (71) can also be rearranged into a single relation, and (u) 1 ,v 1Substitute into the rearranged relational expression to calculate (u) 2 ,v 2 ), and then according to (u 1 ,v 1 ) and (u 2 ,v 2 Calculate the motion vector of each pixel.

[0256] In some embodiments, the motion vector prediction model is determined after the mapping relationship between pixels in the current frame and pixels in the reference frame is determined. For any pixel in the current block, the matching point in the reference frame corresponding to that pixel can be obtained according to the motion vector prediction model, thereby determining the motion vector of that pixel.

[0257] In some embodiments, the mapping relationship between pixels in the current block and pixels in the reference frame is obtained based on the camera parameters of the current frame and the camera parameters of the reference frame, including the following steps I and II:

[0258] Step I: Generate a first observation matrix and a second observation matrix based on the camera extrinsic parameters of the current frame and the camera extrinsic parameters of the reference frame, respectively.

[0259] Step II: Based on the first observation matrix, the second observation matrix, the first parameter, and the second parameter, construct the following fourth relation:

[0260] in,

[0261] Let be the element in the m-th row and n-th column of the first observation matrix. Let f1 and f2 be the elements in the m-th row and n-th column of the second observation matrix, respectively, and let u be the first parameter and the second parameter. 1 and v 1 These are the x and y coordinates of the pixel coordinates of the pixel point in the current block, respectively. 2 and v 2 These are the x and y coordinates of the pixel coordinates of the pixel point in the reference frame, respectively.

[0262] The implementation method and principle of constructing the fourth relation based on the first observation matrix, the second observation matrix, the first parameter and the second parameter can be referred to the above equations (15) to (38) and equations (50) to (63). To avoid redundancy, they will not be repeated here.

[0263] Step III: Determine the fourth relation as the motion vector prediction model for the current block.

[0264] S105. Obtain the prediction block of the current block according to the mapping relationship and the reference frame.

[0265] In some embodiments, obtaining the prediction block of the current block according to the mapping relationship and the reference frame includes: obtaining the prediction pixel points of each pixel point in the current block from the reference frame according to the mapping relationship, and combining the prediction pixel points of each pixel point in the current block into the prediction block of the current block.

[0266] In some embodiments, this disclosure can also obtain a prediction block of the current block based on the motion vector prediction model and the reference frame. This disclosure can determine the predicted pixel points of each pixel in the current block using the motion vector prediction model, and combine the predicted pixel points of each pixel in the current block into a prediction block of the current block.

[0267] S106. Based on the residual block and the prediction block of the current block, reconstruct the current block to obtain the reconstructed block corresponding to the current block.

[0268] In some embodiments, the current block is reconstructed based on the residual block and the prediction block of the current block to obtain the reconstructed block corresponding to the current block, including:

[0269] The values ​​at corresponding positions in the residual block and the prediction block of the current block are summed to obtain the reconstruction block corresponding to the current block.

[0270] The video encoding method provided in this application, after obtaining the encoding data of the current block, first obtains the residual block of the current block and the reference frame of the current frame based on the encoding data of the current block, then obtains the camera parameters of the current frame and the camera parameters of the reference frame, and determines the mapping relationship or motion vector prediction model between the pixels in the current block and the pixels in the reference frame based on the camera parameters of the current frame and the camera parameters of the reference frame, then obtains the prediction block of the current block based on the mapping relationship or motion vector prediction model and the reference frame, and reconstructs the current block based on the residual block of the current block and the prediction block of the current block to obtain the reconstructed block corresponding to the current block. Since the video encoding method provided in this application embodiment can obtain the mapping relationship or motion vector prediction model between the pixels in the current block and the pixels in the reference frame based on the camera parameters of the current frame and the camera parameters of the reference frame, this application embodiment can accurately obtain the motion vector of each pixel in the current block based on the mapping relationship or motion vector prediction model between the pixels in the current block and the pixels in the reference frame. Therefore, this application embodiment can more accurately describe the motion information of the object, thereby improving the quality of the reconstructed block of the current block.

[0271] One embodiment of this application provides a video encoding method. Referring to FIG11, the video encoding method includes the following steps:

[0272] S111. Determine the reference frame for the current frame.

[0273] The current frame is the video frame that contains the current block.

[0274] S112. Obtain the camera parameters of the current frame and the camera parameters of the reference frame.

[0275] In some embodiments, the camera parameters of the current frame and the camera parameters of the reference frame may include the camera extrinsic parameters of the current frame, the camera extrinsic parameters of the reference frame, a first parameter, and a second parameter.

[0276] In some embodiments, the camera parameters of the current frame and the camera parameters of the reference frame may include the camera extrinsic parameters of the current frame, the camera extrinsic parameters of the reference frame, and the camera intrinsic parameters.

[0277] In some embodiments, the camera intrinsic parameters may include the first parameter and / or the second parameter, wherein the first parameter is the ratio of the half-angle cotangent of the camera's field of view to the aspect ratio of the camera, and the second parameter is the half-angle cotangent of the camera's field of view.

[0278] The implementation of obtaining the camera extrinsic parameters of the current frame, the camera extrinsic parameters of the reference frame, the first parameter, and the second parameter can refer to the above embodiments, and will not be repeated here.

[0279] S113. Based on the camera parameters of the current frame and the camera parameters of the reference frame, obtain the mapping relationship between the pixels in the current block and the pixels in the reference frame.

[0280] In some embodiments, the motion vector prediction model can indicate the mapping relationship between pixels in the current block and pixels in the reference frame, that is, the motion vector prediction model can determine the corresponding pixels in the reference frame for pixels in the current block.

[0281] The implementation method for obtaining the mapping relationship between the pixels in the current block and the pixels in the reference frame based on the camera parameters of the current frame and the camera parameters of the reference frame can refer to the above embodiment, and will not be described in detail here.

[0282] S114. Obtain the prediction block of the current block according to the mapping relationship between the pixels in the current block and the pixels in the reference frame.

[0283] In some embodiments, the predicted pixel points of each pixel point in the current block are obtained from the reference frame according to the mapping relationship, and the predicted pixel points of each pixel point in the current block are combined into a predicted block of the current block.

[0284] S115. Obtain the residual block of the current block based on the current block and the prediction block.

[0285] In some embodiments, obtaining the residual block of the current block based on the current block and the predicted block includes:

[0286] Calculate the residuals at corresponding positions in the current block and the prediction block of the current block to obtain the residual block corresponding to the current block.

[0287] S116. Obtain the encoded data of the current block based on the residual block of the current block.

[0288] In some embodiments, this application provides a video encoding apparatus, which includes:

[0289] Memory, configured to store computer programs;

[0290] The processor is configured to cause the video encoding apparatus to implement the video encoding method described in any of the above embodiments when a computer program is invoked.

[0291] In some embodiments, this application provides a video decoding apparatus, which includes:

[0292] Memory, configured to store computer programs;

[0293] The processor is configured to cause the video decoding device to implement the video decoding method described in any of the above embodiments when a computer program is invoked.

[0294] In some embodiments, this application provides a computer-readable storage medium storing a computer program that, when executed by a computing device, causes the computing device to implement the video encoding or video decoding method described in any of the above embodiments.

[0295] In some embodiments, this application provides a computer program product that, when run on a computer, enables the computer to implement the video encoding method or video decoding method described in any of the above embodiments.

[0296] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0297] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. A video decoding method, characterized in that, include: Obtain the encoded data of the current block; the current block is any encoded unit in the video sequence; Based on the encoded data of the current block, obtain the residual block of the current block and the reference frame of the current frame; the current frame is a video frame containing the current block; Obtain the camera parameters of the current frame and the camera parameters of the reference frame; A motion vector prediction model is determined based on the camera parameters of the current frame and the camera parameters of the reference frame; Based on the motion vector prediction model, obtain the predicted block for the current block; Based on the residual block and the prediction block of the current block, obtain the reconstruction block corresponding to the current block.

2. The method according to claim 1, characterized in that, Camera parameters include camera intrinsic parameters, wherein obtaining the camera parameters of the current frame and the camera parameters of the reference frame includes: Obtain the camera extrinsic parameters of the current frame and the camera extrinsic parameters of the reference frame; The camera intrinsic parameters are obtained based on the camera extrinsic parameters of the video frames in the video sequence.

3. The method according to claim 2, characterized in that, in, The camera intrinsic parameters include the ratio of the half-angle cotangent of the camera's field of view to the camera's aspect ratio and / or the half-angle cotangent of the camera's field of view.

4. The method according to claim 2, characterized in that, Obtaining the camera intrinsic parameters includes: Acquire a first video frame and a second video frame, wherein the first video frame is a video frame in the video sequence, and the camera pose corresponding to the first video frame only has rotational motion relative to the camera pose corresponding to the second video frame; Obtain and determine the first pixel coordinates and the second pixel coordinates, where the first pixel coordinates are the pixel coordinates of the first pixel in the first video frame, and the second pixel coordinates are the pixel coordinates of the second pixel in the second video frame that matches the first pixel. The camera intrinsic parameters are obtained based on the camera extrinsic parameters of the first video frame, the camera extrinsic parameters of the second video frame, the first pixel coordinates, and the second pixel coordinates.

5. The method according to claim 4, characterized in that, The step of obtaining the camera intrinsic parameters based on the camera extrinsic parameters of the first video frame, the camera extrinsic parameters of the second video frame, the first pixel coordinates, and the second pixel coordinates includes: The observation matrix of the first video frame and the observation matrix of the second video frame are constructed based on the camera extrinsic parameters of the first video frame and the camera extrinsic parameters of the second video frame, respectively. The camera intrinsic parameters are determined based on the observation matrix of the first video frame, the observation matrix of the second video frame, the first pixel coordinates, and the second pixel coordinates.

6. The method according to claim 5, characterized in that, Determining the camera intrinsic parameters based on the observation matrix of the first video frame, the observation matrix of the second video frame, the first pixel coordinates, and the second pixel coordinates includes: Construct constraint functions: in, The element in the m-th row and n-th column of the observation matrix of the first video frame. Let f1 be the element in the m-th row and n-th column of the observation matrix of the second video frame, and f2 be the first parameter and the second parameter, respectively. 1 and v 1 These are the x and y coordinates of the first pixel, respectively. 2 and v 2 These are the x and y coordinates of the second pixel, respectively; According to min((AB)) 2 +(CB) 2 +(AC) 2 Determine the camera intrinsic parameters.

7. The method according to claim 2, characterized in that, The step of determining the motion vector prediction model based on the camera parameters of the current frame and the camera parameters of the reference frame includes: Obtain the motion vector CPMV of the control point of the current block; The motion vector prediction model is determined based on the motion vector of the control point of the current block, the camera parameters of the current frame, and the camera parameters of the reference frame, wherein the motion vector prediction model includes the mapping relationship between the pixels in the current frame and the pixels in the reference frame.

8. The method according to claim 7, characterized in that, The step of determining the motion vector prediction model based on the motion vector of the control point of the current block, the camera parameters of the current frame, and the camera parameters of the reference frame includes: The pixel coordinates of the predicted point of the target control point of the current block are obtained based on the motion vector of the control point of the current block; the target control point is any control point of the current block; The motion vector prediction model is determined based on the pixel coordinates of the target control point, the pixel coordinates of the predicted point of the target control point, the camera parameters of the current frame, and the camera parameters of the reference frame.

9. The method according to claim 8, characterized in that, The step of determining the motion vector prediction model based on the pixel coordinates of the target control point, the pixel coordinates of the predicted point of the target control point, the camera parameters of the current frame, and the camera parameters of the reference frame includes: A first observation matrix and a second observation matrix are generated based on the camera extrinsic parameters of the current frame and the camera extrinsic parameters of the reference frame, respectively. The motion vector prediction model is determined based on the first observation matrix, the second observation matrix, the camera intrinsic parameters, the pixel coordinates of the target control point, and the pixel coordinates of the predicted point of the target control point.

10. The method according to claim 9, characterized in that, The motion vector prediction model is determined based on the first observation matrix, the second observation matrix, the camera intrinsic parameters, the pixel coordinates of the target control point, and the pixel coordinates of the predicted point of the target control point, including: A first relation is constructed based on the first observation matrix, the second observation matrix, and the camera intrinsic parameters; in, Let be the element in the m-th row and n-th column of the first observation matrix. The element in the m-th row and fourth column of the first observation matrix. The element in the m-th row and n-th column of the second observation matrix. Let f1 and f2 be the elements in the m-th row and fourth column of the second observation matrix, respectively, and u be the first parameter and the second parameter. 1 and v 1 These are the x and y coordinates of the pixel coordinates of the pixel point in the current block, respectively. 2 and v 2 These are the x and y coordinates of the pixel coordinates in the reference frame, respectively. Let be the second coordinate component of the pixel in the current block in the clipping space. The second coordinate component of the pixel in the reference frame in the clipping space; Substitute the pixel coordinates of the target control point and the pixel coordinates of the predicted point of the target control point into the first relational expression to obtain the second coordinate components of the pixels in the current block in the clipping space and the second coordinate components of the corresponding pixels in the reference frame in the clipping space, and determine the mapping relationship between the pixels in the current frame and the pixels in the reference frame. The motion vector prediction model is determined based on the mapping relationship between pixels in the current frame and pixels in the reference frame.

11. The method according to claim 7, characterized in that, The step of determining the motion vector prediction model based on the motion vector of the control point of the current block, the camera parameters of the current frame, and the camera parameters of the reference frame includes: The predicted points of the upper left corner control point, the upper right corner control point, and the lower left corner control point of the current block are determined based on the motion vector of the control point of the current block. The motion vector prediction model is determined based on the depths of the top-left, top-right, and bottom-left control points of the current block, the depths of the predicted points of the top-left, top-right, and bottom-left control points in the reference frame, the camera parameters of the current frame, and the camera parameters of the reference frame.

12. The method according to claim 11, characterized in that, The motion vector prediction model is determined based on the depths of the top-left, top-right, and bottom-left control points of the current block, the predicted depths of the top-left, top-right, and bottom-left control points in the reference frame, the camera parameters of the current frame, and the camera parameters of the reference frame. Based on the depths of the top-left corner control point, the top-right corner control point, and the bottom-left corner control point of the current block, and the depths of the predicted points of the top-left corner control point, the top-right corner control point, and the bottom-left corner control point in the reference frame, the following second relation is constructed; Wherein, α1, α2, and α3 are the depths of the top-left, top-right, and bottom-left control points of the current block, respectively; β1, β2, and β3 are the depths of the predicted points of the top-left, top-right, and bottom-left control points in the reference frame, respectively; λ1 = x / w, λ2 = y / h, where x and y are the coordinates of the pixels in the current block relative to the top-left control point of the current block; and w and h are the width and height of the current block, respectively. Based on the second relation, the camera parameters of the current frame, and the camera parameters of the reference frame, obtain the mapping relationship between the pixels in the current frame and the pixels in the reference frame; The motion vector prediction model is determined based on the mapping relationship between pixels in the current frame and pixels in the reference frame.

13. The method according to claim 12, characterized in that, The step of obtaining the mapping relationship between pixels in the current frame and pixels in the reference frame based on the second relation, the camera parameters of the current frame, and the camera parameters of the reference frame includes: A first observation matrix and a second observation matrix are generated based on the camera extrinsic parameters of the current frame and the camera extrinsic parameters of the reference frame, respectively. Based on the first observation matrix, the second observation matrix, the first parameter, and the second parameter, the following third relation is constructed: in, Let be the element in the m-th row and n-th column of the first observation matrix. The element in the m-th row and fourth column of the first observation matrix. The element in the m-th row and n-th column of the second observation matrix. Let f1 and f2 be the elements in the m-th row and fourth column of the second observation matrix, respectively, and u be the first parameter and the second parameter. 1 and v 1 These are the x and y coordinates of the pixel coordinates of the pixel point in the current block, respectively. 2 and v 2 These are the x and y coordinates of the pixel in the reference frame, respectively; α is the reciprocal of the second coordinate component of the pixel in the current block in the clipping space, and β is the reciprocal of the second coordinate component of the pixel in the reference frame in the clipping space. Based on the second relation and the third relation, the mapping relationship between the pixels in the current frame and the pixels in the reference frame is obtained.

14. The method according to claim 2, characterized in that, Determining the motion vector prediction model based on the camera parameters of the current frame and the camera parameters of the reference frame includes: A first observation matrix and a second observation matrix are generated based on the camera extrinsic parameters of the current frame and the camera extrinsic parameters of the reference frame, respectively. The motion vector prediction model is determined based on the first observation matrix, the second observation matrix, and the camera intrinsic parameters.

15. The method according to claim 14, characterized in that, The motion vector prediction model includes the following fourth relation: in, Let be the element in the m-th row and n-th column of the first observation matrix. Let f1 and f2 be the elements in the m-th row and n-th column of the second observation matrix, respectively, and let u be the first parameter and the second parameter. 1 and v 1 These are the x and y coordinates of the pixel coordinates of the pixel point in the current block, respectively. 2 and v 2 These are the x and y coordinates of the pixel coordinates of the pixel point in the reference frame, respectively.

16. The method according to claim 7, characterized in that, The step of obtaining the motion vector of the control point of the current block includes: Obtain the spatial reference motion vector of the control point of the current block; The spatial reference motion vector is compensated based on the distance from the center of the image block corresponding to the spatial reference motion vector to the control point of the current block, so as to obtain the motion vector of the control point of the current block.

17. A video encoding method, characterized in that, include: Determine the reference frame for the current frame, where the current frame is the video frame containing the current block; Obtain the camera parameters of the current frame and the camera parameters of the reference frame; A motion vector prediction model is determined based on the camera parameters of the current frame and the camera parameters of the reference frame; Based on the motion vector prediction model, obtain the predicted block for the current block; Obtain the residual block of the current block based on the current block and the predicted block; Based on the residual block of the current block, obtain the encoded data of the current block.

18. A video decoding device, characterized in that, include: Memory, configured to store computer programs; The processor is configured to cause the video encoding apparatus to implement the video decoding method according to any one of claims 1-16 when a computer program is invoked.

19. A video encoding apparatus, characterized in that, include: Memory, configured to store computer programs; The processor is configured to cause the video decoding device to implement the video encoding method of claim 17 when a computer program is invoked.

Images