Video decoding method and apparatus and video coding method and apparatus

Abstract

Some embodiments of the present application relate to the technical field of video coding and decoding. Provided are a video decoding method and apparatus and a video coding method and apparatus. The video decoding method comprises: acquiring coded data of the current block, wherein the current block is any coding unit in a video sequence; on the basis of the coded data of the current block, acquiring a residual block of the current block and a reference frame of the current frame, wherein the current frame is a video frame including the current block; acquiring camera parameters of the current frame and camera parameters of the reference frame; on the basis of the camera parameters of the current frame and the camera parameters of the reference frame, acquiring a mapping relationship between a pixel point in the current block and a pixel point in the reference frame; on the basis of the mapping relationship and the reference frame, acquiring a predicted block of the current block; and on the basis of the residual block of the current block and the predicted block of the current block, reconstructing the current block, so as to acquire a reconstructed block corresponding to the current block. Some embodiments of the present application are used for describing motion information of objects more accurately.

Description

Video decoding method, video encoding method and device

[0001] The present disclosure claims priority to a Chinese patent application No. 2024104875129, filed on April 22, 2024, and entitled "Video decoding method, video encoding method and device", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] Some embodiments of the present application relate to the field of video coding technology. More specifically, the present application relates to a video decoding method, a video encoding method and a device. BACKGROUND

[0003] In view of the large amount of temporal redundancy in a video, a traditional encoder encodes a key frame intracodedly, and encodes other video frames intercodedly. When transmitting, only the complete key frame and the residual of the other video frames relative to the key frame are transmitted, so as to achieve redundancy removal. The most core technology of intercoding is motion estimation and motion compensation, that is, finding the best matching coding block for each coding block of the current frame in the coded frame, and obtaining the motion vector corresponding to each coding block.

[0004] Traditional video coding algorithms have good estimation and compensation effects for the translation motion in a video sequence, but are not sensitive to the deformation caused by camera rotation, forward and backward movement. For example, affine transformation does not cover the concept of depth, which is a linear transformation between different two-dimensional coordinates. Therefore, in three-dimensional space, affine transformation can represent relatively limited motion information. However, in three-dimensional animation videos such as cloud games, the motion of objects is more diverse, covering scaling, rotation, perspective motion and other irregular motions, so affine transformation cannot completely describe the non-affine conditions involving depth changes in the above motions. SUMMARY

[0005] The exemplary embodiments of the present application provide a video decoding method, a video encoding method and a device for more accurately describing the motion information of an object.

[0006] Some embodiments of the present application provide the technical solutions as follows:

[0007] In a first aspect, some embodiments of the present application provide a video decoding method, comprising:

[0008] obtaining coding data of a current block; the current block is any coding unit in a video sequence;

[0009] According to the encoding data of the current block, a residual block of the current block and a reference frame of the current frame are obtained; the current frame is a video frame containing the current block;

[0010] Camera parameters of the current frame and camera parameters of the reference frame are obtained.

[0011] According to the camera parameters of the current frame and the camera parameters of the reference frame, a mapping relationship between a pixel point in the current block and a pixel point in the reference frame is obtained.

[0012] According to the mapping relationship and the reference frame, a prediction block of the current block is obtained.

[0013] According to the residual block of the current block and the prediction block of the current block, the current block is reconstructed to obtain a reconstructed block corresponding to the current block.

[0014] In a second aspect, some embodiments of the present application provide a video encoding method, comprising:

[0015] A reference frame of a current frame is determined, the current frame being a video frame containing a current block;

[0016] Camera parameters of the current frame and camera parameters of the reference frame are obtained.

[0017] According to the camera parameters of the current frame and the camera parameters of the reference frame, a mapping relationship between a pixel point in the current block and a pixel point in the reference frame is obtained.

[0018] According to the mapping relationship between the pixel point in the current block and the pixel point in the reference frame, a prediction block of the current block is obtained.

[0019] According to the current block and the prediction block, a residual block of the current block is obtained.

[0020] According to the residual block of the current block, encoding data of the current block is obtained.

[0021] In a third aspect, some embodiments of the present application provide an image decoding device, comprising:

[0022] A memory configured to store a computer program;

[0023] A processor configured to, when the computer program is invoked, cause the video encoding device to implement the image decoding method of the first aspect.

[0024] In a fourth aspect, some embodiments of the present application provide a video encoding device, comprising:

[0025] A memory configured to store a computer program;

[0026] The processor is configured to cause the video decoding apparatus to implement the image encoding method of the first aspect when the computer program is invoked.

[0027] In a fifth aspect, some embodiments of the present application provide a computer readable storage medium, having stored thereon a computer program, which, when executed by a computing device, causes the computing device to implement the method of the first aspect or the second aspect.

[0028] In a sixth aspect, some embodiments of the present application provide a computer program product, which, when running on a computer, causes the computer to implement the method of the first aspect or the second aspect.

[0029] According to the above technical solutions, the video encoding method provided by the embodiments of the present application first acquires the residual block of the current block and the reference frame of the current frame according to the encoding data of the current block after acquiring the encoding data of the current block, then acquires the camera parameters of the current frame and the camera parameters of the reference frame, and acquires the mapping relationship between the pixel points in the current block and the pixel points in the reference frame according to the camera parameters of the current frame and the camera parameters of the reference frame, and then acquires the prediction block of the current block according to the mapping relationship and the reference frame, and reconstructs the current block according to the residual block of the current block and the prediction block of the current block to acquire the reconstructed block corresponding to the current block. Since the video encoding method provided by the embodiments of the present application can acquire the mapping relationship between the pixel points in the current block and the pixel points in the reference frame according to the camera parameters of the current frame and the camera parameters of the reference frame, the embodiments of the present application can accurately acquire the motion vector of each pixel point in the current block according to the mapping relationship between the pixel points in the current block and the pixel points in the reference frame, and thus can more accurately describe the motion information of the object, thereby improving the quality of the reconstructed block of the current block. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the embodiments of some embodiments of the present application or the implementation manners in the related art, the drawings needed to be used in the embodiment or related art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art according to these drawings.

[0031] FIG. 1 shows a structural block diagram of a depth-based image rendering framework in some embodiments of the present application;

[0032] FIG. 2 shows a step flowchart of a motion vector prediction model in some embodiments of the present application;

[0033] FIG. 3 shows a flow diagram of obtaining an observation matrix and a projection matrix in some embodiments of the present application;

[0034] FIG. 4 shows a flow diagram of coordinate space transformation in some embodiments of the present application;

[0035] FIG. 5 shows a diagram of a view frustum in some embodiments of the present application;

[0036] FIG. 6 shows a diagram of the basic principle of 3D transformation in some embodiments of the present application;

[0037] FIG. 7 shows a diagram of a region with uniform depth change in some embodiments of the present application;

[0038] FIG. 8 shows a diagram of a control point of a current block in some embodiments of the present application;

[0039] FIG. 9 shows a diagram of motion vector compensation in some embodiments of the present application;

[0040] FIG. 10 shows a flow diagram of steps of a video decoding method according to some embodiments of the present application;

[0041] FIG. 11 shows a flow diagram of steps of a video encoding method according to some embodiments of the present application. DETAILED DESCRIPTION

[0042] For the purpose of making the object and implementation of the present application more clear, the following will describe the exemplary implementation of the present application in detail with reference to the accompanying drawings of the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only some of the embodiments of the present application, but not all of the embodiments of the present application.

[0043] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the following described embodiments, and is not intended to limit the implementation of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0044] The terms "comprise" and "have" and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that contains a list of components, and it can include other components not clearly listed or inherent to such products or devices.

[0045] References in the specification to "some implementations", "some embodiments", etc. indicate that the description can include a particular feature, structure, or characteristic, but every embodiment can not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same implementation. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of those skilled in the art to effect such feature, structure, or characteristic in connection with other implementations whether or not explicitly described herein.

[0046] Some embodiments of the present application relate to Depth Image Based Rendering (DIBR). The following first describes the Depth Image Based Rendering technology.

[0047] The core of the Depth Image Based Rendering technology is to construct the three-dimensional information of the target viewpoint by using the depth and color information of the old view, and to simulate the rendering process of the graphics engine through three-dimensional mapping, so as to generate a new view of other viewpoints. The essence of the Depth Image Based Rendering technology is the transformation of images in three-dimensional space, so the Depth Image Based Rendering technology is also called 3D Image Warping technology.

[0048] Referring to FIG. 1, the Depth-based image rendering framework mainly includes a three-dimensional transformation module 11, a view fusion module 12, and a hole filling module 13.

[0049] The three-dimensional transformation module 11 is the core component of the Depth-based image rendering framework, and is mainly responsible for realizing the accurate conversion of the reference image to the target image. In the process of realizing the conversion of the reference image to the target image, the three-dimensional transformation module 11 will perform a series of complex mapping transformations, which occur in the three-dimensional space. Specifically, the three-dimensional transformation module 11 will first determine a pixel point on the reference image, and map it from the reference image coordinate system to the three-dimensional world coordinate system through the reference camera coordinate system. Subsequently, the pixel point will be projected again through the target camera coordinate system, and finally accurately mapped to the corresponding point on the target image coordinate system.

[0050] The view fusion module 12 is mainly responsible for making full use of the occlusion relationship of objects in multiple views to generate a more complete and accurate target view. When a camera captures objects in a three-dimensional scene from a fixed position and angle, the objects may be occluded by each other. Therefore, after the left and right reference views corresponding to the virtual views generated by the three-dimensional transformation module 11 are input into the view fusion module 12, the view fusion module 12 can make full use of the different manifestations of the occlusion relationship of objects in multiple views to generate a more complete and accurate target view. The view fusion module 12 can reduce the holes in the target view caused by object occlusion, thereby significantly improving the image quality of the target view and providing users with a more realistic and delicate visual experience.

[0051] The hole filling module 13 is mainly responsible for hole filling of the target view. After the processing of the view fusion module 12, there may still be some pixel points in the target view that cannot find corresponding reference information from the reference view. These pixel points will appear in the form of holes in the target view, which will seriously affect the overall quality of the image. The hole filling module 13 can use the pixel information of the non-empty pixels around the hole to accurately fill the hole, ensuring that the final output target view is complete and of high quality.

[0052] The following briefly describes the video coding technology.

[0053] At present, the commonly used video coding and decoding protocols are still mainly high efficiency video coding (H.265 / HEVC) and advanced video coding (H.264 / AVC). In the best coding template, H.265 / HEVC can save about 36% of the data amount compared with a 1080P film compressed by H.264 / AVC under the premise of maintaining the same peak signal to noise ratio (PSNR), and can further improve the saving range to 50% when the image quality is similar.

[0054] Compared with the previous generation of video coding standard (H.265 / HEVC), the inter-frame coding technology of the Versatile Video Coding (H.266 / VVC) standard basically follows the motion estimation, motion vector prediction and motion compensation technology in H.265 / HEVC. On this basis, H.266 / VVC introduces some new motion vector prediction technologies and improves some motion vector prediction technologies. These improvements mainly target the motion vector (MV) candidate list of the Merge mode, focusing on optimizing its length, perfecting the construction process and adjusting the adjacent coding block checking range. For the improvement of the motion vector prediction technology, it includes affine transformation technology and adaptive motion vector precision technology, etc.

[0055] In H.266 / VVC, the Merge mode has undergone significant changes, especially in the construction of the MV candidate list. One of the most significant changes is the increase in the length of the MV candidate list, from the original 5 candidate MVs to the current 6 candidate MVs, due to the addition of multiple adjacent coding block checking methods. In the process of constructing the MV candidate list, each adjacent coding block of the coding block is still given priority, and then the validity of the temporal candidate is strictly checked. In order to ensure the continuity and stability of the technology, this construction process is consistent with H.265 / HEVC. In H.266 / VVC, if the candidate list fails to reach the predetermined complete length during the construction process, an innovative padding candidate method, the History Based Motion Vector Predictor (HMVP) technology, is introduced. After the construction of the candidate list, the encoder will calculate the rate-distortion cost of each candidate in the list one by one, so as to accurately select the best candidate MV. In H.266 / VVC, in addition to the above-mentioned Merge mode, a variety of new technologies such as Multiple Motion Vector Difference (MMVD) technology are introduced. These technologies independently calculate the candidate MV, take the optimal result as the representative of the current mode, and then evaluate the rate-distortion cost (RD cost) of these modes on the whole level. Finally, through comprehensive comparison, the encoder will determine an optimal motion vector prediction Merge mode and select the best MV candidate corresponding to it.

[0056] In H.265 / HEVC, the motion compensation prediction technique is mainly aimed at translational motion, which limits its ability to describe complex motion in nature. In fact, the types of motion in nature are diverse, including scaling, rotation, and various irregular motions. When the encoder faces these complex motions, it often has the problem of large prediction value error, which leads to the increase of residual of the coding block. In order to solve this problem, H.266 / VVC introduces the sub-block-based affine transformation technology (Affine Mode). Affine mode can be divided into sub-block-based affine transformation and affine motion vector prediction (Affine AMVP), which is mainly aimed at coding blocks (Coding Unit, CU) of different sizes. The gain brought by Affine AMVP mode is not high in the case of small coding unit size, so the minimum coding unit size of Affine AMVP is larger than that of Affine Merge mode. In any Affine mode, it can be divided into two modes of four degrees of freedom and six degrees of freedom. The affine motion of the to-be-encoded block can be represented by two models: one is a four-parameter model composed of two control point motion vectors, and the other is a six-parameter model composed of three control point motion vectors. Because the calculation amount of pixel-level motion compensation is too large, the block is usually taken as the unit to obtain its corresponding motion vector. In H.266 / VVC, for the two Affine modes, the establishment process of the corresponding candidate list is specified, that is, the AMVP candidate list (2 candidate items) is established for Affine Inter Mode, and the Merge candidate list (5 candidate items) is established for Affine Merge Mode. The main function of sub-block-based temporal motion vector prediction (SbTMVP) is to obtain the motion information of the sub-block in the current coding unit from the corresponding sub-block in the same frame as the candidate item of the Merge list. Prediction value correction technology based on optical flow field.

[0057] Bi-directional optical flow (BDOF) technology is developed from BIO technology in JEM reference model. Compared with BIO, BDOF has lower complexity, especially smaller number of multiplication operations and smaller multiplier. The main function of bi-directional optical flow technology is to correct the pixel value of the 4x4 sub-block bi-directional prediction in the coding unit. For each 4x4 sub-block, the motion correction (v x ,v y), and the correction result is then used to adjust bi-predicted pixel values in the 4x4 sub-block. BDOF is applicable to coding units in regular Merge mode or Inter mode under bi-prediction, and requires two reference frames and the current frame to be equidistant.

[0058] There are various motion modes in three-dimensional animation videos such as cloud game videos, such as zooming in, zooming out, rotating, perspective, etc., but the traditional video coding algorithm can only better estimate and compensate for the translation motion occurring in the video sequence, and is not sensitive to the deformation caused by camera rotation, forward and backward. Therefore, in view of the characteristics of three-dimensional animation videos and the shortcomings of traditional encoders, some embodiments of the present application propose a motion vector prediction model based on the idea of three-dimensional transformation, so that the encoder can better adapt to various motion modes and improve the coding performance when coding. Referring to FIG. 2, in some embodiments, the construction process of the motion vector prediction model includes the following steps:

[0059] S21, obtaining the observation matrix and the projection matrix of the virtual camera.

[0060] For three-dimensional animation videos such as cloud game videos, the observation matrix (View Matrix) and the projection matrix (Projection Matrix) are very critical in the process of rendering a three-dimensional model into a video frame and finally presenting it to the screen of a display device. Among them, the observation matrix is used to transform the position information of a point on a three-dimensional model from the corresponding world space to the observation space, and the projection matrix is used to transform the position information of a point on a three-dimensional model from the observation space to the projection space. And the observation matrix and the projection matrix are also commonly called the extrinsic matrix and the intrinsic matrix of the virtual camera.

[0061] S22, determining the transformation relationship between the pixel points in the current frame and the coordinates in the reference frame according to the observation matrix and the projection matrix of the virtual camera.

[0062] S23, constructing a motion vector prediction model according to the transformation relationship between the pixel points in the current frame and the coordinates in the reference frame.

[0063] The implementation of obtaining the observation matrix and the projection matrix is described in detail below.

[0064] Referring to FIG. 3, S30 (obtaining the observation matrix and the projection matrix) can be divided into: S31, obtaining the extrinsic information of the virtual camera and S32, obtaining the intrinsic information of the virtual camera. And S32 (obtaining the intrinsic information of the virtual camera) includes: S321, directly calling an interface to obtain the intrinsic information of the virtual camera and S322, using camera extrinsic information and video frames to estimate intrinsic information.

[0065] In some embodiments, the extrinsic information of the virtual camera is obtained by calling a corresponding interface.

[0066] In the Unity3D game engine, the absolute extrinsic matrix (absolute observation matrix) that changes relative to the origin in the world space can be obtained by directly calling a corresponding interface to obtain the extrinsic information of the virtual camera. Alternatively, the relative observation matrix (extrinsic information) that changes in the camera pose information can be obtained by changing the front and back position information of the camera in the world space coordinate system on the game engine interaction panel.

[0067] It should be noted that if the extrinsic information of the camera is directly obtained by calling a corresponding interface, when the virtual camera only rotates, the elements representing the position of the camera in the world space in the extrinsic matrix will also change, resulting in that the type of motion of the virtual camera cannot be directly determined by the extrinsic information.

[0068] In some embodiments, the intrinsic information of the virtual camera is obtained by calling a corresponding interface to obtain the intrinsic information of the virtual camera or estimating the intrinsic information from the camera extrinsic information and the video frame.

[0069] The intrinsic information of the virtual camera can be directly obtained from the engine by calling a corresponding interface. However, in general, the developer will not publicly disclose the intrinsic information of the virtual camera, so the intrinsic information of the virtual camera cannot be directly obtained by calling a corresponding interface. Therefore, the intrinsic information of the virtual camera can be estimated from the extrinsic information of the virtual camera and the video frame.

[0070] The implementation of estimating the intrinsic information of the virtual camera from the extrinsic information of the virtual camera and the video frame is described in detail below.

[0071] At present, most three-dimensional animation videos are generated by perspective projection technology, and the perspective projection related view frustum parameters can be changed by the field of view (FOV) in the virtual camera component and the near clipping plane and far clipping plane parameters in the clipping plane. The height of the near clipping plane, the height of the far clipping plane, the distance of the near clipping plane, the distance of the far clipping plane, and the field of view exist the following relationships (1) and (2):

[0072] wherein nearClipPlaneHeight is the height of the near clipping plane, Near is the distance of the near clipping plane (the distance from the virtual camera to the near clipping plane), and FOV is the field of view.

[0073] wherein, farClipPlaneHeight is the height of the far clipping plane, Far is the distance of the far clipping plane (the distance from the virtual camera to the far clipping plane), and FOV is the field of view angle.

[0074] The width of the near clipping plane and the width of the far clipping plane can be obtained by the aspect ratio of the virtual camera, and can also be configured by the aspect ratio of the virtual camera. Assuming that the aspect ratio of the virtual camera is Aspect, then:

[0075] wherein, nearClipPlaneWidth is the width of the near clipping plane, and farClipPlaneWidth is the width of the far clipping plane.

[0076] In combination with the above formulae (1) to (3), the projection matrix K of the virtual camera can be expressed as follows:

[0077] Further, the above formula (4) can be simplified and expressed as follows:

[0078] It should be noted that in the process of spatial transformation, the coordinate handedness of each space changes. Among them, the world space is a left-handed coordinate system, the observation space is a right-handed coordinate system, and the projection space reverts to a left-handed coordinate system. Therefore, when performing matrix transformation, the corresponding handedness change between different coordinate systems must be considered.

[0079] On the video sequence of pure rotation of the virtual camera (the virtual camera only rotates and does not displace), the ORB algorithm is used to find matching feature points on two frames of pure rotation of the virtual camera, the constraints between frames are established according to the feature point pairs, and finally the least square method is used to estimate the values of f1 and f2 so as to participate in the subsequent model derivation and calculation. The specific calculation process is as follows:

[0080] Let the virtual camera projection matrix P Projection be:

[0081] Therefore, the virtual camera projection matrix includes four unknown quantities f1, f2, c1 and c2.

[0082] On the video sequence of pure rotation of the virtual camera, the observation matrix only has a rotation matrix, and does not have a translation matrix, so the observation matrix V rotation is:

[0083] wherein, R z , R x , and R yThese are rotation matrices about the z-axis, x-axis, and y-axis, respectively, where z, x, and y are the rotation angles about the z-axis, x-axis, and y-axis, respectively.

[0084] Given two adjacent video frames and their corresponding observation matrices in a video sequence with pure virtual camera rotation. First, based on the ORB-based feature point matching algorithm described above, feature points in two adjacent video frames are matched. Furthermore, according to the principle of coordinate space mapping transformation based on 3D transformation, the matched feature points originate from the same point on the 3D model in world space. Therefore, we can assume the coordinates of a point on the 3D model in world space are P. world =(X,Y,Z,1) T The corresponding matching point in the adjacent video frame is The coordinates in the corresponding clipping space are Then we have:

[0085] Among them, P world The coordinates of the feature point pairs to be matched in world space. P is the observation matrix of the virtual camera. Projection The projection matrix of the virtual camera, The coordinates of the matching feature point pairs in the clipping space. These are the coordinates of the matching feature point pairs in screen space. `pixelwidth` and `pixelheight` are the width and height of the video frame, respectively.

[0086] Will It can be represented as:

[0087] Then, according to equations (9) to (12), equation (8) can be rearranged as follows:

[0088] Based on the principle that the matching feature points in two adjacent video frames in a 3D transformation have the same coordinates in world space, the following constraints can be established for f1 and f2 in equation (13):

[0089] Further utilizing the classical nonlinear least squares method, by min((AB)) 2 +(CB) 2 +(AC) 2 By doing so, the values ​​of f1 and f2 can be obtained.

[0090] In 3D animation videos, especially first-person game videos, there are a large number of video frames with pure rotational motion of the virtual camera. Therefore, the above method can be used to obtain the values ​​of f1 and f2 relatively easily.

[0091] It should be noted that when the different reference views of different rotation modes of the camera are used for calculation, it can be found that when the camera only rotates around one of the x, y and z axes, each of the three constraint terms in formula (14) only contains one of f1 or f2, resulting in reduced accuracy of the final constraint calculation. Therefore, by using the multi-directional composite rotation of the virtual camera motion sequence, more accurate f1 and f2 can be calculated and obtained.

[0092] In some embodiments, the intrinsic parameters of the virtual camera include the ratio of the half-angle cotangent of the field of view angle of the camera to the aspect ratio of the camera and / or the half-angle cotangent of the field of view angle of the camera, i.e., f1 and f2. Therefore, the intrinsic parameter information of the virtual camera includes the values of f1 and f2.

[0093] Therefore, based on the above camera intrinsic parameter estimation algorithm, the unknowns (f1, f2) related to the field of view angle FOV and the aspect ratio Aspect in the projection matrix of the camera can be obtained by different frames of pure rotation of the virtual camera in the video. However, the unknowns (c1, c2) related to the distances of the near clipping plane and the far clipping plane of the virtual camera frustum cannot establish similar constraint relationships, so c1 and c2 cannot be estimated, and the depth information in the three-dimensional animation video cannot be obtained. Therefore, the depth information and c1 and c2 are taken as unknowns in the following model derivation.

[0094] Further, the implementation of converting the coordinates of a point in the current frame into coordinates in the reference frame is described below.

[0095] In the three-dimensional animation video generation process, the points on the three-dimensional model need to undergo multiple coordinate space transformations before they can be finally presented on a two-dimensional screen. Referring to FIG. 4, the transformation process includes:

[0096] S41, transforming the vertex coordinates of the three-dimensional model from the model space to the world space.

[0097] In a virtual three-dimensional scene, each created model has its own independent coordinate space, which is called the model space. The model space changes with the movement or rotation of the model. The world space can describe the absolute position in the virtual three-dimensional scene, so the first step of coordinate space transformation is to transform the vertex coordinates of the three-dimensional model from the model space to the world space. The transformation of the vertex coordinates of the three-dimensional model from the model space to the world space is called model transformation.

[0098] S42, transforming the vertex coordinates of the three-dimensional model from the world space to the observation space.

[0099] In the process of generating three-dimensional animation, the virtual camera determines the user's perspective, i.e. the perspective used for rendering the video, so the coordinates of the world space need to be continuously mapped to the viewing space (camera space). The process of transforming the vertex coordinates of a three-dimensional model from the world space to the viewing space is called view transform, and the matrix that implements the view transform is called view matrix, i.e. the extrinsic matrix of the camera.

[0100] When the vertex coordinates of a three-dimensional model are transformed from the world space to the viewing space, the rule of first translation and then rotation is followed. Therefore, when obtaining the view matrix, the entire viewing space is moved, and the position of the virtual camera in the world space is inversely transformed, i.e. first rotation and then translation.

[0101] Let the coordinates of a certain point on a three-dimensional model in the world space be P world =(X world ,Y world ,Z world ,1) T In the world space coordinate system, let the rotation matrix of the camera be R, the translation matrix be T, and the view matrix be V. Then the view matrix can be described as: V=R·T(15)

[0102] P world After view transform, the coordinates in the viewing space are P view , and there is: P view =V·P world (16)

[0103] S43, transform the vertex coordinates of the three-dimensional model from the viewing space to the clipping space.

[0104] The main goal of the clipping space is to simplify the clipping process of the rendering primitives. The clipping space is determined by the view frustum, which is surrounded by six planes, which are also called clipping planes. Among them, the near clipping plane and the far clipping plane determine the depth range that can be seen by the camera. The coordinates in the viewing space can be converted into the corresponding clipping space by using the projection matrix. The core function of the projection matrix is to scale the x, y, and z components by different proportions and to perform a translation operation on the z component. Such scaling design is mainly to facilitate subsequent clipping operations.

[0105] As shown in FIG. 5, all points on the ray L starting from the position of the camera 50 and located in the view frustum will be rendered on the same point P. The vertical plane where the point P is located is called the projection plane. The view frustum is vertically opened by an angle FOV, the aspect ratio of the video frame is represented by Aspect, the distances of the near clipping plane 51 and the far clipping plane 52 of the view frustum from the camera are represented by Near and Far respectively. Then the projection matrix K can be represented as:

[0106] After the vertex coordinates of the three-dimensional model are transformed from the observation space to the clipping space, the vertex coordinates of the three-dimensional model are still coordinates in the three-dimensional space, and the coordinates in the clipping space are set as Then, P clip = K · P view (18)

[0107] The obtained coordinates in the clipping space are subjected to homogeneous division processing again to obtain normalized device coordinates (NDC), Next, the corresponding pixel coordinates of the output window can be mapped according to the obtained u and v coordinates.

[0108] S44, transforming the vertex coordinates of the three-dimensional model from the clipping space to the screen space.

[0109] In some embodiments, the vertex coordinates of the three-dimensional model in the video cone are projected to the screen space. After the vertex coordinates of the three-dimensional model are transformed from the clipping space to the screen space, the virtual three-dimensional coordinates are converted to actual pixel positions on the screen, thereby presenting the real picture presented by the three-dimensional animation video.

[0110] According to the above space transformation and coordinate mapping process, when a point P world is respectively mapped to different positions in the current frame and the reference frame during the camera motion, the clipping space coordinates of the previous frame are set as P old , and the clipping space coordinates of the next frame are set as P new . Then, P

[0111] wherein V new and V old are the observation matrices corresponding to the current frame and the reference frame, respectively, and K is the projection matrix corresponding to the current frame and the reference frame.

[0112] Therefore, the relationship between the coordinate transformation of the current frame and the reference frame can be established according to formula (19):

[0113] After formula (20) is arranged, formula (21) can be obtained:

[0114] From the above formula (21), it can be seen that the NDC coordinates of a point in the new camera viewpoint projection in the same world space can be calculated from the NDC coordinate information of the last frame, camera motion information and camera intrinsic matrix (projection matrix). According to such a relationship, the positions of most points on a frame image can be calculated pixel by pixel to obtain the motion vector between the current frame and the reference frame image.

[0115] The following describes an implementation of a motion vector prediction model established according to the transformation relationship between the pixel points in the current frame and the coordinates in the reference frame.

[0116] Referring to FIG. 6, based on the basic principle of three-dimensional transformation, the coordinates (x, y) of a point P in the world space are sequentially subjected to the observation matrix V 1 , the projection matrix K and the mapping between the clipping space and the screen space of the first video frame to obtain the pixel coordinates (u 1 , v 1 ) of the point in the first video frame. The coordinates of the point are sequentially subjected to the observation matrix V 2 , the projection matrix K and the mapping between the clipping space and the screen space of the second video frame to obtain the pixel coordinates (u 2 , v 2 ) of the point in the second video frame. Therefore, the coordinates of a point on the current video frame are first inversely mapped from the screen space back to the clipping space, then inversely mapped from the clipping space back to the observation space, and then inversely mapped from the observation space back to the world space. Similarly, the matching point on the current video frame is first inversely mapped from the screen space back to the clipping space, then inversely mapped from the clipping space back to the observation space, and then inversely mapped from the observation space back to the world space to obtain equal world space coordinates.

[0117] In some embodiments, it is assumed that the absolute observation matrices of the cameras corresponding to the current frame and the reference frame are V i , i ∈ {1, 2}, and the coordinates of the matching points in the clipping space of the current frame and the reference frame are The coordinates of the point P world =(X, Y, Z, 1) T in the world space corresponding to the matching points of the current frame and the reference frame, and

[0118] wherein, are the elements of the absolute rotation matrix of the camera relative to the initial direction, are the values of the absolute coordinate position of the camera in the world space after the rotation matrix transformation.

[0119] [Rule 91, 04.12.2025] Combined with the above assumptions and formula (13), the following can be obtained:

[0120] From the above formula (24), it can be seen that A, P1, P2, P3, P4, Q1, Q2, Q3, Q4, S1, S2, S3, S4 are all known quantities that can be obtained from the observation matrix.

[0121] In combination with formula (20), the terms of the coordinate values X and Y in the world space in formula (23) are linked:

[0122] Let the coordinates of a point in the current frame be (u 1 , v 1 ), and the coordinates of the corresponding matching point in the reference frame be (u 2 , v 2 ). According to formula (17), the relationships of u 1 , v 1 , u 2 , and v 2 are as follows:

[0123] In the case where the projection matrix, the observation matrix, the two-dimensional coordinates of the pixels in the current frame, and the depth information are known, a series of points in the two-dimensional coordinate system that are mapped from a point in the same world coordinate system can be calculated by using the above formulas (26) and (27). Therefore, in theory, the transformation relationship of a certain block region where the feature points are located can be estimated by matching the feature points between the video frames, so as to establish a motion vector prediction model and determine the motion vectors of all the pixels in the region.

[0124] For the convenience of analysis and subsequent calculation, formula (26) can be rearranged as follows:

[0125] wherein,

[0126] From the above formula (28), it can be seen that θ, S, and J are parameters related to the depth information and the projection matrix; and E, H, I, and M are parameters related to the observation matrix and the coordinate information of the known matching points.

[0127] At this point, the relationships of u 2 , v 2 , u 1 , and v 1 have been completely represented by the relevant information that has universality. Based on the establishment of the motion relationship between the matching points in the above current frame and reference frame, formula (28) is further transformed according to the objective information that can be obtained, so that the final motion vector prediction model can be flexibly applied to various three-dimensional animation videos.

[0128] For three-dimensional animation videos such as cloud game videos, the observation matrix representing the camera motion and the coordinates of points on the video frame can be obtained, but the projection matrix of the camera and the depth information of the three-dimensional model in the world space are usually unavailable. Therefore, in the derivation process of the motion vector prediction model, the projection matrix and the depth information are taken as unknown quantities, and the observation matrix and the coordinates of points in the current frame are taken as known quantities. In addition, since f1 and f2 in the projection matrix can be estimated according to the matching point pairs in the current frame and the reference frame, f1 and f2 in the embodiment of the present application are also taken as known quantities to participate in the derivation of the subsequent motion vector prediction model.

[0129] In some embodiments, the embodiment of the present application provides a motion vector prediction model of an encoding block in which the depth information of each pixel point is the same.

[0130] When encoding a video frame, the encoder will first divide the video frame into blocks, and divide the video frame into multiple encoding blocks. When the size of the encoding block is small enough, it can be considered that the motion vector of each pixel point in each encoding block is the same, and the depth information of each pixel point is also the same.

[0131] The core idea of the motion vector prediction of the encoding block in which the depth information of each pixel point is the same is that each video frame in the video frame sequence is divided into multiple non-overlapping encoding blocks, and in a given search range, a feature point finds a matching point in a reference frame according to a certain calculation criterion, and the relative displacement of the matching point and the current point is the motion vector of the encoding block.

[0132] Based on the above theory and assumption, the quantities θ, S, and J related to the depth information and the projection matrix in formula (27) and formula (28) are taken as unknown quantities, and the quantities E, H, I, and M related to the observation matrix are taken as known quantities. Therefore, u 2 and v 2 in formula (28) are respectively regarded as a binary first-order equation, and the values of θ, S, and J can be solved by using the coordinates of two pairs of matching points.

[0133] However, it can be obtained from formula (26) that the expression of u 2 contains v 2 , and the expression of v 2 contains u 2 . In order to avoid coupling in the estimation of u 2 and v 2 , u 2 and v 2 need to be transformed into expressions expressed only by the current point u 1 and v 1 in the current frame and known parameters.

[0134] Combining formula (27), formula (28), formula (29), formula (26) can be re-expressed as:

[0135] In the above formula (30), (31), θ, S, J and depth information are related to the projection matrix, and are unknown quantities to be solved. H, M, N are all known quantities which can be solved by the observation matrix. Only E, I are related to the matching point coordinates, and only contain the coordinates u 1 , v 1 of the current point in the current frame. Thus, the coordinates of the matching point in the reference frame u 2 , v 2 can be predicted only by the coordinates of the current point in the current frame u 1 , v 1 , the observation matrix of the current frame and the reference frame, and f1, f2 estimated from the current frame and the reference frame. The coordinate difference between u 2 , v 2 and u 1 , v 1 is the motion vector between the matching point pairs.

[0136] From formula (23), we have:

[0137] Combining formula (27), (29) and (32), the three unknown quantities θ, S, J related to depth information can be expressed as:

[0138] From formula (33) to (35), it can be seen that the motion vector prediction model established at this time only has two unknown parameters, both of which are related to depth information and have nothing to do with other physical quantities.

[0139] Let the coordinates of a point in the world space be P world = [X world , Y xorld , z world , 1] T , then after the observation change, the coordinates of the point become P view = [x view , y view , z view , 1] T , and after the projection transformation and device coordinate normalization transformation, the coordinates are P clip and P NDC respectively. Then we have:

[0140] According to formula (36), we have clip w = z view(37)

[0141] wherein z view is the depth information of the current point in the observation space.

[0142] As can be seen from the above, the motion vector prediction model only needs to solve two unknown parameters to obtain the correspondence between the current point and the matching point between two frames, and thus the unknown parameters solved by the model are the respective depth values of the current matching point in the observation space under different motion states of the camera. Under the assumption that the coding has the same depth, the depth value of the coding block can be solved from a pair of matching points. Therefore, formula (30) can be written as:

[0143] The above embodiment can predict the coordinates of the corresponding pixel points in the next video frame according to the coordinates of the pixel points in the current video frame, and thus the motion vector prediction model of the coding block in which the depth information of the pixel points is uniform can be expressed as formula (38).

[0144] In some embodiments, the present embodiment provides a motion vector prediction model of a coding block in which the depth information of the pixel points is uniform.

[0145] A geometric plane is a basic graphic element in space and is also an element frequently encountered in three-dimensional animation. Therefore, in some cases, the parts with different depths in a three-dimensional scene can be regarded as a plane in which the depth changes uniformly, and a plane equation is established by taking the depth values of the pixel points in the video frame as the change amount, so that the depth information can participate in the process of motion vector prediction, thereby improving the accuracy of motion vector prediction, reducing the residual error, and finally reducing the code rate.

[0146] In a three-dimensional space, any plane Γ can be expressed by a general parameter equation as follows:

[0147] wherein A, B, C and D are constants, and at least one of them is non-zero, is the coordinates of the point, that is, any point satisfying equation (39) is on the plane Γ.

[0148] Referring to FIG. 7, in a region 70 in which the depth changes uniformly, the depth of the upper left corner the depth of the upper right corner the depth of the lower left corner the depth value is set to 1, and the origin of the coordinate system is translated to the projection of the upper left corner control point on the depth plane of the camera, and thus the depth of the current plane can be expressed by the plane equation (40).

[0149] Taking partial derivatives of x, y respectively and using depth, the plane equation (40) can be written as:

[0150] The formula (41) can be arranged to get the depth value of each pixel point in the camera coordinate system as follows:

[0151] where x, y are the coordinates of the current point relative to the top-left control point of the current block, is the depth relative to the top-left control point of the current block, w, h are the width and height of the current block respectively.

[0152] Let λ1=x / w, λ2=y / h, then the above formula (42) can be expressed as follows:

[0153] According to the premise assumption, the derivation of the plane equation is based on a three-dimensional scene, so the coordinate system is established based on the camera space, that is, the observation space. Therefore, is the observation space depth

[0154] Let: According to formula (38), for any point k in the current block, we have:

[0155] According to formula (35), we have:

[0156] Combining formula (43) and formula (44), we have:

[0157] where α k represents the observation space depth of the point k in the current block β k represents the observation space depth of the matching point corresponding to the point k in the reference frame

[0158] As shown in FIG. 8, according to the observation space depths α1, α2, α3 of the top-left control point, the top-right control point and the bottom-left control point of the current block 81, the observation space depths β1, β2, β3 of the predicted point of the top-left control point, the predicted point of the top-right control point and the predicted point of the bottom-left control point in the reference frame 82, the observation space depth α of any point in the current block 81 and the observation space depth β of the corresponding matching point in the reference frame can be obtained.

[0159] According to the above derivation result, in a conventional motion sequence, when the pixel coordinates P(u 1 ,v 1) can be calculated according to the following formula (48) to obtain the pixel coordinates P'(u 2 ,v 2 ) of the corresponding matching point in the reference frame:

[0160] Wherein:

[0161] In some embodiments, the motion vector prediction model of the coding block with uniform depth information change can be represented by the above formula (46)-(49). In some embodiments, the pixel coordinates P'(u 1 ,v 1 ) of the corresponding matching point in the reference frame can be calculated by the above formula (46)-(49) and the pixel coordinates P(u 2 ,v 2 ) in the current frame, so as to obtain the motion vector of each pixel point in the current block by P(u 1 ,v 1 ) and P'(u 2 ,v 2 ).

[0162] Since the depth information is included in the calculation, the motion vector prediction model will be more suitable for the projection transformation existing in the three-dimensional animation video.

[0163] In some embodiments, the present application provides a motion vector prediction model of a coding block of a coding block with pure rotation motion of a camera.

[0164] When the camera only rotates, the absolute position of the camera in the world space does not change, and the translation matrix T remains unchanged in all data in the pure rotation sequence, so there is:

[0165] Wherein, t x ,t y , represents the absolute position of the camera in the world space.

[0166] [Rule 91, 04.12.2025] According to formula (24), in formula (48), only P2 and Q2 are related to the translation matrix, that is:

[0167] Wherein, R 11 ,R 12 ,R 13 ,R 21 ,R 22 ,R 23 ,R 31 ,R 32 ,R 33are the elements of the absolute rotation matrix of the camera relative to the initial direction at each position, respectively, t1, t2, t3are the values of the absolute coordinate position of the camera in the world space after transformation by the rotation matrix, respectively.

[0168] From the above equation (50), we have:

[0169] Therefore, t1, t2, t3may be obtained from t x , t y , may be expressed as:

[0170] Substituting equation (53) into equation (51), and combining and simplifying according to the characteristics of the rotation matrix, we have:

[0171] From the above and equation (50), when the camera only has rotational motion, t x , t y , corresponding to the current frame and the reference frame are constant, so for equation (48), in the case of pure rotation, it can be simplified as:

[0172] In equation (55), there is only one unknown quantity θ, so solving θ can calculate the coordinates of the matching point according to the coordinates of the current point in the current frame. In some embodiments, the vector prediction model in the pure rotation case can be represented as equation (55).

[0173] From the above equation (35), we have:

[0174] where A 1 , A 2 are the determinants of the rotation matrix of the current frame and the reference frame, respectively.

[0175] Since in the case of pure rotation, the determinant of the rotation matrix is always 1, we have:

[0176] In some embodiments, θ is the ratio of the normalization factors used in the normalization of the matching point pair in the current frame and the reference frame.

[0177] According to equation (38), the relationship between the coordinates of the current point and the matching point can be established for any pair of matching points between any two frames. When the camera only has rotational motion, the observation matrix V and the projection matrix K can be represented as:

[0178] Therefore, equation (23) in the case of pure rotation can be represented as:

[0179] For each of X, Y, 1 in formula (59), the following can be obtained after correlation and arrangement:

[0180] Wherein:

[0181] In combination with formula (60) and (61), the following can be obtained:

[0182] Wherein:

[0183] From formula (62) and (63), it can be obtained that by using the pixel coordinates of the known points on the current frame and the extrinsic information of the camera of the current frame and the reference frame, θ can be obtained. After substituting θ into the expression (55) of the motion vector prediction model in the case of pure rotation, the vector prediction model does not include any depth-related information, and there is no unknown quantity. Therefore, for the special motion mode of pure rotation, only the pose information and intrinsic information of the camera are used, and the prediction point can be obtained according to the current point, and finally the pixel-level motion vector is obtained.

[0184] H.266 / VVC further adds an Affine Merge mode on the basis of the conventional Merge of H.265 / HEVC, so that the coding performance is further improved. The motion vector prediction model provided in the above embodiments can be supplemented on this basis, more motion modes are adapted, and the motion vector is more accurately predicted, so that the overall performance is improved, and the code rate is reduced to a certain extent.

[0185] Further, when using the above motion vector prediction model for actual encoding, if the motion vector prediction model constructed based on the same depth of each pixel point of the current block or the motion vector prediction model constructed based on the uniform change of the depth of each pixel point of the current block is used for encoding, the mapping relationship between the pixel points in the current block and the pixel points in the reference frame needs to be determined by using the motion vector of the control point. However, the motion information of the control point usually has a certain error at 1 / 16 precision, and the error will gradually accumulate to form a more serious error in the subsequent calculation process, so in some embodiments, when obtaining the motion information of the control point, a control point compensation technology based on optical flow compensation can also be used to make the calculation of the motion vector prediction model as accurate as possible.

[0186] In some embodiments, as shown in FIG. 9, in the process of constructing the motion vector list, when obtaining the motion vector of the control point from the spatially adjacent coding block, it is no longer only to obtain the motion vector of the center point 91 of the adjacent 4x4 coding block, but to calculate the distance from the center point 91 of the adjacent coding block to the control point, and to compensate the motion vector of the center point 91 of the adjacent coding block based on the distance from the center point 91 of the adjacent coding block to the control point, so as to accurately calculate the motion vector of the current vertex through the motion vector prediction model of the control point at the 16-pixel precision. Compared with the motion vector of the center point 91 of the adjacent 4x4 coding block, the motion vector of the control point is more accurate, so that the calculation of the motion vector of each pixel point in the subsequent coding block is also more accurate, thereby reducing the data amount of the residual of the current block.

[0187] On the basis of the above, the embodiment of the present application provides a video decoding method, as shown in FIG. 10, the video decoding method comprises the following steps:

[0188] S101, obtaining the coding data of a current block.

[0189] Wherein, the current block is any coding unit in a video sequence.

[0190] In some embodiments, obtaining the coding data of the current block comprises: receiving the coding data of the current block sent by a media resource server.

[0191] S102, obtaining a residual block of the current block and a reference frame of a current frame according to the coding data of the current block.

[0192] Wherein, the current frame is a video frame containing the current block.

[0193] S103, obtaining camera parameters of the current frame and camera parameters of the reference frame.

[0194] In some embodiments, the camera parameters of the current frame and the camera parameters of the reference frame comprise one or more of camera extrinsic parameters of the current frame, camera extrinsic parameters of the reference frame and camera intrinsic parameters. In some embodiments, the camera parameters of the current frame and the camera parameters of the reference frame comprise camera extrinsic parameters of the current frame, camera extrinsic parameters of the reference frame, a first parameter and a second parameter. In some embodiments, the camera intrinsic parameters comprise the first parameter and / or the second parameter, wherein the first parameter is a ratio of the cotangent of half angle of field of view of a camera to an aspect ratio of the camera, and the second parameter is the cotangent of half angle of field of view of the camera.

[0195] In some embodiments, the camera parameters of the current frame and the camera parameters of the reference frame can be obtained through corresponding interfaces. Wherein, the camera extrinsic parameters of the current frame and the camera extrinsic parameters of the reference frame can be obtained through corresponding interfaces, and the camera intrinsic parameters can be obtained from the engine through corresponding interfaces.

[0196] In some embodiments, the camera extrinsic parameters of the current frame and the camera extrinsic parameters of the reference frame can be obtained through corresponding interfaces, and the camera intrinsic parameters can be obtained based on the camera extrinsic parameters of the current frame and the camera extrinsic parameters of the reference frame.

[0197] In some embodiments, the camera extrinsic parameters of the video frames can be obtained through corresponding interfaces, the video frames can include the first video frame and the second video frame; and the camera intrinsic parameters can be estimated based on the camera extrinsic parameters of the first video frame and the camera extrinsic parameters of the second video frame.

[0198] The estimation of the camera intrinsic parameters can refer to the following steps a to d:

[0199] Step a, obtaining the first video frame and the second video frame, the first video frame being a video frame in the video sequence, and the camera pose corresponding to the first video frame only having rotational motion relative to the camera pose corresponding to the second video frame, that is, extracting two video frames with pure rotational motion of the camera from the video sequence.

[0200] It should be noted that the first video frame and the second video frame can be adjacent video frames, or can be non-adjacent video frames, and the embodiments of the present application do not limit this, but only require that the camera pose corresponding to the first video frame only has rotational motion relative to the camera pose corresponding to the second video frame.

[0201] Step b, determining the first pixel coordinates and the second pixel coordinates.

[0202] Wherein, the first pixel coordinates are the pixel coordinates of a first pixel point in the first video frame, and the second pixel coordinates are the pixel coordinates of a second pixel point matched with the first pixel point in the second video frame.

[0203] In some embodiments, the matching of the second pixel point with the first pixel point means that the second pixel point is the same point as the first pixel point in the world space.

[0204] In some embodiments, a pair of pixel points matched with each other in the first video frame and the second video frame can be obtained through the ORB algorithm, and the pixel coordinates of the pair of pixel points can be determined as the first pixel coordinates and the second pixel coordinates, respectively.

[0205] Step c, obtaining the first parameter and the second parameter according to the camera extrinsic parameter of the first video frame, the camera extrinsic parameter of the second video frame, the first pixel coordinate and the second pixel coordinate.

[0206] In some embodiments, the camera intrinsic parameter is obtained according to the camera extrinsic parameter of the first video frame, the camera extrinsic parameter of the second video frame, the first pixel coordinate and the second pixel coordinate.

[0207] In some embodiments, the first parameter and the second parameter are obtained according to the camera extrinsic parameter of the first video frame, the camera extrinsic parameter of the second video frame, the first pixel coordinate and the second pixel coordinate, including steps c1 to c3 as follows:

[0208] Step c1, constructing the observation matrix of the first video frame and the observation matrix of the second video frame according to the camera extrinsic parameter of the first video frame and the camera extrinsic parameter of the second video frame respectively.

[0209] Step c2, determining the camera intrinsic parameter according to the observation matrix of the first video frame, the observation matrix of the second video frame, the first pixel coordinate and the second pixel coordinate.

[0210] In some embodiments, the following constraint equation is constructed according to the observation matrix of the first video frame, the observation matrix of the second video frame, the first pixel coordinate and the second pixel coordinate:

[0211] wherein, is the element of the mth row and the nth column in the observation matrix of the first video frame, is the element of the mth row and the nth column in the observation matrix of the second video frame, f1 and f2 are the first parameter and the second parameter respectively, u 1 and v 1 are the horizontal coordinate and the vertical coordinate of the first pixel coordinate respectively, u 2 and v 2 are the horizontal coordinate and the vertical coordinate of the second pixel coordinate respectively.

[0212] The implementation and principle of the above equation (64) constructed according to the observation matrix of the first video frame, the observation matrix of the second video frame, the first pixel coordinate and the second pixel coordinate can refer to the above equations (1) to (14), and the description is not repeated here to avoid repetition.

[0213] Step c3, determining the first parameter and the second parameter according to min((A-B) 2 +(C-B) 2 +(A-C) 2 ).

[0214] In some embodiments, the camera intrinsic is determined according to min((A-B) 2 +(C-B) 2 +(A-C) 2 .

[0215] In some embodiments, the minimum value of (A-B) 2 +(C-B) 2 +(A-C) 2 is obtained, and f1 and f2 at which the value of (A-B) 2 +(C-B) 2 +(A-C) 2 is the minimum are determined as the first parameter and the second parameter respectively.

[0216] In some embodiments, the minimum value of (A-B) 2 +(C-B) 2 +(A-C) 2 is obtained, and f1 and f2 at which the value of (A-B) 2 +(C-B) 2 +(A-C) 2 is the minimum are determined as the camera intrinsic.

[0217] S104, according to the camera parameters of the current frame and the camera parameters of the reference frame, obtaining the mapping relationship between the pixel points in the current block and the pixel points in the reference frame.

[0218] In some embodiments, according to the camera parameters of the current frame and the camera parameters of the reference frame, a motion vector prediction model is obtained.

[0219] Since the mapping relationship between the pixel points in the current block and the pixel points in the reference frame can determine the corresponding prediction points of the pixel points in the current block in the reference frame, and thus the motion vector of the pixel points in the current block, the mapping relationship between the pixel points in the current block and the pixel points in the reference frame can also indicate the motion vector prediction model of the current block.

[0220] In some embodiments, according to the camera parameters of the current frame and the camera parameters of the reference frame, the mapping relationship between the pixel points in the current block and the pixel points in the reference frame is obtained, including: obtaining the motion vector (Control Point Motion Vector, CPMV) of the control points of the current block; according to the motion vector of the control points of the current block, the camera parameters of the current frame and the camera parameters of the reference frame, the mapping relationship between the pixel points in the current frame and the pixel points in the reference frame is obtained.

[0221] In some embodiments, the implementation of obtaining the motion vector of the control point of the current block can include:

[0222] obtaining an index value of a motion vector prediction model according to the encoding data of the current block, obtaining a prediction model of the motion vector of the control point of the current block according to the index value of the motion vector prediction model and the motion vector prediction model candidate list, and predicting the motion vector of the current control point according to the prediction model of the motion vector of the control point of the current block to obtain the motion vector of the control point of the current block.

[0223] In some embodiments, the motion vector of the control point of the current block is obtained according to the spatial reference motion vector, and the spatial reference motion vector is compensated according to the distance from the center of the image block corresponding to the spatial reference motion vector to the control point of the current block to obtain the motion vector of the control point of the current block.

[0224] Compared with directly taking the motion vector of the adjacent coded block as the motion vector of the control point of the current block, the above embodiments can compensate the motion vector of the adjacent coded block according to the distance from the center of the adjacent coded block to the control point of the current block, and take the compensated motion vector as the motion vector of the current block, so that the above embodiments can more accurately obtain the motion vector of the control point of the current block, thereby reducing the data amount of residual data.

[0225] In some embodiments, the mapping relationship between the pixel point in the current frame and the pixel point in the reference frame is obtained according to the motion vector of the control point of the current block, the camera parameter of the current frame and the camera parameter of the reference frame, including the following steps 1 and 2:

[0226] Step 1: obtaining the pixel coordinates of the prediction point of the target control point of the current block according to the motion vector of the control point of the current block.

[0227] The target control point is any control point of the current block.

[0228] Step 2: obtaining the mapping relationship between the pixel point in the current frame and the pixel point in the reference frame according to the pixel coordinates of the target control point, the pixel coordinates of the prediction point of the target control point, the camera parameter of the current frame and the camera parameter of the reference frame.

[0229] In some embodiments, the above step 2 (obtaining the mapping relationship between the pixel point in the current frame and the pixel point in the reference frame according to the pixel coordinates of the target control point, the pixel coordinates of the prediction point of the target control point, the camera parameter of the current frame and the camera parameter of the reference frame) includes the following steps 21 to 22:

[0230] Step 21, generating a first observation matrix and a second observation matrix according to the camera extrinsic parameters of the current frame and the camera extrinsic parameters of the reference frame respectively.

[0231] Step 22, determining the motion vector prediction model, i.e. the mapping relationship between the pixel points in the current frame and the pixel points in the reference frame, according to the first observation matrix, the second observation matrix, the first parameter and the second parameter.

[0232] In some embodiments, the following first relationship can be constructed according to the first observation matrix, the second observation matrix, the first parameter and the second parameter:

[0233] [Rule 91, 04.12.2025] Wherein,

[0234] is the element of the first observation matrix in the mth row and the nth column, is the element of the first observation matrix in the mth row and the fourth column, is the element of the second observation matrix in the mth row and the nth column, is the element of the second observation matrix in the mth row and the fourth column, f1 and f2 are the first parameter and the second parameter respectively, u 1 and v 1 are the horizontal coordinate and the vertical coordinate of the pixel coordinates of the pixel points in the current block respectively, u 2 and v 2 are the horizontal coordinate and the vertical coordinate of the pixel coordinates in the reference frame respectively; is the second coordinate component of the pixel points in the current block in the clipping space, is the second coordinate component of the pixel points in the reference frame in the clipping space.

[0235] The implementation and principle of constructing the first relationship according to the first observation matrix, the second observation matrix, the first parameter and the second parameter can refer to the above formula (15) to formula (38), and details are not repeated here to avoid repetition.

[0236] Step 22, substituting the pixel coordinates of the target control point and the pixel coordinates of the prediction point of the target control point into the first relationship to obtain the second coordinate component of the pixel points in the current block in the clipping space and the second coordinate component of the corresponding pixel points in the reference frame in the clipping space, so as to obtain the mapping relationship between the pixel points in the current frame and the pixel points in the reference frame.

[0237] In some embodiments, the second coordinate component in the clipping space of a pixel point in the current block and the second coordinate component in the clipping space of a corresponding pixel point in the reference frame can be determined by the pixel coordinate of the target control point, the pixel coordinate of the prediction point of the target control point, and the first relationship, and then the motion vector prediction model can be determined.

[0238] There are only two unknown quantities in the above formula (67) and , so it can be understood as a binary linear equation of and . By substituting the pixel coordinate of the target control point and the pixel coordinate of the prediction point of the target control point into the first relationship, the values of and can be calculated. After the values of and are calculated, for any pixel point in the current block, the pixel coordinate (u 1 ,v 1 ) of the pixel point can be substituted into the above formula (67) to calculate the corresponding pixel coordinate (u 2 ,v 2 ) in the reference frame, and then the motion vector of the pixel point can be calculated according to (u 1 ,v 1 ) and (u 2 ,v 2 ).

[0239] In some embodiments, by substituting the pixel coordinate of the target control point and the pixel coordinate of the prediction point of the target control point into the first relationship, the values of and can be calculated. After the values of and are obtained, the motion vector prediction model can be determined according to formula (67). Based on the motion vector prediction model, for the pixel coordinate (u 1 ,v 1 ) of any pixel point in the current block, the corresponding pixel coordinate (u 2 ,v 2 ) in the reference frame can be predicted, so as to obtain the motion vector of the pixel point.

[0240] In some embodiments, according to the motion vector of the control point of the current block, the camera parameters of the current frame, and the camera parameters of the reference frame, the mapping relationship between the pixel point in the current frame and the pixel point in the reference frame is obtained, including the following steps ① and ②:

[0241] Step ①, determining a prediction point of a top-left control point, a prediction point of a top-right control point and a prediction point of a bottom-left control point of the current block according to motion vectors of the control points of the current block.

[0242] Step ②, constructing a second relationship formula according to depths of the top-left control point, the top-right control point and the bottom-left control point of the current block, depths of the prediction point of the top-left control point, the prediction point of the top-right control point and the prediction point of the bottom-left control point in the reference frame.

[0243] wherein, α1, α2, α3 are respectively the depth of the top-left control point, the depth of the top-right control point and the depth of the bottom-left control point of the current block, β1, β2, β3 are respectively the depth of the prediction point of the top-left control point, the depth of the prediction point of the top-right control point and the depth of the prediction point of the bottom-left control point in the reference frame, λ1=x / w, λ2=y / h, x, y are coordinates of a pixel point in the current block relative to the top-left control point of the current block, w, h are width and height of the current block.

[0244] It should be noted that the depth of the top-left control point, the depth of the top-right control point and the depth of the bottom-left control point of the current block, the depth of the prediction point of the top-left control point, the depth of the prediction point of the top-right control point and the depth of the prediction point of the bottom-left control point in the reference frame in step ② are all depth values in the observation space.

[0245] The implementation manner and principle of constructing a third relationship formula according to the depth of the top-left control point, the depth of the top-right control point and the depth of the bottom-left control point of the current block, the depth of the prediction point of the top-left control point, the depth of the prediction point of the top-right control point and the depth of the prediction point of the bottom-left control point in the reference frame can refer to the above formula (39) to formula (49) for the sake of avoiding redundancy, which will not be described herein.

[0246] Step ③, obtaining a mapping relationship between a pixel point in the current frame and a pixel point in the reference frame according to the second relationship formula, camera parameters of the current frame and camera parameters of the reference frame.

[0247] In some embodiments, the above step ③ (obtaining a mapping relationship between a pixel point in the current frame and a pixel point in the reference frame according to the second relationship formula, camera parameters of the current frame and camera parameters of the reference frame) comprises the following steps ③1 to ③3:

[0248] Step ③1, generating a first observation matrix and a second observation matrix according to camera extrinsic parameters of the current frame and camera extrinsic parameters of the reference frame respectively.

[0249] Step ③2, constructing a third relationship according to the first observation matrix, the second observation matrix, the first parameter and the second parameter as follows:

[0250] [Rule 91, 04.12.2025] Wherein,

[0251] is an element of the mth row and nth column of the first observation matrix, is an element of the mth row and fourth column of the first observation matrix, is an element of the mth row and nth column of the second observation matrix, is an element of the mth row and fourth column of the second observation matrix, f1 and f2 are the first parameter and the second parameter respectively, u 1 and v 1 are the horizontal coordinate and the vertical coordinate of the pixel coordinates of the pixel point in the current block respectively, u 2 and v 2 are the horizontal coordinate and the vertical coordinate of the pixel coordinates in the reference frame respectively; a is the inverse of the second coordinate component of the pixel point in the current block in the clipping space, B is the inverse of the second coordinate component of the pixel point in the reference frame in the clipping space.

[0252] The implementation and principle of constructing the third relationship according to the first observation matrix, the second observation matrix, the first parameter and the second parameter can refer to the above formula (15) to formula (38) to avoid repetition, which will not be repeated here.

[0253] Step ③, obtaining the mapping relationship between the pixel point in the current frame and the pixel point in the reference frame according to the second relationship and the third relationship.

[0254] For any pixel point in the current block, the pixel coordinates (u 1 ,v 1 ) of the pixel point can be substituted into the above formula (68) to calculate a and β, and then (u 1 ,v 1 ) a and β can be substituted into the above formula (70) to calculate (u 2 ,v 2 ), and then the motion vector of the pixel point can be calculated according to (u 1 ,v 1 ) and (u 2 ,v 2 ).

[0255] In some embodiments, formula (68) and (71) can also be arranged into a relationship, and (u1 , 1 substitute the relationship formula after arrangement into (u 2 , 2 ), and then calculate the motion vector of the pixel point according to (u 1 , 1 ) and (u 2 , 2 ).

[0256] In some embodiments, after determining the mapping relationship of the pixel point in the current frame and the pixel point in the reference frame, the motion vector prediction model is determined. For any pixel point in the current block, according to the motion vector prediction model, the matching point of the pixel point in the reference frame can be obtained, so as to determine the motion vector of the pixel point.

[0257] In some embodiments, according to the camera parameters of the current frame and the camera parameters of the reference frame, the mapping relationship of the pixel point in the current block and the pixel point in the reference frame is obtained, including the following steps I and step II:

[0258] Step I, respectively according to the camera extrinsic parameters of the current frame and the camera extrinsic parameters of the reference frame, a first observation matrix and a second observation matrix are generated.

[0259] Step II, according to the first observation matrix, the second observation matrix, the first parameter and the second parameter, the following fourth relationship formula is constructed:

[0260] [Rule 91, 04.12.2025] Wherein,

[0261] is the element of the mth row and the nth column in the first observation matrix, is the element of the mth row and the nth column in the second observation matrix, f1 and f2 are the first parameter and the second parameter respectively, u 1 and v 1 are the horizontal coordinate and the vertical coordinate of the pixel coordinate of the pixel point in the current block, u 2 and v 2 are the horizontal coordinate and the vertical coordinate of the pixel coordinate of the pixel point in the reference frame.

[0262] The implementation manner and principle of constructing the fourth relationship formula according to the first observation matrix, the second observation matrix, the first parameter and the second parameter can refer to the above formula (15) to formula (38), and formula (50) to formula (63), and the description is not repeated here to avoid repetition.

[0263] Step III, determining the fourth relationship as a motion vector prediction model of the current block.

[0264] S105, obtaining a prediction block of the current block according to the mapping relationship and the reference frame.

[0265] In some embodiments, obtaining the prediction block of the current block according to the mapping relationship and the reference frame comprises: obtaining a prediction pixel point of each pixel point in the current block from the reference frame according to the mapping relationship, and combining the prediction pixel points of each pixel point in the current block into the prediction block of the current block.

[0266] In some embodiments, the embodiments of the present disclosure can also obtain the prediction block of the current block according to the motion vector prediction model and the reference frame. The present disclosure can determine the prediction pixel point of each pixel point in the current block through the motion vector prediction model, and combine the prediction pixel points of each pixel point in the current block into the prediction block of the current block.

[0267] S106, reconstructing the current block according to the residual block of the current block and the prediction block of the current block to obtain a reconstructed block corresponding to the current block.

[0268] In some embodiments, reconstructing the current block according to the residual block of the current block and the prediction block of the current block to obtain a reconstructed block corresponding to the current block comprises:

[0269] summing values of corresponding positions in the residual block of the current block and the prediction block of the current block to obtain the reconstructed block corresponding to the current block.

[0270] The video coding method provided in the embodiments of the present application comprises the following steps: obtaining the residual block of the current block and the reference frame of the current frame according to the coding data of the current block; obtaining the camera parameters of the current frame and the camera parameters of the reference frame; determining the mapping relationship or the motion vector prediction model between the pixel points in the current block and the pixel points in the reference frame according to the camera parameters of the current frame and the camera parameters of the reference frame; obtaining the prediction block of the current block according to the mapping relationship or the motion vector prediction model and the reference frame; and reconstructing the current block according to the residual block of the current block and the prediction block of the current block to obtain the reconstructed block corresponding to the current block. Since the video coding method provided in the embodiments of the present application can obtain the mapping relationship or the motion vector prediction model between the pixel points in the current block and the pixel points in the reference frame according to the camera parameters of the current frame and the camera parameters of the reference frame, the motion vector of each pixel point in the current block can be accurately obtained according to the mapping relationship or the motion vector prediction model between the pixel points in the current block and the pixel points in the reference frame, and thus the motion information of the object can be more accurately described, thereby improving the quality of the reconstructed block of the current block.

[0271] An embodiment of the present application provides a video coding method, which comprises the following steps with reference to FIG. 11.

[0272] S111, determining the reference frame of the current frame.

[0273] The current frame is a video frame comprising a current block.

[0274] S112, obtaining the camera parameters of the current frame and the camera parameters of the reference frame.

[0275] In some embodiments, the camera parameters of the current frame and the camera parameters of the reference frame can comprise the camera extrinsic parameters of the current frame, the camera extrinsic parameters of the reference frame, a first parameter and a second parameter.

[0276] In some embodiments, the camera parameters of the current frame and the camera parameters of the reference frame can comprise the camera extrinsic parameters of the current frame, the camera extrinsic parameters of the reference frame, and camera intrinsic parameters.

[0277] In some embodiments, the camera intrinsic parameters can comprise the first parameter and / or the second parameter, wherein the first parameter is the ratio of the cotangent of the half angle of the field of view of the camera to the aspect ratio of the camera, and the second parameter is the cotangent of the half angle of the field of view of the camera.

[0278] The implementation manners of obtaining the camera extrinsic parameters of the current frame, the camera extrinsic parameters of the reference frame, the first parameter and the second parameter can refer to the above embodiments, which will not be repeated here.

[0279] S113, obtaining a mapping relationship between the pixel points in the current block and the pixel points in the reference frame according to the camera parameter of the current frame and the camera parameter of the reference frame.

[0280] In some embodiments, the motion vector prediction model can indicate the mapping relationship between the pixel points in the current block and the pixel points in the reference frame, that is, the corresponding pixel points of the pixel points in the current block in the reference frame can be determined through the motion vector prediction model.

[0281] The implementation of obtaining the mapping relationship between the pixel points in the current block and the pixel points in the reference frame according to the camera parameter of the current frame and the camera parameter of the reference frame can refer to the above-mentioned embodiments, which will not be described in detail here.

[0282] S114, obtaining a prediction block of the current block according to the mapping relationship between the pixel points in the current block and the pixel points in the reference frame.

[0283] In some embodiments, the prediction pixel points of each pixel point in the current block are obtained from the reference frame according to the mapping relationship, and the prediction pixel points of each pixel point in the current block are combined into the prediction block of the current block.

[0284] S115, obtaining a residual block of the current block according to the current block and the prediction block.

[0285] In some embodiments, obtaining the residual block of the current block according to the current block and the prediction block comprises:

[0286] obtaining the residual block corresponding to the current block by calculating the residual of the values at the corresponding positions in the current block and the prediction block of the current block.

[0287] S116, obtaining the encoding data of the current block according to the residual block of the current block.

[0288] In some embodiments, some embodiments of the present application provide a video encoding device, which comprises:

[0289] a memory configured to store a computer program;

[0290] a processor configured to, when the computer program is invoked, enable the video encoding device to implement the video encoding method in any of the above-mentioned embodiments.

[0291] In some embodiments, some embodiments of the present application provide a video decoding device, which comprises:

[0292] a memory configured to store a computer program;

[0293] a processor configured to cause the video decoding apparatus to implement the video decoding method according to any one of the preceding embodiments when a computer program is invoked.

[0294] In some embodiments, some embodiments provide a computer readable storage medium, having stored thereon a computer program, which, when executed by a computing device, causes the computing device to implement the video encoding method or the video decoding method according to any one of the preceding embodiments.

[0295] In some embodiments, some embodiments provide a computer program product, which, when running on a computer, causes the computer to implement the video encoding method or the video decoding method according to any one of the preceding embodiments.

[0296] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0297] In order to facilitate explanation, the above description has been made in combination with specific embodiments. However, the above exemplary discussion is not intended to exhaust or limit the embodiments to the specific forms disclosed above. Various modifications and variations can be derived according to the above teachings. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different modified embodiments suitable for specific use considerations.

Claims

1. A method of video decoding, the method comprising: The method comprises: obtaining encoding data of a current block; the current block is any coding unit in a video sequence; obtaining a residual block of the current block and a reference frame of a current frame according to the encoding data of the current block; the current frame is a video frame containing the current block; obtaining camera parameters of the current frame and camera parameters of the reference frame; determining a motion vector prediction model according to the camera parameters of the current frame and the camera parameters of the reference frame; obtaining a prediction block of the current block based on the motion vector prediction model; obtaining a reconstructed block corresponding to the current block according to the residual block of the current block and the prediction block of the current block.

2. The method of claim 1, wherein, The camera parameters include camera intrinsic parameters, wherein the obtaining of the camera parameters of the current frame and the camera parameters of the reference frame comprises: obtaining camera extrinsic parameters of the current frame and camera extrinsic parameters of the reference frame; obtaining the camera intrinsic parameters according to the camera extrinsic parameters of the video frames in the video sequence.

3. The method of claim 2, wherein, The camera intrinsic parameters include a ratio of a half-angle cotangent of a field of view angle of the camera and an aspect ratio of the camera and / or a half-angle cotangent of a field of view angle of the camera. The obtaining of the camera intrinsic parameters comprises:

4. The method of claim 2, wherein, obtaining a first video frame and a second video frame, the first video frame being a video frame in the video sequence, and a camera pose corresponding to the first video frame only having a rotation motion relative to a camera pose corresponding to the second video frame; obtaining a first pixel coordinate and a second pixel coordinate, the first pixel coordinate being a pixel coordinate of a first pixel point in the first video frame, and the second pixel coordinate being a pixel coordinate of a second pixel point matched with the first pixel point in the second video frame; obtaining the camera intrinsic parameters according to the camera extrinsic parameters of the first video frame, the camera extrinsic parameters of the second video frame, the first pixel coordinate, and the second pixel coordinate. The obtaining of the camera intrinsic parameters according to the camera extrinsic parameters of the first video frame, the camera extrinsic parameters of the second video frame, the first pixel coordinate, and the second pixel coordinate comprises:

5. The method of claim 4, wherein, constructing an observation matrix of the first video frame and an observation matrix of the second video frame according to the camera extrinsic parameters of the first video frame and the camera extrinsic parameters of the second video frame respectively; determining the camera intrinsic parameters according to the observation matrix of the first video frame, the observation matrix of the second video frame, the first pixel coordinate, and the second pixel coordinate. The determination of the camera intrinsic parameters according to the observation matrix of the first video frame, the observation matrix of the second video frame, the first pixel coordinate, and the second pixel coordinate comprises:

6. The method of claim 5, wherein, The determination of the motion vector prediction model according to the camera parameters of the current frame and the camera parameters of the reference frame comprises: Constructing the constraint function: wherein, for an element of an mth row and nth column of an observation matrix for the first video frame, for an element of the mth row and nth column of the observation matrix of the second video frame, f1 and f2 are the first parameter and the second parameter respectively, u 1 and v 1 are the horizontal coordinate and the vertical coordinate of the first pixel coordinate respectively, u 2 and v 2 are the horizontal coordinate and the vertical coordinate of the second pixel coordinate respectively; The camera intrinsic parameters are determined according to min((A-B) 2 +(C-B) 2 +(A-C) 2 ).

7. The method of claim 2, wherein, obtaining a control point motion vector CPMV of the current block; determining the motion vector prediction model according to the control point motion vector of the current block, the camera parameters of the current frame, and the camera parameters of the reference frame, wherein the motion vector prediction model includes a mapping relationship between a pixel point in the current frame and a pixel point in the reference frame. ​ 8. The method of claim 7, wherein, The motion vector prediction model is determined according to the motion vector of the control point of the current block, the camera parameter of the current frame and the camera parameter of the reference frame, and the motion vector prediction model comprises: According to the motion vector of the control point of the current block, the pixel coordinates of the target control point of the current block are obtained; the target control point is any control point of the current block; According to the pixel coordinates of the target control point, the pixel coordinates of the prediction point of the target control point, the camera parameter of the current frame and the camera parameter of the reference frame, the motion vector prediction model is determined.

9. The method of claim 8, wherein, The motion vector prediction model is determined according to the pixel coordinates of the target control point, the pixel coordinates of the prediction point of the target control point, the camera parameter of the current frame and the camera parameter of the reference frame, and the motion vector prediction model comprises: First observation matrix and second observation matrix are generated according to the camera external parameter of the current frame and the camera external parameter of the reference frame respectively; The motion vector prediction model is determined according to the first observation matrix, the second observation matrix, the camera internal parameter, the pixel coordinates of the target control point and the pixel coordinates of the prediction point of the target control point.

10. The method according to claim 9, characterized in that The motion vector prediction model is determined according to the first observation matrix, the second observation matrix, the camera internal parameter, the pixel coordinates of the target control point and the pixel coordinates of the prediction point of the target control point, and the motion vector prediction model comprises: constructing a first relationship according to the first observation matrix, the second observation matrix and the camera intrinsic parameter; wherein, for an element in the mth row and nth column of the first observation matrix, for the element of the mth row and fourth column of the first observation matrix, for an element in the mth row and nth column of the second observation matrix, for the element in the mth row and 4th column of the second observation matrix, f1 and f2 are the first parameter and the second parameter respectively, u 1 and v 1 are the horizontal coordinate and the vertical coordinate of the pixel coordinate of the pixel point in the current block respectively, u 2 and v 2 are the horizontal coordinate and the vertical coordinate of the pixel coordinate in the reference frame respectively; for a second coordinate component in the clipping space of a pixel point in the current block, The second coordinate component of the pixel point in the reference frame in the clipping space is obtained; The pixel coordinates of the target control point and the pixel coordinates of the prediction point of the target control point are substituted into the first relationship to obtain the second coordinate component of the pixel point in the current block in the clipping space and the second coordinate component of the corresponding pixel point in the reference frame in the clipping space, and the mapping relationship between the pixel point in the current frame and the pixel point in the reference frame is determined; The motion vector prediction model is determined based on the mapping relationship between the pixel point in the current frame and the pixel point in the reference frame.

11. The method of claim 7, wherein, The motion vector prediction model is determined according to the motion vector of the control point of the current block, the camera parameter of the current frame and the camera parameter of the reference frame, and the motion vector prediction model comprises: The prediction point of the top-left corner control point, the prediction point of the top-right corner control point and the prediction point of the bottom-left corner control point of the current block are determined according to the motion vector of the control point of the current block; The motion vector prediction model is determined according to the depth of the top-left corner control point, the depth of the top-right corner control point, the depth of the bottom-left corner control point of the current block, the depth of the prediction point of the top-left corner control point, the depth of the prediction point of the top-right corner control point, the depth of the prediction point of the bottom-left corner control point in the reference frame, the camera parameter of the current frame and the camera parameter of the reference frame.

12. The method of claim 11, wherein, The motion vector prediction model is determined according to the depth of the top-left corner control point, the depth of the top-right corner control point, the depth of the bottom-left corner control point of the current block, the depth of the prediction point of the top-left corner control point, the depth of the prediction point of the top-right corner control point, the depth of the prediction point of the bottom-left corner control point in the reference frame, the camera parameter of the current frame and the camera parameter of the reference frame, and the motion vector prediction model comprises: According to the depth of the top-left corner control point of the current block, the depth of the top-right corner control point, the depth of the bottom-left corner control point, the depth of the prediction point of the top-left corner control point in the reference frame, the depth of the prediction point of the top-right corner control point, and the depth of the prediction point of the bottom-left corner control point, a second relationship is constructed as follows: wherein, a1, a2, a3 are depths of top-left corner control point, top-right corner control point and bottom-left corner control point of the current block respectively, b1, b2, b3 are depths of predicted point of the top-left corner control point, predicted point of the top-right corner control point and predicted point of the bottom-left corner control point in the reference frame respectively, λ1=x / w, λ2=y / h, x, y are coordinates of a pixel point in the current block relative to the top-left corner control point of the current block, w, h are width and height of the current block; obtaining a mapping relationship between the pixel point in the current frame and the pixel point in the reference frame according to the second relationship, the camera parameter of the current frame and the camera parameter of the reference frame; determining the motion vector prediction model based on the mapping relationship between the pixel point in the current frame and the pixel point in the reference frame.

13. The method according to claim 12, characterized in that The obtaining a mapping relationship between the pixel point in the current frame and the pixel point in the reference frame according to the second relationship, the camera parameter of the current frame and the camera parameter of the reference frame comprises: generating a first observation matrix and a second observation matrix according to the camera extrinsic parameter of the current frame and the camera extrinsic parameter of the reference frame respectively; According to the first observation matrix, the second observation matrix, the first parameter, and the second parameter, a third relationship is constructed as follows: wherein for an element in the mth row and nth column of the first observation matrix, for the element of the mth row and fourth column of the first observation matrix, for an element of the mth row and nth column of the second observation matrix, for the element in the mth row and 4th column of the second observation matrix, f1 and f2 are the first parameter and the second parameter respectively, u 1 and v 1 are the horizontal coordinate and the vertical coordinate of the pixel coordinate of the pixel point in the current block respectively, u 2 and v 2 are the horizontal coordinate and the vertical coordinate of the pixel coordinate in the reference frame respectively; α is the inverse of the second coordinate component of the pixel point in the current block in the clipping space, and β is the inverse of the second coordinate component of the pixel point in the reference frame in the clipping space; obtaining a mapping relationship between the pixel point in the current frame and the pixel point in the reference frame according to the second relationship and the third relationship.

14. The method of claim 2, wherein, The determining the motion vector prediction model according to the camera parameter of the current frame and the camera parameter of the reference frame comprises: generating a first observation matrix and a second observation matrix according to the camera extrinsic parameter of the current frame and the camera extrinsic parameter of the reference frame respectively; determining the motion vector prediction model according to the first observation matrix, the second observation matrix and the camera intrinsic parameter.

15. The method of claim 14, wherein, The motion vector prediction model includes a fourth relationship as follows: wherein, for an element in the mth row and nth column of the first observation matrix, for an element in the mth row and nth column of the second observation matrix, f1 and f2 are the first parameter and the second parameter respectively, u 1 and v 1 are the horizontal coordinate and the vertical coordinate of the pixel coordinate of a pixel point in the current block respectively, u 2 and v 2 are the horizontal coordinate and the vertical coordinate of the pixel coordinate of a pixel point in the reference frame respectively.

16. The method of claim 7, wherein, The obtaining the motion vector of the control point of the current block comprises: obtaining a spatial reference motion vector of the control point of the current block; compensating the spatial reference motion vector according to a distance from a center of an image block corresponding to the spatial reference motion vector to the control point of the current block to obtain the motion vector of the control point of the current block.

17. A method of video encoding, the method comprising: comprising: determining a reference frame of a current frame, the current frame being a video frame containing a current block; obtaining a camera parameter of the current frame and a camera parameter of the reference frame; determining a motion vector prediction model according to the camera parameter of the current frame and the camera parameter of the reference frame; obtaining a predicted block of the current block according to the motion vector prediction model; obtaining a residual block of the current block according to the current block and the predicted block; obtaining encoding data of the current block according to the residual block of the current block.

18. An apparatus for video decoding, the apparatus comprising: comprising: a memory configured to store a computer program; a processor configured to, when the computer program is invoked, cause the video decoding apparatus to implement the video decoding method in any one of claims 1-16.

19. A video encoding apparatus, comprising: comprising: a memory configured to store a computer program; a processor configured to, when the computer program is invoked, cause the video decoding apparatus to implement the video encoding method in claim 17.

Images