Geometric motion compensated prediction

Abstract

A video decoder for decoding a video of a scene from a data stream using motion compensated prediction is configured to derive, for a current block of a current image, a prediction block interior by using video geometry related parameters describing how the scene is projected onto the images of the video to find corresponding locations in a reference image that correspond to pixels in the current block and to sample the reference image at the corresponding locations; and to reconstruct the current block using the prediction block interior.

Description

Technical Field

[0001] According to embodiments of the present invention, there are devices (i.e., video encoders and video decoders) and methods for encoding or decoding videos of scenes using motion compensation prediction. Background Technology

[0002] The hybrid video codec divides the input signal frame by frame into square blocks called codec tree units (CTUs). A CTU can be further subdivided into smaller codec units (CUs). The reconstructed samples of a CU are constructed by superimposing predicted samples with the residual signal transmitted in the bitstream, followed by multiple post-filters that remove codec artifacts and thus improve the quality of the reconstructed samples. Each image is assigned a Picture Order Count (POC) that increases with the display order.

[0003] For CU prediction, two basic modes are distinguished: intra-frame, which predicts samples from the reconstructed region within the current image, typically from the neighboring region; and inter-frame, which uses sample information from previously reconstructed images for temporal sample prediction. A combination of the two modes is called Combined Inter-Frame Intra-Frame Prediction (CIIP). A special mode that can be used for intra-frame prediction is the Intra-Frame Block Copy (IBC) mode, which uses a shift vector pointing to the reconstructed region of the current image to copy the predicted sample from the resulting position.

[0004] In inter-frame prediction, CUs are predicted using weighted hyper-localization with one or more reference images. Previously reconstructed images used as reference images are accessed via a Reference Image List (RPL), where a specific reference image selected for prediction is accessed using a reference index (Ref-Idx) pointing to the list. VVC uses at most two reference lists, L0 and L1. The spatial offset of the location of the prediction sample extracted from the reference image relative to the current block's location is determined by a motion vector (MV) with a resolution accuracy ranging from N × sample to subsample resolution. For prediction from subsample locations, one of a plurality of N-tap interpolation filters is used based on the subsample location.

[0005] To utilize redundancy in motion vector encoding and decoding, each motion vector predictor (MVP) derived from a motion vector prediction procedure is used to predict each motion vector (MV) for extracting prediction samples from a reference image. This procedure searches for spatially and / or temporally adjacent CUs and / or history-based buffers for suitable MVP candidates. MVP candidates are stored in a list, and the MVP used to predict the MV is selected by an MVP index, which is transmitted in the bitstream if not otherwise derived. The final MV is determined by superimposing the MVP with the motion vector difference (MVD) transmitted in the bitstream. For some encoding and decoding modes, such as skip and some merge modes, the motion vector difference is not transmitted in the bitstream, and the final motion vector is derived directly from the MVP.

[0006] A more complex method for predicting time samples is called affine modeling, which uses a multi-parameter affine prediction model to compute the predicted samples. Affine models use two or three motion vectors at certain control points to describe the motion vector field that varies linearly with the sample position within the current block.

[0007] One technique that uses block-by-block prediction weighting (BCW) for bidirectional prediction may be mentioned here, where an index BCW-Idx for each CU is used to address a scaling table that determines the individual weights applied to the hypotheses superimposed in bidirectional prediction.

[0008] Another technique is to use the MV difference merge mode (MMVD), where an index is transmitted in the bitstream. The index determines the direction and spatial distance of the motion vector, where one of the vector components is zero.

[0009] Another technique is Symmetric Motion Vector Difference (SMVD). If bidirectional prediction is used for the current CU and the mode is not merged or skipped, this mode is signaled in the bitstream. In this particular mode, the MVP-Idx of two RPLs are transmitted in the bitstream, but only the MVD is transmitted for RPL L0. The MVD applied to predicting the MV from the L1 assumption is obtained by copying the MVD transmitted for L0 and inverting the sign of the MVD component by component. Reference images are selected from the L0 and L1 lists. In the references stored in the respective RPLs, the reference image from L0 immediately precedes the current image and the reference image from L1 immediately follows the current image, in the order of display.

[0010] Another inter-frame prediction mode is called Geometric Segmentation Mode (GPM), which uses two motion vectors per CU. The region covered by the CU is split into two regions along the tangential direction. The final prediction sample is obtained by applying a pixel-wise weighted matrix to samples of the two prediction hypotheses, and the matrix performs blending from one prediction hypothesis to the other at the region boundaries.

[0011] As mentioned earlier, the MVP derives the program's output delivery of a fixed-size list of MVP candidates, where the final MVP is selected from the list using the MVP index transmitted in the bitstream.

[0012] For the corresponding inter-frame modes, MVP shows differences in details.

[0013] The merged candidate list generation produces a single candidate list of N MVP candidates, containing joint information for prediction from one or two hypotheses (MV and Ref-Idx of the two RPLs and BCW-Idx of each candidate).

[0014] For merge and skip patterns, the direct spatial neighborhood is scanned to find potential motion vector (MV) candidates. If a candidate is found, the MV, the Ref-Idx of both reference lists, and the BCW-Idx are copied to the candidate list, unless it is already stored in the list. If the candidate list is not complete, candidates based on temporal motion vector prediction (TMVP) from previously encoded / decoded reference images may be considered, unless they are already stored in the list. Subsequently, if the candidate list is not completely filled, historical motion vector candidates (HMVP) are considered, unless they are already stored in the list. If the number of MVP candidates in the list is still less than the list size, MVP candidates are constructed from available data, and the list is finally filled with zero motion vector candidates.

[0015] The AMVP list generation procedure generates a candidate list with two MVP candidates for a given RPL and a given Ref-Idx. Therefore, it first scans spatially adjacent CUs to find suitable candidates, with the constraint that candidates must originate from the same RPL and must have the same Ref-Idx as the current prediction hypothesis. If the candidate list contains fewer than two candidates, TMVP candidates are considered. If the list still contains fewer than two candidates, HMVP candidates are considered. If the list still does not contain two candidates, it is populated with zero-motion candidates. To avoid duplication, if a suitable MVP candidate is not already stored in the list, it is simply inserted into the candidate list.

[0016] Other advanced encoding and decoding modes include affine merging, affine prediction, MER merging estimation region, DMVR, and BIM / BDOF.

[0017] Assuming constant motion, depending on the POC difference between image Pm and its reference image Pn, and between the current target image Pt and its reference image Px, temporal scaling of motion is used in the previously encoded and decoded reference image Pn to scale the motion vectors used for prediction from image Pm.

[0018] Temporal Motion Vector Prediction (TMVP) searches for a specific reference image in the L1 RPL. If the corresponding block is encoded and decoded using inter-frame mode, the specific reference image is used to access the co-location and extract the corresponding motion information. The motion vectors of a specific reference list are scaled according to the ratio of the temporal distance derived from the difference in point of view (POC) between co-location frames.

[0019] Time scaling of motion vectors should be applied in the following situations:

[0020] ●Use time motion vector prediction in conjunction with co-location MV prediction

[0021] ● Use MMVD to scale the motion vector difference based on the POC distance between the target image and the reference image.

[0022] Therefore, there is a need to provide concepts that enable more efficient and accurate deriving of motion vectors. Alternatively, improvements to image and / or video codecs are needed to reduce bitstreams and thus lower signaling costs.

[0023] This is achieved through the subject matter of the independent claims of this application.

[0024] Other embodiments of the invention are defined by the subject matter of the dependent claims of this application. Summary of the Invention

[0025] According to a first aspect of the invention, the inventors of this application recognized that a problem encountered when attempting to perform inter-frame prediction stems from the fact that deriving motion vectors involves an exhaustive search to find the best matching block in a reference frame for the current block. According to the first aspect of the application, this difficulty is overcome by using video geometric correlation parameters to derive the interior of the prediction block from the reference image. The video geometric correlation parameters describe how the scene is projected onto the image of the video. The inventors found that knowledge about changes in the projection of the scene onto the current image and the projection of the scene onto the reference image, or changes in the projection between the two images (e.g., due to changes in the position and / or orientation of the camera capturing the video between capturing the reference image and capturing the current image), can enhance the accuracy of inter-frame prediction. This is particularly true for static image content, such as a background, because the video geometric correlation parameters can reliably indicate the position of pixels in the reference image corresponding to the current block. Furthermore, the inventors found that encoding / decoding complexity can be reduced because the video geometric correlation parameters indicate the projection of the scene of the entire image onto the corresponding image; for this reason, computationally intensive motion estimation does not need to be performed for each inter-frame prediction block of the image. In addition, this reduces the bitstream and thus lowers the cost of signaling, because the video geometry-dependent parameters signal the entire image and do not have to signal each block of the image encoded / decoded using the video geometry-dependent parameters.

[0026] Therefore, according to a first aspect of this application, a video decoder / encoder for predicting and decoding a scene from a data stream using motion compensation / encoding the scene's video into a data stream is configured to: for a current block of a current image, find the corresponding position of a pixel in the current block in a reference image (e.g., a previously decoded / encoded, but not necessarily immediately preceding, image in terms of presentation time order) using video geometric correlation parameters (e.g., video capture correlation parameters) on an image describing how the scene is projected onto the video, and sample the reference image at the corresponding position to derive the interior of the predicted block. The video decoder / encoder is configured to reconstruct / encode the current block using the interior of the predicted block, for example, relative to the decoder by adding the prediction residual decoded from the data stream into the interior of the predicted block, or relative to the encoder by subtracting the interior of the predicted block from the actual image content within the current block to obtain the prediction residual and encoding the prediction residual into the data stream. The video geometric correlation parameters may provide information about the position and orientation of the camera capturing the corresponding image and / or the position and / or orientation of the corresponding image relative to the scene for one or more images of the video. Therefore, video geometry parameters can also describe the scene as a projection onto an image of a video synthesized using, for example, artificial intelligence, neural networks, or 3D rendering.

[0027] According to a second aspect of the invention, the inventors recognized that a problem encountered when attempting to perform inter-frame prediction stems from the fact that deriving motion vectors involves an exhaustive search to find the best matching block in a reference frame for the current block. According to the second aspect of the invention, this difficulty is overcome by using video geometric correlation parameters to derive motion vectors. Video geometric correlation parameters describe how a scene is projected onto an image of a video. The inventors found that knowledge about the projection of the scene onto the current image and the projection of the scene onto the reference image can enhance the accuracy of inter-frame prediction, where the projection of the scene onto the current image and the projection of the scene onto the reference image may differ with respect to the position and / or orientation of the camera capturing the respective images. This improvement in accuracy is particularly significant for static image content, such as a background, because the video geometric correlation parameters can reliably indicate how the same scene points within the scene are projected onto both the current and reference images, thereby allowing for efficient and accurate deriving of motion vectors. Furthermore, the inventors found that encoding / decoding complexity can be reduced because the video geometric correlation parameters indicate the projection of the scene onto the corresponding image for the entire image; for this reason, computationally intensive motion estimation does not need to be performed for each inter-frame prediction block of the image. In addition, this reduces the bitstream and thus lowers the cost of signaling, because the video geometry correlation parameter signals the entire image and does not have to signal each block of the image encoded / decoded using the video geometry correlation parameter.

[0028] Therefore, according to a second aspect of this application, a video decoder / encoder for decoding video of a scene from a data stream / encoding video of a scene into a data stream using motion compensation prediction is configured to derive a predetermined motion vector predictor for the current block using video geometric correlation parameters on an image describing how the scene is projected onto the video. The video decoder / encoder is configured to reconstruct / encode the current block using a predetermined MVP.

[0029] According to a third aspect of the invention, the inventors of this application recognized a problem encountered when attempting to predict samples of predetermined blocks of an image inter-frame, stemming from the fact that the reference image used for inter-frame prediction is not always perfectly suited to predicting samples of the current block. According to the third aspect of the application, this difficulty is overcome by generating the reference image synthetically instead of using a decoded / encoded image. The inventors discovered that video geometric correlation parameters can be used to generate a synthetic reference image that better matches the current image than any existing reference image. The video geometric correlation parameters describe how the scene is projected onto the image of the video. This information allows, for example, modification of the reference image associated with a first projection of the scene onto the image, such that a synthetic reference image associated with a second projection is generated. Optionally, the second projection may be similar to or equal to the projection of the scene onto the current image. This will likely improve the accuracy and efficiency of inter-frame prediction because the synthetic reference image is likely to depict the scene in almost the same way as the current image.

[0030] Therefore, according to a third aspect of this application, a video decoder / encoder for decoding a scene from a data stream using motion compensation prediction / encoding a scene's video into a data stream is configured to construct a synthetic reference image by finding corresponding positions in a reference image (e.g., previously decoded / encoded) that correspond to pixels in a predetermined image using video geometric correlation parameters (e.g., video capture correlation parameters) on an image describing how the scene is projected onto the video, and sampling the reference image at those corresponding positions. The synthetic reference image is then synthesized to form a synthetic version of the predetermined image. Additionally, the video decoder / encoder is configured to obtain the interior of a current portion of the current image by sampling the corresponding portion of the synthetic reference image, and to reconstruct / encode the current portion using the interior of the portion. Attached Figure Description

[0031] The accompanying drawings are not necessarily drawn to scale, but rather generally focus on illustrating the principles of the invention. In the following description, various embodiments of the invention are described with reference to the accompanying drawings, wherein:

[0032] Figure 1 This illustrates a device for encoding an image block-by-block into a data stream;

[0033] Figure 2 This illustrates possible implementations of the encoder;

[0034] Figure 3 This illustrates a device for decoding an image block by block from a data stream;

[0035] Figure 4 This illustrates possible implementations of the decoder;

[0036] Figure 5 An example of a decoder using time-sample prediction is shown;

[0037] Figure 6 Examples of non-inherent and inherent camera parameters, which are video geometry-related parameters, are illustrated schematically.

[0038] Figure 7 An example illustrating a homomorphic mapping defined by video geometric correlation parameters is shown;

[0039] Figure 8 An example is shown that was derived using geometric motion vectors from four images;

[0040] Figure 9 An example of a decoder using geometric motion vector prediction is shown;

[0041] Figure 10a illustrates an embodiment of an encoder that encodes video geometrically related parameters into data;

[0042] Figure 10b illustrates an embodiment of an encoder that decodes video geometric correlation parameters from data;

[0043] Figure 11 An example of spatial motion vector prediction is shown;

[0044] Figure 12 A first embodiment of geometric motion vector prediction considering camera roll is shown;

[0045] Figure 13 A second embodiment is shown that predicts the geometric motion vectors taking into account the roll of the camera;

[0046] Figure 14 A third embodiment is shown that predicts the geometric motion vectors taking into account the roll of the camera;

[0047] Figure 15 An example of a scene model is shown;

[0048] Figure 16 An example of a mesh-based scene model is shown;

[0049] Figure 17 An example of a point cloud-based scene model is shown; and

[0050] Figure 18An example of a decoder using a synthetic reference image is shown. Detailed Implementation

[0051] Equivalent or equivalent components, or components having equivalent or equivalent functionality, are indicated by equivalent or equivalent reference numerals or identified by the same name in the following description, and repeated descriptions of components having the same reference numerals or identified by the same name are generally omitted, even if the components appear in different figures. Therefore, the descriptions provided for components having the same or similar reference numerals or identified by the same name can be interchanged in different embodiments or applied to each other.

[0052] In the following description, numerous details are set forth to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other instances, well-known structures and apparatuses are shown in block diagram form rather than in detail in order to avoid obscuring embodiments of the invention. Furthermore, unless otherwise specifically indicated, features of the different embodiments described below may be combined with each other.

[0053] In the following text, the motion vector (MV) is a two-dimensional vector with two distinct associated images (the source image and the destination image). The destination image is the image of the sample to be predicted, and the source image is the image of the sample as input for this prediction.

[0054] The motion vector predictor (MV) to be predicted using motion vector prediction is called the target MV. The MV used as input to the motion vector prediction program is called the input MV. The MV used to predict the target MV by calculating the motion vector difference (MVD) is called the motion vector predictor (MVP).

[0055] For video compression purposes, it can be beneficial to use parameters such as the position, orientation, and internal properties of the camera recording the video. These parameters capture information about the position and manner in which objects in the recorded scene appear in individual video frames. For example, these parameters, including those related to video geometry mentioned in this paper, can be represented as follows: the camera position can be represented by a three-dimensional position vector. The camera orientation can be represented by three Euler angles, or by a four-dimensional rotation quaternion. The camera orientation can also be represented by a normalized direction vector starting at the camera's position and pointing towards the camera's viewing direction, along with the angle of rotation around this direction vector.

[0056] In addition to these so-called non-inherent camera parameters, there are also inherent camera parameters, such as focal length, various possible distortion parameters, and the size of the image produced by the camera, which is equal to the number of samples in the vertical and horizontal image dimensions.

[0057] The projection center of the camera is equal to its position. The image plane of the camera is orthogonal to the viewing direction of the camera, and the focal length is equal to the distance from the image plane to the projection center.

[0058] A pinhole camera is a camera that does not have distortion parameters.

[0059] For pinhole cameras, the scene point can be projected onto the image point on the image plane by taking the line between the scene point and the projection center and calculating the intersection point with the image plane.

[0060] To facilitate understanding of the following examples of this application, the description begins by presenting possible encoders and decoders suitable for it, in which examples of the following overview of this application can be constructed. Figure 1 An apparatus for encoding image 20 block by block into data stream 16 is shown. The apparatus is indicated by reference numeral 1000 and may be a still image encoder or a video encoder. In other words, when encoder 1000 is configured to encode video 12 including image 20 into data stream 16 or encoder 1000 can exclusively encode image 20 into data stream 16, image 20 may be the current image in video 12.

[0061] As mentioned, encoder 1000 performs encoding either block-by-block or block-based. To do this, encoder 1000 subdivides image 20 into blocks, encoding image 20 into data stream 16 in blocks. Possible examples of subdividing image 20 into blocks 18 are described in more detail below. Typically, subdivision may result in blocks 18 of constant size, such as an array of blocks configured in columns and rows, or may result in blocks 18 of varying sizes, where multi-tree subdivision begins with the entire image region of image 20 or begins with an array of pre-divided tree blocks of image 20. These examples should not be considered as excluding other possible ways of subdividing image 20 into blocks 18.

[0062] Additionally, encoder 1000 is a predictive encoder configured to predictively encode image 20 into data stream 16. For a given block 18, this means that encoder 1000 determines the prediction signal for block 18 and encodes the prediction residual (i.e., the prediction error where the prediction signal deviates from the actual image content within block 18) into data stream 16.

[0063] Encoder 1000 can support different prediction modes, such as those described above, to derive a predicted signal for a block 18. An important prediction mode in the following examples is the inter-frame prediction mode, which predicts block 18 from one or more reference images, for example, by determining a motion vector and copying the predicted signal of this block from the location pointed to by the motion vector in the reference image. Other optionally supported prediction modes may be associated with intra-frame prediction modes, which spatially predict the interior of block 18 from adjacent encoded samples of image 20. The encoding and therefore corresponding decoding process in image 20 to data stream 16 can be based on a certain encoding / decoding order 1100 defined within block 18. For example, encoding / decoding order 1100 can traverse block 18 in raster scan order, such as top-down line-by-line, where each line is traversed, for example, from left to right. In the case of hierarchical multi-tree subdivision, raster scan sorting can be applied within each hierarchical level, where a depth-first traversal order can be applied. That is, according to the encoding / decoding order 1100, leaf nodes within a block of a certain hierarchical level can precede blocks of the same hierarchical level that have the same parent block. Depending on the encoding / decoding order 1100, adjacent encoded samples of block 18 can typically be located on one or more sides of block 18. In the example presented herein, for instance, adjacent encoded samples of block 18 are located at the top and left of block 18.

[0064] When encoder 1000 is a video encoder, for example, encoder 1000 may support an inter-frame prediction mode, in which block 18 is temporally predicted from a previously encoded image of video 12. Such an intra-frame prediction mode may be a motion-compensated prediction mode, in which a motion vector is signaled to such block 18, indicating the relative spatial offset of the portion from which the predicted signal of block 18 is derived as a copy. Alternatively, other non-intra-frame prediction modes may also be available, such as the inter-frame prediction mode in the case of encoder 1000 being a multi-view encoder, or a non-predictive mode, in which the internals of block 18 are encoded and decoded as is, i.e., without any prediction.

[0065] Before focusing the description of this application on inter-frame prediction modes, a possible block-based encoder is given (i.e., regarding...). Figure 2 More specific examples of possible implementations of the described encoder 1000 are then presented, each suitable for... Figure 1 and Figure 2 Two corresponding examples of the decoder.

[0066] Figure 2 Show Figure 1Possible implementations of the encoder 1000, i.e., an implementation where the encoder is configured to use transform coding / decoding for encoding prediction residuals, but this implementation is an approximate example and this application is not limited to this category of prediction residual coding / decoding. According to Figure 2 The encoder 1000 includes a subtractor 1022 configured to subtract the corresponding prediction signal 1024 from the input port signal (i.e., image 20) or from the current block 18 on a block-by-block basis to obtain a prediction residual signal 1026. The prediction residual signal is then encoded into the data stream 16 by the prediction residual encoder 1028. The prediction residual encoder 1028 consists of a lossy coding stage 1028a and a lossless coding stage 1028b. The lossy stage 1028a receives the prediction residual signal 1026 and includes a quantizer 1030 that quantizes samples of the prediction residual signal 1026. As mentioned above, this example uses transform encoding and decoding of the prediction residual signal 1026, and therefore, lossy coding stage 1028a includes transform stage 1032 connected between subtractor 1022 and quantizer 1030 to transform such spectrally decomposed prediction residual 1026, wherein quantizer 1030 quantizes the transformed coefficients representing the residual signal 1026. The transform can be DCT, DST, FFT, Hadamard transform, etc. The transformed and quantized prediction residual signal 1034 is then losslessly encoded and decoded through lossless coding stage 1028b, which is an entropy encoder-decoder that entropy-encodes and decodes the quantized prediction residual signal 1034 into data stream 16. Encoder 1000 further includes a prediction residual reconstruction stage 1036 connected to the output of quantizer 1030 to reconstruct the prediction residual signal (see 1034') from the transformed and quantized prediction residual signal 1034 in a manner also available at the decoder; that is, quantizer 1030 takes into account encoding / decoding losses. For this purpose, prediction residual reconstruction stage 1036 includes an inverse quantizer 1038 that performs the inverse operation of the quantization by quantizer 1030, followed by an inverse transform 1040 that performs an inverse transform relative to the transform performed by transform 1032, such as the inverse operation of spectral decomposition, or the inverse operation of any of the specific transform examples mentioned above. Encoder 1000 includes an adder 1042 that adds the reconstructed prediction residual signal output by inverse transform 1040 to the prediction signal 1024 to output the reconstructed signal, i.e., the reconstructed sample. This output is fed into the predictor 1044 of the encoder 1000, which then determines the prediction signal 1024 based on the output. Predictor 1044 supports the features already described above. Figure 1 All the forecasting patterns discussed and will be discussed below. Figure 2It is also illustrated that when the encoder 1000 is a video encoder, the encoder 1000 may also include an in-loop filter 1046, which filters the fully reconstructed image, and the fully reconstructed image, after being filtered, forms a reference image for the predictor 1044 relative to the inter-frame prediction block.

[0067] As mentioned above, encoder 1000 operates on a block-based basis. For the following description, the block basis of interest is the basis for subdividing image 20 into blocks, wherein for each block, an inter-frame prediction mode is selected from a group or multiple inter-frame prediction modes supported by predictor 1044 or encoder 1000, and the selected inter-frame prediction mode is executed individually. However, it is also possible to subdivide image 20 into other categories of blocks. For example, the aforementioned decision regarding whether image 20 is inter-frame encoded or intra-frame encoded can be made at the granularity or unit of blocks deviating from block 18. For example, inter-frame / intra-frame mode decisions can be performed at the level of the encoded / decoded blocks into which image 20 is subdivided, and each encoded / decoded block is subdivided into prediction blocks. Prediction blocks with encoded blocks that have been decided to use inter-frame prediction are each subdivided into inter-frame prediction mode decisions. To this end, for each of these prediction blocks, it is determined which supported inter-frame prediction mode should be used for the corresponding prediction block. These prediction blocks will form block 18 of interest here. Inter-frame prediction blocks are predicted from a reference image, for example, by determining a motion vector and copying the prediction signal of this block from the location pointed to by the motion vector in the reference image. Prediction blocks within the codec blocks associated with intra-frame prediction will be processed differently by predictor 1044. Another block subdivision involves subdividing into transform blocks, which are transformed by transformer 1032 and inverse transformer 1040 on a per-transformer-block basis. Transformed blocks may be, for example, the result of further subdividing the codec blocks. Of course, the examples set forth herein should not be considered limiting, and other examples exist. For completeness only, it should be noted that subdivision into codec blocks may, for example, use multi-tree subdivision, and prediction blocks and / or transform blocks may also be obtained by further subdividing the codec blocks using multi-tree subdivision.

[0068] Figure 3 The description is suitable for Figure 1 The encoder 1000 includes a decoder 10 or device for block-by-block decoding. This decoder 10 performs the opposite operation to the encoder 1000; that is, it decodes the image 20 from the data stream 16 block-by-block and, for this purpose, supports multiple inter-frame prediction modes. For example, the decoder 10 may include a residual provider 156. (The above refers to...) Figure 1All other possibilities discussed also apply to decoder 10. For this purpose, decoder 10 can be a still image decoder or a video decoder, and decoder 10 also supports all prediction modes and prediction possibilities. The main difference between encoder 1000 and decoder 10 lies in the fact that encoder 1000 selects or chooses encoding / decoding decisions based on some optimization, such as minimizing a cost function that may depend on encoding / decoding rate and / or encoding / decoding distortion. One of these encoding / decoding options or parameters may involve selecting an inter-frame prediction mode to be used for the current block 18 from available or supported inter-frame prediction modes. The selected inter-frame prediction mode can then be signaled by encoder 1000 for the current block 18 within data stream 16, where decoder 10 uses this signaling notification for block 18 in data stream 16 to reselect. Similarly, the subdivision of image 20 into block 18 can be optimized within encoder 1000, and the corresponding subdivision information can be transmitted within data stream 16, where decoder 10 recovers the subdivision of image 20 into block 18 based on the subdivision information. In summary, decoder 10 can be a predictive decoder operating on a block-by-block basis, and in addition to inter-frame prediction mode, decoder 10 can also support other prediction modes, such as intra-frame prediction mode, which spatially predicts the interior of block 18 from adjacent coded samples of image 20. During decoding, decoder 10 can also use information about... Figure 1 The encoding / decoding order 1100 is discussed, and since this order is followed at both the encoder 1000 and the decoder 10, the same adjacent samples can be used for intra-frame prediction blocks at both the encoder 1000 and the decoder 10. Therefore, to avoid unnecessary repetition, the description of the operating mode of the encoder 1000 should also apply to the decoder 10 in terms of image 20 subdivision into blocks, for example, in terms of prediction and encoding / decoding of prediction residuals. The difference lies in the fact that the encoder 1000 selects some encoding / decoding options or parameters through optimization, and signals or inserts the encoding / decoding parameters into the data stream 16, which are then obtained from the data stream 16 by the decoder 10 for re-prediction, subdivision, etc.

[0069] Figure 4 Show Figure 3 Possible implementations of decoder 10, i.e., suitable for Figure 1 The implementation of encoder 1000 (e.g.) Figure 2 The implementation shown in the figure. Because... Figure 4 Many components of encoder 10 are similar to those appearing in Figure 2 The components in the corresponding encoder are the same, therefore in Figure 4The same reference numerals with apostrophes are used to indicate these components. Specifically, adder 1042', optional in-loop filter 1046', and predictor 1044' (e.g., output prediction signal 1024') are used in conjunction with it. Figure 2 The encoders are connected in the same manner to form a prediction loop. The reconstructed (i.e., dequantized and retransformed) prediction residual signal applied to adder 1042' is obtained through the following sequence: entropy decoder 56, which performs an inverse operation on the entropy encoding of entropy encoder 1028b; followed by residual signal reconstruction stage 1036', which consists of dequantizer 1038' and inverse transformer 1040', as is the case on the encoding side. The output of the decoder is the reconstruction of image 20. The reconstruction of image 20 can be obtained directly at the output of adder 1042' or alternatively at the output of in-loop filter 1046'. A post-filter can be configured at the output of the decoder to perform some post-filtering on the reconstruction of image 20 in order to improve image quality, but Figure 4 This option is not described in the document.

[0070] Similarly, regarding Figure 4 The above text is about Figure 2 The proposed description is for Figure 4 It should also be effective, except that only the encoder performs optimization tasks and related decisions regarding encoding / decoding options. However, all descriptions of block subdivision, prediction, inverse quantization, and retransformation are not applicable to... Figure 4 Decoder 10 is also effective.

[0071] The embodiments described below will primarily illustrate features and functionality with respect to the decoder. However, it will be apparent that the encoder may contain the same or similar features and functionality; for example, decoding performed by the decoder may correspond to encoding performed by the encoder. Furthermore, the encoder may include the same features described with respect to the decoder in the feedback loop, for example, in prediction stage 36.

[0072] The proposed method, for example, uses image motion parameters, which are contained in video geometric correlation parameters, to perform motion vector prediction or runtime sample prediction. These image motion parameters generalize the motion between images.

[0073] 1. Time-based sample prediction (motion-compensated sample prediction)

[0074] Figure 5 An exemplary illustration shows a video decoder 10 for decoding video 12 of scene 14 from data stream 16 using motion compensation prediction. The video decoder is configured to...

[0075] For the current block 18 of the current image 20a, by using video geometry-related parameters 24 (or using different terms: video capture-related parameters; regardless of the terminology used, and wherein this applies throughout the application and the claims with capital letters) describing how scene 14 is projected 26 onto image 20 of video 12, the parameters should also include the following situation: the video is generated synthetically by, for example, artificial intelligence, neural networks, or using 3D rendering rather than captured by camera 30, as depicted in the figure as possibly moving 32, causing parallax between image pairs such as images 20a and 20b) in (e.g., previously decoded; but in terms of presentation time order 34) (To be precise, the reference image may even be further away from image 20a in time and, in terms of sequence 34, may even be immediately following image 20a, not necessarily immediately preceding it.) Find the corresponding position 22 of the reference image 20b corresponding to pixel 23 in the current block 18 and sample the reference image 20b at the corresponding position 22 to derive the prediction block interior (further note: finding may not involve MV and may involve MVs sent in the data stream for the current block, such as to first pre-offset the pixels of the current block and then find the corresponding pixels relative to the positions of these pre-offset pixels).

[0076] Reconstruct the current block 18 using the inside of the prediction block (e.g., by adding the prediction residual 28 decoded from the data stream to the inside of the prediction block).

[0077] The corresponding video encoder used to encode the video 12 of scene 14 into the data stream 16 using motion-compensated prediction can be configured to find the corresponding position 22 of pixel 23 in the current block 18 of the current image 20a in the reference image 20b (e.g., previously encoded) by using video geometric correlation parameters 24 on the image 20 describing how scene 14 is projected 26 onto video 12 and sampling the reference image 20b at the corresponding position 22, and is configured to use the predicted block interior to encode the current block 18.

[0078] Finding the corresponding location in the reference image may involve...

[0079] - For example, for each pixel 23 in the current block 18, the corresponding pixel 23 is projected onto the corresponding scene point in scene 14 using video geometry correlation parameters (e.g., by obtaining from the video geometry correlation parameters the first scene projection line along which the corresponding scene point is projected onto the corresponding pixel, along with the distance of the corresponding scene point from the current image 20a, such as scene depth, and by determining the first scene projection line at the distance where the corresponding scene point is located), and the corresponding scene point is projected onto the corresponding position 22 in the reference image 20b using video geometry correlation parameters (e.g., by obtaining from the video geometry correlation parameters the second scene projection line along which the corresponding scene point is projected onto the reference image and by determining the intersection point of the second scene projection line and the reference image 20b as the corresponding position 22), or

[0080] - For example, for pixel 23 in the current image, the homomorphic mapping between the current image and the reference image 20b (e.g., regarding...) Figure 7 (As described) the corresponding position 22 in the reference image 20b is obtained, wherein the homomorphic mapping is defined by video geometrically related parameters (e.g., for the current image 20a and not for individual blocks; the homomorphic mapping describes, for example, the mapping of two or more corner points of the current image to the corresponding positions in the reference image, and the homomorphic mapping of pixel 23 in the current image to its corresponding position in the reference image can be determined by interpolation).

[0081] Video geometric parameters 24 can describe a scene-to-image projection 26 of a camera 30, such that scene-to-image projection is performed for each image 20 of video 12, for example, projecting scene 14 onto the images 20 of the video image by image-by-image and image-by-image. The scene-to-image projection 26 of an image 20 can be described by parameters defining the scene projection line along which scene points of scene 14 are projected onto the corresponding pixels within the image 20, along with parameters defining the distance between the scene points and the image 20. Alternatively, the scene-to-image projection 26 of an image 20 can be described by a homomorphic mapping between the image and a reference image (e.g., regarding...). Figure 7 (as described)

[0082] like Figure 6 As described in the text, the video geometry-related parameter 24 can be described by one or more non-inherent camera parameters of a camera (e.g., defining the position of the camera in space (translation vector); i) and / or the orientation of the camera in space (orientation is defined, for example, by a combination of pitch, yaw, and roll; for example, by three Euler angles; or by a four-dimensional rotation quaternion (e.g., a normalized direction vector starting at the camera's location and pointing towards the camera's viewing direction). i along with the vector around this direction The scene is described by one or more of the following parameters: rotation angle α and / or one or more inherent camera parameters (e.g., focal length f and / or FOV angle). One or more non-inherent camera parameters and one or more inherent camera parameters may be parameters of a real camera or a virtual camera (e.g., for video generated in a synthetic manner).

[0083] Figure 6 An exemplary data stream 16 is shown that includes video geometric correlation parameters 24 for each image and for the entire image. For example, the video geometric correlation parameters 24 may include non-inherent camera parameters of the first image 201 of video 12. 1. i and the inherent camera parameters f; the non-inherent camera parameters of the second image 202 of video 12. 2. 2 and the inherent camera parameters f; and the non-inherent camera parameters of the third image 203 of video 12. 3. 3 and inherent camera parameters f. The first image 201 can correspond to Figure 5 The current image 20a and the second image 202 can correspond to Figure 5 Reference image 20b.

[0084] like Figure 7 As shown, alternatively, video geometric correlation parameters can describe the homomorphic mapping between corresponding positions 401 and 402 in the pair of images 201 and 202 of the video, that is, between the first position 401 in the first image 201 and the second position 402 in the second image 202, for example, see 42. 12 This can be a homomorphic mapping between corresponding motion vectors associated with image pairs in the video (e.g., corresponding motion vectors may be associated with a predefined time frame distance). Homomorphic mapping 42 can represent a structure-preserving mapping.

[0085] The video geometric correlation parameters can be described by a homomorphic mapping 42 using vectors 462 or tensors at predetermined control points 442 of the video images (e.g., along with predefined interpolations 48 defined between control points 442). In other words, the video geometric correlation parameters 24 are a set of control point motion vectors describing the MV field between two images, see vector 462. The predetermined control points can be corner points of the video images. For each image, the video geometric correlation parameters can contain only one vector 462 or tensor (e.g., along with predefined interpolations 48 defined between control points 442) for each of the four corner points of the video image.

[0086] According to an embodiment, the video decoder 10 (and corresponding video encoder) is configured to, for a predetermined image, such as the current image 20a, perform a homomorphic mapping (e.g., 42) between corresponding positions 40 in one or more image pairs including the predetermined image and the base image. 12 and 42 23 The sequence is based on both intrinsic and non-intrinsic parameters of the underlying image (e.g., b , b f) to obtain the scene-to-image projection 26 that projects scene 14 onto a predetermined image.

[0087] like Figure 7 As shown, the position and orientation of frame 20 can be signaled using a set of motion vectors 46 at a specific control point 44. The motion vectors 46 describe the displacement of the current image (e.g., 202) relative to a reference image (e.g., 401) at the control point (e.g., 442). That is, according to this option, for image 20, the video geometric correlation parameters 24 are applied to each corner of image 20, i.e., each control point 44 contains a vector 46, and these vectors 46 describe the position of the corner and its offset from its corresponding position in a certain "reference image". In terms of encoding / decoding or presentation timing, the reference image may always be the immediately preceding image, or it may be another image. Therefore, images form pairs (a, b), where a is, for example, the point of view (POC) of image 20, for which vectors 46 are transmitted in data stream 16 as part of the video geometric correlation parameters 24, and b is the associated "reference" image. For example, such pairs can be defined as following the GOP structural interdependencies between images in a GOP. To obtain knowledge of the effective vector 46 of a corner point of an image 'a' relative to a predetermined reference image 'x', the decoder and encoder can simply concatenate (add) the vectors 46 of the video geometric correlation parameters of the corner points of image pairs (a,b), (b,c), (c,...) ... (...,x), where the smallest such sequence of image pairs containing vectors 46 in the video geometric correlation parameters 24 is selected. This concatenation is unnecessary when the video geometric correlation parameters 24 contain corner vectors of (a,b) for the current image 'a' and the reference image 'b' of the current block.

[0088] For example, vector mapping or homomorphism can be achieved by using an effective vector 46 as the support vector for some interpolation 48 to generate mapping vectors to certain locations 40 to be mapped. For example, homomorphism achieved by the corner vectors of images (a, b) can represent mapping 42, which maps image points in image a to corresponding points in image b, where, for example, it is assumed that these corresponding points in images a and b lie in scene plane 50 within the scene. This scene plane 50 can be, for example, the background plane of the captured scene 14, such as walls, etc. Figure 5 and Figure 6 The bookshelf behind the head in the foreground shown in scene 14.

[0089] When the inherent parameters of the camera are given, it is also possible to derive (potentially multiple) solutions from the homomorphism for non-inherent camera parameters and parameters defining the scene plane 50. The same mapping exists between any point in the scene plane 50 and the image points defined by the projection from the camera to the origin, as defined by the homomorphism. Multiple solutions can be generated by deriving the non-inherent camera parameters (i.e., vector 46 at the image corners) from the homomorphism, but one of these solutions can be selected at both the decoder and encoder using predetermined rules.

[0090] This seems to contradict the view that homomorphism is defined only between two images 20, and that camera parameters appear to represent one image 20 and be independent of the other images 20. However, this is merely a matter of reference point; that is, any coordinate system can be chosen for the (camera) parameters: for example, the parameters of the first image (of a group) can always be placed at the origin, meaning all camera parameters are now relative to that image. The same applies to homomorphic parameters. Using an origin that is not equivalent to a useful set of parameters would actually be bad, as it would waste description length (or bits) to have this useless origin. In other words, entropy encoding / decoding will compute the difference and thus derive relative parameters anyway.

[0091] Therefore, control point vector 46 can be considered an alternative representation of camera parameters defining camera position and orientation. This alternative representation may be more suitable for entropy encoding / decoding, which can be used to encode / decode parameter 24 into stream 16. Encoding / decoding may include parameter quantization, and quantization errors in control point vector 46 proportionally cause errors in the image plane. Quantization errors in rotation angles or depth-dependent camera position errors are more difficult to control in terms of their consequences. However, video geometry-dependent parameters 24 may also provide a combination of non-intrinsic and / or intrinsic camera parameters with parameters defining homomorphic mappings (e.g., vector 46) for the image 20 of video 12.

[0092] Regarding the difference between the homomorphic corner vectors defined between image pairs and camera projection parameters (such as non-intrinsic parameters defined individually for each image), the following should be noted: It is correct that the homomorphism is defined only between two images, while the camera parameters appear to represent one image and are independent of the others. However, this is merely a matter of reference point. Any coordinate system can be chosen for the (camera) parameters: for example, the parameters of the first image (of a group of images) can always be placed at the origin, meaning that all camera parameters are now relative to that image. The same applies to homomorphic parameters. Using an origin that is not equivalent to a useful set of parameters would actually be bad, because it would waste description length (or bits) to have this useless origin. In other words, entropy encoding / decoding will calculate the difference and thus will always result in relative parameters.

[0093] like Figures 5 to 7 As can be seen, the video decoder 10 is configured to decode the video geometric correlation parameter 24 from the data stream 16, and the video encoder is configured to encode the video geometric correlation parameter 24 into the data stream 16. The video decoder 10 is configured to determine the video geometric correlation parameter 24 based on the decoded portion of the video 12, and the video encoder is configured to determine the video geometric correlation parameter 24 based on the encoded portion of the video 12.

[0094] about Figures 5 to 7 The described video decoder 10 can be configured to decode syntax elements from data stream 16 (e.g., at the block level, such as an MVP candidate list index or pattern syntax element, or at a higher level, tuning motion compensation prediction tools), and if the syntax element has a first state, then

[0095] For the current block 18, the corresponding position 22 of pixel 23 in the current block 18 is found in the reference image 20b using video geometric correlation parameters 24 that describe how scene 14 is projected onto image 20 of video 12, and the reference image 20b is sampled at the corresponding position 22 to obtain the predicted block interior.

[0096] Reconstruct the current block 18 using the predicted block.

[0097] The corresponding video encoder can be configured to encode syntax elements into data stream 16, and, as described for the decoder, derive the predicted block interior for the current block 18 and use the predicted block interior to encode the current block 18.

[0098] Optionally, if the syntax element has a second state, the video decoder 10 can be configured to reconstruct the current block 18 independently of the video geometric correlation parameter 24, or to reconstruct the current block 18 by copying the decoded video portion with regular inter-pixel spacing (e.g., without distortion). Similarly, in this case, the corresponding video encoder can be configured to encode the current block 18 independently of the video geometric correlation parameter 24, or to encode the current block 18 by copying the encoded video portion with regular inter-pixel spacing (e.g., without distortion).

[0099] According to an embodiment, the video decoder 10 is configured to generate a list of MVP candidates for the current block 18, one of which corresponds to a specific motion-vector inter-frame encoding / decoding mode. A selected MVP is chosen from the list of MVP candidates, and the current block 18 is reconstructed using the selected MVP through the following operations:

[0100] If the selected MVP corresponds to a specific motion vectorless inter-frame encoding / decoding mode, then

[0101] For the current block 18, the corresponding position 22 of pixel 23 in the reference image 20b (e.g., previously decoded) is found using video geometric correlation parameters 24 on the image 20 describing how scene 14 is projected onto video 12, and the reference image 20b is sampled at the corresponding position 22 to deduce the interior of the predicted block.

[0102] Reconstruct the current block 18 using the predicted block.

[0103] If the selected MVP corresponds to an MVP candidate other than a specific motion vectorless inter-frame encoding / decoding mode, then

[0104] Use the selected MVP to reconstruct the current block 18 using motion vector compensation prediction.

[0105] Similarly, the video encoder can be configured to generate a list of MVP candidates for the current block 18, one of which corresponds to a specific motion vectorless inter-frame encoding / decoding mode. A selected MVP is chosen from the list of MVP candidates, and the current block 18 is reconstructed using the selected MVP through the following operations:

[0106] If the selected MVP corresponds to a specific motion vectorless inter-frame encoding / decoding mode, then

[0107] For the current block 18, the corresponding position 22 of pixel 23 in the current block 18 is found in the reference image 20b (e.g., previously decoded (e.g., stored in a decoder buffer) or encoded) using video geometric correlation parameters 24 describing how scene 14 is projected onto video 12, and the reference image 20b is sampled at the corresponding position 22 to deduce the interior of the predicted block.

[0108] The current block 18 is encoded using the internal code of the predicted block.

[0109] If the selected MVP corresponds to an MVP candidate other than a specific motion vectorless inter-frame encoding / decoding mode, then

[0110] Use the selected MVP to encode the current block 18 using motion vector compensation prediction.

[0111] For additional embodiments, such as those describing video geometry-related parameters, refer to the following description.

[0112] These parameters can be encoded or decoded within the bitstream (i.e., data stream 16), or they can be derived at the decoder from previously decoded information.

[0113] As described above, in one embodiment of the invention, the image motion parameters, i.e., the video geometric correlation parameters 24, are pinhole camera parameters associated with each image 20 of video 12 (e.g., see...). Figure 6 And a description of inherent and non-inherent camera parameters). In another embodiment, the image motion parameters are a set of control point motion vectors associated with each image, which describe the MV field between the two images (e.g., see...). Figure 7 And a description of homomorphic mappings.

[0114] 2. Motion Vector Prediction

[0115] When the image motion parameters (i.e., video geometric correlation parameters 24) are camera parameters, for example, the proposed method combines camera parameters with MV to reconstruct the estimated 3D scene points, as described below. Figure 8 As described.

[0116] Figure 8 This shows the use of, for example, the current image 20a (also denoted as F). dt ), source image 20b (also represented as F) di ), reference image 20c (also denoted as F) st ) and another reference image 20d (also denoted as F) siThis embodiment derives geometric motion vectors from four images. The source image 20b, reference image 20c, and another reference image 20d may represent previously decoded / encoded images of the video. In some situations, such as spatial motion vector prediction, which will be described in more detail below, geometric motion vector derivation can be performed using fewer than four images.

[0117] As described above, a motion vector (MV) is a two-dimensional vector that additionally has two distinct associated images (a source image and a destination image). The tail of the motion vector can be placed in the destination image, and the head of the motion vector can point to the source image. In the motion vector derivation proposed in this paper, the first MVP 130 (also referred to below as the input motion vector MV) can be obtained. i The first MVP 130 can be compared with the source image 20b (also represented as F). di ) and another reference image 20d (also denoted as F) si For this reason, source image 20b can also be referred to hereinafter as the associated destination image, and another reference image 20d can also be referred to as the associated source image. In this paper, a predetermined motion vector predictor 126 is proposed to be determined geometrically based on a first MVP 130. The predetermined motion vector predictor 126 can be associated with the current image 20a (also denoted as F...). dt ) and reference image 20c (also denoted as F) st As they are associated, for this reason, the current image 20a may also be referred to as an associated destination image or an arbitrary destination image in the following text, and the reference image 20c may also be referred to as an associated source image or an arbitrary source image.

[0118] For the input motion vector MV i 130 (with associated source image F) si and destination image F di This can form two rays: a source ray and a destination ray, which pass through the F... si / F di The projection center 31 of the associated camera passes through a source / destination point in the image plane of the corresponding camera, for example, the first scene projection line 26b passes through the first image position 132 in the source image 20b (also referred to as q in this document). di (or destination point) and the second scene projection line 26d passes through the second image position 134 in another reference image 20d (also referred to as q in this document). si (or source point). In the image plane (i.e., in source image 20b), for the target location q of the translational motion. di 132 is related to MV iThe top-left corner of block 128 associated with 130, or the center of block 128, or the bottom-right corner of block 128. Source point q si 134 is the sum of destination 132 and input MV 130; q si =q di +MV i For example, see also Figure 9 The corresponding co-location block 128 of the current block 124 in the current image 20a can be obtained from the input motion vector MV. i Block 128 in the associated source image 20b is represented by 130.

[0119] For an affine prediction block, the destination point q di 132 can be one of the control points in the parametric affine model (the top-left and top-right corners for a 4-parameter model, the bottom-left corner for a 6-parameter model, and the bottom-right corner for an 8-parameter model). For each affine prediction block, each of the desired destination points 132 is considered individually. Destination point q di Each of the 132 and its associated motion vector MV i The sum produces the source point q for each of the control points. si 134.

[0120] If q di and q si If the projections are onto the same 3D scene point, then the two rays 26b and 26d intersect at the scene point. For any MV i 132 (selected by encoder control) and the estimated camera parameters may not be as described. To estimate scene points given this uncertainty, the proposed method considers at least three scene point estimates derived using the shortest distance 27 between two rays 26b and 26d in 3D space, see 52a (f d ), 52b (f m ) and 52c (f s The points under consideration are two points f. s 52c and f d 52a, a point on each ray, where the shortest distance between any two such points is 27. The third point f m 52b is the midpoint between the first two points, that is, f m = (f s +f d ) / 2. f s with f d The distance between them is d min .

[0121] It can also be obtained in different ways, such as f mThe three-dimensional scene point (hereinafter referred to as Q) is determined, for example, by solving a system of equations or by minimizing the point q. di and q si Reprojection error at that location.

[0122] For example,

[0123] 1) According to the embodiment, scene point 52 can be determined by determining and solving a system of linear equations. The system of linear equations can be determined using video geometric correlation parameters 24, a first image position 132 of reference block 128, and a second image position 134 of another reference image 20d, wherein the first MVP 130 is positioned at the tail of the first image position 132, and the first MVP 130 points to the second image position 134 of the other reference image 20d.

[0124] For example, scene point 52 can be determined by solving the homogeneous linear system of equations AQ=0 (finding the solution with the minimum error), for example by using the singular value decomposition (SVD) method.

[0125] Here, Q is the 3d point to be determined. In homogeneous coordinates, Q = [XYZ 1].

[0126] A is a 3×4 matrix with the following rows:

[0127] A.row(0) = q di (0)*cam di .row(2) - cam di .row(0)

[0128] A.row(1) = q di (1)*cam di .row(2) - cam di .row(1)

[0129] A.row(2) = q si (0)*cam si .row(2) - cam si .row(0)

[0130] A.row(3) = q si (1)*cam si .row(2) - cam si .row(1).

[0131] Among them cam x .row(.) is F si and F diThe 3×4 camera matrix of associated cameras consists of rows, each horizontally connected by a rotation matrix R and a translation vector t. In other words, cam x = [R|t],

[0132] And q di and q si For image F di and F si 2D image points, such as q xi (0) represents the image coordinates x and q xi (1) represents the image coordinate y in image 1.

[0133] If the singular value decomposition (SVD) of A produces three matrices U, S, and V that satisfy the equation A = USV, then the fourth column of V, denoted by V.col(3), determines the solution to Q:

[0134] Q = V.col(3) / V.col(3)(3), where Q is used instead of f. m As the intersection point of the reprojection.

[0135] 2) According to an embodiment, scene point 52 can be determined by minimizing a cost function based on the reprojection error. The cost function can be the sum of two squared reprojection errors, the sum of cube powers of the errors, the mean square error, the root mean square error, the mean absolute error, the mean logarithmic error, or any other cost function. The reprojection error can include the distance between the first image position 132 of the reference block 128 in the source image 20b and the projection of scene point 52 onto the source image 20b, and the distance between the second image position 134 of another reference image 20d and the projection of scene point 52 onto the other reference image 20d, wherein the first MVP 130 points to the second image position 134 of the other reference image 20d by placing the tail of the first MVP 130 onto the first image position 132.

[0136] For example, this can be achieved by minimizing the sum of the two squared reprojection errors, i.e., at point q. si With 3D point Q to image F si The squared distance between the image coordinates of the projections on the surface plus point f di With Q to image F di The scene point 52 is determined by the squared distance between the image coordinates of the projections on the screen. This can be done by first determining the scene point 52 from f. si and f di And calculate the optimal image point f using camera parameters. si 'and f di This can be achieved, for example, through the algorithm described in P. Lindstrom's "Triangulation Made Easy" or by other means. (The last sentence appears to be incomplete and possibly refers to a different topic.)si 'and f di The created rays will intersect, and the 3d point Q will be the intersection point.

[0137] Therefore, as Figure 9 As shown, the video decoder 100 for decoding video 12 of scene 14 from data stream 16 using motion-compensated prediction can be configured to derive a predetermined motion vector predictor (MVP) 126 for the current block 124 using video geometric correlation parameters 24 describing how scene 14 is projected 26 onto image 20 of video 12, and to reconstruct the current block 124 using the predetermined MVP 126. The MVP 126 can describe the spatial offset in the image plane between the image position of the current block 124 in the current image 20a and the corresponding position in the reference image 20c.

[0138] The corresponding video encoder used to encode video 12 of scene 14 into data stream 16 using motion compensation prediction can be configured to derive a predetermined motion vector predictor (MVP) 126 for the current block 124 using video geometric correlation parameters 24 on image 20 describing how scene 14 is projected 26 onto video 12, and to encode the current block 124 using the modified MVP 126.

[0139] The video decoder can be configured to use a predetermined MVP 126 to obtain a final motion vector by superimposing the predetermined MVP 126 and the motion vector difference (MVD) decoded from the data stream 16, thereby extracting prediction samples from the reference image 20c of the current block 124 and adding the prediction residual decoded from the data stream 16 to the prediction samples to reconstruct the current block 124. The video encoder can be configured to use the predetermined MVP 126 to determine and encode the motion vector difference (MVD) between the final motion vector and the predetermined MVP 126, and to encode the prediction residual into the data stream 16 to encode the current block 124, wherein the combination of the prediction samples extracted from the reference image 20c of the current block 124 using the final motion vector and the prediction residual represents the current block 124.

[0140] The video decoder 100 can be configured to derive a first MVP 130 from a previously decoded portion 128 of the video 12 for the current block 124 and modify the first MVP 130 using video geometric correlation parameters 24 to obtain a predetermined MVP 130. That is, the predetermined MVP 130 is determined geometrically based on the first MVP 130 using the video geometric correlation parameters 24 to derive the predetermined motion vector predictor (MVP) 126. A corresponding encoder can similarly perform the derivation of the predetermined motion vector predictor 126, the difference being that the first MVP 130 is derived from a previously encoded portion of the video 12. However, the encoder may also be configured to derive the first MVP 130 from a previously decoded portion of the video 12, such as a decoder buffer of the encoder, for example, stored in the decoding unit of the video encoder.

[0141] In the context of motion vectors, the term "modification" can encompass determining a new motion vector based on another motion vector. For example, modifying the first MVP 130 to obtain a predetermined MVP 126 could mean determining the predetermined MVP 126 based on the first MVP 130. The predetermined MVP 126 can replace the first MVP 130; that is, the predetermined MVP 126 can be used instead of the first MVP 130. Figure 8 As exemplarily shown, the first MVP 130 can be "modified" by mapping the first MVP 130 (e.g., using a geometric mapping defined as video geometric correlation parameter 24) onto a predetermined MVP 126. Therefore, the predetermined MVP 126 can be considered a modified version of the first MVP 130. After the motion vector is "modified," the modified motion vector may not be associated with the same source and destination images as the unmodified motion vector.

[0142] Such as about Figure 6 and Figure 7 As described, the video geometric correlation parameter 24 describes a scene-to-image projection 26 for a camera, which, for example, projects scene 14 onto image 20 of video 12 for each image of the video, such as image-by-image and image-by-image. (See also: Regarding...) Figure 6 As described, the video geometrical parameters 24 can describe the scene to image projection 26 through one or more non-inherent camera parameters of a camera and / or one or more inherent camera parameters (e.g., focal length or FOV angle). (See also: Regarding...) Figure 7 As described, the video geometric correlation parameter 24 can describe the homomorphic mapping 42 between corresponding positions 40 in the image 20 pair of video 12 or the homomorphic mapping between corresponding motion vectors associated with the image pair of the video (e.g., the corresponding motion vectors can be associated with a predefined time frame distance).

[0143] Homomorphic mapping 42 can be achieved by using the vector 46 of the corner point of the current image 20a relative to the reference image 20c as the support vector of some interpolation 48 to generate mapping vectors to certain locations to be mapped. For example, in order to obtain the predetermined MVP 126 of the current block 124 of the current image 20a, interpolation can be used to map the position of the current block 124 to the corresponding position in the reference image 20c and the difference can be used as the predetermined MVP 126.

[0144] Given the inherent parameters of the camera, it is also possible to derive (possibly multiple) solutions from the homomorphic parameters for the non-inherent camera parameters and the parameters defining the scene plane 50. The same mapping exists between any point in the scene plane 50 and the image points defined by the projections from that point to the camera and to the origin in the camera, as defined by the homomorphism. However, a predetermined MVP 126 can be derived using these camera parameters through triangulation as described below. That is, using the homomorphic parameters, the encoder and decoder can perform the triangulation-based tasks described below to derive the predetermined MVP. It is not limited to using only the homomorphic parameters to trace the homomorphic mapping between two images. The homomorphic parameters can also be used for triangulation of the input MV, as described below.

[0145] The video encoder can be configured to encode video geometric correlation parameters 24 into data stream 16, and the video decoder 100 can be configured to decode video geometric correlation parameters 24 from data stream 16. The video encoder can be configured to determine video geometric correlation parameters 24 based on the encoded portion of video 12, and the video decoder 100 can be configured to determine video geometric correlation parameters 24 based on the decoded portion of video 12.

[0146] According to the embodiments shown in Figures 10a and 10b, the video encoder is configured to determine 200 video geometric correlation parameters 24 based on video 12 and encode the video geometric correlation parameters 24 into data stream 16. Alternatively, the video encoder may be configured to determine the video geometric correlation parameters 24 based on a version of the video generated by the decoded data stream, for example, from a decoding buffer of the decoding unit of the video encoder.

[0147] The video encoder can be configured to determine video geometry-related parameters 24 using one or more of the following:

[0148] ●Optimize video geometric correlation parameter 24 or a portion thereof through rate-distortion optimization;

[0149] ●Use stereo matching; and

[0150] ● Background / Foreground splitting.

[0151] about Figure 8 and Figure 9The described video decoder 100 and corresponding video encoder can be configured to derive one or more of the following from the video geometric correlation parameters 24:

[0152] Current image 20a (F dt The first scene is projected onto the image, and the current block 124 is a part of the current image 20a.

[0153] Reference image 20c (F) related to the planned MVP 126 st The second scene to image projection,

[0154] From this, the source image 20b (F) of the first MVP 130 was derived. di The third scene to image projection,

[0155] Another reference image 20d (F) related to the first MVP 130 si The fourth scene is projected onto the image.

[0156] As will be described in more detail below, one or more of the first to fourth scene-to-image projections can be used to geometrically derive a predetermined motion vector predictor 126.

[0157] The video decoder 100 and its corresponding video encoder can be configured to modify the first MVP 130 using video geometry-related parameters 24 through the following operations:

[0158] Using video geometric correlation parameters 24, from which the first image position 132 of the reference block 128 of the first MVP 130 is derived, and the first MVP 130, one or more scene points 52 in the scene are determined (e.g., f m f d f s ),as well as

[0159] The predetermined MVP 126 is determined using video geometry correlation parameters 24 and one or more scene points 52.

[0160] To determine one or more scene points 52, for example, they can be derived from video geometric correlation parameters 24 and a third scene-to-image projection and a fourth scene-to-image projection can be used. To determine a predetermined MVP 126, for example, it can be derived from video geometric correlation parameters 24 and a first scene-to-image projection and a second scene-to-image projection can be used.

[0161] The reference block 128 in the source image 20b can be co-located with the current block 124 in the current image 20a, and the first image position 132 can correspond to the upper left corner of the reference block 128, or the center of the reference block 128, or the lower right corner of the reference block 128.

[0162] The following is an example of the implementation of the newly proposed method to derive the MVP.

[0163] First, the MVP of the current block 124 in the current image 20a can be generated through the video geometric correlation parameters 24, that is, the MVP 126 is predetermined, such that the predetermined MVP 126 describes the spatial offset in the image plane between the image position of the current block 124 in the previous image 20a and the corresponding position in the reference image 20c, for example, see Figure 9 .

[0164] Secondly, the video geometric correlation parameter 24 can be used to "improve" the (first) MVP 130 predicted in other ways. In other words, any MV (with the associated source image F) that can be derived at decoder 100 si and destination image F di For example, the first MVP 130 and the source image 20b (also denoted as F) di ) and another reference image 20d (also denoted as F) si (Related) can be used as input MV MV i This is used to estimate scene point 52, as described above. This is achieved by using scene points (e.g., midpoint f) m 52b) Projected onto F st F dt On the image plane of the associated camera, the projection output point q is thus obtained in the two-dimensional image space. so (For example, referenced by 136) and q do (For example, from 138 references), while arbitrary source image F is derived from scene point estimation 52. st and destination image F dt The predefined MVP 126 (associated with the target MV). The term "arbitrary" means that the geometry described by the predefined MVP 126 can be performed on any image 20 of the video. To be able to describe the process of an image, F... dt In this paper, it is considered to be the current image 20a and F st The reference image 20c is considered as the current image 20a. The predetermined MVP 126 is calculated, for example, as the difference between two projection points 136 and 138, where MVP = q. so - q do .

[0165] The shortest distance between rays 26b and 26d is 27d. min Can be used as a music video i130 tracks the quality of static scene objects and thus serves as a measure of the suitability of the resulting predefined MVP 126. The geometry described herein for the predefined MVP 126 is derived, for example, only for blocks of the current image 20a associated with static scene objects such as background, furniture, buildings, and plants.

[0166] In current advanced video encoding and decoding technologies, it depends on F si F di F st and F dt The proposed method handles motion vector prediction differently depending on the identifier used. The following sections describe these different situations and how the proposed method is applied in these situations.

[0167] Regarding nomenclature and wording, note the following: Typically, the verb "projection" is used with the object / adverb phrase "scene point" or "scene" and another object / adverb phrase "...the...position of the image." In such cases, projection means specific to the "...image" being referred to, as defined by the video geometrical correlation parameter 24 (because the mapping from scene 14 to image 20 changes during video 12). On the other hand, sometimes "offset" or "difference" between the image positions of different images 20 is mentioned. In this case, projection no longer plays a role. Assume that images 20 form a common image region that can be fixed across the entire video (i.e., fixed inherent camera parameters), or that at least an image region of one of images 20 can be correlated with an image region of another image by lateral (in-plane) offset and lateral (in-plane) stretching (e.g., isotropic or anisotropic) by a panner, such that the difference or offset of the image positions is invariant relative to the absolute position of the image positions, but depends only on their relative positions.

[0168] Time motion vector prediction

[0169] If the current image is 20a / F dt Not equal to the source image 20b / F di Currently, advanced video encoding and decoding technologies use temporal motion vector scaling (TMS) for motion vector prediction to derive the motion vector prediction sub-MVP. temp Here, the source image is 20b / F. di Also known as co-positioned images. In this situation, MVP temp Equal to scaled input MV MV i And a scaling factor was selected to match the reference image 20c / F. st With current image 20a / F dt The time difference between them. That is, if the image sequence count (POC) of the images involved is POC. di POC si POCdt and POC st MVP temp =MV i *( POC st - POC dt ) / (POC si - POC di ).

[0170] In contrast, the newly proposed method described in the previous chapters starts from the input MV MV i The scene point 52b / f is obtained. m Set scene point 52b / f m Projected onto reference image 20c / F st and current image 20a / F dt Up, thus obtaining the projection point 136 / q so and 138 / q do And calculate MVP temp,G = q so - q do Motion Vector MVP temp,G This can indicate that MVP 126 is reserved.

[0171] about Figure 8 and Figure 9 The described video decoder 100 and corresponding video encoder can be configured to read from the source image 20b / F di The reference block 128 (e.g., previously decoded / encoded) yields the first MVP 130, and the first MVP 130 is modified using video geometry-related parameters 24 by the following operations:

[0172] Using video geometric correlation parameter 24, the first image position 132 / q of reference block 128 di And the first MVP 130,

[0173] Scene 14 is determined to be projected onto the first image position 132 / q based on video geometric correlation parameters 24 of source image 20b. di The first scene projection line 26b along the top (e.g., connecting q) di The line with the center of the camera projection 31b, such as Figure 8 As can be seen, the line crosses scene point 52a / f. d ),

[0174] Determine the second image position 134 / q of scene 14 (e.g., based on video geometric correlation parameters 24 of another reference image 20d) projected onto the other reference image 20d. siThe second scene projection line 26d along the top is achieved by placing the tail of the first MVP 130 at the first image position 132 / q. di Above, the head of the first MVP 130 points to the second image position (it should be noted that when referencing another image 20d from one image 20b, any motion vector such as 130 corresponds to the in-plane reference image vector 130' which points from the control / block position 132 of the inter-frame prediction block 128 relative to the reference image 20b to the position 134 pointed to by the motion vector 130), and

[0175] Determine the shortest line 51 connecting any point on the first scene projection line 26b and any point on the second scene projection line 26d (e.g., connecting scene points 52a / f). d and scene point 52c / f s Scene points on line 51 (e.g., 52b / f) m 52a / f d Or 52c / f s (For example, the shortest line 51 can connect the first scene projection line 26b and the second scene projection line 26d with the shortest distance between them in 3D space), and

[0176] Using video geometry-related parameters 24 and scene points 52.

[0177] Determine the reference image 20c / F st The third image position 136 / q so Scene point 52 (e.g., based on video geometric correlation parameters 24 of reference image 20c) is projected onto the third image position.

[0178] Determine the current image 20a / F dt The fourth image position 138 / qdo, scene point 52 (e.g., based on the video geometric correlation parameter 24 of the current image 20a) is projected onto the fourth image position.

[0179] Determine the predetermined MVP 126 to define the offset (e.g., difference) between the third image position 136 and the fourth image position 138.

[0180] like Figure 8 As shown, scene point 52 can be determined as the midpoint 52b / f on the shortest line 51, located in the middle of the shortest line. m That is, the distance from scene point 52 to the first scene projection line 26b can be the same as the distance from the second scene projection line 26d. Scene point 52, for example, f mAlternatively, it can be selected as another point on the shortest line 51. For example, scene point 52 may also be located elsewhere on the shortest line 51, such as being closer to the first scene projection line 26b than to the second scene projection line 26d, closer to the second scene projection line 26d than to the first scene projection line 26b, or directly on the first scene projection line 26b (see scene point 52a / f). d ) or directly on the second scene projection line 26d (see scene point 52c / f) s ).

[0181] According to the embodiments described in more detail below,

[0182] ●If the source image 20b is equal to the reference image 20c / F st Then the intersection point 52a / f of the shortest line 51 and the first scene projection line 26b is... d It can be considered as scene point 52, and

[0183] ●If another reference image 20d is equal to reference image 20c, then the intersection point 52c / f of the shortest line 51 and the second scene projection line 26d is... s It can be considered as scene point 52, and

[0184] ●If the source image 20b, the reference image 20c, and another reference image 20d are different images, then the midpoint 52b on the shortest line 51 can be regarded as scene point 52.

[0185] In special cases, Temporal Motion Vector Prediction (TMVP) involves exactly three distinct images; that is, if F st =F si or F st =F di (Note that F) si It may not be equal to F. dt In this situation, the alternative is f. m Projected to F st The projection calculation can be omitted. More specifically, if F st =F si , then q so It can be selected as equal to q si And if F st =F di , then q so It can be selected as equal to q di Alternatively, instead of using q directly. si and q di The midpoint of the shortest line can be used for reprojection onto F. st In the middle. For projection to F dt f can still be used in the middle. m Alternatively, if F st=F si , then q do It can be selected as equal to f s To F dt The projection on, and if F st =F di q do It can be selected as equal to f d To F dt The projection on the surface.

[0186] In any of these situations, the video decoder 100 and the corresponding video encoder can be configured (e.g., using video geometric correlation parameters 24, the first image position 132 of the reference block 128, and from the source image 20b / F) di The first MVP (130) derived from reference block 128,

[0187] Scene 14 is determined to be projected onto the first image position 132 / q based on video geometric correlation parameters 24 of source image 20b. di The first scene projection line 26b along the top,

[0188] Scene 14 (e.g., based on video geometric correlation parameters 24 of another reference image 20d) is determined to be projected onto another reference image 20d / F. si Second image position 134 / q si The second scene projection line 26d along the top is achieved by placing the tail of the first MVP 130 on the first image position 132, with the head of the first MVP 130 pointing towards the second image position, and...

[0189] Determine a scene point 52 (e.g., one of 52a to 52c) on the shortest line 51 connecting any point on the first scene projection line 26b and any point on the second scene projection line 26d.

[0190] For example, if the source image is 20b / F di Equal to reference image 20c / F st Then the video decoder 100 and the corresponding video encoder can be further configured (for example, using video geometry correlation parameters 24 and scene points 52).

[0191] Determine the fourth image position 138 / q of the current image 20a. do Scene point 52 (e.g., based on video geometric correlation parameters 24 of the current image 20a) is projected onto the fourth image position, and

[0192] Determine the reserved MVP 126 in order to define the fourth image position 138 and the first image position 132 / q diThe offset between or between the fourth image position 138 and the intermediate image position, the intermediate image position being determined by the intersection of scene point 52 on the shortest line 51 and the first scene projection line 26b (see 52a / f). d The midpoint between the two points is the source image 20b / F. di It is generated by projection onto the surface.

[0193] According to the embodiment, the intersection point 52a / f of the shortest line 51 and the first scene projection line 26b is... d This can be considered as scene point 52, and the video decoder 100 and the corresponding video encoder can be configured to determine the fourth image position 138 / q of the current image 20a. do Scene point 52 is projected onto the fourth image position, and a predetermined MVP 126 is determined to define the fourth image position 138 and the first image position 132 / q. di The offset between them. Alternatively, the midpoint 52b or any other point on the shortest line 51 may also be considered scene point 52.

[0194] According to another embodiment, the midpoint 52b can be regarded as scene point 52, or the intersection of the shortest line 51 on the shortest line 51 and the first scene projection line 26b (see 52a / f). d The intersection point of the shortest line 51 and the second scene projection line 26d (see 52c / f) s Any point between (including intersection point 52c / f) s ) can be considered as scene point 52 (e.g., intersection point 52c / f) s It could also be considered a scene point, and the intersection point 52a / f d (Not considered a scene point). In this situation, the video decoder 100 and the corresponding video encoder can be configured to determine the fourth image position 138 / q of the current image 20a. do Scene point 52 is projected onto the fourth image position, and a predetermined MVP 126 is determined to define the offset between the fourth image position 138 and the intermediate image position, which is the intersection of scene point 52 on the shortest line 51 and the first scene projection line 26b (see 52a / f). d The midpoint between the two points is the source image 20b / F. di It is produced by projection onto the surface.

[0195] For example, if another reference image is 20d / F si Equal to reference image 20c / F st Then the video decoder 100 and the corresponding video encoder can be configured (for example, using video geometry correlation parameter 24 and scene point 52).

[0196] Determine the fourth image position 138 / q of the current image 20a. do Scene point 52 (e.g., based on video geometric correlation parameters 24 of the current image 20a) is projected onto the fourth image position, and

[0197] Determine the reserved MVP 126 in order to define the fourth image position 138 and the second image position 134 / q si The offset between or between the fourth image position 138 and another intermediate image position, the other intermediate image position being the intersection of scene point 52 on the shortest line 51 and the intersection of the shortest line 51 and the second scene projection line 26d (see 52c / f). s Another midpoint between ) and another reference image 20d / F si It is produced by projection onto the surface.

[0198] According to the embodiment, the intersection point 52c / f of the shortest line 51 and the second scene projection line 26d is... s This can be considered as scene point 52, and the video decoder 100 and the corresponding video encoder can be configured to determine the fourth image position 138 / q of the current image 20a. do Scene point 52 is projected onto the fourth image position, and a predetermined MVP 126 is determined to define the fourth image position 138 and the second image position 134 / q. si The offset between them. Alternatively, the midpoint 52b or any other point on the shortest line 51 may also be considered scene point 52.

[0199] According to another embodiment, the midpoint 52b can be regarded as scene point 52, or the intersection of the shortest line 51 on the shortest line 51 and the first scene projection line 26b (see 52a / f). d The intersection point of the shortest line 51 and the second scene projection line 26d (see 52c / f) s Any point between (including intersection point 52a / f) d ) can be considered as scene point 52 (e.g., intersection point 52a / f) d It could also be considered a scene point, and the intersection point 52c / f s (Not considered a scene point). In this situation, the video decoder 100 and the corresponding video encoder can be configured to determine the fourth image position 138 / q of the current image 20a. do Scene point 52 is projected onto the fourth image position, and a predetermined MVP 126 is determined to define the offset between the fourth image position 138 and another intermediate image position, which is the intersection of scene point 52 on the shortest line 51 and the second scene projection line 26d (see 52c / f). s Another midpoint between ) and another reference image 20d / F siIt is produced by projection onto the surface.

[0200] For example, if the source image 20b, the reference image 20c, and another reference image 20d are different images from each other, the video decoder 100 and the corresponding video encoder can be configured (e.g., using video geometric correlation parameter 24 and scene point 52).

[0201] Determine the third image position 136 / q of reference image 20c. so Scene point 52 (e.g., based on video geometric correlation parameters 24 of reference image 20c) is projected onto the third image position.

[0202] Determine the fourth image position 138 / q of the current image 20a. do Scene point 52 (e.g., based on video geometric correlation parameters 24 of the current image 20a) is projected onto the fourth image position, and

[0203] Determine the reserved MVP 126 in order to define the offset between the third image position 136 and the fourth image position 138.

[0204] The pre-defined MVP 126 generated by the newly proposed method is also referred to as MVP in this paper. temp,G It can also be used in other MVPs, for example, it can be added to the MVP list or it can be used as a time motion vector prediction sub-MVP. temp Alternatives. For example, the pre-selected MVP 126 can be inserted into the list of MVP candidates as a replacement for the time-predicted MVP or the first MVP 130.

[0205] For example, video decoder 100 can be configured to reconstruct current block 124 by inserting predetermined MVP 126 into a list of MVP candidates, selecting a selected MVP from the list of MVP candidates, and reconstructing the current block 124 using the selected MVP. Similarly, the corresponding encoder can be configured to encode current block 124 by inserting predetermined MVP 126 into a list of MVP candidates, selecting a selected MVP from the list of MVP candidates, and encoding the current block 124 using the selected MVP. The encoder is optionally configured to encode an index into data stream 16, wherein the index is indexed to the selected MVP in the list of MVP candidates. Specifically, video decoder 100 can be configured to decode, for example, an index pointing to a list of MVP candidates from data stream 16, and use the index to select a selected MVP from the list of MVP candidates.

[0206] Use threshold d active , condition d min <d active It can be used to determine MVPtemp,G Whether to use it as an additional MVP. temp,G Used as MVP temp In the case of a replacement, such conditions can determine the MVP. temp,G Should the MVP be replaced? temp .

[0207] For example, the video decoder 100 and the corresponding video encoder can be configured based on a length metric of the shortest line 51 (e.g., the length of the shortest line 51, see, for example, [reference needed]). Figure 8 27 in the middle, or the shortest line 51 to the source image 20b / F di Above and / or to another reference image 20d / F si Up or up to the current image 20a / F dt Above or to reference image 20c / F st A quality metric is determined by the length of the projection on any other image, and if the quality metric meets a predetermined criterion (e.g., less than a predetermined criterion, such as less than a predetermined length metric), the predetermined MVP 126 is inserted into the list of MVP candidates. According to an embodiment, if the quality metric meets the predetermined criterion, the predetermined MVP 126 may be inserted into the list of MVP candidates as a replacement for the temporal prediction MVP or the first MVP 130.

[0208] Spatial motion vector prediction

[0209] Situation F dt =F di That is, the current image 20a is equal to the source image 20b, which is called spatial motion vector prediction, i.e., the input MV and the target MV have the same destination image. This situation involves two or three different images, i.e., a single shared destination image, i.e., the current image 20a, and one or two other source images (if F st =F si If one is true, then there is one; otherwise there are two; that is, reference image 20c and / or another reference image 20d).

[0210] In the case of three images (i.e., different source images; that is, F...), st ≠F si VVC does not use this type of input MV for MVP. In contrast, the newly proposed method described in the above sections reduces the first MVP to 130 MV. i The scene point is 52, for example, 52b / f. m Project scene point 52 onto reference image 20c / F st and current image 20a / F dt Up, thus obtaining the projection point q so and q doThat is, the third image position 136 and the fourth image position 138, and calculate a predetermined MVP 126, for example, to define the offset (e.g., difference) between the third image position 136 and the fourth image position 138, for example, through the MVP. spatial,G =q so - q do .

[0211] Spatial motion vector prediction can be applied to generate MVPs for translational and affine prediction blocks. That is, for each control point of the affine prediction block, such as certain corner points of the affine prediction block of the current image, the spatial prediction vector will be modified in the manner described above, for example, the first MVP 130, and the program will even compensate for spatial prediction motion vectors that may originate from different reference images.

[0212] As mentioned above Figure 8 and Figure 9 As described, the video decoder 100 and the corresponding video encoder can be configured to read from the source image 20b / F. di The reference block 128 (e.g., previously decoded / encoded) yields the first MVP 130, and the first MVP 130 is modified using video geometry-related parameters 24 by the following operations:

[0213] Using video geometric correlation parameter 24, the first image position 132 / q of reference block 128 di And the first MVP 130,

[0214] Scene 14 is determined to be projected onto the first image position 132 / q based on video geometric correlation parameters 24 of source image 20b. di The first scene projection line 26b along the top (e.g., connecting q) di The line with the center of the camera projection 31b, such as Figure 8 As can be seen, the line crosses scene point 52a / f. d ),

[0215] Determine the second image position 134 / q of scene 14 (e.g., based on video geometric correlation parameters 24 of another reference image 20d) projected onto the other reference image 20d. si The second scene projection line 26d along the top is achieved by placing the tail of the first MVP 130 at the first image position 132 / q. diAbove, the head of the first MVP 130 points to the second image position (it should be noted that when referencing another image 20d from one image 20b, any motion vector such as 130 corresponds to the in-plane reference image vector 130' which points from the control / block position 132 of the inter-frame prediction block 128 relative to the reference image 20b to the position 134 pointed to by the motion vector 130), and

[0216] Determine the shortest line 51 connecting any point on the first scene projection line 26b and any point on the second scene projection line 26d (e.g., connecting scene points 52a / f). d and scene point 52c / f s Scene points on line 51 (e.g., 52b / f) m 52a / f d Or 52c / f s (For example, the shortest line 51 can connect the first scene projection line 26b and the second scene projection line 26d with the shortest distance between them in 3D space), and

[0217] Using video geometry-related parameters 24 and scene points 52.

[0218] Determine the reference image 20c / F st The third image position 136 / q so Scene point 52 (e.g., based on video geometric correlation parameters 24 of reference image 20c) is projected onto the third image position.

[0219] Determine the current image 20a / F dt The fourth image position 138 / qdo, scene point 52 (e.g., based on the video geometric correlation parameter 24 of the current image 20a) is projected onto the fourth image position.

[0220] Determine the predefined MVP 126 to define the offset (e.g., difference) between the third image position 136 and the fourth image position 138.

[0221] The current block 124 and the reference block 128 are within the same image, and the current image 20a is equal to the source image 20b. Optionally, the reference block 128 may be equal to the current block 124.

[0222] like Figure 11As exemplarily shown, video decoder 100 and corresponding video encoder can be configured to derive a first MVP 130 from reference block 128 of the current image 20a (e.g., a block in the previously decoded / encoded spatial neighborhood of the current block 124), and modify the first MVP 130 using video geometry correlation parameters 24 by:

[0223] Using video geometric correlation parameters 24, the first image position 132 of reference block 128, and the first MVP 130,

[0224] Determine the first scene projection line 26b along which scene 14 (e.g., based on the video geometric correlation parameters 24 of the current image 20a) is projected onto the first image position 132.

[0225] Determine the second scene projection line 26d along which scene 14 (e.g., based on video geometric correlation parameters 24 of another reference image 20d) is projected onto the second image position 134 of the other reference image 20d, by placing the tail of the first MVP 130 onto the first image position 132, with the head of the first MVP 130 pointing towards the second image position (note that when referencing another image 20d from one image 20a, any motion vector such as 130 corresponds to an in-plane reference image vector pointing from the control / block position 132 of the inter-frame prediction block 128 relative to the reference image 20b to the position 134 pointed to by the motion vector 130), and

[0226] Determine scene point 52 (e.g., one of 52a to 52c) on the shortest line 51 connecting any point on the first scene projection line 26b and any point on the second scene projection line 26d, and

[0227] Using video geometry-related parameters 24 and scene points 52.

[0228] The third image position 136 of the reference image 20c is determined, and scene point 52 (e.g., based on the video geometric correlation parameters 24 of the reference image 20c) is projected onto the third image position.

[0229] The fourth image position 138 of the current image 20a is determined, and scene point 52 (e.g., based on the video geometric correlation parameter 24 of the current image 20a) is projected onto the fourth image position.

[0230] Determine the predetermined MVP 126 to define the offset (e.g., difference) between the third image position 136 and the fourth image position 138.

[0231] As described above and Figure 8 and Figure 11As shown, scene point 52 can be determined as the midpoint 52b / f on the shortest line 51, located in the middle of the shortest line. m That is, the distance from scene point 52 to the first scene projection line 26b can be the same as the distance from the second scene projection line 26d. Scene point 52, for example, f m Alternatively, it can be selected as another point on the shortest line 51. For example, scene point 52 may also be located elsewhere on the shortest line 51, such as being closer to the first scene projection line 26b than to the second scene projection line 26d, closer to the second scene projection line 26d than to the first scene projection line 26b, or directly on the first scene projection line 26b (see scene point 52a / f). d ) or directly on the second scene projection line 26d (see scene point 52c / f) s ).

[0232] In the three-image scenario, the midpoint 52b / f is replaced. m Projected onto current image 20a / F dt Above, by q do Choose to be equal to q di That is, by selecting the fourth image position 138 as equal to the first image position 132, this projection calculation can be omitted. In this case, the video decoder 100 and the corresponding video encoder can be configured to determine a predetermined MVP 126 to define the offset between the third image position 136 and the first image position 132 or between the third image position 136 and the intermediate image position. The intermediate image position is generated by the projection of the midpoint between the scene point 52 on the shortest line 51 and the intersection point 52a of the shortest line 51 and the first scene projection line 26b onto the current image 20a. Furthermore, instead of using the midpoint 52b / f... m Projected onto reference image 20c / F st Up, q so It can be selected as equal to f d To F st The projection onto the reference image 20c, that is, the third image position 136 can be selected as equal to the projection from scene point 52a to the reference image 20c. In other words, scene point 52 can be the intersection point 52a of the shortest line 51 and the first scene projection line 26d.

[0233] In the two-image scenario, the MVP derived from VVC is equal to the input MV. The proposed method can compute multiple predefined MVPs126, for example, also denoted as MVP. spatial,G , in f d Not equal to f s In the case where scene point 52a is not equal to scene point 52c, the pre-defined MVP is not equal to the input MV. This can be resolved by connecting 52a / f. d and 52c / f sAny point p on the shortest line 51 is projected onto 20c / F st and 20a / F dt To obtain the projection point 136 / q so and 138 / q do This leads to the predetermined MVP 126. Specifically, p = f can be chosen. d p = f s Or p = f m .

[0234] According to an embodiment, the video decoder 100 and the corresponding video encoder can be configured to distinguish between three-image states and two-image states.

[0235] For example, video decoder 100 and its corresponding video encoder can be configured to modify the first MVP 130 using video geometry-related parameters 24 through the following operations:

[0236] Using video geometric correlation parameters 24, the first image position 132 of reference block 128, and the first MVP 130,

[0237] Scene 14 (e.g., based on video geometric correlation parameters 24 of the source image represented by the current image 20a) is determined to be projected onto the first image position 132 / q. di The first scene projection line 26b along the top,

[0238] Scene 14 (e.g., based on video geometric correlation parameters 24 of another reference image 20d) is determined to be projected onto another reference image 20d / F. si Second image position 134 / q si The second scene projection line 26d along the top is achieved by placing the tail of the first MVP 130 on the first image position 132, with the head of the first MVP 130 pointing towards the second image position, and...

[0239] Determine the shortest line 51 connecting any point on the first scene projection line 26b and any point on the second scene projection line 26d, and

[0240] Using video geometry-related parameters 24 and the shortest line 51.

[0241] If another reference image is 20d / F si Unlike the reference image 20c / F associated with the pre-ordered MVP 126 st ,but

[0242] Scene point 52 is defined as the midpoint 52a of the shortest line 51, or as the intersection point 52a / f of the shortest line 51 and the first scene projection line 26b. d ,

[0243] Determine the third image position 136 / q of reference image 20c. so Scene point 52 (e.g., based on video geometric correlation parameters 24 of reference image 20c) is projected onto the third image position.

[0244] MVP 126 is scheduled for definition.

[0245] The third image position 136 and the first image position 132 / q di offset between, or

[0246] The offset between the third image position 136 and the intermediate image position, where the intermediate image position is determined by the intersection point 52a / f of the scene point 52 on the shortest line 51 and the first scene projection line 26b. d The midpoint between the two points to the source image 20b / F di Produced by projection on, or

[0247] The third image position 136 and the fourth image position 138 / q do The offset between them, the fourth image position is generated by the projection of scene point 52 (e.g., midpoint 52b) onto the current image 20a, and

[0248] If another reference image is 20d / F si Equal to reference image 20c / F st ,but

[0249] Define scene point 52 as a point on the shortest line 51.

[0250] Determine the third image position 136 / q of reference image 20c. so Scene point 52 (e.g., based on video geometric correlation parameters 24 of reference image 20c) is projected onto the third image position.

[0251] Determine the fourth image position 138 / q of the current image 20a. do Scene point 52 (e.g., based on video geometric correlation parameters 24 of the current image 20a) is projected onto the fourth image position.

[0252] Determine the pre-defined MVP to define the offset between the third image position 136 and the fourth image position 138.

[0253] Optionally, if another reference image is 20d / F si Equal to reference image 20c / F st Then scene point 52 can be the intersection point 52a / f of the shortest line 51 and the first scene projection line 26b. d The intersection point 52c / f of the shortest line 51 and the second scene projection line 26d.s , or the midpoint 52b on the shortest line 51.

[0254] The motion vector prediction sub-MVP generated by the proposed method spatial,G It can also be used for other MVPs; for example, it can be added to the MVP list. Use the threshold d. active , condition d min <d active It can be used to determine MVP spatial,G Whether to use it as an additional MVP. This option can be implemented as described in the subsection on temporal motion vector prediction.

[0255] The motion vectors derived considering the camera's rotation without any change in position are as follows.

[0256] In the case where all cameras involved in the prediction of image points are at the same position, it is impossible to obtain two images at 20d / F. si and 20b / F di Determine the specific 3D intersection point, namely scene point 52. Conversely, originate from image 20d / F. si and 20b / F di Rays from either of the two associated first cameras, such as first scene projection line 26b or second scene projection line 26d, are associated with 20c / F. st and 20a / F dt The images from the third camera intersect the camera plane, and the resulting point is the predicted image point 136 / q. so and 138 / q do As an alternative example of this case, by calculating the direction vector dir i = 0.5*(dir si + dir di The third ray is used to intersect the camera plane of the third camera, using two rays 26b and 26d (with direction vector dir) originating from the first two cameras. si and dir di ), where the intersection point is the predicted image point 136 / q so and 138 / q do .

[0257] Therefore, the video decoder 100 and corresponding video encoder described herein can be configured to derive a predetermined motion vector predictor 126 by: deriving a first MVP 130 from a previously decoded portion 128 of the video for the current block 124; and modifying the first MVP 130 using video geometry correlation parameters 24 to obtain the predetermined MVP 126 by examining the camera position indicated by the video geometry correlation parameters 24 and performing geometric derivation depending on the camera position (see [link to documentation]). Figures 12 to 14 ):

[0258] For example, such as Figure 12 As shown, if the current image 20a, the reference image 20c, and the source image 20b are associated with the same camera position (but not, for example, with the same camera orientation; that is, there is no translational change between the cameras in the images, only a rotational change), then the geometry of the predetermined MVP 126 can be obtained by the following operation:

[0259] The scene 14 (e.g., based on the video geometric correlation parameters 24 of the source image 20b) is determined to be projected onto the first image position 132 of the reference block 128 in the source image 20b along the first scene projection line 26b (as per the context of the first image position 132 of the reference block 128 in the source image 20b). Figure 9 As explained, reference block 128 in source image 20b can be co-located with current block 124 in current image 20a.

[0260] The third image position 136 of the reference image 20c is determined as the intersection point of the first scene projection line 26b and the reference image 20c.

[0261] The fourth image position 138 of the current image 20a is determined as the intersection point of the first scene projection line 26b and the current image 20a, and

[0262] Determine the predetermined MVP 126 to define the offset (e.g., difference) between the third image position 136 and the fourth image position 138.

[0263] For example, such as Figure 13 As shown, if the current image 20a, reference image 20c, and another reference image 20d are associated with the same camera position (but not, for example, with the same camera orientation; that is, there is no translational change between the cameras in the images, only a rotational change), then the geometry of the predetermined MVP 126 can be obtained by the following operation:

[0264] Determine the second scene projection line 26d along which scene 14 (e.g., based on video geometric correlation parameters 24 of another reference image 20d) is projected onto the second image position 134 of the other reference image 20d, by placing the tail of the first MVP 130 derived from reference block 128 onto the first image position 132, with the head of the first MVP 130 pointing towards the second image position.

[0265] The third image position 136 of reference image 20c is determined as the intersection point of the second scene projection line 26d and reference image 20c.

[0266] The fourth image position 138 of the current image 20a is determined as the intersection point of the second scene projection line 26d and the current image 20a, and

[0267] Determine the predetermined MVP 126 to define the offset (e.g., difference) between the third image position 136 and the fourth image position 138.

[0268] For example, such as Figure 14 As shown, if the current image 20a, reference image 20c, source image 20b, and another reference image 20d are associated with the same camera position (but not, for example, with the same camera orientation; that is, there is no translational change between the cameras in the images, only a rotational change), then the geometry of the predetermined MVP 126 can be obtained by the following operation:

[0269] Option a) Determine the first scene projection line 26b along which scene 14 (e.g., based on video geometric correlation parameters 24 of source image 20b) is projected onto the first image position 132 of reference block 128 in source image 20b (as per the context of the first scene projection line 26b). Figure 9 As explained, reference block 128 in source image 20b can be co-located with current block 124 in current image 20a.

[0270] The scene (e.g., based on video geometric correlation parameters 24 of another reference image 20d) is projected onto the second image position 134 of the other reference image 20d along the second scene projection line 26d, by placing the tail of the first MVP 130 derived from the reference block 128 onto the first image position 132, with the head of the first MVP 130 pointing towards the second image position.

[0271] A third scene projection line 26' is determined based on the first scene projection line 26b and the second scene projection line 26d, wherein the third scene projection line 26' is correlated with the arithmetic mean of the first scene projection line 26b and the second scene projection line 26d.

[0272] The third image position 136 of reference image 20c is determined as the intersection point of the third scene projection line 26' and reference image 20c, and

[0273] The fourth image position 138 of the current image 20a is determined as the intersection point of the third scene projection line 26' and the current image 20a.

[0274] Determine the predefined MVP 126 to define the offset (e.g., difference) between the third image position 136 and the fourth image position 138;

[0275] or

[0276] Option b) Determine the first scene projection line 26b along which scene 14 (e.g., based on video geometric correlation parameters 24 of source image 20b) is projected onto the first image position 132 of reference block 128 in source image 20b.

[0277] The third image position 136 of the reference image 20c is determined as the intersection point of the first scene projection line 26b and the reference image 20c.

[0278] The fourth image position 138 of the current image 20a is determined as the intersection point of the first scene projection line 26b and the current image 20a, and

[0279] Determine the predefined MVP 126 to define the offset (e.g., difference) between the third image position 136 and the fourth image position 138;

[0280] or

[0281] Option c) Determine the second scene projection line 26d along which scene 14 (e.g., based on video geometric correlation parameters 24 of another reference image 20d) is projected onto the second image position 134 of the other reference image 20d, by placing the tail of the first MVP 130 derived from reference block 128 onto the first image position 132, with the head of the first MVP 130 pointing towards the second image position.

[0282] The third image position 136 of reference image 20c is determined as the intersection point of the second scene projection line 26d and reference image 20c.

[0283] The fourth image position 138 of the current image 20a is determined as the intersection point of the second scene projection line 26d and the current image 20a, and

[0284] Determine the predetermined MVP 126 to define the offset (e.g., difference) between the third image position 136 and the fourth image position 138.

[0285] The video decoder 100 and its corresponding video encoder can be configured, for example, to execute only one of options a) to c) by default. For instance, options available to the video decoder 100 and its corresponding video encoder can be predefined, and other options may not be selectable. Alternatively, the video decoder 100 and its corresponding video encoder can be configured to select one of options a) to c) block by block. Therefore, all options a) to c) are available to the video decoder 100 and its corresponding video encoder.

[0286] For example, otherwise, that is, if none of the camera position conditions defined above apply, the video decoder 100 and the corresponding video encoder can be configured to modify the first MVP 130 using the video geometry-related parameters 24 to obtain the predetermined MVP 126 described above, such as in this section (i.e., in section "2 Motion Vector Prediction"), see, for example, "Temporal Motion Vector Prediction" and "Spatial Motion Vector Prediction". Alternatively, the video decoder 100 and the corresponding video encoder can be configured to check only one of the camera position conditions, that is, regarding Figure 12 or Figure 13 or Figure 14 The explained camera position conditions, and otherwise, if the corresponding camera position conditions are not met, the first MVP 130 is modified using video geometry related parameters 24 to obtain the predetermined MVP 126 described above, such as in this section (i.e., in section “2 Motion Vector Prediction”), see, for example, “Temporal Motion Vector Prediction” and “Spatial Motion Vector Prediction”.

[0287] 3. Motion-compensated sample prediction (e.g., using scene models)

[0288] The following describes an example of a novel motion-compensated sample prediction method.

[0289] Scene Model

[0290] In the variants, see, for example Figure 15 An intermediate scene model 300 (such as a mesh or point cloud) is used to collect information about scene 14 during the decoding / encoding process.

[0291] Scene point estimation, such as that described above in Section 2 Motion Vector Prediction, can be used to determine one or more scene points 52 for each inter-frame prediction block, depending on the inter-frame prediction mode used (e.g., those scene model points 302 from which scene model 300 can be derived). More precisely, scene point estimation assumes or is achieved by selecting, among the motion vectors of the corresponding images, such that these motion vectors primarily describe the parallax between the two camera viewpoints of their base images (i.e., the corresponding image and the reference image) (e.g., see the section on...). Figure 7The motion vector 46 described is used to determine the scene model 300 (or 3D scene model) for each decoded image 20 based on the motion vector. The motion vector (MV) describing the (primarily) real scene motion can be excluded from the estimation. Therefore, the scene model 300 can be constructed after decoding an image (e.g., after decoding the current image 20a) for use in subsequent images to be decoded. As described above, the MV is estimated using inter-frame prediction blocks. Advantageously, scene model construction can help estimate scene model regions for which, for example, no MV may be available due to the selection of intra-frame prediction modes for blocks in such regions. For this purpose, inter-frame / extrapolation can be used. Scene model estimation can rely on finding scene point 52b / f in scene 14. m (or similar points) of the above concept. Scene point 52a / f d With 52c / f s The distance between them can be used to exclude certain MVs from contributing to the scene model estimation.

[0292] In the case of translational motion, a scene point 52 estimated from the scene point can be used to determine the distance d between the object in scene 14 and the camera or image. SP In the next step, the corner points of the predicted block (or source block) are set to the same d. SP (The obtained scene point distance (a rectangle spanning a line parallel to the image plane)) is projected into scene 14, see [link / reference]. Figure 15 vectors in to Therefore, scene model 300 can be composed of multiple projected rectangles or vectors. to The scene model point 302 it points to can be defined by multiple scene points 52 representing the scene model point 302.

[0293] For affine prediction blocks, depending on the affine model used (4 or 6-parameter model), the model is extended to produce an 8-parameter model, and scene point estimation is used for the block corners against 4 control points (see about...). Figure 7 Each control point in the described control point 44) individually yields a scene point 52. The encoder performs the same operation on the encoded image based on the encoded motion vector, so that the decoder and encoder can use the same scene model 300 for MVP determination and / or sample prediction.

[0294] For example, the video decoder and corresponding video encoder described herein can be configured to determine the scene model 300 based on image motion vectors, such as a first MVP 130 based on the current image 20a or a first MVP 130 based on a previously decoded / encoded image (e.g., a first MVP based on an inter-frame prediction block of the current or previously decoded / encoded image) and based on video geometric correlation parameters 24. (See also: ...) Figures 8 to 14 As described, the video decoder 100 and the corresponding video encoder can be configured to use the scene model 300 and video geometry-related parameters 24 to determine a predetermined MVP 126. (See also: Regarding...) Figures 5 to 7 As described, the video decoder 10 and the corresponding video encoder can be configured to use the scene model 300 and video geometry-related parameters 24 to find the corresponding pixel 23 in the (e.g., previously decoded / encoded) reference image 20b.

[0295] By determining each of one or more control points (CPs) (e.g., the first image position 132 described herein may represent a control point) of an inter-frame prediction block (e.g., reference block 128 as described herein) for a predetermined image 20b, a scene model point 302 for forming the basis of the scene model 300 is determined (e.g., the scene model 300 is then determined based on the scene model point 302, such as by scene point 52 that contributes to the point cloud of the scene model 300, or by the vertices of facial regions (e.g., triangles) that form the mesh of the scene model 300). The scene model 300 can be determined (e.g., by a video decoder or video encoder as described herein) based on the MV of a predetermined image (pP) (e.g., the current image after decoding / encoding), or based on its MV and the MV of one or more previous (e.g., previously decoded / encoded) images. The determination of the scene model point 302 for each of the one or more control points of the inter-frame prediction block can be performed by:

[0296] The motion vector (e.g., the first MVP 130 described herein) is determined from the corresponding control point to the source image position (sPP) (e.g., the second image position 134 described herein may represent the source image position) in the corresponding reference image (cRP) (e.g., another reference image 20d described herein may represent the cRP). The motion vector is then encoded and decoded in the data stream of the inter-frame prediction block at the corresponding control point.

[0297] Using video geometry correlation parameter 24,

[0298] Scene 14 (e.g., based on video geometric correlation parameters 24 of a predetermined image (e.g., 20b)) is projected onto corresponding control points (e.g., q). diThe first scene projection line 26b along / 132),

[0299] Scene 14 (e.g., based on video geometric correlation parameters 24 of the corresponding reference image (e.g., 20d)) is projected onto the source image location (see q) si / 134)(by placing the tail of MV 130 on the corresponding control point, with the head of MV 130 pointing towards the source image position) along the second scene projection line 26d, and

[0300] Determine the shortest line 51 connecting any point on the first scene projection line 26b and any point on the second scene projection line 26d (e.g., connecting 52a / f). d and 52c / f s Scene point 52 on the line (e.g., 52b / f) m 52a / f d Or 52c / f s ),as well as

[0301] Scene model point 302 is determined as follows:

[0302] Determine scene model point 302 as scene point 52, or

[0303] The scene model point 302 is determined by determining the distance between scene point 52 and the predetermined image (see 20b) and by determining that scene model point 302 is located on the first scene projection line 26b at that distance.

[0304] For example, scene model 300 is a mesh or point cloud.

[0305] Optionally, the quality metric determined based on the length of the shortest line 51 does not conform to a predetermined criterion (e.g., not less than a predetermined maximum length, see condition d described herein). min <d active Scene model point 302.

[0306] According to an embodiment, one control point (CP) (e.g., a corner point of these blocks, such as the top left corner point) is used to translate the inter-frame prediction block, and / or more than one control point (CP) (e.g., two or more corner points of these blocks) is used to affine the inter-frame prediction block.

[0307] Scene points using the corner points of the projection block, which are triangular corner points, and the obtained scene points 52 associated with the prediction block can be stored as a grid with two triangles. When using a point cloud, each of the scene points 52 can be stored in the point cloud, or alternatively, the sample-by-sample back projection surrounded by the projection block can be stored in the point cloud.

[0308] The intermediate scene model can optionally be updated using decoder motion information that only considers the currently decoded image, or alternatively, a scene model 300 can be constructed that has motion information from several decoded images that may be bounded by a time layer (e.g., only T). emporalLayer Threshold TL (Can contribute to the model). In the variant, scene model 300 is optimized and used within an intra-frame period and reset at the next random access point (see below, for example).

[0309] Scene model 300 can be additionally determined based on motion vectors of inter-frame prediction blocks from one or more previous images (e.g., previously decoded / encoded images).

[0310] Optionally, the determination of scene model 300 can be constrained by inter-frame prediction blocks of images that meet predetermined criteria (e.g., time base layer up to predetermined maximum time layer).

[0311] Scene model optimization

[0312] When using motion information from multiple decoded images to construct a scene model 300, it can be beneficial to weight scene points 52 according to their relevance. For example, scene points 52 originating from affine prediction blocks are superior to scene points 52 originating from translation prediction blocks within the same scene region. Furthermore, translation patterns, due to merging or skipping patterns, and assuming RD-optimized encoding / decoding, produce more accurate predictions with less distortion, even with higher rate costs, resulting in a better scene model 300. In contrast, evaluating the presence or energy of transmitted residual signals, such as a higher number of residual coefficients or a higher sum of absolute values ​​associated with prediction blocks, will reduce the preference for the corresponding scene points 52.

[0313] For example,

[0314] ● Compared to scene model points 302 derived from motion vectors in translation mode, scene model 300 is determined based primarily on scene model points 302 derived from inter-frame prediction blocks encoded and decoded in affine mode, and / or

[0315] ● Compared to scene model points 302 derived from motion vectors of inter-prediction blocks encoded and decoded individually in skip, direct, or merge modes, scene model 300 is determined primarily based on scene model points 302 derived from motion vectors of these inter-prediction blocks encoded and decoded individually for inter-prediction blocks in the data stream. And / or

[0316] ● Determine scene model 300 based on preferences varying between scene model points 302, such that: higher preferences result in lower prediction residual signals encoded into the data stream according to predetermined metrics of inter-frame prediction blocks, scene model points 302 originating from motion vectors of inter-frame prediction blocks, and / or

[0317] ● Determine scene model 300 based on preferences varying between scene model points 302, such that: higher preferences result in images of inter-frame prediction blocks being temporally closer, scene model points 302 originate from motion vectors of inter-frame prediction blocks, and / or

[0318] ●The scene model 300 is determined by the preference that varies between scene model points, such that: the higher the preference, the smaller the quantizer step size of the inter-frame prediction block, and the scene model point 302 is derived from the motion vector of the inter-frame prediction block.

[0319] The video decoder and corresponding video encoder described in this paper can be configured to determine the predetermined MVP 126 using the scene model 300 and video geometry-related parameters 24 (e.g., regarding...). Figures 8 to 14 The described video decoder 100 and corresponding video encoder) or when using scene model 300 and video geometry related parameters 24 to derive the interior of a block (e.g., regarding...) Figures 5 to 7 The described video decoder 10 and corresponding video encoder are weighted according to parts of the scene model 300 (e.g., points in the point cloud or facial regions of the mesh).

[0320] ● The portion of motion vectors originating from inter-frame prediction blocks encoded and decoded in affine mode has a higher weight compared to the portion originating from translation mode, and / or

[0321] ● Compared to the portion of motion vectors originating from inter-prediction blocks encoded and decoded individually for each inter-prediction block in the data stream, the portion of motion vectors originating from these inter-prediction blocks has a higher weight, and / or

[0322] ● The higher the weight, the lower the prediction residual signal encoded into the data stream according to the predetermined metric of the inter-frame prediction block, which partly originates from the motion vectors of the inter-frame prediction block, and / or

[0323] ● Higher weights mean that the images of the inter-frame prediction blocks are temporally closer, partly due to the motion vectors of the inter-frame prediction blocks, and / or

[0324] ● The higher the weight, the smaller the quantizer step size of the inter-frame prediction block, which is partly derived from the motion vector of the inter-frame prediction block.

[0325] It should be noted that the construction of scene model 300 may be based on all scene model points 302, and for example, for each image 20, but the construction may favor certain scene model points 302 compared to other scene model points, such as making the scene model 300 "sparse" in areas where several scene model points 302 are close to each other. The resulting scene model 300 can then be used as is for MVP creation or deformation, that is, without distinguishing the origin of individual parts of scene model 300. However, scene model 300 may steadily increase in size with additional scene model points 302, and its use in MVP creation or deformation depends on the origin of individual parts.

[0326] When using a mesh-based scene model (e.g., scene model 300 with mesh 590, such as...) Figure 16 When using the (shown) for prediction, the destination location (see 132 / q) is taken into account. di The image is projected onto scene model 300, and the intersection point of the projection ray in front of image 20 and the mesh triangle of scene model 300 is selected as scene point P for back projection to determine the location of scene point. BPi 608, see 134 / q si The intersection point has the minimum distance to the camera.

[0327] like Figure 16 As shown, regarding Figures 8 to 14 The described video decoder 100 and corresponding video encoder can be configured to determine the current image 602 (corresponding to the usage scenario model 300 and video geometric correlation parameters 24) based on the usage scenario model 300. Figure 9 The current block 600 of the current image 20a) (corresponding to Figure 9 When the current block 124 is scheduled to be MVP 126,

[0328] Determine the scene projection line 604 along which scene 14 (e.g., based on the video geometry correlation parameters 24 of the current image 602) is projected onto the image position 606 of the current block 600 (e.g., the top left corner of the current block 600, the center of the current block 600, or the bottom right corner of the current block 600).

[0329] A final use point 608 on the scene projection line 604 is determined based on one or more intersection points 608 between the scene projection line 604 and the face regions 310 (e.g., intersecting with one or more of them, each face region 310 having one intersection point; for example, a face region 310 may also be represented as a surface region, a rectangular region, a triangular region, or a region spanned by the scene model point 302 of the mesh 590) of the scene model 300 (e.g., the most recent one).

[0330] The final point 608 is projected 26 onto the reference image 610 referenced by the predetermined MVP 126 (corresponding to...). Figure 9 The reference image 20c is used to obtain the corresponding image position 612 in the reference image 610, and the predetermined MVP 126 is determined as the offset between the corresponding image position 612 and the image position 606.

[0331] like Figure 16 As shown, regarding Figures 5 to 7 The described video decoder 10 and corresponding video encoder can be configured to use scenario model 300 and video geometry related parameters 24 on (e.g., previously decoded / encoded) reference image 610 (corresponding to) Figure 5 Find the corresponding position 612 in reference image 20b (corresponding to Figure 2 At the corresponding position 22),

[0332] For the current block 600 (corresponding to Figure 5 Each pixel 606 in the current block 18) (corresponding to Figure 5 (pixel 23 in the middle)

[0333] Determine scene 14 (e.g., based on current image 602 (corresponding to...) Figure 5 The video geometry-related parameters 24) of the current image 20a) are projected onto the scene projection line 604 along the corresponding pixel 606 of the current block 600.

[0334] A final use point 608 on the scene projection line 604 is determined based on one or more intersection points 608 of the face regions 310 (e.g., intersecting with one or more of them, each intersection having one intersection point; for example, a face region 310 may also be represented as a surface region, a rectangular region, a triangular region, or a region spanned by the scene model point 302 of the mesh 590 of the scene model 300) and the scene projection line 604.

[0335] The final point 608 is projected onto the reference image 610 to obtain the corresponding position 612 corresponding to the corresponding pixel 606.

[0336] Alternatively, when using a point cloud-based model (e.g., a scene model 300 with point cloud 580), such as... Figure 17 When shown in the figure, the destination location is 606, also see 132 / q diIt can traverse a virtual pyramid or cone 622, the virtual pyramid or cone having a vertex at the camera position and passing through the destination position 606 (see also 132 / q). di The center line of the point cloud within the cone 622 or pyramid 602 is considered as the point point 608 P in the scene. BPi Valid candidates, scene points are used for back projection to determine the source point location 612, see also 134 / q si 608P scene points used for back projection. BPi This can be obtained by averaging the N points with the minimum distance to the camera, or alternatively by selecting the point with the minimum distance to the center line 604 of the cone or pyramid. From scene point 608 P BPi The back projection ray 26 of the camera point to the source image 610 is at position 612 (see also 134 / q) si The image intersects at point i, thus generating the final motion vector MV at position i. i =q si - q di If the destination is location 606 (see also 132 / q) di If the projection into scene 14 does not intersect any of the mesh triangles of scene model 300, or if none of the intersecting scene points are in front of the image, then the source point location 612 cannot be determined. See also 134 / q. si And no music video i This can be used in this location.

[0337] like Figure 17 As shown, regarding Figures 8 to 14 The described video decoder 100 and corresponding video encoder can be configured to determine the current image 602 (corresponding to the usage scenario model 300 and video geometric correlation parameters 24) based on the usage scenario model 300. Figure 9 The current block 600 of the current image 20a) (corresponding to Figure 9 When the current block 124 is scheduled to be MVP 126,

[0338] Determine the scene projection line 604 along which scene 14 (e.g., based on the video geometry correlation parameters 24 of the current image 602) is projected onto the image position 606 of the current block 600 (e.g., the top left corner of the current block 600, the center of the current block 600, or the bottom right corner of the current block 600).

[0339] A final use point 608 is determined based on one or more points 620 (which may be directly derived from scene model points 302) falling into a cone 622 or pyramid based on the point cloud 580 of scene model 300. The cone or pyramid is widened away from image point 606 (or away from camera points associated with the current image 602, where the central axis of the cone 622 (e.g., scene projection line 604) intersects with image point 606) and surrounds scene projection line 604.

[0340] The final point 608 is projected 26 onto the reference image 610 referenced by the predetermined MVP 126 (corresponding to...). Figure 9 The corresponding point 612 in the reference image 610 is obtained on the reference image 20c, and the predetermined MVP 126 is determined as the offset between the corresponding image position point 612 and the image position 606.

[0341] like Figure 17 As shown, regarding Figures 5 to 7 The described video decoder 10 and corresponding video encoder can be configured to use scenario model 300 and video geometry related parameters 24 on (e.g., previously decoded / encoded) reference image 610 (corresponding to) Figure 5 Find the corresponding position 612 in reference image 20b (corresponding to Figure 2 At the corresponding position 22),

[0342] For the current block 600 (corresponding to Figure 5 Each pixel 606 in the current block 18) (corresponding to Figure 5 (pixel 23 in the middle)

[0343] Determine scene 14 (e.g., based on current image 602 (corresponding to...) Figure 5 The video geometry-related parameters 24) of the current image 20a) are projected onto the scene projection line 604 along the corresponding pixel 606 of the current block 600.

[0344] A final use point 608 is determined based on one or more points 620 of the point cloud 580 of the scene model 300 that fall into the cone 622 or pyramid, wherein the cone or pyramid is widened away from the corresponding pixel 606 (or away from the camera point associated with the current image 602, wherein the central axis of the cone 622 (e.g., scene projection line 604) intersects the corresponding pixel 606) and surrounds the scene projection line 604.

[0345] The final point 608 is projected onto the reference image 610 to obtain the corresponding position 612 corresponding to the corresponding pixel 606.

[0346] Z buffer

[0347] The Z-buffer is a data structure that tracks the depth information of each pixel in an image. It stores the depth value (z-value) of each pixel in the image. The depth value represents the distance from the camera to the surface of an object in scene 14 projected onto that pixel. The Z-buffer is typically implemented as a 2D array, where each element corresponds to a pixel in the image and stores the depth value of the closest object to the camera mapped to that pixel.

[0348] In this variant, scene model 300 is used to derive the z-values ​​of all sample locations in a given image and stores these z-values ​​in a z-buffer. Combining these with camera parameters, such as those contained in video geometry-related parameters 24, for each sample location in the given image, a projection from the camera location to scene 14 is performed. If the specific sample location (e.g., see...) Figure 16 and Figure 17 The projected ray of 606 in (e.g., see 606) Figure 16 and Figure 17 If the camera point (604) intersects with one or more mesh triangles in scene 14, then the nearest intersection point in front of the projection plane (e.g., see [reference]) is the camera point. Figure 16 China and Figure 17 The distance between the 608 samples in the camera direction is stored in the z buffer as the z-value of the sample position. For samples whose projection does not intersect any grid triangle or whose intersection point is behind the projection plane of the image, the z-value is considered unknown.

[0349] Prediction from the z-buffer can be improved by predicting effective z-values ​​into regions with unknown z-values. The z-buffer can be organized as an image plane buffer with width and height based on the image size.

[0350] One variant can fill the hole using bilinear interpolation of surrounding valid z-samples and extrapolate the z-values ​​at the z-buffer boundaries in the x and y directions. This is particularly beneficial for motion vector prediction, where extrapolation to the region of unknown z-values ​​with a fixed or dynamic margin can be advantageous.

[0351] Dynamic margin extrapolation can be implemented as a quadtree. The z-buffer is divided into initial blocks. For each block, an approximation of the z-plane is estimated using the effective z-values ​​covered by the block, thereby minimizing the sum of squared errors between all effective z-values ​​in the block and the approximate z-plane. The sum of squared errors expanded by the adjustment term is considered the cost of the block. The examined block is then divided into four quadtree sub-blocks, thus recursively performing the described approximation. At the block level, the minimum cost of the sub-blocks and the sum of the costs of the current block is selected to determine the optimal quadtree. The quadtree estimation is recursively performed on each initial block until a minimum block size is reached. Finally, for the determined optimal quadtree, the z-values ​​in the z-buffer are replaced with the estimated z-plane values ​​of the found optimal sub-segmented blocks.

[0352] The adjustment term is used to balance the accuracy of the approximate plane with the expansion of the effective z-value into the region with invalid z-values.

[0353] Location (see) Figures 8 to 14 132 / q di ) motion vector MV i (see Figures 8 to 14 (130) is obtained by selecting the z value from the z buffer of the destination image, and is performed from the camera point (see Figure 8 31b) passes through point 132 / q di Projected onto scene 14 and the z-value is used to determine the projected ray in the camera direction (see [link]). Figures 8 to 14 Scene point P on 26b) BPi (see Figures 8 to 14 The distance between point 52a) and camera point 31b. The obtained scene point P BPi 52a is then back-projected onto the source image (see...) Figures 8 to 14 The camera point in 20d (see) Figure 8 In 31d), and at position 134 / q si The point intersects with the image, thus generating the final motion vector MV. i = q si - q di That is, the predetermined MVP 126 for position i. This method can be used by the video decoder 100 and the corresponding video encoder described above in section "2 Motion Vector Prediction" to derive the predetermined MVP 126 for the current block 124 in the current image 20a.

[0354] This method can also be used by the video decoder 10 and corresponding video encoder described above in section 1, "Temporal Sample Prediction," to find the corresponding position 22 for each pixel 23 in the current block 18. For example, the video decoder 10 and corresponding video encoder can be configured to select a z-value from the z-buffer of the current image 20a for each pixel 23 of the current image 20a, and perform a projection 26 from the camera point (e.g., derived from the video geometric correlation parameters 24 of the current image 20a) through the corresponding pixel 23 into the scene 14, and use the z-value to determine the distance between the scene point and the camera point on the projection ray in the camera direction. The obtained corresponding scene point is then back-projected onto the camera point of the reference image 20b (e.g., derived from the video geometric correlation parameters 24 of the reference image 20b), and intersects with the reference image 20b at the corresponding position 22 corresponding to the corresponding pixel 23.

[0355] For example, the video decoder and corresponding video encoder discussed in this paper can be configured to determine the predetermined MVP 126 using the scene model 300 and video geometric correlation parameters 24 (e.g., Figures 8 to 14 The video decoder 100 and the corresponding video encoder) or when using the scene model 300 and video geometry related parameters 24 to determine the interior of the block (e.g., Figures 5 to 7 (video decoder 10 and corresponding video encoder),

[0356] A depth map (e.g., see the z-buffer described above) is constructed by measuring the distance from each pixel in the current image 20a to the scene model 300 to determine the depth value at the corresponding pixel. Scene 14 (e.g., based on the video geometry correlation parameters 24 of the current image 20a) is projected onto the corresponding pixel along the scene model.

[0357] Interpolation is applied to the depth map to determine the depth values ​​of pixels in scene model 300 that are not hit or are not sufficiently close to scene projection lines (e.g., along scene projection lines, scene 14 is projected onto the corresponding pixels).

[0358] The predetermined MVP is determined using depth map and video geometry-related parameters 24 (e.g., by...). Figures 8 to 14 (executed by the video decoder 100 and the corresponding video encoder), or

[0359] The depth map and video geometry parameters 24 are used to determine the interior of the block (e.g., by...). Figures 5 to 7 (The video decoder 10 and the corresponding video encoder are executed).

[0360] Sample prediction

[0361] The embodiments described in this subsection may include the features and / or functionality described with respect to the video decoder 10 and the corresponding video encoder in section "1 Temporal Sample Prediction", and vice versa.

[0362] Scene model 300 can also be used for direct sample prediction. The sample location in the source image can be determined by using any of the methods described above. si That is, the corresponding position 22 in reference image 20b (see Figure 5 This allows us to obtain predicted sample blocks. For example, 134 / q si The integer part determines the sample location in pixels (e.g., Figure 5 The corresponding position in the text is 22), and 134 / q siThe fractional part is used, for example, to select the corresponding interpolation filters in the x and y directions to obtain predicted samples from the subsample locations. The interpolation filters used to generate samples at the subsample locations can be, for example, interpolation filters of VTM 8Tap or ECM 12Tap filters used for motion compensation, or other interpolation filters.

[0363] In the variant, sampling of the predicted block is performed sample-by-sample. If 134 / q cannot be determined... si For example, the affected prediction samples are marked as invalid.

[0364] In another variant, the predicted block is divided into sub-blocks, and for each sub-block, a single 134 / q is determined. si , of which 132 / q di (For example, corresponding to) Figure 5 Pixel 23 in the image represents the position within the sub-block (e.g., the center position within the sub-block). Figure 5 In the current block 18 (within the sub-blocks), thus generating a motion vector MV for each sub-block. i = q si - q di Using motion vectors, predict all samples in the sub-block. If 134 / q cannot be determined... si If so, the entire affected sub-block is marked as invalid.

[0365] After predicting the samples, for example, bilinear interpolation and / or extrapolation are used to derive the samples marked as invalid in the predicted block from the valid neighboring samples.

[0366] Explicit mode

[0367] Scene model sample prediction can be signaled by using flags transmitted in the bitstream that indicate the prediction mode used for individual frames.

[0368] Generate reference image

[0369] Scene model 300 can be used to create additional scene model-based reference images (SMBRPs), such as composite reference image 504 hereinafter (see [link]). Figure 18 Reference images are inserted into one or two lists of reference images. SMBRP can be accessed via the reference image generated by addressing the syntax element ref-idx. Samples of the reference images can be created, as described above (for example, see the first part of this subsection, i.e., the first part of the subsection “Sample Prediction”), thus treating the reference image as a large prediction block. In the first step, valid samples are stored, for example, in a buffer.

[0370] like Figure 18As shown, the video decoder 500 for decoding video 12 of scene 14 from data stream 16 using motion-compensated prediction (and the corresponding video encoder for encoding video 12 of scene 14 into data stream 16 using motion-compensated prediction) can be configured to use video geometric correlation parameters 24 on image 20 describing how scene 14 is projected onto video 12. (Or use different terms: video capture related parameters; regardless of the terminology used, and wherein this is valid throughout the application and claims, the parameters should also include the situation where video 12 is generated synthetically, for example by artificial intelligence, neural networks, or using 3D rendering) In (e.g., previously decoded) a reference image (e.g., 20c), find the corresponding position 508 corresponding to pixel 506 in a predetermined image (e.g., 20a) and sample the reference image 20c at the corresponding position 508 to construct 502 a synthetic reference image 504, the reference image being synthesized to form a synthetic version of the predetermined image (e.g., the current image 20a or some other previous image, such as 20b, such that the pixels of these images (i.e., the synthetic image and the image representing its synthetic version) become synonymous). Additionally, the video decoder 500 is configured to obtain the interior of a current portion of the current image (e.g., block 510 in image 20a) by sampling the corresponding portion of the synthetic reference image 504, and reconstruct the current portion 510 using the interior of the portion. The corresponding encoder can be configured to obtain the interior of the current portion 510 by sampling the corresponding portion of the synthetic reference image 504, and then use the interior of the current portion to encode the current portion 510.

[0371] The video decoder 500 and the corresponding video encoder can be configured to determine the scene model 300 based on the motion vectors and video geometric parameters of the image (e.g., as described above in this section, i.e., in section “3 Motion Compensation Sample Prediction”), and use the scene model 300 and the video geometric parameters 24 to find the corresponding position 508 in the (e.g., previously decoded / encoded) reference image 20c.

[0372] The synthesized reference image 504 can be used to find the corresponding position 508 in the (e.g., previously decoded / encoded) reference image 20c using scene model 300 and video geometry-related parameters 24 (e.g., each of which can correspond to...). Figure 16 and Figure 17 The corresponding image position 612 shown is constructed by sampling the reference image 20c at the corresponding position 508. The corresponding position 508 can be constructed for each pixel 506 (e.g., corresponding to...). Figure 16 and Figure 17 The following operation is performed on the image position 606 shown in the figure, while on the predetermined image (e.g., the current image 20a, for example, corresponding to...). Figure 16and Figure 17 In the current image 602 shown, the scene projection line along which scene 14 (e.g., based on the video geometry correlation parameters 24 of the predetermined image) is projected onto the corresponding pixel 506 is found (e.g., see...). Figure 16 and Figure 17 604 in the middle); based on scene projection lines (e.g., see 604). Figure 16 and Figure 17 (604 in the original text) and scene model 300 (for example, see 604 in the original text) Figure 17 580 or Figure 16 The intersection point of 590 in the middle determines the final use case point (for example, see 590). Figure 16 and Figure 17 (608 in the example) (e.g., using cones and point clouds or triangles from a scene model); and will ultimately use points (e.g., see 608 in the example) Figure 16 and Figure 17 608) is projected onto reference image 20c (e.g., see...) Figure 16 and Figure 17 The corresponding position 508 corresponding to the corresponding pixel 506 is obtained on 610).

[0373] The current portion 510 can be reconstructed or encoded using the synthetic reference image 504 by: copying the co-located portion of the synthetic reference image 504 that is co-located with the current portion 510; or copying / sampling the portion of the synthetic reference image 504 pointed to by the motion vectors encoded or decoded for the current portion 510 (which may be an inter-frame prediction block).

[0374] Alternatively, the current portion 510 can be reconstructed using the composite reference image 504 by selecting the composite reference image 504 from the list of reference images in the decoded image buffer DPB 509 using a reference index encoded into the data stream 16 for the current portion 510. The current portion 510 can be encoded using the composite reference image 504 by encoding the reference index into the data stream 16 for selecting the composite reference image from the list of reference images in the decoded image buffer DPB 509, i.e., making it possible to select the composite reference image from the list of reference images in the decoded image buffer DPB 509.

[0375] Alternatively, a synthetic reference image 504 may be reconstructed or constructed using one or more other reference images to assist in determining the synthetic reference image 504 at pixels in the current image 20a where a corresponding pixel is not found in the reference image 20c.

[0376] To handle effects such as occlusion, SMBRP can be derived from more than one source image, i.e., from two or more reference images (e.g., including reference image 20c mentioned above and one or more other reference images), i.e., synthesizing reference image 504. For each source image (reference image), temporary buffers can be populated as described above (e.g., in this subsection "Sample Prediction"), thus treating the temporary buffers as prediction blocks. In the next step, SMBRP samples are derived by selecting the first valid sample entry from all temporary buffers at the same sample location (note that the order of the source images is important for reconstruction). If all temporary buffers contain invalid sample values ​​at the sample location, the corresponding SMBRP sample is also marked as invalid. In the final step, the samples marked as invalid at the sample location are populated with valid sample values ​​using bilinear interpolation and extrapolation of sample values ​​from valid neighboring locations.

[0377] In other words, a synthetic reference image 504 can be constructed using interpolation / extrapolation to help determine the synthetic reference image 504 in any reference image, such as pixel 506 of the current image 20a where no corresponding pixel / position 508 is found in the current image 20a or one or more other reference images.

[0378] In the bitstream, that is, in the data stream 16, a signal is sent to indicate the source image from which the composite reference image 504 is derived, that is, the reference image. In another variant, the source image is the first image in the reference list.

[0379] The position where the composite reference image 504 is inserted into one or more reference image lists can be signaled in bitstream 16. If no signal is given in bitstream 16, the position is a predetermined position in the list (e.g., the last position in the list). The composite reference image 504 can replace the reference image at a specific position or add to the reference list, which implies a modification of num_ref_idx (to access all reference images).

[0380] A preferred configuration for using the composite reference image 504 is to append a composite reference image 504 to the end of the reference_picture_list for each reference image list. If available, the composite reference image 504 inserted in RPL_L0 is derived from the first image in RPL_L0 and the first image in RPL_L1. If available, the composite reference image 504 inserted in RPL_L1 is derived from the first image in RPL_L1 and the first image in RPL_L0.

[0381] Motion vector prediction

[0382] The scene-model-based reference image (SMBRP), i.e., the composite reference image 504, is a motion-compensated representation of the current image 20a derived from the reconstructed reference image 20c from the reference buffer. It should be noted that "motion-compensated representation" here means the composite reference image 504 created via camera-based motion compensation (MC) or homography—the composite reference image 504 is a quasi-deformed version of the source / reference image 20c that further utilizes the current image 20a (i.e., currFrame) and the camera orientation and position of the source / reference frame 20c.

[0383] When using samples from the synthetic reference image 504, the motion vectors of the reference source samples implicitly reference, for example, the motion-compensated representation of the reconstructed reference image 20c.

[0384] If the synthetic reference image 504 is used for prediction, the actual motion vector field used for prediction is a superposition of motion compensation based on the scene model and the motion vectors used to offset the source blocks in the synthetic reference image 504.

[0385] According to an embodiment, in order to perform temporal motion vector prediction for one or more inter-frame prediction blocks of other images and / or spatial motion vector prediction for one or more inter-frame prediction blocks of the current image, the motion vector of the reference composite reference image 504 can be modified by adding a motion vector, determined by the video geometric correlation parameter 24 and describing the disparity between the portion of the composite reference image 504 referenced by the motion vector and the block of the reference image to which the motion vector is applied, to the motion vector of the reference composite reference image 504.

[0386] The superimposed MV field can be used for motion vector prediction, for example when access is used as a source to derive a reference image 20c for a synthetic reference image 504.

[0387] The temporal motion vector prediction can be redirected for prediction from synthetic reference image 504 to reference image 20c (e.g., to alternatively derive MVP candidates, such as 126, from the reference images).

[0388] A general approach to correcting motion vectors is to superimpose the base motion onto a fixed grid of blocks within the prediction block, thereby dividing the prediction block into sub-blocks (e.g., as small as possible for each 4×4 block). The sample-by-sample motion vector of the base motion vector field associated with each sub-block is averaged sub-block by sub-block and superimposed with the motion vector used to address the sample blocks in the synthetic reference image 504.

[0389] For affine prediction blocks, the sub-block stacking method combines the sample-wise averaged motion vector of the underlying motion vector field with the affine motion vector obtained at the sub-block location.

[0390] When using common translation prediction to predict the current block 510 from the source image (i.e., reference image 20c) used as the synthetic reference image 504 and to predict adjacent blocks from the synthetic reference image 504, the superimposed MV can be used as an MVP candidate (MV prediction treats the synthetic reference image 504 and the source image of the synthetic reference image 504 (i.e., reference image 20c) as predictions from the same image).

[0391] In contrast, when the current block 510 is predicted from the synthetic reference image 504 and the adjacent blocks are not predicted from it, the prediction source is considered a different image, and the MV of this adjacent block cannot be used for MV prediction.

[0392] For example, the current block 510 can be reconstructed or encoded using the predefined MVP 126 by deforming the pixel positions in the current block 510 to the corresponding pixel positions (e.g., 506 or 508) in the reference image (e.g., composite reference image 504 or reference image 20b) using the video geometry correlation parameter 24 and the predefined MVP 126.

[0393] 4. Encoding and decoding of camera parameters

[0394] In variations of the proposed method, camera parameters, for example, contained in video geometric correlation parameters 24, are not derived at the decoder and are therefore transmitted in bitstream 16. Camera parameters for a specific frame (e.g., the current image 20a) may include camera position (x, y, z) and camera orientation expressed in Euler angles (α, β, γ) or as quaternions (x0, x1, x2, x3). Furthermore, focal length (f) and optional parameters for more complex camera models may be included in the camera parameters. Camera parameters may, for example, have a floating-point value range.

[0395] Quantification of camera parameters

[0396] The values ​​associated with the camera parameters read from bitstream 16 must be mapped from binary codewords to the camera parameters on the decoding side. In a variant of the embodiment, the binary code mapping uses the meta-VLC and signed VLC used in VVC to produce intermediate integer values. A dequantizer with a uniform quantization step size can be used to map the intermediate integer values ​​to the range of camera parameter values.

[0397] The quantization step size (QP) of the camera parameters can be obtained by sending the quantization parameters in the video parameter set. Cam QP Cam =f(QP Video )+DeltaQP Cam DeltaQP camIt can be transmitted in bitstream 16 or derived by a suitable parameter (e.g., the TemporalLayer of the associated image).

[0398] A variation of quantizing camera parameters involves transmitting the exponent and mantissa of the floating-point values ​​of the camera parameters as two fixed-length codes.

[0399] Prediction of camera parameters

[0400] In video scenarios, camera parameters associated with consecutive frames may be similar, and therefore, predicting camera parameters from previously reconstructed camera parameters can help reduce the number of bits required for transmission. Assuming camera parameters are transmitted in display order and camera movement is stable, the parameters from frame f... n-1 Associated previously reconstructed camera parameters p x,n-1 Camera parameter prediction for frame f n Transmitted specific camera parameters p x,n The difference d px,n Transmission in bitstream (p x,n = p x,n-1 + d px,n ).

[0401] In another variant, assuming the camera movement is smooth and stable, d px,n DP can also be predicted from its predecessor. x,n =dp x,n-1 +dp2 x,n This leads to the second-level prediction, in which dp2 is transmitted in the bitstream. x,n Instead of dp x,n .

[0402] Encode camera parameters at the image level

[0403] Since camera parameters are associated with a specific frame, such as the current image 20a, variations of the embodiment propose transmitting camera parameters within a slice header or image header.

[0404] In other words, the video geometric correlation parameter 24 can be encoded into or decoded from the slice header or image header in the access unit of the data stream 16 associated with the image 20 of the video 12.

[0405] Flags in the image header or slice header can be used to signal the presence of camera parameter syntax elements in the header. If camera parameters exist, one of the quantization and / or prediction schemes is used to transmit the camera parameters.

[0406] Encoding and decoding camera parameters at the GOP level

[0407] In this variant of the embodiment, camera parameters for multiple images (e.g., intra-frame time periods or GOPs) are transmitted in a single data block. The data block can be indicated by flags transmitted in the image header or slice header, or in another variant, the block can be encapsulated within an APS. However, the block-by-block approach provides the option of directly combining the prediction and quantization schemes mentioned above.

[0408] MV encoding and decoding at control points

[0409] Instead of using camera parameters, a set of motion vectors 46 at a specific control point 44 can be used to signal information about the position and orientation of the frame, for example, see [link to relevant documentation]. Figure 7Motion vector 46 describes the displacement of the current image (e.g., 402) at control point 44 relative to a reference image (e.g., 401). That is, according to this option, for each corner of an image, the video geometry correlation parameter 24 contains a vector 46, and these vectors 46 describe the offset of the corner's position from its corresponding position in a certain "reference image." In terms of encoding / decoding or presentation timing, the reference image may always be the immediately preceding image, or it may be another image. Therefore, these images form a pair (a, b), where a and b are, for example, the point of view (POC) of the image, and for the image, vector 46 is transmitted in data stream 16 as part of the video geometry correlation parameter 24, and b is the associated "reference" image. For example, such a pair can be defined as following the GOP structural interdependence between images in a GOP. To obtain knowledge of the effective vector 46 of a corner point of an image a relative to a predetermined reference image x, the decoder and encoder can simply concatenate (add) the vectors 46 of the video geometric correlation parameters of the corner points of image pairs (a,b), (b,c), (c,...) ... (...,x), where the smallest such sequence of image pairs containing vector 46 in the video geometric correlation parameters 24 is selected. This concatenation is unnecessary when the video geometric correlation parameters 24 contain corner vectors of (a,b) for the current image a and reference image b of the current block. Vector mapping or homomorphism is achieved by using the effective vector as the support vector of some interpolation 48 to generate mapping vectors to certain locations to be mapped. For example, a homomorphism achieved by the vectors of the corner points of images (a,b) can represent mapping 42, which maps image points in image a to corresponding points in image b, where, for example, it is assumed that these corresponding points in images a and b are located in scene plane 50 in scene 14. This scene plane 50 can be, for example, the background plane of the captured scene 14, such as a wall, or a bookshelf behind a head in the foreground shown in scene 14. For example, to obtain the MVP of the current block of the current image a, which is related to the reference image b, interpolation can be used to map the image position of the current block to the corresponding position and the difference can be used as a pre-defined MVP that may be added to the list of MVP candidates described above (e.g., see section “2 Motion Vector Prediction”).

[0410] Given the intrinsic parameters of the camera, it is also possible to derive (potentially multiple) solutions from the homomorphism to the non-intrinsic camera parameters and the parameters defining this scene plane 50. The same mapping exists between any point in scene plane 50 and the resulting image points projected onto the camera and onto the camera at the origin, as defined by the homomorphism. However, the MVP can be derived using these camera parameters through triangulation exactly as described below. That is, it is not limited to using only the homomorphic parameters to follow the homomorphic mapping between two images. The homomorphic parameters can also be used for triangulation using the input MV, as described above.

[0411] This seems to contradict the idea that homomorphism is defined only between two images, and that camera parameters appear to represent one image and be independent of others. However, this is merely a matter of reference point; that is, any coordinate system can be chosen for the (camera) parameters: for example, the parameters of the first image (of a group) can always be placed at the origin, meaning all camera parameters are now relative to that image. The same applies to homomorphic parameters. Using an origin that is not equivalent to a useful set of parameters would actually be bad, as it would waste description length (or bits) to have this useless origin. In other words, entropy encoding / decoding will compute the difference and therefore will always result in relative parameters.

[0412] Therefore, control point vector 46 can be considered an alternative representation of camera parameters defining camera position and orientation. This alternative representation may be more suitable for entropy encoding / decoding, which can be used to encode / decode parameter 24 into stream 16. Encoding / decoding may include parameter quantization, and quantization errors in the control point vector proportionally cause errors in the image plane 50. Quantization errors in rotation angles or depth-dependent camera position errors are more difficult to control in terms of their consequences.

[0413] The predetermined motion vector predictor (MVP) 126 can be derived, for example, by using video geometric correlation parameters 24 to map the image position of the current block 124 to the corresponding position in the reference image referenced by the predetermined MVP 126 and using the offset between the image position and the corresponding position as the predetermined MVP 126.

[0414] Given intrinsic camera parameters, it is also possible to derive non-intrinsic camera parameters and parameters defining the scene plane 50 from homomorphism. Multiple solutions can be generated by deriving non-intrinsic camera parameters (i.e., vectors at image corners) from homomorphism, but one of these solutions can be selected at both the decoder and encoder using predetermined rules. That is, using homomorphic parameters, the encoder and decoder can perform the triangulation-based task described above to derive the predetermined MVP126.

[0415] Video geometric parameters, such as the homomorphic mapping 42 describing the corresponding positions in the image pair of video 12, and the video decoder described herein is configured to, for example, the current image 20a, derive a scene-to-image projection of scene 14 onto the predetermined image based on the homomorphic mapping 42 between corresponding positions in the image pair including the predetermined image.

[0416] Regarding the difference between the homomorphic corner vectors defined between image pairs and camera projection parameters (such as non-intrinsic parameters defined individually for each image), the following should be noted: It is correct that the homomorphism is defined only between two images, while the camera parameters appear to represent one image and are independent of the others. However, this is merely a matter of reference point. Any coordinate system can be chosen for the (camera) parameters: for example, the parameters of the first image (of a group of images) can always be placed at the origin, meaning that all camera parameters are now relative to that image. The same applies to homomorphic parameters. Using an origin that is not equivalent to a useful set of parameters would actually be bad, because it would waste description length (or bits) to have this useless origin. In other words, entropy encoding / decoding will compute the difference and thus will always result in relative parameters.

[0417] Although some aspects have been described in the context of the device, it is clear that these aspects also represent a description of the corresponding method, where blocks or devices correspond to method steps or features of method steps. Similarly, aspects described in the context of method steps also represent a description of corresponding blocks, items, or features of the corresponding device. Some or all of the method steps may be performed by (or using) hardware devices (such as microprocessors, programmable computers, or electronic circuits). In some embodiments, one or more of the most important method steps may be performed by such devices. Similarly, the device may include hardware devices, such as microprocessors, programmable computers, or electronic circuits, configured to perform one or more method steps.

[0418] Depending on certain implementation requirements, embodiments of the present invention can be implemented in hardware or software. Implementation can be performed using digital storage media such as floppy disks, DVDs, Blu-ray discs, CDs, ROMs, PROMs, EPROMs, EEPROMs, or flash memory, the digital storage media having electronically readable control signals stored thereon that cooperate (or are capable of cooperating with) a programmable computer system to perform the corresponding methods. Therefore, the digital storage media can be computer-readable.

[0419] According to some embodiments of the invention, a data carrier having electronically readable control signals is included, which are capable of cooperating with a programmable computer system to perform one of the methods described herein.

[0420] Generally, embodiments of the present invention can be implemented as a computer program product having program code that, when run on a computer, is operatively used to perform one of the methods. The program code may, for example, be stored on a machine-readable medium.

[0421] Other embodiments include a computer program stored on a machine-readable medium for performing one of the methods described herein.

[0422] In other words, embodiments of the methods of the present invention are therefore computer programs having program code for executing one of the methods described herein when the computer program is run on a computer.

[0423] Therefore, another embodiment of the method of the present invention is a data carrier (or digital storage medium, or computer-readable medium) containing a computer program recorded thereon for performing one of the methods described herein. The data carrier, digital storage medium, or recording medium is generally tangible and / or non-transitory.

[0424] Therefore, another embodiment of the method of the present invention is a data stream or signal sequence representing a computer program for performing one of the methods described herein. The data stream or signal sequence may, for example, be configured to be transmitted via a data communication connection (e.g., via the Internet).

[0425] Another embodiment includes a processing element, such as a computer or programmable logic device, configured or adapted to perform one of the methods described herein.

[0426] Another embodiment includes a computer having a computer program installed thereon for performing one of the methods described herein.

[0427] Another embodiment of the invention includes an apparatus or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. For example, the receiver may be a computer, a mobile device, a memory device, or the like. The apparatus or system may, for example, include a file server for transmitting the computer program to the receiver.

[0428] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware device.

[0429] The device described herein may be implemented using hardware devices or using a computer or a combination of hardware devices and a computer.

[0430] The device described herein or any component thereof may be implemented, at least in part, in hardware and / or in software.

[0431] The methods described in this article can be implemented using hardware devices, computers, or a combination of hardware devices and computers.

[0432] The methods described herein or any component of the devices described herein may be performed at least in part by hardware and / or software.

[0433] The embodiments described above are merely illustrative of the principles of the invention. It should be understood that modifications and variations of the configurations and details described herein will be readily apparent to those skilled in the art. Therefore, it is intended to be limited only by the scope of the following claims, and not by the specific details presented through the description and explanation of the embodiments herein.

[0434] [1]Motion Vector Coding and Block Merging in Versatile Video CodingStandard; Wei-Jung Chien, Li Zhang, Martin Winken, Xiang Li, Ru-Ling Liao,HanGao,Chih-Wie Hsu, Hongbin Liu, Chun-Chi Chen; IEEE Transaction on Circuitsand Systems for Video Technology Vol.31 No.10; Oct 2021

[0435] [2]ITU-T and ISO / IEC JTC 1, “Versatile video coding (ITU-T Rec. H.266and ISO / IEC 23090-3),” Aug. 2020.

Claims

1. A video decoder (10) for using motion compensation to predict and decode video (12) of a scene (14) from a data stream (16), configured to For the current block (18) of the current image (20a), the prediction block interior is obtained through the following operations: Using the video geometry-related parameters (24) on the image (20) describing how scene (14) is projected (26) onto video (12), the corresponding position (22) of the pixel (23) in the current block (18) is found in the reference image (20b), and Sample the reference image (20b) at the corresponding position (22); and Reconstruct the current block using the predicted block (18).

2. The video decoder of claim 1, wherein the video geometric correlation parameters describe the scene-to-image projection of a camera (30) onto an image (20) of a video.

3. The video decoder as described in any one of claims 1 or 2, wherein the video geometric correlation parameters describe the scene-to-image projection of the scene onto the video image for each image of the video by the camera.

4. The video decoder as described in any one of claims 1 to 3, wherein the video geometric correlation parameters describe a scene-to-image projection of a camera onto the video image by image and over the entire image.

5. The video decoder as described in any one of claims 1 to 4, wherein the video geometric correlation parameters describe the scene to the image projection via one or more of the following: One or more non-inherent camera parameters of a camera; and One or more inherent camera parameters of a camera.

6. The video decoder of claim 5, wherein one or more non-inherent camera parameters define the position of a camera. and / or the orientation of a camera .

7. The video decoder as described in any one of claims 5 or 6, wherein one or more inherent camera parameters define the focal length and / or FOV angle of a camera.

8. The video decoder as described in any one of claims 1 to 7, wherein the video geometric correlation parameters describe a homomorphic mapping (42) between corresponding positions (401, 402) in a pair of images (201, 202) of the video, or a homomorphic mapping between corresponding motion vectors associated with the pair of images of the video.

9. The video decoder of claim 8, wherein the video geometric correlation parameters describe the homomorphic mapping via a vector (462) or tensor at a predetermined control point (442) of the image of the video.

10. The video decoder of claim 9, wherein the predetermined control point is a corner point of the video image.

11. The video decoder as described in any one of claims 8 to 10, wherein for each image, the video geometric correlation parameters contain only one vector or tensor for each of the four corner points of the image of the video.

12. The video decoder of any one of claims 1 to 11, wherein video geometric correlation parameters describe a homomorphic mapping between corresponding positions in a pair of images of the video, and the video decoder is configured to, for a predetermined image, derive a scene-to-image projection onto the predetermined image based on the homomorphic mapping between corresponding positions in a pair of images including the predetermined image.

13. The video decoder as described in any one of claims 1 to 12 is configured to decode video geometric correlation parameters from a data stream.

14. The video decoder as described in any one of claims 1 to 13 is configured to determine video geometric correlation parameters based on the decoded portion of the video.

15. The video decoder of any one of claims 1 to 14, wherein video geometric correlation parameters describe homomorphic mappings between corresponding positions in image pairs of the video, and the video decoder is configured to, for a predetermined image, derive a scene-to-image projection onto the predetermined image based on a sequence of homomorphic mappings between corresponding positions in one or more image pairs including the predetermined image and a base image, and based on non-inherent parameters and inherent parameters of the base image.

16. The video decoder as described in any one of claims 1 to 15, configured to decode syntax elements from a data stream, and If the syntax element has a first state, then For the current block, video geometric parameters describing how the scene is projected onto the video are used to find the corresponding position of the pixel in the reference image within the current block. The reference image is then sampled at the corresponding position to derive the predicted block's interior. Reconstruct the current block using the predicted block.

17. The video decoder of claim 16, configured to, if the syntax element has a second state, then Reconstruct the current block independently of video geometry parameters, or The current block is reconstructed by copying portions of the decoded video at regular pixel intervals.

18. The video decoder as described in any one of claims 1 to 17, configured to This yields a list of MVP candidates for the current block, where each MVP candidate corresponds to a specific motion-vector inter-frame encoding / decoding mode. Select the MVP from the list of MVP candidates, and Rebuild the current block using the selected MVP: If the selected MVP corresponds to a specific motion vectorless inter-frame encoding / decoding mode, then For the current block, video geometric parameters describing how the scene is projected onto the video are used to find the corresponding position of the pixel in the reference image within the current block. The reference image is then sampled at the corresponding position to derive the predicted block's interior. Reconstruct the current block using the predicted block. If the selected MVP corresponds to an MVP candidate other than a specific motion vectorless inter-frame encoding / decoding mode, then Use the selected MVP to reconstruct the current block using motion vector compensation prediction.

19. The video decoder as described in any one of claims 1 to 18, configured to The scene model (300) is determined based on the motion vectors (126) of the image and the geometric parameters (24) of the video. The corresponding position (22) is found in the reference image using the scene model (300) and video geometry parameters (24).

20. The video decoder as described in any one of claims 1 to 19 is configured to determine a scene model (300) by: For each control point in one or more control points (132) of an inter-frame prediction block (128) of a predetermined image (20b), scene model points (302) for forming the basis of the scene model are determined by the following operation: The source image position (134) in the corresponding reference image (20d) from which the motion vector points (130) is determined, and the motion vector is encoded and decoded in the data stream of the inter-frame prediction block of the corresponding control point. Using video geometry-related parameters (24), Determine the first scene projection line along which the scene is projected onto the corresponding control point (132). Determine the second scene projection line along which the scene is projected onto the source image position (134), and Determine the scene point on the shortest line connecting any point on the first scene projection line and any point on the second scene projection line, and The scene model points are determined as follows: Define scene model points as scene points, or The scene model point is determined by determining the distance between the scene point and the predetermined image, and by determining that the scene model point is located on the first scene projection line at that distance.

21. The video decoder as described in any one of claims 1 to 19 is configured to determine a scene model (300) by: For each control point in one or more control points (132) of an inter-frame prediction block (128) of a predetermined image (20b), scene model points (302) for forming the basis of the scene model are determined by the following operation: The source image position (134) in the corresponding reference image (20d) from which the motion vector points (130) is determined, and the motion vector is encoded and decoded in the data stream of the inter-frame prediction block of the corresponding control point. Using video geometry-related parameters (24), Determine the scene point using the following steps: The linear equations are determined using video geometric parameters (24), corresponding control points (132), and source image location (134). Solve the system of linear equations. The scene model points are determined as follows: Define scene model points as scene points, or The scene model point is determined by determining the distance between the scene point and the predetermined image, and by determining that the scene model point is located on the first scene projection line at that distance.

22. The video decoder of claim 21, wherein the system of linear equations is defined by the equation AQ=0, wherein Q represents a vector containing the coordinates of scene points to be determined, and A represents a matrix defined by video geometric correlation parameters (24), corresponding control points (132), and source image position (134).

23. The video decoder of claim 22, wherein matrix A is defined as follows: A.row(0) = q di (0)*cam di .row(2) - cam di .row(0) A.row(1) = q di (1)*cam di .row(2) - cam di .row(1) A.row(2) = q si (0)*cam si .row(2) - cam si .row(0) A.row(3) = q si (1)*cam si .row(2) - cam si .row(1), The rows of matrix A are represented by A.row(0), A.row(1), A.row(2), and A.row(3). The coordinates of the corresponding control point (132) are given by q. di (0) and q di (1) indicates that The coordinates of the source image position (134) are determined by q. si (0) and q si (1) indicates, and The video geometric correlation parameters (24) are derived from cam di .row(2), cam di .row(0), cam di .row(2), cam di .row(1), cam si .row(2), cam si .row(0), cam si .row(2) and cam si .row(1) represents.

24. The video decoder of any one of claims 22 to 23, configured to perform the task of solving a system of linear equations using a singular value decomposition method.

25. The video decoder as described in any one of claims 1 to 19 is configured to determine a scene model (300) by: For each control point in one or more control points (132) of an inter-frame prediction block (128) of a predetermined image (20b), scene model points (302) for forming the basis of the scene model are determined by the following operation: The source image position (134) in the corresponding reference image (20d) from which the motion vector points (130) is determined, and the motion vector is encoded and decoded in the data stream of the inter-frame prediction block of the corresponding control point. Using video geometry-related parameters (24), Scene points are determined by minimizing the cost function based on the reprojection error. The scene model points are determined as follows: Define scene model points as scene points, or The scene model point is determined by determining the distance between the scene point and the predetermined image, and by determining that the scene model point is located on the scene projection line at the distance.

26. The video decoder of claim 25, wherein the reprojection error associated with a corresponding control point (132) of an inter-frame prediction block (128) of a predetermined image (20b) includes: The distance between the corresponding control point (132) of the inter-frame prediction block (128) and the projection of the scene point onto the predetermined image (20b), and The distance between the source image position (134) of the corresponding reference image (20d) and the projection of the scene point onto the corresponding reference image (20d).

27. The video decoder of claim 26, wherein the scene model is a mesh or a point cloud.

28. The video decoder of any one of claims 20 and subsequent claims, configured to exclude scene model points whose quality metric determined based on the length metric of the shortest line does not conform to a predetermined criterion.

29. The video decoder as claimed in any one of claims 20 and subsequent claims, configured to Use a control point to translate the inter-frame prediction block, and / or Use more than one control point for the affine inter-frame prediction block.

30. The video decoder of any one of claims 20 and subsequent claims, configured to determine the scene model additionally based on motion vectors of inter-frame prediction blocks of one or more previous images when determining the scene model.

31. The video decoder of any one of claims 20 and subsequent claims, configured to limit inter-frame prediction blocks to determine scene models relative to images at time layers that satisfy predetermined criteria.

32. The video decoder of claim 19 or subsequent claims, configured to, when determining a scene model, Compared to scene model points (302) derived from motion vectors in translation mode, scene model points (302) derived from inter-frame prediction blocks encoded and decoded in affine mode are preferentially used to determine the scene model, and / or Compared to scene model points (302) derived from motion vectors of inter-prediction blocks encoded and decoded individually in the data stream for inter-prediction blocks, the scene model is determined preferentially based on scene model points (302) derived from motion vectors of inter-prediction blocks encoded and decoded individually for inter-prediction blocks in the data stream, and / or The scene model is determined by the preference varying between scene model points (302) such that: the higher the preference, the lower the prediction residual signal encoded into the data stream according to a predetermined metric of the inter-frame prediction block; the scene model points (302) originate from the motion vectors of the inter-frame prediction blocks; and / or The scene model is determined by a preference that varies between scene model points (302), such that: the higher the preference, the closer the images of the inter-frame prediction blocks are in time; the scene model points (302) originate from the motion vectors of the inter-frame prediction blocks; and / or The scene model is determined by the preference that varies between scene model points, such that: the higher the preference, the smaller the quantizer step size of the inter-frame prediction block, and the scene model point (302) originates from the motion vector of the inter-frame prediction block.

33. The video decoder as claimed in claim 19 or subsequent claims, configured to When using scene models and video geometry parameters to derive the interior of a block... The weights depend on the part of the scene model. Compared to the portion of motion vectors originating from translation mode, the portion of motion vectors originating from inter-prediction blocks encoded and decoded in affine mode has a higher weight, and / or Compared to the portion of motion vectors originating from inter-prediction blocks encoded and decoded individually in the data stream for each inter-prediction block, the portion of motion vectors originating from these inter-prediction blocks has a higher weight, and / or The higher the weight, the lower the prediction residual signal encoded into the data stream according to the predetermined metric of the inter-frame prediction block, which partly originates from the motion vectors of the inter-frame prediction block, and / or The higher the weight, the closer the images of the inter-frame prediction blocks are in time, partly due to the motion vectors of the inter-frame prediction blocks, and / or The higher the weight, the smaller the quantizer step size of the inter-frame prediction block, which is partly derived from the motion vector of the inter-frame prediction block.

34. The video decoder as described in any of claim 19 or any of the following claims, configured to When using the scene model (300) and video geometry parameters (24) to find the corresponding position (612) in the reference image (610; 20b), For each pixel (606; 23) in the current block (600; 18), Determine the scene projection line (604) along which the scene (14) is projected onto the corresponding pixel (606; 23) of the current block. A final use point (608) on the scene projection line is determined based on one or more intersection points (608) between the scene projection line (604) and the facial region (310) of the scene model's mesh, and The final point (608) is projected onto the reference image (610; 20b) to obtain the corresponding position (612; 22) corresponding to the corresponding pixel (606; 23).

35. The video decoder as described in any of claim 19 or any of the subsequent claims, configured to When using scene models and video geometry parameters to find corresponding pixels in a reference image For each pixel (606; 23) in the current block (600; 18), Determine the scene projection line along which the scene is projected onto the corresponding pixel (606). A final use point is determined based on one or more points (620) of the point cloud (580) of the scene model (300) falling into a cone (622) or pyramid, the cone (622) or pyramid being widened away from the corresponding pixel (606) and surrounding the scene projection line. The final point is projected onto the reference image (610) to obtain the corresponding position (612; 22) corresponding to the corresponding pixel (606; 23).

36. The video decoder as claimed in claim 19 or subsequent claims, configured to When using scene models and video geometry parameters to derive the interior of a block... A depth map is constructed by measuring the distance from each pixel in the current image to the scene model along which the scene is projected onto the corresponding pixel, in order to determine the depth value of the depth map at each pixel. Interpolation is applied to the depth map to determine the depth values ​​of pixels in the scene model that are not hit or are close enough to the scene projection lines. The interior of the block is determined using depth maps and video geometry parameters.

37. A video encoder for encoding video (12) of a scene (14) into a data stream (16) using motion compensation prediction, configured to For the current block (18) of the current image (20a), the prediction block interior is obtained through the following operations: Using the video geometry-related parameters (24) describing how scene (14) is projected (26) onto the image (20) of video (12), find the corresponding position (22) in the reference image (20b) corresponding to the pixel (23) in the current block (18), and The reference image (20b) is sampled at the corresponding position (22), and The current block is encoded using the internal code of the predicted block (818).

38. The video encoder of claim 37, wherein the video geometric correlation parameters describe the scene-to-image projection of a camera (30) onto an image (20) of a video.

39. The video encoder as described in any one of claims 37 to 38, wherein the video geometric correlation parameters describe the scene-to-image projection of a camera onto the video for each image of the video.

40. The video encoder of any one of claims 37 to 39, wherein the video geometric correlation parameters describe a scene-to-image projection of a camera onto the video image by image-by-image and whole-image projection of the scene.

41. The video encoder of any one of claims 38 to 40, wherein the video geometric correlation parameters describe the scene to the image projection via one or more of the following: One or more non-inherent camera parameters of a camera; and One or more inherent camera parameters of a camera.

42. The video encoder of claim 41, wherein one or more non-inherent camera parameters define the position of a camera. and / or the orientation of a camera .

43. The video encoder as described in any one of claims 41 or 42, wherein one or more inherent camera parameters define at least one of the focal length and FOV angle of a camera.

44. The video encoder as described in any one of claims 37 to 43, wherein the video geometric correlation parameters describe a homomorphic mapping (42) between corresponding positions (401, 402) in a pair of images (201, 202) of the video, or a homomorphic mapping between corresponding motion vectors associated with the pair of images of the video.

45. The video encoder of claim 44, wherein the video geometric correlation parameters describe the homomorphic mapping via a vector (462) or tensor at a predetermined control point (442) of the image of the video.

46. ​​The video encoder of claim 45, wherein the predetermined control point is a corner point of the video image.

47. The video encoder of any one of claims 44 to 46, wherein for each image, the video geometric correlation parameters contain only one vector or tensor for each of the four corner points of the image of the video.

48. The video encoder of any one of claims 37 to 47, wherein the video geometric correlation parameters describe the homomorphic mapping between corresponding positions in a pair of images of the video, and the video encoder is configured to, for a predetermined image, derive a scene-to-image projection onto the predetermined image based on the homomorphic mapping between corresponding positions in a pair of images including the predetermined image.

49. The video encoder as described in any one of claims 37 to 48 is configured to encode video geometrical correlation parameters into a data stream.

50. The video decoder as described in any one of claims 37 to 49 is configured to determine video geometric correlation parameters based on the encoded portion of the video.

51. The video encoder of any one of claims 37 to 50, wherein video geometric correlation parameters describe homomorphic mappings between corresponding positions in image pairs of the video, and the video encoder is configured to, for a predetermined image, derive a scene-to-image projection onto the predetermined image based on a sequence of homomorphic mappings between corresponding positions in one or more image pairs including the predetermined image and a base image, and based on non-inherent parameters and inherent parameters of the base image.

52. The video encoder as described in any one of claims 37 to 51 is configured to encode syntax elements into a data stream, and If the syntax element has a first state. For the current block, the corresponding position of the pixel in the reference image is found in the reference image using video geometric correlation parameters describing how the scene is projected onto the video, and the reference image is sampled at the corresponding position to deduce the interior of the predicted block. The current block is encoded using the internal code of the predicted block.

53. The video encoder of claim 52, configured to, if the syntax element has a second state, Encoding the current block independently of video geometry-related parameters, or The current block is encoded by copying the encoded video portion at regular pixel intervals.

54. The video encoder as described in any one of claims 37 to 53, configured to This yields a list of MVP candidates for the current block, where each MVP candidate corresponds to a specific motion-vector inter-frame encoding / decoding mode. Select the MVP from the list of MVP candidates, and Encode the current block using the selected MVP through the following operations: If the selected MVP corresponds to a specific motion-vector inter-frame encoding / decoding mode. For the current block, the corresponding position of the pixel in the reference image is found in the reference image using video geometric correlation parameters describing how the scene is projected onto the video, and the reference image is sampled at the corresponding position to deduce the interior of the predicted block. Encode the current block using the internal code of the predicted block. If the selected MVP corresponds to an MVP candidate other than a specific motion vectorless inter-frame encoding / decoding mode. Use the selected MVP to encode the current block using motion vector compensation prediction.

55. The video encoder as described in any one of claims 37 to 54, configured to The scene model (300) is determined based on the motion vectors (126) of the image and the geometric parameters (24) of the video. The corresponding position (22) is found in the reference image using the scene model (300) and video geometry parameters (24).

56. The video encoder as described in any one of claims 37 to 55 is configured to determine a scene model (300) by: For each control point in one or more control points (132) of an inter-frame prediction block (128) of a predetermined image (20b), scene model points (302) for forming the basis of the scene model are determined by the following operation: The source image position (134) in the corresponding reference image (20d) from which the motion vector points (130) is determined, and the motion vector is encoded and decoded in the data stream of the inter-frame prediction block of the corresponding control point. Using video geometry-related parameters (24), Determine the first scene projection line along which the scene is projected onto the corresponding control point (132). Determine the second scene projection line along which the scene is projected onto the source image position (134), and Determine the scene point on the shortest line connecting any point on the first scene projection line and any point on the second scene projection line, and The scene model points are determined as follows: Define scene model points as scene points, or The scene model point is determined by determining the distance between the scene point and the predetermined image, and by determining that the scene model point is located on the first scene projection line at that distance.

57. The video encoder as described in any one of claims 37 to 55 is configured to determine a scene model (300) by: For each control point in one or more control points (132) of an inter-frame prediction block (128) of a predetermined image (20b), scene model points (302) for forming the basis of the scene model are determined by the following operation: The source image position (134) in the corresponding reference image (20d) from which the motion vector points (130) is determined, and the motion vector is encoded and decoded in the data stream of the inter-frame prediction block of the corresponding control point. Using video geometry-related parameters (24), Determine the scene point using the following steps: The linear equations are determined using video geometric parameters (24), corresponding control points (132), and source image location (134). Solve the system of linear equations. The scene model points are determined as follows: Define scene model points as scene points, or The scene model point is determined by determining the distance between the scene point and the predetermined image, and by determining that the scene model point is located on the first scene projection line at that distance.

58. The video encoder of claim 57, wherein the system of linear equations is defined by the equation AQ=0, wherein Q represents a vector containing the coordinates of scene points to be determined, and A represents a matrix defined by video geometric correlation parameters (24), corresponding control points (132), and source image position (134).

59. The video encoder of claim 58, wherein matrix A is defined as follows: A.row(0) = q di (0)*cam di .row(2) - cam di .row(0) A.row(1) = q di (1)*cam di .row(2) - cam di .row(1) A.row(2) = q si (0)*cam si .row(2) - cam si .row(0) A.row(3) = q si (1)*cam si .row(2) - cam si .row(1), The rows of matrix A are represented by A.row(0), A.row(1), A.row(2), and A.row(3). The coordinates of the corresponding control point (132) are given by q. di (0) and q di (1) indicates that The coordinates of the source image position (134) are determined by q. si (0) and q si (1) indicates, and The video geometric correlation parameters (24) are derived from cam di .row(2), cam di .row(0), cam di .row(2), cam di .row(1), cam si .row(2), cam si .row(0), cam si .row(2) and cam si .row(1) represents.

60. The video encoder of any one of claims 57 to 59, configured to perform the solution of a system of linear equations using a singular value decomposition method.

61. The video encoder as described in any one of claims 37 to 55 is configured to determine a scene model (300) by: For each control point in one or more control points (132) of an inter-frame prediction block (128) of a predetermined image (20b), scene model points (302) for forming the basis of the scene model are determined by the following operation: The source image position (134) in the corresponding reference image (20d) from which the motion vector points (130) is determined, and the motion vector is encoded and decoded in the data stream of the inter-frame prediction block of the corresponding control point. Using video geometry-related parameters (24), Scene points are determined by minimizing the cost function based on the reprojection error. The scene model points are determined as follows: Define scene model points as scene points, or The scene model point is determined by determining the distance between the scene point and the predetermined image, and by determining that the scene model point is located on the first scene projection line at that distance.

62. The video encoder of claim 61, wherein the reprojection error associated with a corresponding control point (132) of an inter-frame prediction block (128) of a predetermined image (20b) includes: The distance between the corresponding control point (132) of the inter-frame prediction block (128) and the projection of the scene point onto the predetermined image (20b), and The distance between the source image position (134) of the corresponding reference image (20d) and the projection of the scene point onto the corresponding reference image (20d).

63. The video encoder of claim 56, wherein the scene model is a mesh or a point cloud.

64. The video encoder of any one of claims 56 and subsequent claims, configured to exclude scene model points whose quality metric determined based on the length metric of the shortest line does not conform to a predetermined criterion.

65. The video encoder of any one of claims 56 and subsequent claims, configured to Use a control point to translate the inter-frame prediction block, and / or Use more than one control point for the affine inter-frame prediction block.

66. The video encoder of any one of claims 56 and subsequent claims, configured to determine the scene model additionally based on motion vectors of inter-frame prediction blocks of one or more previous images when determining the scene model.

67. The video encoder of any one of claims 56 and subsequent claims, configured to limit inter-frame prediction blocks to determine scene models relative to images at time layers that satisfy predetermined criteria.

68. The video encoder of claim 55 or a subsequent claim, configured to, when determining a scene model, perform the following operations depending on the determination of the scene model: Compared to scene model points (302) derived from motion vectors in translation mode, scene model points (302) based on motion vectors derived from inter-frame prediction blocks encoded and decoded in affine mode are preferred, and / or Compared to scene model points (302) derived from motion vectors of inter-prediction blocks encoded and decoded individually in the data stream for inter-prediction blocks, scene model points (302) are preferentially based on motion vectors of these inter-prediction blocks whose motion vectors are encoded and decoded individually for inter-prediction blocks, and / or With a preference varying among scene model points (302), such that: the higher the preference, the lower the prediction residual signal encoded into the data stream according to a predetermined metric of the inter-frame prediction block, the scene model points (302) originate from the motion vectors of the inter-frame prediction blocks, and / or With a preference varying between scene model points (302), such that: the higher the preference, the closer the images of the inter-frame prediction blocks are in time, the scene model points (302) originate from the motion vectors of the inter-frame prediction blocks, and / or The preference varies between scene model points such that: the higher the preference, the smaller the quantizer step size of the inter-frame prediction block, and the scene model point (302) originates from the motion vector of the inter-frame prediction block.

69. The video encoder of claim 55 or subsequent claims, configured to When using scene models and video geometry parameters to derive the interior of a block... The weights depend on the part of the scene model. Compared to the portion of motion vectors originating from translation mode, the portion of motion vectors originating from inter-prediction blocks encoded and decoded in affine mode has a higher weight, and / or Compared to the portion of motion vectors originating from inter-prediction blocks encoded and decoded individually in the data stream for each inter-prediction block, the portion of motion vectors originating from these inter-prediction blocks has a higher weight, and / or The higher the weight, the lower the prediction residual signal encoded into the data stream according to the predetermined metric of the inter-frame prediction block, which partly originates from the motion vectors of the inter-frame prediction block, and / or The higher the weight, the closer the images of the inter-frame prediction blocks are in time, partly due to the motion vectors of the inter-frame prediction blocks, and / or The higher the weight, the smaller the quantizer step size of the inter-frame prediction block, which is partly derived from the motion vector of the inter-frame prediction block.

70. The video encoder as described in any of claims 55 or subsequent claims, configured to When using the scene model (300) and video geometry parameters (24) to find the corresponding position (612) in the reference image (610; 20b), For each pixel (606; 23) in the current block (600; 18), Determine the scene projection line (604) along which the scene (14) is projected onto the corresponding pixel (606; 23) of the current block. A final use point (608) on the scene projection line is determined based on one or more intersection points (608) between the scene projection line (604) and the facial region (310) of the scene model's mesh, and The final point (608) is projected onto the reference image (610; 20b) to obtain the corresponding position (612; 22) corresponding to the corresponding pixel (606; 23).

71. The video encoder as described in any of the preceding claims 55 or subsequent claims, configured to When using scene models and video geometry parameters to find corresponding pixels in a reference image For each pixel (606; 23) in the current block (600; 18), Determine the scene projection line along which the scene is projected onto the corresponding pixel (606). A final use point is determined based on one or more points (620) of the point cloud (580) of the scene model (300) falling into a cone (622) or pyramid, the cone or pyramid being widened away from the corresponding pixel (606) and surrounding the scene projection line. The final point is projected onto the reference image (610) to obtain the corresponding position (612; 22) corresponding to the corresponding pixel (606; 23).

72. The video encoder of claim 55 or subsequent claims, configured to When using scene models and video geometry parameters to derive the interior of a block... A depth map is constructed by measuring the distance from each pixel in the current image to the scene model along which the scene is projected onto the corresponding pixel, in order to determine the depth value of the depth map at each pixel. Interpolation is applied to the depth map to determine the depth values ​​of pixels in the scene model that are not hit or are close enough to the scene projection lines. The interior of the block is determined using depth maps and video geometry parameters.

73. A video decoder (100) for decoding video (12) of a scene (14) from a data stream (16) using motion compensation prediction is configured to: For the current block (124), a predetermined motion vector predictor (126) is derived using video geometric correlation parameters (24) describing how the scene (14) is projected (26) onto the image (20) of the video (12), and Rebuild the current block (124) using the pre-defined MVP (126).

74. The video decoder of claim 73, configured to derive a predetermined motion vector predictor (126) by: For the current block (124), the first MVP (130) is derived from the previously decoded portion (128) of the video, and The first MVP (130) is modified using video geometry-related parameters (24) in order to obtain the predetermined MVP (130).

75. The video decoder as described in any one of claims 73 to 74, wherein the video geometric correlation parameter (24) describes a scene-to-image projection of a camera onto an image of the video.

76. The video decoder as described in any one of claims 73 to 75, wherein the video geometric correlation parameters describe the scene-to-image projection of a camera onto the video for each image of the video.

77. The video decoder of any one of claims 73 to 76, wherein video geometric correlation parameters describe a scene-to-image projection of a camera onto the video image by image and image-by-image.

78. The video decoder of any one of claims 74 to 77, wherein video geometric correlation parameters describe the scene to image projection via one or more of the following: One or more non-inherent camera parameters of a camera; and One or more inherent camera parameters of a camera.

79. The video decoder of claim 78, wherein one or more non-inherent camera parameters define the position of a camera and / or the orientation of a camera.

80. The video decoder as described in any one of claims 78 or 79, wherein one or more inherent camera parameters define the focal length and / or FOV angle of a camera.

81. The video decoder as described in any one of claims 73 to 80, wherein the video geometric correlation parameters describe a homomorphic mapping between corresponding positions in a pair of images of the video, or a homomorphic mapping between corresponding motion vectors associated with a pair of images of the video.

82. The video decoder of claim 81, wherein the video geometric correlation parameters describe the homomorphic mapping through vectors or tensors at predetermined control points of the video image.

83. The video decoder of claim 82, wherein the predetermined control point is a corner point of the video image.

84. The video decoder as described in any one of claims 81 to 83, wherein for each image, the video geometric correlation parameters contain only one vector or tensor for each of the four corner points of the image of the video.

85. The video decoder of any one of claims 73 to 83, wherein video geometric correlation parameters describe a homomorphic mapping between corresponding positions in a pair of images of the video, and the video decoder is configured to, for a predetermined image, derive a scene-to-image projection onto the predetermined image based on the homomorphic mapping between corresponding positions in a pair of images including the predetermined image.

86. The video decoder as described in any one of claims 73 to 85 is configured to decode video geometric correlation parameters from a data stream.

87. The video decoder as described in any one of claims 73 to 86 is configured to determine video geometric correlation parameters based on the decoded portion of the video.

88. The video decoder of any one of claims 73 to 87, wherein video geometric correlation parameters describe homomorphic mappings between corresponding positions in image pairs of the video, and the video decoder is configured to, for a predetermined image, derive a scene-to-image projection onto the predetermined image based on a sequence of homomorphic mappings between corresponding positions in one or more image pairs including the predetermined image and a base image, and based on non-inherent parameters and inherent parameters of the base image.

89. The video decoder as described in any one of claims 73 to 88 is configured to modify the first motion vector predictor (130) using video geometric correlation parameters by using one or more of the following derived from video geometric correlation parameters: The first scene-to-image projection of the current image (20a), the current block is a part of the current image. The second scene to image projection of the reference image (20c) associated with the pre-defined MVP (126), From the third scene of the source image (20b) from which the first MVP (130) is derived, to the image projection, The fourth scene to image projection of another reference image (20d) associated with the first MVP (130).

90. The video decoder as described in any one of claims 73 to 89 is configured to modify the first MVP (130) using video geometric correlation parameters by: Using video geometric correlation parameters (24), the first image position (132) of the reference block (128) from which the first MVP (130) is derived, and the first MVP (130), one or more scene points in the scene are determined, and Using video geometry-related parameters (24) and one or more scene points, determine the pre-defined MVP (126).

91. The video decoder of claim 90, wherein the video decoder is configured to determine one or more scene points in a scene by: The linear equations are determined using video geometric correlation parameters (24), the first image position (132) of the reference block (128), and the second image position (134) of another reference image, wherein the first MVP is positioned by placing the tail of the first MVP at the first image position, and the first MVP points to the second image position (134) of the other reference image. Solve the system of linear equations.

92. The video decoder of claim 91, wherein the system of linear equations is defined by the equation AQ=0, wherein Q represents a vector containing the coordinates of scene points to be determined, and A represents a matrix defined by video geometric correlation parameters (24), the first image position (132), and the second image position (134).

93. The video decoder of claim 92, wherein matrix A is defined as follows: A.row(0) = q di (0)*cam di .row(2) - cam di .row(0) A.row(1) = q di (1)*cam di .row(2) - cam di .row(1) A.row(2) = q si (0)*cam si .row(2) - cam si .row(0) A.row(3) = q si (1)*cam si .row(2) - cam si .row(1), The rows of matrix A are represented by A.row(0), A.row(1), A.row(2), and A.row(3). The coordinates of the first image position (132) are determined by q. di (0) and q di (1) indicates that The coordinates of the second image position (134) are determined by q. si (0) and q si (1) indicates, and The video geometric correlation parameters (24) are derived from cam di .row(2), cam di .row(0), cam di .row(2), cam di .row(1), cam si .row(2), cam si .row(0), cam si .row(2) and cam si .row(1) represents.

94. The video decoder of any one of claims 91 to 93, configured to perform the task of solving a system of linear equations using a singular value decomposition method.

95. The video decoder of claim 90, wherein the video decoder is configured to determine one or more scene points in a scene by minimizing a cost function based on a reprojection error.

96. The video decoder of claim 95, wherein the reprojection error includes: The distance between the first image position (132) of the reference block (128) in the source image (20b) and the projection of the scene point onto the source image (20b), and The distance between the second image position (134) of the other reference image and the projection of the scene point onto the other reference image, wherein the first MVP points to the second image position (134) of the other reference image by placing the tail of the first MVP on the first image position.

97. The video decoder as described in any one of claims 73 to 96, configured to The first MVP is derived from the reference block of the source image, and Modify the first MVP using video geometry-related parameters through the following steps: Using video geometry-related parameters, the first image position (132) of the reference block, and the first MVP, Determine the scene point using the following steps: The linear equations are determined using video geometric correlation parameters (24), the first image position (132) of the reference block (128), and the second image position (134) of another reference image, wherein the first MVP is positioned by placing the tail of the first MVP at the first image position, and the first MVP points to the second image position (134) of the other reference image. Solve the system of linear equations. Using video geometry-related parameters and scene points The location of the third image in the reference image is determined, and the scene points are projected onto that location. The fourth image position of the current image is determined, and the scene point is projected onto the fourth image position. Determine the predefined MVP to define the offset between the third image position and the fourth image position.

98. The video decoder as described in any one of claims 73 to 97, configured to The first MVP is derived from the reference block of the source image, and Modify the first MVP using video geometry-related parameters through the following steps: Using video geometry-related parameters, the first image position (132) of the reference block, and the first MVP, Scene points are determined by minimizing the sum of the two squared reprojection errors. Using video geometry-related parameters and scene points The location of the third image in the reference image is determined, and the scene points are projected onto that location. The fourth image position of the current image is determined, and the scene point is projected onto the fourth image position. Determine the predefined MVP to define the offset between the third image position and the fourth image position.

99. The video decoder as described in any one of claims 73 to 98, configured to The first MVP is derived from the reference block of the source image, and Modify the first MVP using video geometry-related parameters through the following steps: Using video geometry-related parameters, the first image position (132) of the reference block, and the first MVP, Determine the first scene projection line along which the scene is projected onto the first image position (132). Determine the second scene projection line along the second image position (134) where the scene is projected onto another reference image, by placing the tail of the first MVP at the first image position and pointing the head of the first MVP towards the second image position, and Determine the scene point on the shortest line connecting any point on the first scene projection line and any point on the second scene projection line, and Using video geometry-related parameters and scene points The location of the third image in the reference image is determined, and the scene points are projected onto that location. The fourth image position of the current image is determined, and the scene point is projected onto the fourth image position. Determine the predefined MVP to define the offset between the third image position and the fourth image position.

100. The video decoder of claim 99, configured to determine the scene point as the midpoint of the shortest line in the middle of the shortest line.

101. The video decoder as described in any one of claims 73 to 100, configured to The first MVP is derived from the reference block of the source image, and Modify the first MVP using video geometry-related parameters through the following steps: Using video geometry-related parameters, the first image position of the reference block, and the first MVP. Determine the first scene projection line along which the scene is projected onto the first image position. Determine the second scene projection line along the second image position where the scene is projected onto another reference image, by placing the tail of the first MVP at the first image position and pointing the head of the first MVP towards the second image position, and Determine the scene point on the shortest line connecting any point on the first scene projection line and any point on the second scene projection line, and Using video geometry-related parameters and scene points If the source image is equal to the reference image associated with the predefined MVP, then The fourth image position of the current image is determined, and the scene point is projected onto the fourth image position. The predetermined MVP is determined to define the offset between the fourth image position and the first image position, or between the fourth image position and the intermediate image position. The intermediate image position is generated by the projection of the midpoint between the scene point on the shortest line and the intersection of the shortest line and the first scene projection line onto the source image. If another reference image is equal to the reference image, then The fourth image position of the current image is determined, and the scene point is projected onto the fourth image position. Determine the predetermined MVP to define the offset between the fourth image position and the second image position, or between the fourth image position and another intermediate image position, where the other intermediate image position is generated by the projection of another intermediate point from the intersection of the scene point on the shortest line and the intersection of the shortest line and the second scene projection line onto another reference image. If the source image, the reference image, and another reference image are different from each other, then The location of the third image in the reference image is determined, and the scene points are projected onto that location. The fourth image position of the current image is determined, and the scene point is projected onto the fourth image position. Determine the predefined MVP to define the offset between the third image position and the fourth image position.

102. The video decoder of claim 101, configured to determine the scene point as the midpoint of the shortest line in the middle of the shortest line.

103. The video decoder as described in any of the preceding claims 101 or subsequent claims, configured to If the source image is equal to the reference image associated with the predefined MVP, then The scene point is defined as the intersection of the shortest line and the first scene projection line. If another reference image is equal to the reference image, then The scene point is defined as the intersection of the shortest line and the second scene projection line. If the source image, the reference image, and another reference image are different from each other, then The scene point is defined as the midpoint of the shortest line within the shortest line.

104. The video decoder as described in any one of claims 73 to 103 is configured to reconstruct the current block using a predetermined MVP by: Insert the pre-selected MVP into the list of MVP candidates. Select the MVP from the list of MVP candidates, and Rebuild the current block using the selected MVP.

105. The video decoder of claim 104, configured to Decode the indexes pointing to the list of MVP candidates from the data stream, and Use an index to select the chosen MVP from the list of MVP candidates.

106. The video decoder as described in any of claims 104 or subsequent claims is configured to insert a predetermined MVP into a list of MVP candidates as a replacement for a time-predicted MVP or a first MVP.

107. The video decoder as described in any of the preceding claims 104 or subsequent claims, configured to The quality metric is determined based on the length of the shortest line. If the quality metrics meet the predetermined criteria, the predetermined MVP will be inserted into the list of MVP candidates.

108. The video decoder of claim 99 or 100, wherein the current block and the reference block are within one image and the current image is equal to the source image.

109. The video decoder as described in any one of claims 73 to 108, configured to The first MVP is derived from the reference block of the current image, and Modify the first MVP using video geometry-related parameters through the following steps: Using video geometry-related parameters, the first image position of the reference block, and the MVP. Determine the first scene projection line along which the scene is projected onto the first image position. Determine the second scene projection line along the second image position where the scene is projected onto another reference image, by placing the tail of the first MVP at the first image position and pointing the head of the first MVP towards the second image position, and Determine the scene point on the shortest line connecting any point on the first scene projection line and any point on the second scene projection line, and Using video geometry-related parameters and scene points The location of the third image in the reference image is determined, and the scene points are projected onto that location. Determine the predetermined MVP in order to define the offset between the third image position and the first image position or between the third image position and the intermediate image position. The intermediate image position is generated by the projection of the midpoint between the scene point on the shortest line and the intersection of the shortest line and the first scene projection line onto the source image.

110. The video decoder of claim 109, configured to determine the scene point as the midpoint of the shortest line in the middle of the shortest line.

111. The video decoder of claim 109, configured to determine scene points as the intersection of the shortest line and the first scene projection line.

112. The video decoder as described in any one of claims 73 to 111 is configured to The first MVP is derived from the reference block of the current image, and Modify the first MVP using video geometry-related parameters through the following steps: Using video geometry-related parameters, the first image position of the reference block, and the MVP. Determine the first scene projection line along which the scene is projected onto the first image position. Determine the second scene projection line along the second image position where the scene is projected onto another reference image, by placing the tail of the first MVP at the first image position and pointing the head of the first MVP towards the second image position, and Determine the shortest line connecting any point on the first scene projection line and any point on the second scene projection line, and Using video geometry parameters and the shortest line If the other reference image is different from the reference image associated with the predetermined MVP, then The scene point is determined as the midpoint of the shortest line, or as the intersection point of the shortest line and the first scene projection line. The location of the third image in the reference image is determined, and the scene points are projected onto that location. The predetermined MVP is determined to define the offset between the third image position and the first image position, or between the third image position and the intermediate image position. The intermediate image position is generated by the projection of the midpoint between the scene point on the shortest line and the intersection of the shortest line and the first scene projection line onto the source image. If another reference image is equal to the reference image, then Define the scene points as points on the shortest line. The location of the third image in the reference image is determined, and the scene points are projected onto that location. The fourth image position of the current image is determined, and the scene point is projected onto the fourth image position. Determine the predefined MVP to define the offset between the third image position and the fourth image position.

113. The video decoder of claim 112, configured to determine the scene point as if another reference image is equal to the reference image. The intersection point of the shortest line and the projection line of the first scene. The intersection point of the shortest line and the projection line of the second scene, or The midpoint of the shortest line in the middle.

114. The video decoder as described in any one of claims 73 to 113 is configured to reconstruct the current block using a predetermined MVP by: Using video geometry parameters and a predefined MVP, the pixel positions in the current block are deformed to the corresponding pixel positions in the reference image.

115. The video decoder as described in any one of claims 73 to 114 is configured to decode video geometric correlation parameters from a slice header or image header within an access unit of an image-related data stream of the video.

116. The video decoder as described in any one of claims 73 to 115 is configured to derive a predetermined motion vector predictor by: Video geometry parameters are used to map the image position of the current block to the corresponding position in the reference image to be referenced by the predetermined MVP, and the offset between the image position and the corresponding position is used as the predetermined MVP.

117. The video decoder as described in any one of claims 73 to 116, configured to The scene model (300) is determined based on the motion vectors (126) of the image and the geometric parameters (24) of the video. The scene model (300) and video geometry-related parameters (24) are used to determine the pre-defined MVP (130).

118. The video decoder as described in any one of claims 73 to 117 is configured to determine a scene model (300) by: For each control point in one or more control points (132) of an inter-frame prediction block (128) of a predetermined image (20b), scene model points (302) for forming the basis of the scene model are determined by the following operation: The source image position (134) in the corresponding reference image (20d) from which the motion vector points (130) is determined, and the motion vector is encoded and decoded in the data stream of the inter-frame prediction block of the corresponding control point. Using video geometry-related parameters (24), Determine the first scene projection line along which the scene is projected onto the corresponding control point (132). Determine the second scene projection line along which the scene is projected onto the source image position (134), and Determine the scene point on the shortest line connecting any point on the first scene projection line and any point on the second scene projection line, and The scene model points are determined as follows: Define scene model points as scene points, or The scene model point is determined by determining the distance between the scene point and the predetermined image, and by determining that the scene model point is located on the first scene projection line at that distance.

119. The video decoder as described in any one of claims 73 to 117 is configured to determine a scene model (300) by: For each control point in one or more control points (132) of an inter-frame prediction block (128) of a predetermined image (20b), scene model points (302) for forming the basis of the scene model are determined by the following operation: The source image position (134) in the corresponding reference image (20d) from which the motion vector points (130) is determined, and the motion vector is encoded and decoded in the data stream of the inter-frame prediction block of the corresponding control point. Using video geometry-related parameters (24), Determine the scene point using the following steps: The linear equations are determined using video geometric parameters (24), corresponding control points (132), and source image location (134). Solve the system of linear equations. Determine scene model points using the following methods Define scene model points as scene points, or The scene model point is determined by determining the distance between the scene point and the predetermined image, and by determining that the scene model point is located on the first scene projection line at that distance.

120. The video decoder of claim 119, wherein the system of linear equations is defined by the equation AQ=0, wherein Q represents a vector containing the coordinates of scene points to be determined, and A represents a matrix defined by video geometric correlation parameters (24), corresponding control points (132), and source image position (134).

121. The video decoder of claim 120, wherein matrix A is defined as follows: A.row(0) = q di (0)*cam di .row(2) - cam di .row(0) A.row(1) = q di (1)*cam di .row(2) - cam di .row(1) A.row(2) = q si (0)*cam si .row(2) - cam si .row(0) A.row(3) = q si (1)*cam si .row(2) - cam si .row(1), The rows of matrix A are represented by A.row(0), A.row(1), A.row(2), and A.row(3). The coordinates of the corresponding control point (132) are given by q. di (0) and q di (1) indicates that The coordinates of the source image position (134) are determined by q. si (0) and q si (1) indicates that, and The video geometric correlation parameters (24) are derived from cam di .row(2), cam di .row(0), cam di .row(2), cam di .row(1), cam si .row(2), cam si .row(0), cam si .row(2) and cam si .row(1) represents.

122. The video decoder of any one of claims 118 to 121 is configured to perform the task of solving a system of linear equations using a singular value decomposition method.

123. The video decoder as described in any one of claims 73 to 117 is configured to determine a scene model (300) by: For each control point in one or more control points (132) of an inter-frame prediction block (128) of a predetermined image (20b), scene model points (302) for forming the basis of the scene model are determined by the following operation: The source image position (134) in the corresponding reference image (20d) from which the motion vector points (130) is determined, and the motion vector is encoded and decoded in the data stream of the inter-frame prediction block of the corresponding control point. Using video geometry-related parameters (24), Scene points are determined by minimizing the cost function based on the reprojection error. The scene model points are determined as follows: Define scene model points as scene points, or The scene model point is determined by determining the distance between the scene point and the predetermined image, and by determining that the scene model point is located on the first scene projection line at that distance.

124. The video decoder of claim 123, wherein the reprojection error associated with a corresponding control point (132) of an inter-frame prediction block (128) of a predetermined image (20b) includes: The distance between the corresponding control point (132) of the inter-frame prediction block (128) and the projection of the scene point onto the predetermined image (20b), and The distance between the source image position (134) of the corresponding reference image (20d) and the projection of the scene point onto the corresponding reference image (20d).

125. The video decoder of claim 118, wherein the scene model is a mesh or a point cloud.

126. The video decoder of any one of claims 118 and subsequent claims, configured to exclude scene model points whose quality metric determined based on the length metric of the shortest line does not conform to a predetermined criterion.

127. The video decoder as claimed in any one of claims 117 and subsequent claims, configured to Use a control point to translate the inter-frame prediction block, and / or Use more than one control point for the affine inter-frame prediction block.

128. The video decoder of any one of claims 118 and subsequent claims, configured to determine the scene model additionally based on motion vectors of inter-frame prediction blocks of one or more previous images when determining the scene model.

129. The video decoder of any one of claims 118 and subsequent claims, configured to limit inter-frame prediction blocks to determine scene models relative to images at time layers that satisfy predetermined criteria.

130. The video decoder of claim 117 or a subsequent claim, configured to, when determining a scene model, Compared to scene model points (302) derived from motion vectors in translation mode, scene model points (302) derived from inter-frame prediction blocks encoded and decoded in affine mode are preferentially used to determine the scene model, and / or Compared to scene model points (302) derived from motion vectors of inter-prediction blocks encoded and decoded individually in the data stream for inter-prediction blocks, the scene model is determined preferentially based on scene model points (302) derived from motion vectors of inter-prediction blocks encoded and decoded individually for inter-prediction blocks in the data stream, and / or The scene model is determined by the preference varying between scene model points (302), such that: the higher the preference, the lower the prediction residual signal encoded into the data stream according to a predetermined metric of the inter-frame prediction block; the scene model points (302) originate from the motion vectors of the inter-frame prediction blocks; and / or The scene model is determined by a preference that varies between scene model points (302), such that: the higher the preference, the closer the images of the inter-frame prediction blocks are in time; the scene model points (302) originate from the motion vectors of the inter-frame prediction blocks; and / or The scene model is determined by the preference that varies between scene model points, such that: the higher the preference, the smaller the quantizer step size of the inter-frame prediction block, and the scene model point (302) originates from the motion vector of the inter-frame prediction block.

131. The video decoder as claimed in claim 117 or a subsequent claim, configured to When using scene models and video geometry parameters to determine the predetermined MVP The weights depend on the part of the scene model. Compared to the portion of motion vectors originating from translation mode, the portion of motion vectors originating from inter-prediction blocks encoded and decoded in affine mode has a higher weight, and / or Compared to the portion of motion vectors originating from inter-prediction blocks encoded and decoded individually in the data stream for each inter-prediction block, the portion of motion vectors originating from these inter-prediction blocks has a higher weight, and / or The higher the weight, the lower the prediction residual signal encoded into the data stream according to the predetermined metric of the inter-frame prediction block, which partly originates from the motion vectors of the inter-frame prediction block, and / or The higher the weight, the closer the images of the inter-frame prediction blocks are in time, partly due to the motion vectors of the inter-frame prediction blocks, and / or The higher the weight, the smaller the quantizer step size of the inter-frame prediction block, which is partly derived from the motion vector of the inter-frame prediction block.

132. The video decoder as described in any of claims 117 or 118, configured to determine a predetermined MVP of the current block (600; 124) of the current image (602; 20a) using the scene model (300) and video geometric parameters, Determine the scene projection line (604) along which the scene (14) is projected (26) onto the image position (606) of the current block (600; 124). A final use point (608) on the scene projection line (604) is determined based on one or more intersection points (608) of the face region (310) of the mesh of the scene model (300), and The final point (608) is projected (26) onto the reference image (610; 20c) referenced by the predetermined MVP in order to obtain the corresponding image position (612) in the reference image (610), and the predetermined MVP (126) is determined as the offset between the corresponding image position (612) and the image position (606).

133. The video decoder as described in claim 117 or any of the following claims, configured to When using the scene model (300) and video geometry-related parameters (24) to determine the pre-defined MVP (126) of the current block (600; 124) of the current image, Determine the scene projection line (604) along which the scene is projected (26) onto the image position (606) of the current block. A final use point (608) is determined based on one or more points (620) of the point cloud (580) of the scene model (300) falling into a cone (622) or pyramid, the cone or pyramid being widened away from the image point (606) and surrounding the scene projection line. The final point (608) is projected onto the reference image (602) referenced by the predetermined MVP (126) to obtain the corresponding point in the reference image, and the predetermined MVP is determined as the offset between the corresponding image position (612) and the image position (606).

134. The video decoder as claimed in claim 117 or a subsequent claim, configured to When using scene models and video geometry parameters to determine the predetermined MVP A depth map is constructed by measuring the distance from each pixel in the current image to the scene model along which the scene is projected onto the corresponding pixel, in order to determine the depth value of the depth map at each pixel. Interpolation is applied to the depth map to determine the depth values ​​of pixels that the scene model misses or is close enough to the scene projection lines. The predetermined MVP is determined using depth maps and video geometry parameters.

135. The video decoder as described in any one of claims 73 to 134 is configured to derive a predetermined motion vector predictor (126) by: For the current block (124), the first MVP (130) is derived from the previously decoded portion (128) of the video, and The first MVP (130) is modified using the video geometry-related parameters (24) to obtain the predetermined MVP (126): Check whether the video geometry-related parameters (24) indicate that the current image (20a), the reference image associated with the predetermined MVP (126), the source image from which the first MVP (130) is derived (20b), and another reference image associated with the first MVP (130) are associated with the same camera position. If the current image (20a), the reference image, the source image (20b), and another reference image are associated with the same camera location, then Determine the first scene projection line (26b) along the first image position (132) of the reference block (128) in the source image (20b) where the scene is projected. The third image position (136) of the reference image (20c) is determined as the intersection point of the first scene projection line (26b) and the reference image (20c). The fourth image position (138) of the current image (20a) is determined as the intersection point of the first scene projection line (26b) and the current image (20a). Determine the predefined MVP (126) to define the offset between the third image position (136) and the fourth image position (138). or The second scene projection line (26d) along which the scene is projected onto the second image position (134) of another reference image (20d) is determined, by placing the tail of the first MVP (130) derived from the reference block (128) onto the first image position (132), with the head of the first MVP (130) pointing towards the second image position. The third image position (136) of the reference image (20c) is determined as the intersection point of the second scene projection line (26d) and the reference image (20c). The fourth image position (138) of the current image (20a) is determined as the intersection point of the second scene projection line (26d) and the current image (20a). Determine the predefined MVP (126) to define the offset between the third image position (136) and the fourth image position (138). or Determine the first scene projection line (26b) along the first image position (132) of the reference block (128) in the source image (20b) where the scene is projected. The second scene projection line (26d) along which the scene is projected onto the second image position (134) of another reference image (20d) is determined, by placing the tail of the first MVP (130) derived from the reference block (128) onto the first image position (132), with the head of the first MVP (130) pointing towards the second image position. A third scene projection line (26') is determined based on the first scene projection line (26b) and the second scene projection line (26d), wherein the third scene projection line (26') is correlated with the arithmetic mean of the first scene projection line (26b) and the second scene projection line (26d). The third image position (136) of the reference image (20c) is determined as the intersection point of the third scene projection line (26') and the reference image (20c). The fourth image position (138) of the current image (20a) is determined as the intersection point of the third scene projection line (26') and the current image (20a). Determine the pre-defined MVP (126) to define the offset between the third image position (136) and the fourth image position (138).

136. A video encoder for encoding video (12) of a scene (14) into a data stream (16) using motion compensation prediction, configured to For the current block (124), a predetermined motion vector predictor (126) is derived using video geometric correlation parameters (24) describing how the scene (14) is projected (26) onto the image (20) of the video (12), and The current block (124) is encoded using a predefined motion vector predictor (126).

137. The video encoder of claim 136, configured to derive a predetermined motion vector predictor (126) by: For the current block (124), the first MVP (130) is derived from the previously encoded portion (128) of the video, and The first MVP (130) is modified using video geometry-related parameters (24) in order to obtain the predetermined MVP (130).

138. The video encoder as described in any one of claims 136 to 137, wherein video geometric correlation parameters describe scene-to-image projection of a camera onto an image of the video.

139. The video encoder of any one of claims 136 to 138, wherein video geometric correlation parameters describe scene-to-image projection of a camera onto the video for each image of the video.

140. The video encoder of any one of claims 136 to 139, wherein video geometric correlation parameters describe a scene-to-image projection of a camera onto the video image on an image-by-image and whole-image basis.

141. The video encoder of any one of claims 137 to 140, wherein the video geometric correlation parameters describe the scene to the image projection via one or more of the following: One or more non-inherent camera parameters of a camera; and One or more inherent camera parameters of a camera.

142. The video encoder of claim 141, wherein one or more non-inherent camera parameters define the position of a camera and / or the orientation of a camera.

143. The video encoder as described in any one of claims 141 or 142, wherein one or more inherent camera parameters define the focal length or FOV angle of a camera.

144. The video encoder of any one of claims 136 to 143, wherein the video geometric correlation parameters describe a homomorphic mapping between corresponding positions in a pair of images of the video, or a homomorphic mapping between corresponding motion vectors associated with a pair of images of the video.

145. The video encoder of claim 144, wherein the video geometric correlation parameters describe the homomorphic mapping through vectors or tensors at predetermined control points of the video image.

146. The video encoder of claim 145, wherein the predetermined control point is a corner point of the video image.

147. The video encoder as described in any one of claims 144 to 146, wherein for each image, the video geometric correlation parameters contain only one vector or tensor for each of the four corner points of the image of the video.

148. The video encoder of any one of claims 136 to 147, wherein video geometric correlation parameters describe a homomorphic mapping between corresponding positions in a pair of images of the video, and the video encoder is configured to derive a scene-to-image projection onto a predetermined image based on the homomorphic mapping between corresponding positions in a pair of images including the predetermined image.

149. The video encoder as described in any one of claims 136 to 148 is configured to encode video geometrically related parameters into a data stream.

150. The video encoder as described in any one of claims 136 to 149 is configured to determine video geometric correlation parameters based on the encoded portion of the video.

151. The video encoder of any one of claims 136 to 150, wherein video geometric correlation parameters describe homomorphic mappings between corresponding positions in image pairs of the video, and the video encoder is configured to derive a scene-to-image projection onto the predetermined image based on a sequence of homomorphic mappings between corresponding positions in one or more image pairs including the predetermined image and a base image, and based on non-inherent parameters and inherent parameters of the base image.

152. The video encoder as described in any one of claims 136 to 151 is configured to modify the first motion vector predictor (130) using video geometric correlation parameters by using one or more of the following derived from the video geometric correlation parameters: The first scene-to-image projection of the current image (20a), the current block is a part of the current image. The second scene to image projection of the reference image (20c) associated with the pre-defined MVP (126), From the third scene of the source image (20b) from which the first MVP (130) is derived, to the image projection, The fourth scene to image projection of another reference image (20d) associated with the first MVP (130).

153. The video encoder as described in any one of claims 136 to 151 is configured to modify the first MVP (130) using video geometric correlation parameters by: Using video geometric correlation parameters (24), the first image position (132) of the reference block (128) from which the first MVP (130) is derived, and the first MVP (130), one or more scene points in the scene are determined, and Using video geometry-related parameters (24) and one or more scene points, determine the pre-defined MVP (126).

154. The video encoder of claim 153, wherein the video encoder is configured to determine one or more scene points in a scene by: The linear equations are determined using video geometric correlation parameters (24), the first image position (132) of the reference block (128), and the second image position (134) of another reference image, wherein the first MVP is positioned by placing the tail of the first MVP at the first image position, and the first MVP points to the second image position (134) of the other reference image. Solve the system of linear equations.

155. The video encoder of claim 154, wherein the system of linear equations is defined by the equation AQ=0, wherein Q represents a vector containing the coordinates of scene points to be determined, and A represents a matrix defined by video geometric correlation parameters (24), the first image position (132), and the second image position (134).

156. The video encoder of claim 155, wherein matrix A is defined as follows: A.row(0) = q di (0)*cam di .row(2) - cam di .row(0) A.row(1) = q di (1)*cam di .row(2) - cam di .row(1) A.row(2) = q si (0)*cam si .row(2) - cam si .row(0) A.row(3) = q si (1)*cam si .row(2) - cam si .row(1), The rows of matrix A are represented by A.row(0), A.row(1), A.row(2), and A.row(3). The coordinates of the first image position (132) are determined by q. di (0) and q di (1) indicates that The coordinates of the second image position (134) are determined by q. si (0) and q si (1) indicates, and The video geometric correlation parameters (24) are derived from cam di .row(2), cam di .row(0), cam di .row(2), cam di .row(1), cam si .row(2), cam si .row(0), cam si .row(2) and cam si .row(1) represents.

157. The video encoder of any one of claims 154 to 156, configured to perform the solution of a system of linear equations using a singular value decomposition method.

158. The video encoder of claim 153, wherein the video encoder is configured to determine one or more scene points in a scene by minimizing a cost function based on a reprojection error.

159. The video encoder of claim 158, wherein the reprojection error includes: The distance between the first image position (132) of the reference block (128) in the source image (20b) and the projection of the scene point onto the source image (20b), and The distance between the second image position (134) of the other reference image and the projection of the scene point onto the other reference image, wherein the first MVP points to the second image position (134) of the other reference image by placing the tail of the first MVP on the first image position.

160. The video encoder as described in any one of claims 136 to 159 is configured to determine (200) video geometric correlation parameters based on the video (12) and encode the video geometric correlation parameters into a data stream.

161. The video encoder as described in any one of claims 136 to 160 is configured to determine video geometric correlation parameters based on the version of the video generated from the decoded data stream.

162. The video encoder as described in any one of claims 136 to 161 is configured to determine video geometric correlation parameters using one or more of the following: Optimize video geometric parameters or a portion thereof through rate distortion optimization. Using stereo matching, Background / Foreground splitting.

163. The video encoder as described in any one of claims 136 to 162, configured to The first MVP is derived from the reference block of the source image, and Modify the first MVP using video geometry-related parameters through the following steps: Using video geometry-related parameters, the first image position of the reference block (132), and MVP, Determine the scene point using the following steps: The linear equations are determined using video geometric correlation parameters (24), the first image position (132) of the reference block (128), and the second image position (134) of another reference image, wherein the first MVP is positioned by placing the tail of the first MVP at the first image position, and the first MVP points to the second image position (134) of the other reference image. Solve the system of linear equations. Using video geometry-related parameters and scene points The location of the third image in the reference image is determined, and the scene points are projected onto that location. The fourth image position of the current image is determined, and the scene point is projected onto the fourth image position. Determine the predefined MVP to define the offset between the third image position and the fourth image position.

164. The video encoder as described in any one of claims 136 to 163, configured to The first MVP is derived from the reference block of the source image, and Modify the first MVP using video geometry-related parameters through the following steps: Using video geometry-related parameters, the first image position of the reference block (132), and MVP, Scene points are determined by minimizing the sum of the two squared reprojection errors. Using video geometry-related parameters and scene points The location of the third image in the reference image is determined, and the scene points are projected onto that location. The fourth image position of the current image is determined, and the scene point is projected onto the fourth image position. Determine the predefined MVP to define the offset between the third image position and the fourth image position.

165. The video encoder as described in any one of claims 136 to 164, configured to The first MVP is derived from the reference block of the source image, and Modify the first MVP using video geometry-related parameters through the following steps: Using video geometry-related parameters, the first image position of the reference block (132), and MVP, Determine the first scene projection line along which the scene is projected onto the first image position (132). Determine the second scene projection line along the second image position (134) where the scene is projected onto another reference image, by placing the tail of the first MVP at the first image position and pointing the head of the first MVP towards the second image position, and Determine the scene point on the shortest line connecting any point on the first scene projection line and any point on the second scene projection line, and Using video geometry-related parameters and scene points The location of the third image in the reference image is determined, and the scene points are projected onto that location. The fourth image position of the current image is determined, and the scene point is projected onto the fourth image position. Determine the predefined MVP to define the offset between the third image position and the fourth image position.

166. The video encoder of claim 165, configured to determine a scene point as the midpoint of the shortest line in the middle of the shortest line.

167. The video encoder as described in any one of claims 136 to 166, configured to The first MVP is derived from the reference block of the source image, and Modify the first MVP using video geometry-related parameters through the following steps: Using video geometry-related parameters, the first image position of the reference block, and the first MVP. Determine the first scene projection line along which the scene is projected onto the first image position. Determine the second scene projection line along the second image position where the scene is projected onto another reference image, by placing the tail of the first MVP at the first image position and pointing the head of the first MVP towards the second image position, and Determine the scene point on the shortest line connecting any point on the first scene projection line and any point on the second scene projection line, and Using video geometry-related parameters and scene points If the source image is equal to the reference image associated with the predefined MVP, then The fourth image position of the current image is determined, and the scene point is projected onto the fourth image position. The predetermined MVP is determined to define the offset between the fourth image position and the first image position, or between the fourth image position and the intermediate image position. The intermediate image position is generated by the projection of the midpoint on the shortest line between the scene point and the intersection of the shortest line and the first scene projection line onto the source image. If another reference image is equal to the reference image, then The fourth image position of the current image is determined, and the scene point is projected onto the fourth image position. Determine the predetermined MVP to define the offset between the fourth image position and the second image position, or between the fourth image position and another intermediate image position, where the other intermediate image position is generated by the projection of another intermediate point from the intersection of the scene point on the shortest line and the intersection of the shortest line and the second scene projection line onto another reference image. If the source image, the reference image, and another reference image are different from each other, then The location of the third image in the reference image is determined, and the scene points are projected onto that location. The fourth image position of the current image is determined, and the scene point is projected onto the fourth image position. Determine the predefined MVP to define the offset between the third image position and the fourth image position.

168. The video encoder of claim 167, configured to determine a scene point as the midpoint of the shortest line in the middle of the shortest line.

169. The video encoder as described in any of the preceding claims 167 or subsequent claims, configured to If the source image is equal to the reference image associated with the predefined MVP, then The scene point is defined as the intersection of the shortest line and the first scene projection line. If another reference image is equal to the reference image, then The scene point is defined as the intersection of the shortest line and the second scene projection line. If the source image, the reference image, and another reference image are different from each other, then The scene point is defined as the midpoint of the shortest line within the shortest line.

170. The video encoder as described in any one of claims 136 to 169 is configured to encode the current block using a predetermined MVP by: Insert the pre-selected MVP into the list of MVP candidates. Select the MVP from the list of MVP candidates, and Encode the current block using the selected MVP.

171. The video encoder of claim 170, configured to encode an index pointing to a list of MVP candidates into a data stream, wherein the index is used to index a selected MVP in the list of MVP candidates.

172. The video encoder as described in any of claims 170 or subsequent claims is configured to insert a predetermined MVP into a list of MVP candidates as a substitute for a time-predicted MVP or a first MVP.

173. The video encoder as described in any of the preceding claims 170 or subsequent claims, configured to The quality metric is determined based on the length of the shortest line. If the quality metrics meet the predetermined criteria, the predetermined MVP will be inserted into the list of MVP candidates.

174. The video encoder of claim 165 or 166, wherein the current block and the reference block are within one image and the current image is equal to the source image.

175. The video encoder as described in any one of claims 136 to 174, configured to The first MVP is derived from the reference block of the current image, and Modify the first MVP using video geometry-related parameters through the following steps: Using video geometry-related parameters, the first image position of the reference block, and the MVP. Determine the first scene projection line along which the scene is projected onto the first image position. Determine the second scene projection line along the second image position where the scene is projected onto another reference image, by placing the tail of the first MVP at the first image position and pointing the head of the first MVP towards the second image position, and Determine the scene point on the shortest line connecting any point on the first scene projection line and any point on the second scene projection line, and Using video geometry-related parameters and scene points The location of the third image in the reference image is determined, and the scene points are projected onto that location. Determine the predetermined MVP in order to define the offset between the third image position and the first image position or between the third image position and the intermediate image position. The intermediate image position is generated by the projection of the midpoint between the scene point on the shortest line and the intersection of the shortest line and the first scene projection line onto the source image.

176. The video encoder of claim 175, configured to determine a scene point as the midpoint of the shortest line in the middle of the shortest line.

177. The video encoder of claim 175, configured to determine scene points as the intersection of the shortest line and the first scene projection line.

178. The video encoder as described in any one of claims 136 to 177, configured to The first MVP is derived from the reference block of the current image, and Modify the first MVP using video geometry-related parameters through the following steps: Using video geometry-related parameters, the first image position of the reference block, and the MVP. Determine the first scene projection line along which the scene is projected onto the first image position. Determine the second scene projection line along the second image position where the scene is projected onto another reference image, by placing the tail of the first MVP at the first image position and pointing the head of the first MVP towards the second image position, and Determine the shortest line connecting any point on the first scene projection and any point on the second scene projection line, and Using video geometry parameters and the shortest line If the other reference image is different from the reference image associated with the predetermined MVP, then The scene point is determined as the midpoint of the shortest line, or as the intersection point of the shortest line and the first scene projection line. The location of the third image in the reference image is determined, and the scene points are projected onto that location. The predetermined MVP is determined to define the offset between the third image position and the first image position, or between the third image position and the intermediate image position. The intermediate image position is generated by the projection of the midpoint between the scene point on the shortest line and the intersection of the shortest line and the first scene projection line onto the source image. If another reference image is equal to the reference image, then Define the scene points as points on the shortest line. The location of the third image in the reference image is determined, and the scene points are projected onto that location. The fourth image position of the current image is determined, and the scene point is projected onto the fourth image position. Determine the predefined MVP to define the offset between the third image position and the fourth image position.

179. The video encoder of claim 178, configured to determine the scene point as if another reference image is equal to the reference image. The intersection point of the shortest line and the projection line of the first scene. The intersection point of the shortest line and the projection line of the second scene, or The midpoint of the shortest line in the middle.

180. The video encoder as described in any one of claims 136 to 179 is configured to encode the current block using a predetermined MVP by: Using video geometry parameters and a predefined MVP, the pixel positions in the current block are deformed to the corresponding pixel positions in the reference image.

181. The video encoder as described in any one of claims 136 to 180 is configured to encode video geometrical parameters into a slice header or image header within an access unit of an image-related data stream of the video.

182. The video encoder as described in any one of claims 136 to 181 is configured to derive a predetermined motion vector predictor by: Video geometry parameters are used to map the image position of the current block to the corresponding position in the reference image to be referenced by the predetermined MVP, and the offset between the image position and the corresponding position is used as the predetermined MVP.

183. The video encoder as described in any one of claims 136 to 182, configured to The scene model (300) is determined based on the motion vectors (126) of the image and the geometric parameters (24) of the video. The scene model (300) and video geometry-related parameters (24) are used to determine the pre-defined MVP (130).

184. The video encoder as described in any one of claims 136 to 183 is configured to determine a scene model (300) by: For each control point in one or more control points (132) of an inter-frame prediction block (128) of a predetermined image (20b), scene model points (302) for forming the basis of the scene model are determined by the following operation: The source image position (134) in the corresponding reference image (20d) from which the motion vector points (130) is determined, and the motion vector is encoded and decoded in the data stream of the inter-frame prediction block of the corresponding control point. Using video geometry-related parameters (24), Determine the first scene projection line along which the scene is projected onto the corresponding control point (132). Determine the second scene projection line along which the scene is projected onto the source image position (134), and Determine the scene point on the shortest line connecting any point on the first scene projection line and any point on the second scene projection line, and The scene model points are determined as follows: Define scene model points as scene points, or The scene model point is determined by determining the distance between the scene point and the predetermined image, and by determining that the scene model point is located on the first scene projection line at that distance.

185. The video encoder as described in any one of claims 136 to 183 is configured to determine a scene model (300) by: For each control point in one or more control points (132) of an inter-frame prediction block (128) of a predetermined image (20b), scene model points (302) for forming the basis of the scene model are determined by the following operation: The source image position (134) in the corresponding reference image (20d) from which the motion vector points (130) is determined, and the motion vector is encoded and decoded in the data stream of the inter-frame prediction block of the corresponding control point. Using video geometry-related parameters (24), Determine the scene point using the following steps: The linear equations are determined using video geometric parameters (24), corresponding control points (132), and source image location (134). Solve the system of linear equations. The scene model points are determined as follows: Define scene model points as scene points, or The scene model point is determined by determining the distance between the scene point and the predetermined image, and by determining that the scene model point is located on the first scene projection line at that distance.

186. The video encoder of claim 185, wherein the system of linear equations is defined by the equation AQ=0, wherein Q represents a vector containing the coordinates of scene points to be determined, and A represents a matrix defined by video geometric correlation parameters (24), corresponding control points (132), and source image position (134).

187. The video encoder of claim 186, wherein matrix A is defined as follows: A.row(0) = q di (0)*cam di .row(2) - cam di .row(0) A.row(1) = q di (1)*cam di .row(2) - cam di .row(1) A.row(2) = q si (0)*cam si .row(2) - cam si .row(0) A.row(3) = q si (1)*cam si .row(2) - cam si .row(1), The rows of matrix A are represented by A.row(0), A.row(1), A.row(2), and A.row(3). The coordinates of the corresponding control point (132) are given by q. di (0) and q di (1) indicates that The coordinates of the source image position (134) are determined by q. si (0) and q si (1) indicates, and The video geometric correlation parameters (24) are derived from cam di .row(2), cam di .row(0), cam di .row(2), cam di .row(1), cam si .row(2), cam si .row(0), cam si .row(2) and cam si .row(1) represents.

188. The video encoder of any one of claims 186 to 187, configured to perform the solution of a system of linear equations using a singular value decomposition method.

189. The video encoder as described in any one of claims 136 to 183 is configured to determine a scene model (300) by: For each control point in one or more control points (132) of an inter-frame prediction block (128) of a predetermined image (20b), scene model points (302) for forming the basis of the scene model are determined by the following operation: The source image position (134) in the corresponding reference image (20d) from which the motion vector points (130) is determined, and the motion vector is encoded and decoded in the data stream of the inter-frame prediction block of the corresponding control point. Using video geometry-related parameters (24), Scene points are determined by minimizing the cost function based on the reprojection error. The scene model points are determined as follows: Define scene model points as scene points, or The scene model point is determined by determining the distance between the scene point and the predetermined image, and by determining that the scene model point is located on the first scene projection line at that distance.

190. The video decoder of claim 189, wherein the reprojection error associated with a corresponding control point (132) of an inter-frame prediction block (128) of a predetermined image (20b) includes: The distance between the corresponding control point (132) of the inter-frame prediction block (128) and the projection of the scene point onto the predetermined image (20b), and The distance between the source image position (134) of the corresponding reference image (20d) and the projection of the scene point onto the corresponding reference image (20d).

191. The video encoder of claim 190, wherein the scene model is a mesh or a point cloud.

192. The video encoder of any one of claims 190 and subsequent claims, configured to exclude scene model points whose quality metric determined based on the length metric of the shortest line does not conform to a predetermined criterion.

193. The video encoder of any one of claims 190 and subsequent claims, configured to Use a control point to translate the inter-frame prediction block, and / or Use more than one control point for the affine inter-frame prediction block.

194. The video encoder of any one of claims 190 and subsequent claims, configured to determine the scene model additionally based on motion vectors of inter-frame prediction blocks of one or more previous images when determining the scene model.

195. The video encoder of any one of claims 190 and subsequent claims, configured to limit inter-frame prediction blocks to determine scene models relative to images at time layers that satisfy predetermined criteria.

196. The video encoder of claim 183 or a subsequent claim, configured to, when determining a scene model, utilize the following portions dependent on the determination of the scene model: Compared to scene model points (302) derived from motion vectors in translation mode, scene model points (302) based on motion vectors derived from inter-frame prediction blocks encoded and decoded in affine mode are preferred, and / or Compared to scene model points (302) derived from motion vectors of inter-prediction blocks encoded and decoded individually in the data stream for inter-prediction blocks, scene model points (302) are preferentially based on motion vectors of these inter-prediction blocks whose motion vectors are encoded and decoded individually for inter-prediction blocks, and / or With a preference varying among scene model points (302), such that: the higher the preference, the lower the prediction residual signal encoded into the data stream according to a predetermined metric of the inter-frame prediction block, the scene model points (302) originate from the motion vectors of the inter-frame prediction blocks, and / or With a preference varying between scene model points (302), such that: the higher the preference, the closer the images of the inter-frame prediction blocks are in time, the scene model points (302) originate from the motion vectors of the inter-frame prediction blocks, and / or The preference varies between scene model points such that: the higher the preference, the smaller the quantizer step size of the inter-frame prediction block, and the scene model point (302) originates from the motion vector of the inter-frame prediction block.

197. The video encoder as described in claim 183 or subsequent claims, configured to When using scene models and video geometry parameters to determine the predetermined MVP The weights depend on the part of the scene model. Compared to the portion of motion vectors originating from translation mode, the portion of motion vectors originating from inter-prediction blocks encoded and decoded in affine mode has a higher weight, and / or Compared to the portion of motion vectors originating from inter-prediction blocks encoded and decoded individually in the data stream for each inter-prediction block, the portion of motion vectors originating from these inter-prediction blocks has a higher weight, and / or The higher the weight, the lower the prediction residual signal encoded into the data stream according to the predetermined metric of the inter-frame prediction block, which partly originates from the motion vectors of the inter-frame prediction block, and / or The higher the weight, the closer the images of the inter-frame prediction blocks are in time, partly due to the motion vectors of the inter-frame prediction blocks, and / or The higher the weight, the smaller the quantizer step size of the inter-frame prediction block, which is partly derived from the motion vector of the inter-frame prediction block.

198. The video encoder as described in any of claims 183 or the following claims, configured to determine a predetermined MVP of the current block (600; 124) of the current image (602; 20a) using the scene model (300) and video geometrical parameters, Determine the scene projection line (604) along which the scene (14) is projected (26) onto the image position (606) of the current block (600; 124). A final use point (608) on the scene projection line (604) is determined based on one or more intersection points (608) of the face region (310) of the mesh (590) of the scene model (300), and The final point (608) is projected (26) onto the reference image (610; 20c) referenced by the predetermined MVP in order to obtain the corresponding image position (612) in the reference image (610), and the predetermined MVP (126) is determined as the offset between the corresponding image position (612) and the image position (606).

199. The video encoder as described in claim 183 or any of the following claims, configured to When using the scene model (300) and video geometry-related parameters (24) to determine the pre-defined MVP (126) of the current block (600; 124) of the current image, Determine the scene projection line (604) along which the scene is projected (26) onto the image position (606) of the current block. A final use point (608) is determined based on one or more points (620) of the point cloud (580) of the scene model (300) falling into a cone (622) or pyramid, the cone or pyramid being widened away from the image point (606) and surrounding the scene projection line. The final point (608) is projected onto the reference image (602) referenced by the predetermined MVP (126) to obtain the corresponding point in the reference image, and the predetermined MVP is determined as the offset between the corresponding image position point (612) and the image position (606).

200. The video encoder as claimed in claim 183 or subsequent claims, configured to When using scene models and video geometry parameters to determine the predetermined MVP A depth map is constructed by measuring the distance from each pixel in the current image to the scene model along which the scene is projected onto the corresponding pixel, in order to determine the depth value of the depth map at each pixel. Interpolation is applied to the depth map to determine the depth values ​​of pixels in the scene model that are not hit or are close enough to the scene projection lines. The predetermined MVP is determined using depth maps and video geometry parameters.

201. The video encoder as described in any one of claims 136 to 200 is configured to derive a predetermined motion vector predictor (126) by: For the current block (124), the first MVP (130) is derived from the previously encoded portion (128) of the video, and The first MVP (130) is modified using the video geometry-related parameters (24) in order to obtain the predetermined MVP (130): Check whether the video geometry-related parameters (24) indicate that the current block is part of the current image (20a), the reference image associated with the predetermined MVP (126), the source image from which the first MVP (130) is derived (20b), and another reference image associated with the first MVP (130) are associated with the same camera position. If the current image (20a), the reference image, the source image (20b), and another reference image are associated with the same camera location, then The first scene projection line along the first image position (132) of the reference block (128) in the source image (20b) is determined. The position of the third image in the reference image is determined as the intersection point of the projection lines of the first scene and the reference image. The position of the fourth image in the current image is determined as the intersection point of the projection line of the first scene and the current image. Determine the predefined MVP to define the offset between the third and fourth image positions. or The second scene projection line along which the scene is projected onto the second image position of another reference image is determined by placing the tail of the first MVP derived from the reference block (128) onto the first image position (132), with the head of the first MVP pointing towards the second image position. The position of the third image in the reference image is determined as the intersection point of the second scene projection line and the reference image. The position of the fourth image in the current image is determined as the intersection point of the second scene projection line and the current image. Determine the predefined MVP to define the offset between the third and fourth image positions. or The first scene projection line along the first image position (132) of the reference block (128) in the source image (20b) is determined. The second scene projection line along which the scene is projected onto the second image position of another reference image is determined by placing the tail of the first MVP derived from the reference block (128) onto the first image position (132), with the head of the first MVP pointing towards the second image position. A third scene projection line is determined based on the first and second scene projection lines, wherein the third scene projection line is correlated with the arithmetic mean of the first and second scene projection lines. The position of the third image in the reference image is determined as the intersection point of the third scene projection line and the reference image. The position of the fourth image in the current image is determined as the intersection point of the projection line of the third scene and the current image. Determine the predefined MVP to define the offset between the third image position and the fourth image position.

202. A video decoder (500) for using motion compensation to predict and decode video (12) of a scene (14) from a data stream (16), configured to A synthetic reference image is constructed (502) and (504) through the following operations, which are then synthesized to form a synthetic version of the predetermined image: Using the video geometry-related parameters (24) describing how scene (14) is projected onto the image (20) of video (12), the corresponding position (508) of the pixel (506) in the predetermined image (20a) is found in the reference image (20c), and The reference image (20c) is sampled at the corresponding position (508), and For the current part of the current image, the interior of the part is obtained by sampling the corresponding part of the synthetic reference image (504), and Rebuild the current part using the internal reconstruction (510).

203. The video decoder of claim 202, configured to The scene model is determined based on the motion vectors of the image and the geometric parameters of the video. The corresponding position (508) is found in the reference image (20c) using the scene model and video geometry parameters.

204. The video decoder as described in any one of claims 202 to 203, wherein video geometric correlation parameters describe a scene-to-image projection of a camera (30) onto an image (20) of a video.

205. The video decoder as described in any one of claims 202 to 204, wherein video geometric correlation parameters describe a scene-to-image projection of a camera onto the video for each image of the video.

206. The video decoder of any one of claims 202 to 205, wherein video geometric correlation parameters describe a scene-to-image projection of a camera onto the video image by image and image-by-image.

207. The video decoder of any one of claims 203 to 206, wherein video geometric correlation parameters describe the scene to image projection via one or more of the following: One or more non-inherent camera parameters of a camera; and One or more inherent camera parameters of a camera.

208. The video decoder of claim 207, wherein one or more non-inherent camera parameters define the position of a camera. and / or the orientation of a camera .

209. The video decoder as described in any one of claims 207 or 208, wherein one or more inherent camera parameters define the focal length and / or FOV angle of a camera.

210. The video decoder as described in any one of claims 202 to 209, wherein the video geometric correlation parameters describe a homomorphic mapping (42) between corresponding positions (401, 402) in a pair of images (201, 202) of the video, or a homomorphic mapping between corresponding motion vectors associated with the pair of images of the video.

211. The video decoder of claim 210, wherein the video geometric correlation parameters describe the homomorphic mapping via a vector (462) or tensor at a predetermined control point (442) of the video image.

212. The video decoder of claim 211, wherein the predetermined control point is a corner point of the video image.

213. The video decoder as described in any one of claims 210 to 212, wherein for each image, the video geometric correlation parameters contain only one vector or tensor for each of the four corner points of the image of the video.

214. The video decoder of any one of claims 202 to 213, wherein video geometric correlation parameters describe a homomorphic mapping between corresponding positions in a pair of images of the video, and the video decoder is configured to derive a scene-to-image projection onto a predetermined image based on the homomorphic mapping between corresponding positions in a pair of images including the predetermined image.

215. The video decoder as described in any one of claims 202 to 214 is configured to decode video geometric correlation parameters from a data stream.

216. The video decoder as described in any one of claims 202 to 215 is configured to determine video geometric correlation parameters based on the decoded portion of the video.

217. The video decoder of any one of claims 202 to 216, wherein video geometric correlation parameters describe homomorphic mappings between corresponding positions in image pairs of the video, and the video decoder is configured to derive a scene-to-image projection onto the predetermined image based on a sequence of homomorphic mappings between corresponding positions in one or more image pairs including the predetermined image and a base image, and based on non-inherent parameters and inherent parameters of the base image.

218. The video decoder as described in any one of claims 202 to 217, configured to decode syntax elements from a data stream, and If the syntax element has a first state, then A synthetic reference image is constructed (502) and (504) through the following operations, which are then synthesized to form a synthetic version of the predetermined image: Using the video geometry-related parameters (24) describing how scene (14) is projected onto the image (20) of video (12), the corresponding position (508) of the pixel (506) in the predetermined image (20a) is found in the reference image (20c), and The reference image (20c) is sampled at the corresponding position (508), and For the current part of the current image, the interior of the part is obtained by sampling the corresponding part of the synthetic reference image (504), and Rebuild the current part using the internal reconstruction (510).

219. The video decoder of claim 218, configured to, if the syntax element has a second state, then Reconstruct the current portion independently of video geometric parameters, or The current segment is reconstructed by copying the decoded video portion at regular pixel intervals.

220. The video decoder as described in any one of claims 202 to 219, configured to This yields a list of MVP candidates for the current section, where each MVP candidate corresponds to a specific motion vectorless inter-frame encoding / decoding mode. Select the MVP from the list of MVP candidates, and Rebuild the current section using the selected MVP by performing the following operations: If the selected MVP corresponds to a specific motion vectorless inter-frame encoding / decoding mode, then A synthetic reference image is constructed (502) and (504) through the following operations, which are then synthesized to form a synthetic version of the predetermined image: Using the video geometry-related parameters (24) describing how scene (14) is projected onto the image (20) of video (12), the corresponding position (508) of the pixel (506) in the predetermined image (20a) is found in the reference image (20c), and The reference image (20c) is sampled at the corresponding position (508), and For the current part of the current image, the interior of the part is obtained by sampling the corresponding part of the synthetic reference image (504), and Rebuild the current part using a partial internal reconstruction (510). If the selected MVP corresponds to an MVP candidate other than a specific motion vectorless inter-frame encoding / decoding mode, then Use the selected MVP to reconstruct the current part using motion vector compensation prediction.

221. The video decoder as described in any one of claims 202 to 210, configured to The scene model (300) is determined based on the motion vectors (126) of the image and the geometric parameters (24) of the video. The corresponding pixel (23) is found in the reference image using the scene model (300) and video geometry parameters (24).

222. The video decoder as described in any one of claims 202 to 221 is configured to determine a scene model (300) by: For each control point in one or more control points (132) of an inter-frame prediction block (128) of a predetermined image (20b), scene model points (302) for forming the basis of the scene model are determined by the following operation: The source image position (134) in the corresponding reference image (20d) from which the motion vector points (130) is determined, and the motion vector is encoded and decoded in the data stream of the inter-frame prediction block of the corresponding control point. Using video geometry-related parameters (24), Determine the first scene projection line along which the scene is projected onto the corresponding control point (132). Determine the second scene projection line along which the scene is projected onto the source image position (134), and Determine the scene point on the shortest line connecting any point on the first scene projection line and any point on the second scene projection line, and The scene model points are determined as follows: Define scene model points as scene points, or The scene model point is determined by determining the distance between the scene point and the predetermined image, and by determining that the scene model point is located on the first scene projection line at that distance.

223. The video decoder as described in any one of claims 202 to 221 is configured to determine a scene model (300) by: For each control point in one or more control points (132) of an inter-frame prediction block (128) of a predetermined image (20b), scene model points (302) for forming the basis of the scene model are determined by the following operation: The source image position (134) in the corresponding reference image (20d) from which the motion vector points (130) is determined, and the motion vector is encoded and decoded in the data stream of the inter-frame prediction block of the corresponding control point. Using video geometry-related parameters (24), Determine the scene point using the following steps: The linear equations are determined using video geometric parameters (24), corresponding control points (132), and source image location (134). Solve the system of linear equations. The scene model points are determined as follows: Define scene model points as scene points, or The scene model point is determined by determining the distance between the scene point and the predetermined image, and by determining that the scene model point is located on the first scene projection line at that distance.

224. The video decoder of claim 223, wherein the system of linear equations is defined by the equation AQ=0, wherein Q represents a vector containing the coordinates of scene points to be determined, and A represents a matrix defined by video geometric correlation parameters (24), corresponding control points (132), and source image position (134).

225. The video decoder of claim 224, wherein matrix A is defined as follows: A.row(0) = q di (0)*cam di .row(2) - cam di .row(0) A.row(1) = q di (1)*cam di .row(2) - cam di .row(1) A.row(2) = q si (0)*cam si .row(2) - cam si .row(0) A.row(3) = q si (1)*cam si .row(2) - cam si .row(1), The rows of matrix A are represented by A.row(0), A.row(1), A.row(2), and A.row(3). The coordinates of the corresponding control point (132) are given by q. di (0) and q di (1) indicates that The coordinates of the source image position (134) are determined by q. si (0) and q si (1) indicates, and The video geometric correlation parameters (24) are derived from cam di .row(2), cam di .row(0), cam di .row(2), cam di .row(1), cam si .row(2), cam si .row(0), cam si .row(2) and cam si .row(1) represents.

226. The video decoder of any one of claims 223 to 225, configured to perform the task of solving a system of linear equations using a singular value decomposition method.

227. The video decoder as described in any one of claims 202 to 221 is configured to determine a scene model (300) by: For each control point in one or more control points (132) of an inter-frame prediction block (128) of a predetermined image (20b), scene model points (302) for forming the basis of the scene model are determined by the following operation: The source image position (134) in the corresponding reference image (20d) from which the motion vector points (130) is determined, and the motion vector is encoded and decoded in the data stream of the inter-frame prediction block of the corresponding control point. Using video geometry-related parameters (24), Scene points are determined by minimizing the cost function based on the reprojection error. The scene model points are determined as follows: Define the scene model points as scene points. The scene model point is determined by determining the distance between the scene point and the predetermined image, and by determining that the scene model point is located on the first scene projection line at that distance.

228. The video decoder of claim 227, wherein the reprojection error associated with a corresponding control point (132) of an inter-frame prediction block (128) of a predetermined image (20b) includes: The distance between the corresponding control point (132) of the inter-frame prediction block (128) and the projection of the scene point onto the predetermined image (20b), and The distance between the source image position (134) of the corresponding reference image (20d) and the projection of the scene point onto the corresponding reference image (20d).

229. The video decoder of claim 222, wherein the scene model is a mesh or a point cloud.

230. The video decoder of any one of claims 222 and subsequent claims, configured to exclude scene model points whose quality metric determined based on the length metric of the shortest line does not conform to a predetermined criterion.

231. The video decoder as claimed in any one of claims 222 and subsequent claims, configured to Use a control point to translate the inter-frame prediction block, and / or Use more than one control point for the affine inter-frame prediction block.

232. The video decoder of any one of claims 222 and subsequent claims, configured to determine the scene model additionally based on motion vectors of inter-frame prediction blocks of one or more previous images when determining the scene model.

233. The video decoder of any one of claims 222 and subsequent claims, configured to limit inter-frame prediction blocks to determine scene models relative to images at time layers that satisfy predetermined criteria.

234. The video decoder as described in claim 221 or a subsequent claim, configured to, when determining a scene model, Compared to scene model points (302) derived from motion vectors in translation mode, scene model points (302) derived from inter-frame prediction blocks encoded and decoded in affine mode are preferentially used to determine the scene model, and / or Compared to scene model points (302) derived from motion vectors of inter-prediction blocks encoded and decoded individually in the data stream for inter-prediction blocks, the scene model is determined preferentially based on scene model points (302) derived from motion vectors of inter-prediction blocks encoded and decoded individually for inter-prediction blocks in the data stream, and / or The scene model is determined by the preference varying between scene model points (302), such that: the higher the preference, the lower the prediction residual signal encoded into the data stream according to a predetermined metric of the inter-frame prediction block, the scene model points (302) originate from the motion vectors of the inter-frame prediction blocks, and / or The scene model is determined by a preference that varies between scene model points (302), such that: the higher the preference, the closer the images of the inter-frame prediction blocks are in time; the scene model points (302) originate from the motion vectors of the inter-frame prediction blocks; and / or The scene model is determined by the preference that varies between scene model points, such that: the higher the preference, the smaller the quantizer step size of the inter-frame prediction block, and the scene model point (302) originates from the motion vector of the inter-frame prediction block.

235. The video decoder as claimed in claim 221 or a subsequent claim, configured to When using scene model and video geometry parameters to find the corresponding position (508) in the reference image (20c), The weights depend on the part of the scene model. Compared to the portion of motion vectors originating from translation mode, the portion of motion vectors originating from inter-prediction blocks encoded and decoded in affine mode has a higher weight, and / or Compared to the portion of motion vectors originating from inter-prediction blocks encoded and decoded individually in the data stream for each inter-prediction block, the portion of motion vectors originating from these inter-prediction blocks has a higher weight, and / or The higher the weight, the lower the prediction residual signal encoded into the data stream according to the predetermined metric of the inter-frame prediction block, which partly originates from the motion vectors of the inter-frame prediction block, and / or The higher the weight, the closer the images of the inter-frame prediction blocks are in time, partly due to the motion vectors of the inter-frame prediction blocks, and / or The higher the weight, the smaller the quantizer step size of the inter-frame prediction block, which is partly derived from the motion vector of the inter-frame prediction block.

236. The video decoder as described in any of the preceding claims 221 or subsequent claims, configured to When using the scene model (300) and video geometry parameters (24) to find the corresponding position (612) in the reference image (610; 20b), For each pixel (606; 23) in the current portion (600; 18), Determine the scene projection line (604) along which the scene (14) is projected onto the corresponding pixel (606; 23) of the current block. A final use point (608) on the scene projection line is determined based on one or more intersection points (608) between the scene projection line (604) and the facial region (310) of the scene model's mesh, and The final point (608) is projected onto the reference image (610; 20b) to obtain the corresponding position (612; 22) corresponding to the corresponding pixel (606; 23).

237. The video decoder as described in any one of claims 202 to 236, configured to The synthetic reference image is constructed using the following operations: Using scene model and video geometry parameters, locate the corresponding position (508; 612) in the reference image (20c) using the following operations: For each pixel (506; 606) in the predetermined image (602), Determine the scene projection line (604) along which the scene is projected onto the corresponding pixel. The final scene point (608) is determined based on the intersection of the scene projection line and the scene model (300; 580 or 590). The final point (608) is projected onto the reference image (610; 20c) to obtain the corresponding position corresponding to the respective pixel, and The reference image is sampled at the corresponding location.

238. The video decoder as described in any one of claims 202 to 237, configured to By copying the co-located portion of the composite reference image that is in the same position as the current portion; or The portion of the reference image pointed to by motion vectors encoded and decoded for the current portion is synthesized by copying / sampling. Use the synthesized reference image to reconstruct the current part.

239. The video decoder as described in any one of claims 202 to 238, configured to The current part is reconstructed using a synthetic reference image by selecting a synthetic reference image from a list of reference images in the decoded image buffer DPB (509) using a reference index encoded to the data stream for the current part.

240. The video decoder as described in any one of claims 202 to 239 is configured to construct a synthetic reference image by using one or more other reference images to assist in determining the synthetic reference image at pixels of the current image where no corresponding pixel is found in the reference images.

241. The video decoder as described in any one of claims 202 to 240, configured to A synthetic reference image is constructed by using interpolation / extrapolation to help determine the synthetic reference image at pixels in the current image where no corresponding pixel is found in any reference image.

242. The video decoder as described in any one of claims 202 to 241, configured to Redirected temporal motion vector prediction is used for prediction from the synthetic reference image (504) to the reference image (20c).

243. The video decoder as described in any one of claims 202 to 242, configured to To perform temporal motion vector prediction for inter-frame prediction blocks of other images and / or spatial motion vector prediction for inter-frame blocks of the current image, the motion vector of the reference synthetic reference image is modified by adding a motion vector, determined by video geometric correlation parameters and describing the disparity between the portion of the synthetic reference image referenced by the motion vector and the block of the reference image to which the motion vector is applied, to the motion vector of the reference synthetic reference image.

244. A video encoder (500) for encoding video (12) of a scene (14) into a data stream (16) using motion compensation prediction, configured to A synthetic reference image is constructed (502) and (504) through the following operations, which are then synthesized to form a synthetic version of the predetermined image: Using the video geometry-related parameters (24) describing how scene (14) is projected onto the image (20) of video (12), the corresponding position (508) of the pixel (506) in the predetermined image (20a) is found in the reference image (20c), and The reference image (20c) is sampled at the corresponding position (508), and For the current part of the current image, the interior of the part is obtained by sampling the corresponding part of the synthetic reference image (504), and Use part of the internal encoding for the current part (510).

245. The video encoder of claim 244, configured to The scene model is determined based on the motion vectors of the image and the geometric parameters of the video. The corresponding position (508) is found in the reference image (20c) using the scene model and video geometry parameters.

246. The video encoder as described in any one of claims 244 to 245, wherein video geometric correlation parameters describe a scene-to-image projection of a camera (30) onto an image (20) of a video.

247. The video encoder of any one of claims 244 to 246, wherein the video geometric correlation parameters describe the scene-to-image projection of a camera onto the video for each image of the video.

248. The video encoder of any one of claims 244 to 247, wherein video geometric correlation parameters describe a scene-to-image projection of a camera onto the video image by image-by-image and whole-image projection.

249. The video encoder of any one of claims 245 to 248, wherein the video geometric correlation parameters describe the scene to the image projection via one or more of the following: One or more non-inherent camera parameters of a camera; and One or more inherent camera parameters of a camera.

250. The video encoder of any one of claims 249, wherein one or more non-inherent camera parameters define the position of a camera. and / or the orientation of a camera .

251. The video encoder as described in any of claims 249 or 250, wherein one or more inherent camera parameters define at least one of the focal length and FOV angle of a camera.

252. The video encoder as described in any one of claims 244 to 251, wherein the video geometric correlation parameters describe a homomorphic mapping (42) between corresponding positions (401, 402) in a pair of images (201, 202) of the video, or a homomorphic mapping between corresponding motion vectors associated with the pair of images of the video.

253. The video encoder of claim 252, wherein the video geometric correlation parameters describe the homomorphic mapping via a vector (462) or tensor at a predetermined control point (442) of the image of the video.

254. The video encoder of claim 253, wherein the predetermined control point is a corner point of the video image.

255. The video encoder as described in any one of claims 252 to 254, wherein for each image, the video geometric correlation parameters contain only one vector or tensor for each of the four corner points of the image of the video.

256. The video encoder of any one of claims 244 to 255, wherein video geometric correlation parameters describe a homomorphic mapping between corresponding positions in a pair of images of the video, and the video encoder is configured to derive a scene-to-image projection onto a predetermined image based on the homomorphic mapping between corresponding positions in a pair of images including the predetermined image.

257. The video encoder as described in any one of claims 244 to 256 is configured to encode video geometrically related parameters into a data stream.

258. The video encoder as described in any one of claims 244 to 257 is configured to determine video geometric correlation parameters based on the encoded portion of the video.

259. The video encoder of any one of claims 244 to 258, wherein the video geometric correlation parameters describe the homomorphic mapping between corresponding positions in the image pairs of the video, and the video encoder is configured to derive a scene-to-image projection onto the predetermined image based on a sequence of homomorphic mappings between corresponding positions in one or more image pairs including the predetermined image and a base image, and based on non-inherent parameters and inherent parameters of the base image.

260. The video encoder as described in any one of claims 244 to 259, configured to encode syntax elements into a data stream, and If the syntax element has a first state, then A synthetic reference image is constructed (502) and (504) through the following operations, which are then synthesized to form a synthetic version of the predetermined image: Using the video geometry-related parameters (24) describing how scene (14) is projected onto the image (20) of video (12), the corresponding position (508) of the pixel (506) in the predetermined image (20a) is found in the reference image (20c), and The reference image (20c) is sampled at the corresponding position (508), and For the current part of the current image, the interior of the part is obtained by sampling the corresponding part of the synthetic reference image (504), and Use part of the internal encoding for the current part (510).

261. The video encoder of claim 260, configured to, if the syntax element has a second state, then Encoding the current portion independently of video geometry-related parameters, or The current portion is encoded by copying the already encoded video portion at regular pixel intervals.

262. The video encoder as described in any one of claims 244 to 261, configured to This yields a list of MVP candidates for the current section, where each MVP candidate corresponds to a specific motion vectorless inter-frame encoding / decoding mode. Select the MVP from the list of MVP candidates, and Encode the current section using the selected MVP by performing the following operations: If the selected MVP corresponds to a specific motion vectorless inter-frame encoding / decoding mode, then A synthetic reference image is constructed (502) and (504) through the following operations, which are then synthesized to form a synthetic version of the predetermined image: Using the video geometry-related parameters (24) describing how scene (14) is projected onto the image (20) of video (12), the corresponding position (508) of the pixel (506) in the predetermined image (20a) is found in the reference image (20c), and The reference image (20c) is sampled at the corresponding position (508), and For the current part of the current image, the interior of the part is obtained by sampling the corresponding part of the synthetic reference image (504), and Using part of the internal encoding of the current part (510), If the selected MVP corresponds to an MVP candidate other than a specific motion vectorless inter-frame encoding / decoding mode, then Use the selected MVP to encode the current part using motion vector compensation prediction.

263. The video encoder as described in any one of claims 244 to 262, configured to The scene model (300) is determined based on the motion vectors (126) of the image and the geometric parameters (24) of the video. The corresponding pixel (23) is found in the reference image using the scene model (300) and video geometry parameters (24).

264. The video encoder as described in any one of claims 244 to 263 is configured to determine a scene model (300) by: For each control point in one or more control points (132) of an inter-frame prediction block (128) of a predetermined image (20b), scene model points (302) for forming the basis of the scene model are determined by the following operation: The source image position (134) in the corresponding reference image (20d) from which the motion vector points (130) is determined, and the motion vector is encoded and decoded in the data stream of the inter-frame prediction block of the corresponding control point. Using video geometry-related parameters (24), Determine the first scene projection line along which the scene is projected onto the corresponding control point (132). Determine the second scene projection line along which the scene is projected onto the source image position (134), and Determine the scene point on the shortest line connecting any point on the first scene projection line and any point on the second scene projection line, and The scene model points are determined as follows: Define scene model points as scene points, or The scene model point is determined by determining the distance between the scene point and the predetermined image, and by determining that the scene model point is located on the first scene projection line at that distance.

265. The video encoder as described in any one of claims 244 to 263 is configured to determine a scene model (300) by: For each control point in one or more control points (132) of an inter-frame prediction block (128) of a predetermined image (20b), scene model points (302) for forming the basis of the scene model are determined by the following operation: The source image position (134) in the corresponding reference image (20d) from which the motion vector points (130) is determined, and the motion vector is encoded and decoded in the data stream of the inter-frame prediction block of the corresponding control point. Using video geometry-related parameters (24), Determine the scene point using the following steps: The linear equations are determined using video geometric parameters (24), corresponding control points (132), and source image location (134). Solve the system of linear equations. The scene model points are determined as follows: Define scene model points as scene points, or The scene model point is determined by determining the distance between the scene point and the predetermined image, and by determining that the scene model point is located on the first scene projection line at that distance.

266. The video encoder of claim 265, wherein the system of linear equations is defined by the equation AQ=0, wherein Q represents a vector containing the coordinates of scene points to be determined, and A represents a matrix defined by video geometric correlation parameters (24), corresponding control points (132), and source image position (134).

267. The video encoder of claim 266, wherein matrix A is defined as follows: A.row(0) = q di (0)*cam di .row(2) - cam di .row(0) A.row(1) = q di (1)*cam di .row(2) - cam di .row(1) A.row(2) = q si (0)*cam si .row(2) - cam si .row(0) A.row(3) = q si (1)*cam si .row(2) - cam si .row(1), The rows of matrix A are represented by A.row(0), A.row(1), A.row(2), and A.row(3). The coordinates of the corresponding control point (132) are given by q. di (0) and q di (1) indicates that The coordinates of the source image position (134) are determined by q. si (0) and q si (1) indicates, and The video geometric correlation parameters (24) are derived from cam di .row(2), cam di .row(0), cam di .row(2), cam di .row(1), cam si .row(2), cam si .row(0), cam si .row(2) and cam si .row(1) represents.

268. The video encoder of any one of claims 265 to 267, configured to perform the solution of a system of linear equations using a singular value decomposition method.

269. The video encoder as described in any one of claims 244 to 263 is configured to determine a scene model (300) by: For each control point in one or more control points (132) of an inter-frame prediction block (128) of a predetermined image (20b), scene model points (302) for forming the basis of the scene model are determined by the following operation: The source image position (134) in the corresponding reference image (20d) from which the motion vector points (130) is determined, and the motion vector is encoded and decoded in the data stream of the inter-frame prediction block of the corresponding control point. Using video geometry-related parameters (24), Scene points are determined by minimizing the cost function based on the reprojection error. The scene model points are determined as follows: Define scene model points as scene points, or The scene model point is determined by determining the distance between the scene point and the predetermined image, and by determining that the scene model point is located on the first scene projection line at that distance.

270. The video encoder of claim 269, wherein the reprojection error associated with a corresponding control point (132) of an inter-frame prediction block (128) of a predetermined image (20b) comprises: The distance between the corresponding control point (132) of the inter-frame prediction block (128) and the projection of the scene point onto the predetermined image (20b), and The distance between the source image position (134) of the corresponding reference image (20d) and the projection of the scene point onto the corresponding reference image (20d).

271. The video encoder of claim 264, wherein the scene model is a mesh or a point cloud.

272. The video encoder of any one of claims 264 and subsequent claims, configured to exclude scene model points whose quality metric determined based on the length metric of the shortest line does not conform to a predetermined criterion.

273. The video encoder of any one of claims 264 and subsequent claims, configured to Use a control point to translate the inter-frame prediction block, and / or Use more than one control point for the affine inter-frame prediction block.

274. The video encoder of claim 264 and any subsequent claim, configured to determine the scene model additionally based on the motion vectors of inter-frame prediction blocks of one or more previous images when determining the scene model.

275. The video encoder of any one of claims 264 and subsequent claims, configured to limit inter-frame prediction blocks to determine scene models relative to images at time layers that satisfy predetermined criteria.

276. The video encoder of claim 263 or a subsequent claim, configured to, when determining a scene model, utilize the following portions dependent on the determination of the scene model: Compared to scene model points (302) derived from motion vectors in translation mode, scene model points (302) based on motion vectors derived from inter-frame prediction blocks encoded and decoded in affine mode are preferred, and / or Compared to scene model points (302) derived from motion vectors of inter-prediction blocks encoded and decoded in skip, direct, or merge modes, scene model points (302) preferentially based on motion vectors of these inter-prediction blocks whose motion vectors are individually encoded and decoded for inter-prediction blocks in the data stream, and / or With a preference varying among scene model points (302), such that: the higher the preference, the lower the prediction residual signal encoded into the data stream according to a predetermined metric of the inter-frame prediction block, the scene model points (302) originate from the motion vectors of the inter-frame prediction blocks, and / or With a preference varying between scene model points (302), such that: the higher the preference, the closer the images of the inter-frame prediction blocks are in time, the scene model points (302) originate from the motion vectors of the inter-frame prediction blocks, and / or The preference varies between scene model points such that: the higher the preference, the smaller the quantizer step size of the inter-frame prediction block, and the scene model point (302) originates from the motion vector of the inter-frame prediction block.

277. The video encoder as claimed in claim 263 or a subsequent claim, configured to When using scene model and video geometry parameters to find the corresponding position (508) in the reference image (20c), The weights depend on the part of the scene model. Compared to the portion of motion vectors originating from translation mode, the portion of motion vectors originating from inter-prediction blocks encoded and decoded in affine mode has a higher weight, and / or Compared to the portion of motion vectors originating from inter-prediction blocks encoded and decoded individually in the data stream for each inter-prediction block, the portion of motion vectors originating from these inter-prediction blocks has a higher weight, and / or The higher the weight, the lower the prediction residual signal encoded into the data stream according to the predetermined metric of the inter-frame prediction block, which partly originates from the motion vectors of the inter-frame prediction block, and / or The higher the weight, the closer the images of the inter-frame prediction blocks are in time, partly due to the motion vectors of the inter-frame prediction blocks, and / or The higher the weight, the smaller the quantizer step size of the inter-frame prediction block, which is partly derived from the motion vector of the inter-frame prediction block.

278. The video encoder as described in any of the preceding claims 263 or subsequent claims, configured to When using the scene model (300) and video geometry parameters (24) to find the corresponding position (612) in the reference image (610; 20b), For each pixel (606; 23) in the current portion (600; 18), Determine the scene projection line (604) along which the scene (14) is projected onto the corresponding pixel (606; 23) of the current block. A final use point (608) on the scene projection line is determined based on one or more intersection points (608) between the scene projection line (604) and the facial region (310) of the scene model's mesh, and The final point (608) is projected onto the reference image (610; 20b) to obtain the corresponding position (612; 22) corresponding to the corresponding pixel (606; 23).

279. The video encoder as described in any one of claims 244 to 278, configured to The synthetic reference image is constructed using the following operations: Using scene model and video geometry parameters, locate the corresponding position (508; 612) in the reference image (20c) using the following operations: For each pixel (506; 606) in the predetermined image (602), Determine the scene projection line (604) along which the scene is projected onto the corresponding pixel. The final usage scene point (608) is determined based on the intersection point of the scene projection line and the scene model (300), and The final point (608) is projected onto the reference image (610; 20c) to obtain the corresponding position corresponding to the respective pixel, and The reference image is sampled at the corresponding location.

280. The video encoder as described in any one of claims 244 to 279, configured to The current portion is encoded using a synthetic reference image through the following operations: Copy the co-located portion of the composite reference image that is in the same position as the current portion; or The part of the copy / sample composite reference image pointed to by the motion vector encoded and decoded for the current part.

281. The video encoder as described in any one of claims 244 to 280, configured to The current portion is encoded using a synthetic reference image through the following operations: A composite reference image is selected from the list of reference images in the decoded image buffer DPB (509) using the reference index encoded to the data stream for the current part.

282. The video encoder as described in any one of claims 244 to 281, configured to A synthetic reference image is constructed by using one or more other reference images to assist in determining the synthetic reference image at pixels in the current image where no corresponding pixel is found in the reference image.

283. The video encoder as described in any one of claims 244 to 282, configured to A synthetic reference image is constructed by using interpolation / extrapolation to help determine the synthetic reference image at pixels in the current image where no corresponding pixel is found in any reference image.

284. The video encoder as described in any one of claims 244 to 283, configured to Redirected temporal motion vector prediction is used for prediction from the synthetic reference image (504) to the reference image (20c).

285. The video encoder as described in any one of claims 244 to 284, configured to To perform temporal motion vector prediction for inter-frame prediction blocks of other images and / or spatial motion vector prediction for inter-frame blocks of the current image, the motion vector of the reference synthetic reference image is modified by adding a motion vector, determined by video geometric correlation parameters and describing the disparity between the portion of the synthetic reference image referenced by the motion vector and the block of the reference image to which the motion vector is applied, to the motion vector of the reference synthetic reference image.

286. A method for decoding video (12) of a scene (14) from a data stream (16) using motion compensation prediction, comprising: For the current block (18) of the current image (20a), the prediction block interior is obtained through the following operations: Using the video geometry-related parameters (24) on the image (20) describing how scene (14) is projected (26) onto video (12), the corresponding position (22) of the pixel (23) in the current block (18) is found in the reference image (20b), and The reference image (20b) is sampled at the corresponding position (22), and Reconstruct the current block using the predicted block (18).

287. A method for decoding video (12) of a scene (14) from a data stream (16) using motion compensation prediction, comprising: For the current block (124), a predetermined motion vector predictor (126) is derived using video geometric correlation parameters (24) describing how the scene (14) is projected (26) onto the image (20) of the video (12), and Reconstruct the current block (124) using the predefined motion vector predictor (126).

288. A method for decoding video (12) of a scene (14) from a data stream (16) using motion compensation prediction, comprising: A synthetic reference image is constructed (502) and (504) through the following operations, which are then synthesized to form a synthetic version of the predetermined image: Using the video geometry-related parameters (24) describing how scene (14) is projected onto the image (20) of video (12), the corresponding position (508) of the pixel (506) in the predetermined image (20a) is found in the reference image (20c), and The reference image (20c) is sampled at the corresponding position (508), and For the current part of the current image, the interior of the part is obtained by sampling the corresponding part of the synthetic reference image (504), and Rebuild the current part using the internal reconstruction (510).

289. A method for encoding video (12) of a scene (14) into a data stream (16) using motion compensation prediction, comprising: For the current block (18) of the current image (20a), the prediction block interior is obtained through the following operations: Using the video geometry-related parameters (24) on the image (20) describing how scene (14) is projected (26) onto video (12), the corresponding position (22) of the pixel (23) in the current block (18) is found in the reference image (20b), and The reference image (20b) is sampled at the corresponding position (22), and The current block is encoded using the internal code of the predicted block (818).

290. A method for encoding video (12) of a scene (14) into a data stream (16) using motion compensation prediction, comprising: For the current block (124), a predetermined motion vector predictor (126) is derived using video geometric correlation parameters (24) describing how the scene (14) is projected (26) onto the image (20) of the video (12), and The current block (124) is encoded using a predefined motion vector predictor (126).

291. A method for encoding video (12) of a scene (14) into a data stream (16) using motion compensation prediction, comprising: A synthetic reference image is constructed (502) and (504) through the following operations, which are then synthesized to form a synthetic version of the predetermined image: Using the video geometry-related parameters (24) describing how scene (14) is projected onto the image (20) of video (12), the corresponding position (508) of the pixel (506) in the predetermined image (20a) is found in the reference image (20c), and The reference image (20c) is sampled at the corresponding position (508), and For the current portion of the current image (510), the interior of the portion is obtained by sampling the corresponding portion of the synthetic reference image (504), and Use part of the internal encoding for the current part (510).

292. A computer program having program code, which, when the computer program is run on a computer, performs one of the methods as claimed in claims 286 to 291.

293. A data stream generated by an apparatus as described in any one of claims 37 to 72, 136 to 200 and 244 to 285 or by a method as described in any one of claims 289 to 291.

Images