Video encoding method, computer system and medium

Abstract

A method, computer program, and computer system for encoding video data are provided. The video data includes (1) a first frame including a first current viewpoint and a first reference viewpoint, and (2) a second frame including a second current viewpoint and a second reference viewpoint. For a current block in the first current viewpoint, a co-occurring block in the first reference viewpoint is identified. A predicted offset vector is calculated based on the motion vector between the co-occurring block and a reference image of the co-occurring block in the second reference viewpoint. The video data is encoded / decoded based on the predicted offset vector.

Description

[0001] Incorporation

[0002] This application claims the priority benefit of U.S. Patent Application No. 17 / 026,922, filed September 21, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to the field of data processing, and more specifically to video encoding and decoding. Background Technology

[0004] Block-based motion compensation is commonly used in modern video coding systems such as Moving Picture Experts Group-2 (MPEG-2), Moving Picture Experts Group-4 (MPEG-4), Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC). The predictor for the current coded block is derived from previously coded images, referred to as reference images. Motion vectors are used to point to the position of the current block in the reference image relative to its position in the current image. If more than one reference image is available for generating the predictor, the motion information may additionally include reference image indices as selection indices from a set of candidate reference images. In some prediction schemes, such as bidirectional prediction methods, more than one prediction block can be used to generate the final predictor for the current block.

[0005] For each inter-frame prediction CU (Coding Unit or Coding Block), motion parameters may be required, consisting of motion vectors, reference picture indices, and reference picture list usage indices, along with additional information for generating new coded features for the VVC based on the inter-frame prediction samples. Motion parameters can be signaled explicitly or implicitly. When a CU is encoded using skip or merge mode, the CU is associated with a set of inferred motion information and has no significant residual coefficients, no encoded motion vector increments, or reference picture indices. A merge mode is specified, thereby inferring and obtaining the motion parameters (including spatial and temporal candidates) of the current CU from neighboring CUs. Merge mode can be applied to any inter-frame prediction CU, not just skip mode. An alternative to merge mode is explicit transmission of motion parameters, where for each CU, the motion vector (or predicted motion vector difference), the corresponding reference picture index for each reference picture list, the reference picture list usage flag, and other necessary information are explicitly signaled. Summary of the Invention

[0006] The embodiments relate to methods, systems, and computer-readable media for encoding video data. According to one aspect, a method for encoding video data is provided. The method may include receiving video data comprising (1) a first frame including a first current viewpoint and a first reference viewpoint and (2) a second frame including a second current viewpoint and a second reference viewpoint. For a current block in the first current viewpoint, a co-occurring block in the first reference viewpoint is identified. A predicted offset vector is calculated based on a motion vector between the co-occurring block and a reference block of the co-occurring block in the second reference viewpoint. The video data is encoded / decoded based on the predicted offset vector.

[0007] According to another aspect, a computer system for encoding video data is provided. The computer system may include one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage devices, and program instructions stored on at least one of the one or more storage devices, which are executed by at least one of the one or more processors via at least one of the one or more memories, thereby enabling the computer system to perform a method. The method may include receiving video data comprising (1) a first frame including a first current viewpoint and a first reference viewpoint and (2) a second frame including a second current viewpoint and a second reference viewpoint. For a current block in the first current viewpoint, a co-occurring block in the first reference viewpoint is identified. A predicted offset vector is calculated based on a motion vector between the co-occurring block and a reference block of the co-occurring block in the second reference viewpoint. The video data is encoded / decoded based on the predicted offset vector.

[0008] According to another aspect, a computer-readable medium for encoding video data is provided. The computer-readable medium may include one or more computer-readable storage devices and program instructions stored on at least one of the one or more tangible storage devices, the program instructions being executable by a processor. The program instructions, executable by the processor, are used to perform a method that may accordingly include receiving video data, the video data including (1) a first frame including a first current viewpoint and a first reference viewpoint and (2) a second frame including a second current viewpoint and a second reference viewpoint. For a current block in the first current viewpoint, a co-occurring block in the first reference viewpoint is identified. A predicted offset vector is calculated based on a motion vector between the co-occurring block and a reference block of the co-occurring block in the second reference viewpoint. The video data is encoded / decoded based on the predicted offset vector. Attached Figure Description

[0009] These objects, features, and advantages, as well as others, will become apparent from the following detailed description of illustrative embodiments, which are read in conjunction with the accompanying drawings. Various features in the drawings are not drawn to scale because they are intended to facilitate understanding by those skilled in the art in relation to specific embodiments. In the drawings:

[0010] Figure 1 A networked computer environment according to at least one embodiment is shown.

[0011] Figure 2 It is an illustration of frames of stitched video data (each frame having multiple viewpoints) according to at least one embodiment.

[0012] Figure 3 It is an operation flowchart of the steps performed by a program that encodes video data according to at least one embodiment.

[0013] Figure 4 According to at least one embodiment Figure 1 A block diagram depicting the internal and external components of a computer and server.

[0014] Figure 5 It includes, according to at least one embodiment Figure 1 A block diagram illustrating a cloud computing environment for a computer system.

[0015] Figure 6 According to at least one embodiment Figure 5 A block diagram illustrating the functional layers of an illustrative cloud computing environment. Detailed Implementation

[0016] This document discloses detailed embodiments of the claimed structures and methods. However, it is to be understood that the disclosed embodiments are merely illustrative of the claimed structures and methods, which can be implemented in various forms. These structures and methods can be implemented in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art. In the embodiments, details of well-known features and techniques may be omitted to avoid unnecessarily obscuring the presented embodiments.

[0017] The embodiments generally relate to the field of data processing, and more specifically to video encoding and decoding. The exemplary embodiments described below provide a system, method, and computer program for encoding video data based on inter-frame motion vector prediction. Therefore, some embodiments have the capability to improve computational efficiency through improved, inter-frame prediction-based video data encoding.

[0018] As previously mentioned, for each inter-frame prediction CU (coding unit or coding block), motion parameters consisting of motion vectors, reference picture indices, and reference picture list usage indices, along with additional information, may be required for new coded features of the VVC generated for the inter-frame prediction samples. Motion parameters can be signaled explicitly or implicitly. When a CU is encoded with a skip mode or a merge mode, the CU is associated with a set of inferred motion information and has no significant residual coefficients, no encoded motion vector increments, or reference picture indices. A merge mode is specified, thereby inferring and obtaining the motion parameters (including spatial and temporal candidates) of the current CU from neighboring CUs. A merge mode can be applied to any inter-frame prediction CU, not just skip mode. An alternative to the merge mode is explicit transmission of motion parameters, where, for each CU, the motion vector (or predicted motion vector difference), the corresponding reference picture index for each reference picture list, the reference picture list usage flag, and other necessary information are explicitly signaled.

[0019] In traditional methods, multiple viewpoints within the same scene are stitched together into a single signal video frame. Therefore, the stitched video can be encoded using conventional 2D video encoding and decoding techniques (such as AVC, HEVC, etc.). However, encoding very large images in this approach requires a significant increase in computational complexity. Typically, for inter-frame motion estimation, it is generally assumed that objects move smoothly from a reference image to the current image with continuous motion. Therefore, the best matching block in the reference image will appear around the projection center, which can have coordinates similar to the current block in the current image. Many motion search algorithms in the literature are designed to focus the search effort near the search center and reduce search effort when moving away from it. Therefore, leveraging the similarity between stitched viewpoints within the same image and across images can be advantageous for improving encoding efficiency and speed. In this way, inter-frame motion estimation can be used to efficiently find the best matching block in the context of multi-view video coding.

[0020] This document describes aspects with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer-readable media according to various embodiments. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0021] Now for reference Figure 1 A functional block diagram of a networked computer environment illustrates a video coding system 100 (hereinafter referred to as the "System") for encoding video data based on inter-frame motion vector prediction of video data. It should be understood that... Figure 1 This illustration is merely one example of an implementation and does not imply any limitation regarding the environment in which different embodiments may be implemented. Many modifications can be made to the depicted environment based on design and implementation requirements.

[0022] System 100 may include computer 102 and server computer 114. Computer 102 may communicate with server computer 114 via communication network 110 (hereinafter referred to as the "network"). Computer 102 may include processor 104 and software program 108, which is stored on data storage device 106 and is capable of interfacing with a user and communicating with server computer 114. Reference will be made below. Figure 4 As discussed, computer 102 may include internal component 800A and external component 900A, and server computer 114 may include internal component 800B and external component 900B. Computer 102 may be, for example, a mobile device, telephone, personal digital assistant, netbook, laptop computer, tablet computer, desktop computer, or any type of computing device capable of running programs, accessing networks, and accessing databases.

[0023] Server computer 114 can also operate within a cloud computing service model, such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS), as described below. Figure 5 and 6 The server computer 114 can also be located in a cloud computing deployment model, such as a private cloud, community cloud, public cloud, or hybrid cloud.

[0024] The server computer 114, which can be used to encode video data, is capable of running a video encoding program 116 (hereinafter referred to as the "program") that can interact with the database 112. See below for further details. Figure 3 The video encoding program method will be described in more detail below. In one embodiment, computer 102 may operate as an input device including a user interface, while program 116 may run primarily on server computer 114. In an alternative embodiment, program 116 may run primarily on one or more computers 102, while server computer 114 may be used to process and store data used by program 116. It should be noted that program 116 may be a standalone program, or program 116 may be integrated into a larger video encoding program.

[0025] However, it should be noted that in some cases, the processing of program 116 can be shared between computer 102 and server computer 114 in any proportion. In another embodiment, program 116 can run on more than one computer, server computer, or some combination of computers and server computers. For example, program 116 can run on multiple computers 102 communicating with a single server computer 114 via network 110. In another embodiment, for example, program 116 can run on multiple server computers 114 communicating with multiple client computers via network 110. Alternatively, the program can run on a network server communicating with a server and multiple client computers via a network.

[0026] Network 110 may include wired connections, wireless connections, fiber optic connections, or combinations thereof. Typically, network 110 may be any combination of connections and protocols supporting communication between computer 102 and server computer 114. Network 110 may include various types of networks, such as local area networks (LANs), wide area networks (WANs) such as the Internet, telecommunications networks such as the Public Switched Telephone Network (PSTN), wireless networks, public switched networks, satellite networks, cellular networks (e.g., 5G, LTE, 3G, CDMA networks, etc.), public land mobile networks (PLMNs), metropolitan area networks (MANs), private networks, ad hoc networks, intranets, fiber optic networks, etc.), and / or combinations of these or other types of networks.

[0027] supply Figure 1 The number and arrangement of devices and networks shown are for illustrative purposes. In practice, with... Figure 1 Compared to the devices and / or networks shown, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or devices and / or networks with different arrangements. Furthermore, it can be implemented within a single device. Figure 1 The two or more devices shown, or Figure 1 The single device shown can be implemented as multiple distributed devices. Alternatively, a group of devices in system 100 (e.g., one or more devices) can perform one or more functions described as being performed by another group of devices in system 100.

[0028] Now for reference Figure 2 The diagram depicts frames of stitched video data 200A and 200B. The stitched video data 200A and 200B may include multiple viewpoints (viewpoint 0 to viewpoint 5) that are spatially stitched together to form a two-dimensional image. This is intended as an example, not a limitation. Figure 2The image depicts six viewpoints. However, it's understandable that any number of viewpoints can be stitched together. Furthermore, it can be understood that there are several ways to stitch viewpoints together spatially. For example, with six viewpoints, each row can stitch together one, two, or three viewpoints, resulting in 1x6, 2x3, and 3x2 stitches respectively. For multi-view video, viewpoints can be stitched together using horizontal or vertical raster scan sequences, zigzag scan sequences, or other scan sequences (such as serpentine scan sequences).

[0029] Stitching can be designed to result in a large image with a desired size. For example, the large image can be close to a square, a 4:3 aspect ratio rectangle, a 16:9 aspect ratio rectangle, etc. The frame resolution for each viewpoint can be W×H, where W and H can be the width and height of the viewpoint's frame resolution, respectively. When the large image is close to a square shape, we can search for factorization pairs (L, M) of N, where N = L*M is the number of viewpoints, such that |W*LH*M| can be minimized. The minimizer can be represented as (L0, M0), and each row can stitch L0 viewpoints. This might result in an L0*M0 stitch, with a large image size of W*L0 width and H*M0 height.

[0030] For P-slices or B-slices in a very large image, motion vectors from previously encoded viewpoints within the same image will be highly correlated with the motion vectors in the current viewpoint being encoded. Therefore, motion vectors from previously encoded viewpoints within the same image are well-suited as the starting point for a Motion Vector Predictor (MVP) or motion estimation of the current viewpoint. Furthermore, the MVP can be computed using the viewpoint transformation between two viewpoints. When the camera's intrinsic parameters are identical across all viewpoints, the viewpoint transformation from the reference viewpoint to the current viewpoint can be [R|t]. The viewpoint transformation from the current viewpoint to the reference viewpoint can be [R'|-R't], where R can be a rotation matrix, t can be a translation value, and R' can be the transpose of R. The motion vector in the reference viewpoint can be written as a translation step [I|m], where I can be the identity matrix and m can be the translation step. Therefore, in the reference viewpoint:

[0031] X cur =[I|m]X ref (Equation 1)

[0032] Among them, X cur and X ref These can be blocks within the reference viewpoints of the current image and the reference image, respectively. Within the current viewpoint:

[0033] Y cur =[R|t]X cur=[R|t][I|m]X ref =[R|t][I|m][R'|-R't]Y ref =[I|Rm]Y ref (Equation 2)

[0034] Among them, Y cur and Y ref These can be blocks within the current viewpoint of the current image and the reference image, respectively. cur It can also be Y ref The translation, and the translation step size (motion vector) can be Rm (i.e., the rotation matrix R multiplied by the translation step size m).

[0035] Generally, the camera intrinsic parameters may differ from different viewpoints. The camera intrinsic parameter matrices for the reference viewpoint and the current viewpoint can be K and K, respectively. ref and K cur Therefore, we can conclude that:

[0036] (K cur ) -1 Y cur =[R|t](K ref ) -1 X cur =[R|t][I|m](K ref ) -1 X ref =[R|t][I|m][R'|-R't](K cur ) -1 Y ref =[I|Rm](K cur ) -1 Y ref (Equation 3)

[0037] as well as,

[0038] Y cur =[I|Rm]Y ref (Equation 4)

[0039] This means that the translation step size (motion vector) is still Rm, which is the rotation matrix R multiplied by the translation step size m.

[0040] The translation step size of the current viewpoint can be calculated based on the rotation matrix R and the translation step size m of the reference viewpoint. Furthermore, the translation step size of the current viewpoint can be independent of the translation value t of the cross-viewpoint perspective transformation. The value Rm of the translation step size can be used as a motion vector predictor for the current viewpoint.

[0041] You can P the current image cur Current viewpoint V cur The current block B0 in the algorithm derives motion vector predictor candidates. Given from the current viewpoint V...cur To reference viewpoint V ref The viewpoint transformation, assuming its rotation matrix is:

[0042]

[0043] Under the change of perspective, the current image P cur Current viewpoint V cur Block B0 in the image can be mapped to the current image P. cur Reference viewpoint V ref Block B1 in the image. If the motion vector of block B1 is (Mx, My), then the reference block B2 of block B1 resides in the reference image P. ref same viewpoint V ref From this, we can conclude that:

[0044] X2 = X3 + Mx (Equation 6)

[0045] Y2 = Y3 + My (Equation 7)

[0046] Here, (X2, Y2) are the coordinates in reference block B2, and (X3, Y3) can be the coordinates in co-located block B3 of B1. Both B3 and B1 can be referenced from viewpoint V. ref In the middle, and B3 can reside in the reference image P. ref In the middle. The current image can be edited. cur Current viewpoint V cur The motion vector predictor candidates (Mxc, Myc) for block B0 are calculated as follows:

[0047]

[0048] Mxc = r 1，1 *Mx+r 1，2 *My (Equation 9)

[0049] Myc = r 2，1 *Mx+r 2，2 *My (Equation 10)

[0050] If (X4, Y4) are coordinates in block B5, which is a co-location of B0, where B5 can reside in reference image P. ref In the current viewpoint Vcur, (X5, Y5) are the coordinates of block B4 (B4 is the reference block of B0), from which we can deduce:

[0051] X5 = X4 + Mxc = X4 + r 1，1 *Mx+r 1，2 *My (Equation 11)

[0052] Y5 = Y4 + Myc = Y4 + r 2，1 *Mx+r 2，2 *My (Equation 12)

[0053] When performing a motion search on block B0 in the current viewpoint, the derived motion vector predictor candidates (Mxc, Myc) can be used as a starting point.

[0054] Additionally, reference blocks can be derived from other viewpoints. When the current block of viewpoint 2 can be encoded, a good-matching reference for that block can also be derived from viewpoints other than viewpoint 2 in the reference image. For example, blocks in viewpoints 0 and 1 that may have been encoded before viewpoint 2.

[0055] The disparity vector DV(Dx, Dy) can be used to find the co-location of the current block in the same image from a reference viewpoint, assuming that samples in the block share the same disparity.

[0056] A positional offset can be established between the current viewpoint and its reference viewpoint. For example, the offset could be twice the viewpoint width in the x-direction and zero in the y-direction. A disparity vector can be added to such a viewpoint offset to find a co-occurring block in the reference viewpoint of the current block. The disparity vector can be derived using virtually any method. Since a co-occurring block indicated by the disparity vector in the reference viewpoint (viewpoint 0) may already be encoded, its motion vector (if present) can point to the reference block in viewpoint 0 of the reference image. The reference block can also be used as a reference block for the current block (in viewpoint 2) or as a starting point in motion estimation.

[0057] If the current block position can be (X) curr Y curr Furthermore, the derived reference block can be located in the reference image at a position that is (X...). ref Y ref If ), then the suggested MV predictor could be: (Mxc, Myc) = (X ref Y ref )-(X curr Y curr ).

[0058] Motion vector predictors can be used to improve motion vector differential coding. Similar to any of the methods described above, motion vector predictor candidates (Mxc, Myc) can be derived for block B0 in the current viewpoint using motion information from existing block B1. (Mxc, Myc) can be either List_0 or List_1MV predictors.

[0059] After the encoder finds the true motion vector MV for encoding, the difference MVD = MV - motion vector predictor can be encoded. The derived (Mxc, Myc) can be used as one of the possible motion vector prediction candidates.

[0060] On the decoder side, a motion vector predictor from above can be selected as the predictor, and then the MVD can be decoded and added to the MVD to form the motion vector of the current block.

[0061] In one or more embodiments, the derived motion vector predictor candidate (Mxc, Myc) can be added to the merge candidate list for inter-frame merging or skipping modes. In one example, the motion vector predictor candidate can be placed after the spatially adjacent candidate.

[0062] In one or more embodiments, multiple such derived motion vector predictor candidates (Mxc, Myc) may be available, for example, multiple derived motion vector predictor candidates from different viewpoints. These motion vector predictor candidates can be added together to a merged candidate list for inter-frame merging or skipping modes. In one example, motion vector predictor candidates can be placed in a list where candidates from closer viewpoints (relative to the current viewpoint) are placed before candidates from farther viewpoints.

[0063] In one or more embodiments, the derived motion vector predictor candidate (Mxc, Myc) can be added to the motion vector predictor candidate list for inter-frame motion vector coding modes. In one example, the motion vector predictor candidate (Mxc, Myc) can be placed after the spatially adjacent candidate.

[0064] In one or more embodiments, the encoder uses a derived reference location indicated by (Mxc, Myc) as a starting point or search center to perform motion estimation and can continue searching around that location, where more computational resources (e.g., more search points) are used for locations near the center and fewer computational resources (e.g., fewer search points) are used for more distant locations.

[0065] In one or more embodiments, the encoder uses a derived reference position indicated by (Mxc, Myc) as one of the possible best matching positions to perform motion estimation. If a matching result from that position is returned at a sufficiently low cost, the motion estimation process can be terminated early without further checking within the search window or checking more candidates.

[0066] In one or more embodiments, the encoder performs tile-based encoding. Each viewpoint can be encoded as a tile, and each viewpoint can be decoded as a tile. The method may include receiving video data including (1) a first frame including a first current tile and a first reference tile; and (2) a second frame including a second current tile and a second reference tile. For a current tile in the first current tile, a co-occurring tile in the first reference tile is identified. A predicted offset vector is calculated based on the motion vector between the co-occurring tile and a reference tile in the second reference tile. The video data is encoded / decoded based on the predicted offset vector.

[0067] In one or more embodiments, the encoder performs candidate motion vector index signal notification, such as signal notification in HEVC.

[0068] In one or more embodiments, the encoder performs an ordered search among motion vector candidates, wherein when more than one motion vector candidate is added, the candidate vector list is sorted based on the proximity between a first current viewpoint and a viewpoint corresponding to a candidate vector or based on quality sorting in a reference viewpoint.

[0069] In one or more embodiments, the encoder performs dual prediction using both inter-frame prediction and intra-frame prediction. Inter-frame prediction may be a reference image from a list of previously encoded frames. Intra-frame prediction may be a reference viewpoint from the same frame from a list of previously encoded viewpoints.

[0070] Now for reference Figure 3 This describes an operational flowchart illustrating the steps of a method 300 for encoding video data. In some embodiments, this can be performed by a computer 102 ( Figure 1 ) and server computer 114 ( Figure 1 )implement Figure 3 One or more processing blocks. In some implementations, this can be performed by another device or a group of devices that are separate from or include computer 102 and server computer 114. Figure 3 One or more processing blocks.

[0071] At 302, method 300 includes receiving video data, the video data including (1) a first frame including a first current viewpoint and a first reference viewpoint, and (2) a second frame including a second current viewpoint and a second reference viewpoint.

[0072] At 304, method 300 includes identifying a co-occurring block in a first reference viewpoint for the current block in a first current viewpoint.

[0073] At 306, method 300 includes calculating the predicted offset vector based on the motion vector between the co-block and the reference block in the reference viewpoint.

[0074] At 308, method 300 includes encoding the video data based on the predicted offset vector.

[0075] Understandable. Figure 3 This illustration is provided for illustrative purposes only and does not imply any limitations on how different embodiments may be implemented. Many modifications may be made to the depicted environment based on design and implementation requirements.

[0076] Figure 4 According to the illustrative embodiments Figure 1 Block diagram 400 depicts the internal and external components of a computer. It should be understood that... Figure 4 This illustration is provided only for one implementation method and does not imply any limitation regarding the environment in which different embodiments may be implemented. Many modifications can be made to the depicted environment based on design and implementation requirements.

[0077] Computer 102 ( Figure 1 ) and server computer 114 ( Figure 1 ) can include Figure 4 The diagram shows a corresponding set of internal components 800A, 800B and external components 900A, 900B. Each set of internal components 800 includes one or more processors 820, one or more computer-readable RAMs 822 and one or more computer-readable ROMs 824 on one or more buses 826, one or more operating systems 828, and one or more computer-readable tangible storage devices 830.

[0078] Processor 820 is implemented in hardware, firmware, or a combination of hardware and software. Processor 820 is a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Accelerated Processing Unit (APU), microprocessor, microcontroller, Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), Application-Specific Integrated Circuit (ASIC), or another type of processing component. In some embodiments, processor 820 includes one or more processors that can be programmed to perform functions. Bus 826 includes components that allow communication between internal components 800A, 800B.

[0079] Server computer 114 ( Figure 1 One or more operating systems 828 and software programs 108 on ) Figure 1 ) and video encoding program 116 ( Figure 1 The data is stored on one or more corresponding computer-readable tangible storage devices 830 for execution by one or more corresponding processors 820 via one or more corresponding RAMs 822 (typically including cache memory). Figure 4 In the illustrated embodiment, each computer-readable tangible storage device 830 is a disk storage device of an internal hard disk drive. Alternatively, each computer-readable tangible storage device 830 is a semiconductor storage device, such as a ROM 824, EPROM, flash memory, optical disc, magneto-optical disc, solid-state drive, compact disc (CD), digital versatile disc (DVD), floppy disk, cassette tape, magnetic tape, and / or another type of non-transitory computer-readable tangible storage device capable of storing computer programs and digital information.

[0080] Each set of internal components 800A, 800B also includes an R / W drive or interface 832 for performing read and write operations on one or more portable computer-readable tangible storage devices 936, such as CD-ROMs, DVDs, Memory Sticks, magnetic tapes, disks, optical discs, or semiconductor storage devices. Software program 108 (...) Figure 1 ) and video encoding program 116 ( Figure 1 The software program can be stored on one or more corresponding portable computer-readable tangible storage devices 936, and read and loaded into the corresponding hard disk drive 830 via the corresponding R / W drive or interface 832.

[0081] Each group of internal components 800A and 800B also includes a network adapter or interface 836, such as a TCP / IP adapter card, a wireless Wi-Fi interface card, a 3G, 4G, or 5G wireless interface card, or other wired or wireless communication links. Server computer 114 ( Figure 1 Software program 108 on ) Figure 1 ) and video encoding program 116 ( Figure 1 ) can be downloaded from an external computer to computer 102 via a network (e.g., the Internet, a local area network, or other wide area networks) and a corresponding network adapter or interface 836. Figure 1 The network includes a network adapter or interface 836 and a server computer 114. Software programs 108 and video encoding programs 116 on the server computer 114 are loaded into the corresponding hard disk drives 830. The network may include copper wire, fiber optic, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers.

[0082] Each set of external components 900A, 900B may include a computer monitor 920, a keyboard 930, and a computer mouse 934. External components 900A, 900B may also include a touchscreen, virtual keyboard, touchpad, pointing device, and other human-computer interface devices. Each set of internal components 800A, 800B also includes a device driver 840 for interfacing with the computer monitor 920, keyboard 930, and computer mouse 934. Device driver 840, R / W driver or interface 832, and network adapter or interface 836 include hardware and software (stored in storage device 830 and / or ROM 824).

[0083] It is understood in advance that although this disclosure includes a detailed description of cloud computing, the implementations of the teachings listed herein are not limited to cloud computing environments. Rather, some embodiments can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0084] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing power, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with service providers. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0085] The characteristics are as follows:

[0086] On-demand self-service: Cloud consumers can automatically and unilaterally allocate computing power, such as server time and network storage, as needed, without having to interact manually with service providers.

[0087] Broad network access: Capabilities are available via the network and can be accessed through standard mechanisms that facilitate the use of heterogeneous thin client platforms or heterogeneous thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0088] Resource pooling: Pooling a provider's computing resources to use a multi-tenant model and dynamically allocate and reallocate different physical and virtual resources on demand to serve multiple consumers. The location independence of this approach lies in the fact that consumers typically cannot control or know the exact location of the resources provided, but may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0089] Rapid elasticity: Computing power can be rapidly and flexibly allocated, and in certain situations, it can be automatically allocated to quickly scale outwards and quickly release to scale inwards. For consumers, the available supply capacity often appears unlimited and can be purchased in any quantity at any time.

[0090] Metering services: Cloud systems automatically control and optimize resource usage by utilizing metrics at a certain level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and activated user accounts). Resource usage can be identified, controlled, and reported, providing transparency to both service providers and consumers.

[0091] The service model is as follows:

[0092] Software as a Service (SaaS): The functionality provided to consumers is the use of a provider's applications running on cloud infrastructure. These applications can be accessed from various client devices via a thin client interface such as a web browser (e.g., web-based email). Aside from limited user-specific application configuration settings, consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application functionalities.

[0093] Platform as a Service (PaaS): This provides consumers with the ability to deploy consumer-created or acquired applications on cloud infrastructure using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they can control the deployed applications and, if any, the configuration of the application hosting environment.

[0094] Infrastructure as a Service (IaaS): This provides consumers with processing, storage, networking, and other basic computing resources on which they can deploy and run arbitrary software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they can control the operating system, storage, deployed applications, and may have limited control over selected network components (such as host firewalls).

[0095] The deployment modes are as follows:

[0096] Private cloud: The cloud infrastructure is operated solely by the organization. A private cloud may only be managed by the organization or a third party, and it may exist internally or externally.

[0097] Community cloud: A cloud infrastructure shared by multiple organizations that supports a specific community with common concerns (e.g., mission, security needs, policy, and compliance considerations). A community cloud may be managed by an organization or a third party and may reside internally or externally.

[0098] Public cloud: Cloud infrastructure that is available to the public or large industry groups and is owned by the organization that sells cloud services.

[0099] Hybrid cloud: A cloud infrastructure consisting of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursts for load balancing between clouds).

[0100] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure that includes a network of interconnected nodes.

[0101] refer to Figure 5 An illustrative cloud computing environment 500 is shown. As illustrated, the cloud computing environment 500 includes one or more cloud computing nodes 10, and local computing devices used by cloud consumers can communicate with the cloud computing nodes 10. These local computing devices can be, for example, a personal digital assistant (PDA) or cellular phone 54A, a desktop computer 54B, a laptop computer 54C, and / or an automotive computer system 54N. The cloud computing nodes 10 can communicate with each other. In one or more networks, such as a private cloud, community cloud, public cloud, or hybrid cloud, or a combination thereof, as described above, the cloud computing nodes 10 can be physically or virtually grouped (not shown). This allows the cloud computing environment 500 to provide infrastructure, platform, and / or software as a service, without requiring cloud consumers to maintain resources on their local computing devices. It should be understood that... Figure 5 The types of computing devices 54A-54N shown are intended to be illustrative only, and cloud computing node 10 and cloud computing environment 500 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0102] refer to Figure 6 This demonstrates the 500 (cloud computing environment) Figure 5 The provided functional abstraction layer 600 should be understood in advance. Figure 6 The components, layers, and functions shown are intended to be illustrative only, and the embodiments are not limited thereto. As shown, the following layers and corresponding functions are provided.

[0103] The hardware and software layer 60 includes hardware components and software components. Examples of hardware components include: a mainframe 61, servers 62 and 63 based on RISC (Reduced Instruction Set Computer) infrastructure, blade servers 64, storage devices 65, and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0104] The virtualization layer 70 provides an abstraction layer from which examples of the following virtual entities can be provided: virtual server 71, virtual storage 72, virtual network including virtual private network 73, virtual application and operating system 74, and virtual client 75.

[0105] In one example, management layer 80 can provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and pricing 82 provides cost determination and billing or invoicing for the consumption of these resources when they are used in the cloud computing environment. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks and protects data and other resources. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 provides cloud resource allocation and management to meet the required service level. Service Level Agreement (SLA) planning and implementation 85 provides pre-scheduling and procurement of cloud resources for future needs anticipated according to the SLA.

[0106] Workload layer 90 provides examples of functionalities available in a cloud computing environment. Examples of workloads and functionalities that can be provided from workload layer 90 include: mapping and navigation 91, software development and lifecycle management 92, virtual classroom education delivery 93, data analytics and processing 94, transaction processing 95, and video encoding 96. Video encoding 96 can encode video data based on motion vector prediction between frames of video data.

[0107] Some embodiments may relate to systems, methods, and / or computer-readable media at any possible level of integration technical detail. The computer-readable medium may include a computer-readable non-transitory storage medium having computer-readable program instructions thereon for causing a processor to perform operations.

[0108] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or raised structures in recesses on which instructions are recorded, and any suitable combination of the foregoing. The computer-readable storage medium used in this document should not be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses through fiber optic cables), or electrical signals transmitted through wires.

[0109] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0110] Computer-readable program code / instructions used to perform operations can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). Computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server as a standalone software package. In the latter case, the remote computer can be connected to the user's computer via any type of network (including local area network (LAN) or wide area network (WAN)), or can be connected to an external computer (e.g., via the internet provided by an internet service provider). In some embodiments, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can be personalized to execute computer-readable program instructions by utilizing state information of computer-readable program instructions in order to perform various aspects or operations.

[0111] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in flowchart and / or block diagram blocks or blocks. These computer-readable program instructions can also be stored in a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other apparatus to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture comprising instructions for implementing aspects of the functions / actions specified in the flowchart and / or block diagram blocks or blocks.

[0112] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other apparatus to cause a series of operational steps to be performed on the computer, other programmable apparatus or other apparatus, thereby producing a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other apparatus, implement the functions / actions specified in flowchart and / or block diagram blocks or blocks.

[0113] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. The methods, computer systems, and computer-readable media may include additional blocks, fewer blocks, different blocks, or blocks arranged differently than those shown in the figures. In some alternative implementations, the functions marked in the blocks may appear in the order indicated in the figures. For example, two blocks shown consecutively may actually be executed simultaneously or substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block illustrated in the block diagrams and / or flowcharts, as well as combinations of blocks illustrated in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs a specified function or action or executes a combination of dedicated hardware and computer instructions.

[0114] Clearly, the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or combinations of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to any specific software code—it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0115] Unless explicitly stated otherwise, no element, action, or instruction used herein should be construed as critical or necessary. Furthermore, as used herein, the term "a / an" is intended to include one or more items and may be used interchangeably with "one or more." Additionally, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, etc.) and may be used interchangeably with the term "one or more." If only one item is intended to be used, the term "one" or similar language is used. Furthermore, as used herein, the terms "has / have / having," etc., are intended to be open-ended terms. Moreover, unless explicitly stated otherwise, the word "based on" means "at least partially based on."

[0116] For illustrative purposes, various aspects and embodiments have been described, but are not intended to be exhaustive or limited to the disclosed embodiments. Even though combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways not specifically recited in the claims and / or not disclosed in the specification. Although each dependent claim listed below may be directly dependent on only one claim, the disclosure of possible implementations includes combinations of each dependent claim with each other claim in the claims. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein is chosen to best explain the principles of the embodiments, practical applications of techniques found in the market, or improvements to the technology, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method of video coding, the method comprising: The method includes: Receive video data, the video data including (1) a first frame including a first current viewpoint and a first reference viewpoint, and (2) a second frame including a second current viewpoint and a second reference viewpoint; For the current block in the first current viewpoint, identify the corresponding block in the first reference viewpoint; The predicted motion vector is calculated based on the motion vector between the co-op block and a reference image of the co-op block in the second reference viewpoint of the second frame; and The predicted motion vector is added to the candidate vector list for inter-frame merge mode or inter-frame skip mode. When more than one predicted motion vector is added, the candidate vector list is sorted based on the proximity between the first current viewpoint and the viewpoint corresponding to the candidate vector, or based on the quality ranking in the reference viewpoint. The video data of the current block is encoded based on the predicted motion vectors from the candidate vector list.

2. The method according to claim 1, characterized in that, The method further includes: The video data is encoded by predicting the position of the co-location block in the first reference viewpoint by applying the disparity vector from the viewpoint transformation to the current block in the first current viewpoint.

3. The method according to claim 2, characterized in that, The encoder performs motion estimation based on the position indicated by the predicted motion vector as a possible matching position.

4. The method according to claim 1 or 2, characterized in that, The method further includes: The predicted motion vector is added to the candidate vector list for inter-frame motion vector coding mode.

5. The method according to claim 4, characterized in that, The method further includes: When one or more of the predicted motion vectors are added, the candidate vector list is sorted based on the proximity between the first current viewpoint and the viewpoint corresponding to the candidate vector, or based on the quality ranking in the reference viewpoint.

6. The method according to claim 1 or 2, characterized in that, The encoder performs motion estimation based on the endpoints of the predicted motion vectors as the search center.

7. The method according to claim 1 or 2, characterized in that, The predicted motion vector is derived by applying a geometric transformation of the motion vector of the co-position block.

8. A computer system for encoding video data, characterized in that, The computer system includes: One or more computer-readable non-transitory storage media configured to store computer program code; and One or more computer processors configured to access the computer program code and perform the method of any one of claims 1 to 7.

9. A non-transitory computer-readable medium, characterized in that, It stores a computer program for encoding video data, the computer program being configured to cause one or more computer processors to perform the method of any one of claims 1 to 7.

Images