Video encoding by providing a geometry proxy

Abstract

Compressing a video frame includes receiving a video frame, identifying a three-dimensional (3D) object in the frame, matching the 3D object to a stored 3D object, compressing the video frame using a color prediction scheme based on the 3D object and the stored 3D object, and storing the compressed frame with metadata that identifies the 3D object, indicates a position of the 3D object in the video frame, and indicates an orientation of the 3D object in the video frame.

Description

[0001] Related Applications

[0002] This application is a continuation of and claims the benefit of U.S. Non-Provisional Application No. 16 / 143,165, filed on September 26, 2018, the entire contents of which are incorporated herein by reference. Technical Field

[0003] Embodiments relate to compressing and decompressing three-dimensional (3D) video data. Background Art

[0004] Techniques for video compression are all related by a general approach. Typically, a frame of a video is temporally compressed by defining blocks of the frame as residuals (e.g., in terms of displacements from previous or future frames). This compression technique is generally acceptable (e.g., has minimal artifacts or errors when decompressed) for objects within a frame that have residuals that can be characterized by an in-plane rigid transformation (e.g., objects that shift and rotate in the image plane over time). Although this model captures many sources of video dynamics (e.g., camera or scene translation), there are common scenarios for which this is not the optimal model (either inefficiently decompressed or contains too many artifacts or errors).

[0005] In other words, typical prediction schemes can reliably predict pixels / blocks / patches in previous and / or future frames (e.g., key frames) for when objects move mostly linearly and / or have predictable motion from frame to frame. However, when objects have dynamic, non-linear motion from frame to frame, typical prediction schemes may not be able to reliably predict pixels / blocks / patches in previous and / or future frames (e.g., key frames) for computing residuals. Therefore, when objects have dynamic, non-linear motion from frame to frame, using a displacement prediction model will likely result in little compression. Summary of the invention

[0006] Example implementations describe systems and methods for compressing frames of video using color prediction through a geometry proxy.

[0007] In a general aspect, a method and non-transitory computer-readable storage medium having computer-executable program code stored thereon, the computer-executable program code, when executed on a computer system, causes the computer system to perform steps. The steps include receiving a frame of a video, identifying a three-dimensional (3D) object in the frame, matching the 3D object to stored 3D objects, compressing the video frame using a color prediction scheme based on the 3D object and the stored 3D objects, and storing the compressed frame with metadata identifying the 3D object, indicating a position of the 3D object in the video frame, and indicating an orientation of the 3D object in the video frame.

[0008] Embodiments may include one or more of the following features. For example, compressing frames of a video using a color prediction scheme based on a 3D object and a stored 3D object may include generating a first 3D object proxy based on the stored 3D object, transforming the first 3D object proxy based on a 3D object identified in the frame, generating a second 3D object proxy based on the stored 3D object, identifying the 3D object in a key frame of the video, transforming the second 3D object proxy based on the 3D object identified in the key frame, mapping color properties from the 3D object to the transformed first 3D object proxy, mapping color properties from the 3D object identified in the key frame to the transformed second 3D object proxy, and generating a residual of the 3D object based on the color properties of the transformed first object and the color properties of the transformed second 3D object proxy.

[0009] For example, compressing frames of a video using a color prediction scheme based on a 3D object and a stored 3D object may include generating a first 3D object proxy based on the stored 3D object, transforming the first 3D object proxy based on a 3D object identified in the frame, generating a second 3D object proxy based on the stored 3D object, identifying the 3D object in a key frame of the video, transforming the second 3D object proxy based on the 3D object identified in the key frame, mapping color properties from the 3D object to the transformed first 3D object proxy, and generating a residual of the 3D object based on color properties of the transformed first 3D object proxy and default color properties of the transformed second 3D object proxy.

[0010] For example, compressing frames of a video using a color prediction scheme based on a 3D object and a stored 3D object may include generating a first 3D object proxy based on the stored 3D object, encoding the first 3D object proxy using an autoencoder, transforming the encoded first 3D object proxy based on a 3D object identified in the frame, generating a second 3D object proxy based on the stored 3D object, encoding the second 3D object proxy using an autoencoder, identifying a 3D object in a key frame of the video, transforming the encoded second 3D object proxy based on the 3D object identified in the key frame, mapping color properties from the 3D object to the transformed first 3D object proxy, mapping color properties from the 3D object identified in the key frame to the transformed second 3D object proxy, and generating a residual for the 3D object based on the color properties for the transformed first 3D object proxy and the color properties for the transformed second 3D object proxy.

[0011] For example, compressing frames of a video using a color prediction scheme based on a 3D object and a stored 3D object may include generating a first 3D object proxy based on the stored 3D object, encoding the first 3D object proxy using an autoencoder, transforming the encoded first 3D object proxy based on a 3D object identified in the frame, generating a second 3D object proxy based on the stored 3D object, encoding the second 3D object proxy using an autoencoder, identifying the 3D object in a key frame of the video, transforming the encoded second 3D object proxy based on the 3D object identified in the key frame, mapping color properties from the 3D object to the transformed first 3D object proxy, and generating a residual of the 3D object based on color properties of the transformed first 3D object proxy and default color properties of the transformed second 3D object proxy.

[0012] For example, before storing the 3D object, the steps may further include: identifying at least one 3D object of interest associated with the video, determining a plurality of mesh attributes associated with the 3D object of interest, determining a position associated with the 3D object of interest, determining an orientation associated with the 3D object of interest, determining a plurality of color attributes associated with the 3D object of interest, and reducing the number of variables associated with the mesh attributes of the 3D object of interest using an autoencoder. Compressing frames of the video may include determining the position coordinates of the 3D object relative to the origin coordinates of the background 3D object in the keyframe. The stored 3D object may include a default color attribute, and the color prediction scheme may use the default color attribute. The steps may further include: identifying at least one 3D object of interest associated with the video; based on the at least one 3D object of interest, generating at least one stored 3D object, each of the at least one stored 3D objects being defined by a mesh including a set of points connected by faces, each point storing at least one attribute, the at least one attribute including the position coordinates of the corresponding point, and storing the at least one stored 3D object in association with the video.

[0013] In another general aspect, a method and non-transitory computer-readable storage medium having computer-executable program code stored thereon, the computer-executable program code, when executed on a computer system, causes the computer system to perform steps. The steps include receiving a frame of a video, identifying a three-dimensional (3D) object in the frame, matching the 3D object with a stored 3D object, decompressing the frame of the video using a color prediction scheme based on the 3D object and the stored 3D object, and rendering the frame of the video.

[0014] Embodiments may include one or more of the following features. For example, decompressing a frame of a video using a color prediction scheme based on a 3D object and a stored 3D object may include generating a first 3D object proxy based on the stored 3D object, transforming the first 3D object proxy based on a 3D object identified in the frame, identifying the 3D object in a key frame of the video, transforming a second 3D object proxy based on the 3D object identified in the key frame, mapping color properties from the 3D object to the transformed first 3D object proxy, mapping color properties from the 3D object identified in the key frame to the transformed second 3D object proxy, and generating color properties of the 3D object based on the color properties of the transformed first 3D object proxy and the color properties of the transformed second 3D object proxy.

[0015] For example, decompressing a frame of a video using a color prediction scheme based on a 3D object and a stored 3D object may include generating a first 3D object proxy based on the stored 3D object, transforming the first 3D object proxy based on a 3D object identified in the frame, identifying the 3D object in a key frame of the video, transforming a second 3D object proxy based on the 3D object identified in the key frame, mapping color properties from the 3D object to the transformed first 3D object proxy, and generating color properties of the 3D object based on the color properties of the transformed first 3D object proxy and default color properties of the transformed second 3D object proxy.

[0016] For example, decompressing frames of a video using a color prediction scheme based on a 3D object and a stored 3D object may include generating a first 3D object proxy based on the stored 3D object, decoding the first 3D object proxy using an autoencoder, transforming the decoded first 3D object proxy based on metadata associated with the 3D object, generating a second 3D object proxy based on the stored 3D object, decoding the second 3D object proxy using an autoencoder, identifying the 3D object in a key frame of the video, transforming the decoded second 3D object proxy based on metadata associated with the 3D object identified in the key frame, mapping color properties from the 3D object to the transformed first 3D object proxy, mapping color properties from the 3D object identified in the key frame to the transformed second 3D object proxy, and generating color properties of the 3D object based on the color properties of the transformed first 3D object proxy and the color properties of the transformed second 3D object proxy.

[0017] For example, decompressing frames of a video using a color prediction scheme based on a 3D object and a stored 3D object may include generating a first 3D object proxy based on the stored 3D object, decoding the first 3D object proxy using an autoencoder, transforming the decoded first 3D object proxy based on metadata associated with the 3D object, generating a second 3D object proxy based on the stored 3D object, decoding the second 3D object proxy using an autoencoder, identifying the 3D object in a key frame of the video, transforming the decoded second 3D object proxy based on metadata associated with the 3D object identified in the key frame, mapping color properties from the 3D object to the transformed first 3D object proxy, and generating color properties of the 3D object based on color properties of the transformed first 3D object proxy and default properties of the transformed second 3D object proxy.

[0018] For example, the steps may further include receiving at least one latent representation of a 3D shape using a machine-trained generative modeling technique to: determine a plurality of mesh attributes associated with the 3D shape; determine a position associated with the 3D shape; determine an orientation associated with the 3D shape; and determine a plurality of color attributes associated with the 3D shape, and storing the 3D shape as a stored 3D object. Rendering a frame of the video may include: receiving position coordinates of the 3D object relative to the origin coordinates of the background 3D object in the keyframe, and placing the 3D object in the frame using the position coordinates. The steps may further include: receiving a neural network used by an encoder of an autoencoder to reduce the number of variables associated with mesh attributes, position, orientation, and color attributes of at least one 3D object of interest, regenerating points associated with a mesh of at least one 3D object of interest using the neural network in a decoder of the autoencoder, the regenerated points including regenerating position attributes, orientation attributes, and color attributes, and storing at least one 3D object of interest as a stored 3D object.

[0019] In yet another general aspect, a method and a non-transitory computer-readable storage medium having computer-executable program code stored thereon, the computer-executable program code, when executed on a computer system, causes the computer system to perform steps for predicting color variance using a proxy. The steps include: generating a first 3D object proxy based on a stored 3D object; generating a second 3D object proxy based on the stored 3D object; transforming the first 3D object proxy based on a 3D object identified in a frame of a video; transforming the second 3D object proxy based on a 3D object identified in a key frame of the video, mapping color properties from the 3D object identified in the video frame to the transformed first 3D object proxy, mapping color properties from the 3D object identified in the key frame to the transformed second 3D object proxy, and generating color data for the 3D object based on the color properties of the transformed first 3D object proxy and the color properties of the transformed second 3D object proxy.

[0020] Implementations may include one or more of the following features. For example, the steps may further include encoding the first 3D object proxy using an autoencoder before transforming the first 3D object proxy, and encoding the second 3D object proxy using an autoencoder before transforming the second 3D object proxy. The steps may further include decoding the first 3D object proxy using an autoencoder after transforming the first 3D object proxy, and decoding the second 3D object proxy using an autoencoder after transforming the second 3D object proxy. Generating color data for the 3D object may include subtracting color properties of the transformed first 3D object proxy from color properties of the transformed second 3D object proxy. Generating color data for the 3D object may include adding color properties of the transformed first 3D object proxy to color properties of the transformed second 3D object proxy. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Example embodiments will become more fully understood from the following detailed description given herein and the accompanying drawings, wherein like elements are denoted by like reference numerals, which are given by way of illustration only and thus do not limit example embodiments and wherein:

[0022] Figure 1 A block diagram illustrating signal flow for compressing video according to an example implementation.

[0023] Figure 2 A block diagram illustrating signal flow for storing compressed video according to an example implementation.

[0024] Figure 3A A block diagram of an encoder prediction module is illustrated according to an example implementation.

[0025] Figure 3BA block diagram of another encoder prediction module is illustrated according to an example implementation.

[0026] Figure 4A A block diagram of a decoder prediction module is illustrated according to an example implementation.

[0027] Figure 4B A block diagram of another decoder prediction module is illustrated according to an example embodiment.

[0028] Figure 5A A block diagram illustrating a signal flow for encoding a 3D object according to an example embodiment.

[0029] Figure 5B A block diagram illustrating a signal flow for decoding a 3D object according to an example embodiment.

[0030] Fig. 6A A block diagram illustrating signal flow for streaming video and rendering the video on a client device.

[0031] Figure 6B A block diagram illustrating another signal flow for streaming video and rendering the video on a client device.

[0032] Figure 7 A block diagram illustrates a method for compressing frames of a video according to at least one example embodiment.

[0033] Figure 8 A block diagram illustrating another method for compressing frames of a video according to at least one example embodiment.

[0034] Fig. 9 A block diagram illustrating a method for decompressing and rendering frames of a video according to at least one example embodiment.

[0035] Fig.10 A block diagram illustrates a method for compressing a 3D object according to at least one example embodiment.

[0036] Fig.11 A block diagram illustrates a method for decompressing a 3D object according to at least one example embodiment.

[0037] Fig.12 A video encoder system according to at least one example embodiment is illustrated.

[0038] Fig.13 A video decoder system according to at least one example embodiment is illustrated.

[0039] Fig.14 An example of a computer device and a mobile computer device according to at least one example embodiment is illustrated.

[0040] It should be noted that these figures are intended to illustrate the general characteristics of methods and / or structures utilized in certain example embodiments, and are intended to supplement the written description provided below. However, these drawings are not to scale and may not accurately reflect the precise structure or performance characteristics of any given embodiment, and should not be interpreted as defining or limiting the range of values ​​or properties encompassed by the example embodiments. The use of similar or identical reference numerals in the various drawings is intended to indicate the presence of similar or identical elements or features. DETAILED DESCRIPTION

[0041] Although the example embodiments may include various modifications and alternative forms, embodiments thereof are shown by way of example in the drawings and will be described in detail herein. However, it should be understood that there is no intention to limit the example embodiments to the particular forms disclosed, but on the contrary, the example embodiments will cover all modifications, equivalents, and alternative forms falling within the scope of the claims. Throughout the description of the drawings, the same reference numerals refer to the same elements.

[0042] In a 3D dynamic environment, 3D objects that undergo complex 3D transformations may not allow simple rigid transformations based on previous frames and / or future frames, previously compressed images or frames, etc. For example, a human actor that undergoes a transformation consisting of rigidity, clarity, and deformation components will result in a highly nonlinear transformation in pixel space. Subsequently, there may be no suitable corresponding blocks in nearby key frames. These 3D objects are referred to herein as dynamic 3D objects.

[0043] Non-dynamic can include 3D objects that appear to move from frame to frame due to camera or scene translation, which can be encoded / decoded using color prediction through geometric agents. In this example, because the camera capturing the scene is moving (e.g., in a predictable manner and / or direction), a stationary 3D object can appear to be moving from frame to frame. In this example, a 3D object (e.g., a vehicle, i.e., a train, car, or airplane) can move from frame to frame in a predictable manner (e.g., at a constant speed and / or direction). These objects are sometimes referred to as translational 3D objects in this article. Another example of a non-dynamic 3D object is a 3D object that does not appear to move within a scene. In this example, a stationary or fixed 3D object (e.g., the background of a scene, furniture in a fixed position within a scene, or a slow-moving object in the distance) may appear to be stationary from frame to frame (e.g., without any camera or scene translation). These objects are sometimes referred to as fixed or background 3D objects in this article.

[0044] Example embodiments use the stored 3D objects as geometric proxies for objects that appear in a video. The stored 3D objects may be deformable 3D shape models (e.g., meshes with given properties that may be manipulated when necessary) that may be used in a prediction scheme. For example, the positions of pixels associated with dynamic 3D objects in a frame of a video may be predicted based on the stored 3D objects. Similarly, the prediction scheme may be used to compress background images in a frame of a video, 3D objects that move (e.g., predictably) within an image plane of a frame of a video, portions of 3D objects, layers of a video frame that include 3D objects (e.g., as part of a Z-order image), containers that include 3D objects (e.g., standard mesh objects), and the like.

[0045] One or more implementations may include using the stored 3D object to locate pixels, blocks and / or patches in a key frame. One or more implementations may include matching pixels, blocks and / or patches in a frame of a video with pixels, blocks and / or patches of a stored 3D object. One or more implementations may include compressing the stored 3D object before storing the stored 3D object, and decompressing the stored 3D object during the prediction process. One or more implementations may include compressing the stored 3D object before storing the stored 3D object; compressing the 3D object associated with the frame of the video; and using the compressed 3D object in the prediction process.

[0046] Figure 1 A block diagram illustrating a signal flow for compressing video according to an example implementation. Figure 1 As shown in , encoder 105 includes frames 110 , prediction module 115 , and encoding module 120 . In addition, video storage 125 includes metadata 130 , stored 3D objects 135 , and compressed frames 140 .

[0047] The video data 5 is input to the encoder 105, where a frame 110 is selected from a plurality of frames included in the video data 5. The video data 5 may correspond to a 3D video (e.g., a monocular view), a 2D video, a portion of a video (e.g., less than all frames of a video), etc. Therefore, the frame 110 may include data corresponding to a frame of the video. The encoder 105 may be configured to use a color prediction scheme based on using a 3D object as a geometric proxy. The encoder 105 may compress the frame 110 using a color prediction scheme based on using the 3D object as a geometric proxy. Compressing the frame 110 may reduce the amount of data used to store and / or communicate the frame 110. The compressed frame 110 may include a prediction step, a quantization step, a transform step, and an entropy encoding step.

[0048] Inter prediction can exploit spatial redundancy (e.g., correlation between pixels between frames) by calculating delta values ​​expressed in terms of one or more neighboring frames. Incremental coding can include locating relevant pixels / blocks / patches in a key frame (e.g., a previous neighboring key frame, an upcoming neighboring key frame), and then calculating delta values ​​for pixels in the frame being encoded. The delta values ​​can be referred to as residuals. Thus, inter prediction can produce residuals for pixels / blocks / patches (e.g., 3D objects) in a frame. In example embodiments, delta values ​​can be expressed in terms of explicit textures (colors), default textures, predefined textures, textures of identified 3D objects, and the like.

[0049] The prediction module 115 can be configured to locate the 3D object in the frame 110 and / or the keyframe. For example, machine vision, computer vision and / or computer image recognition techniques can be used to identify and locate the 3D object. Once the 3D object has been identified, the 3D object can be located using a coordinate system (e.g., 2D Cartesian, 3D Cartesian, polar coordinates, etc.) as an attribute of a point associated with the mesh of the identified 3D object.

[0050] In an example embodiment, computer image recognition techniques based on training (machine learning) convolutional neural networks using multiple known images can be used to identify and locate 3D objects. For example, a block, multiple blocks, and / or patches are selected and / or identified from a selected frame. The trained convolutional neural network can operate on the selected block, multiple blocks, and / or patches. The results can be tested (e.g., error tests, loss tests, divergence tests, etc.). If the test produces a value below a threshold (or alternatively, a value above a threshold depending on the type of test), the selected block, multiple blocks, and / or patches can be identified as a 3D object.

[0051] In an example embodiment, a frame of a video may include a tag indicating that a previously identified 3D object of interest is included in the frame. The tag may include the identity and location of the 3D object. For example, a video may be generated using a computer generated image (CGI) tool (e.g., a computer animated film). Computer generated characters may be identified and labeled in each frame. In addition, a model for each identified 3D object of interest (e.g., an identified character) may be stored as a stored 3D object 135.

[0052] In an example embodiment, the stored 3D objects 135 and 3D objects may be defined by a triangular mesh. A triangular mesh may be a collection of points connected by triangular faces. Each point may store various attributes. For example, the attributes may include the position, color, texture coordinates, etc. of each point. The attributes may include and / or indicate (e.g., multiple attributes may indicate) the orientation of a corresponding 3D object and / or the position of a corresponding 3D object in a frame (e.g., frame 110) of a video and / or an image.

[0053] Thus, in an example real-time manner, the mesh properties of a 3D object may be sufficient to identify and locate the 3D object. A model including mesh properties for a plurality of 3D objects of interest may be stored as a stored 3D object 135. The model may be standardized. For example, a model for a man, a woman, a teenager, a child, or more generally a human or a portion of a human (e.g., body, head, hand, etc.) may be stored as a 3D object 135. For example, a model for a dog, a cat, a deer, or more generally a quadruped or a portion of a quadruped (e.g., body, head, leg, etc.) may be stored as a stored 3D object 135. The properties of the model may then be used to search for 3D objects with similar properties in a frame.

[0054] A 3D object (hereinafter referred to as a 3D object) located in the frame 110 may be matched with one of the stored 3D objects 135. In an example embodiment, the identity of the 3D object generated by computer image recognition techniques may be used to search for the stored 3D objects 135. In an example embodiment, a label of the 3D object found in the frame may be matched with a label of one of the stored 3D objects 135. In an example embodiment, a model of one of the stored 3D objects 135 and the 3D object having similar properties may be identified as a match.

[0055] The matched 3D object in the stored 3D objects 135 (hereinafter referred to as the stored 3D object) can then be used in a color prediction scheme for the frame 110. For example, a mesh corresponding to the stored 3D object can be translated and oriented to align with the orientation of the 3D object. Points corresponding to the translated and oriented stored 3D object can then be matched with points of a corresponding (e.g., the same) 3D object in a nearby (temporary), previously encoded key frame. The prediction module 115 can then use the matching points of the corresponding 3D object to select or predict pixels / blocks / patches in the key frame for calculating the residual of the 3D object in the frame 110 (e.g., color displacement relative to the key frame).

[0056] Additionally, the prediction module 115 may generate metadata 130 associated with the frame 110. The metadata 130 may include data associated with at least one 3D object located in the frame 110 that has been predicted using one of the stored 3D objects 135. The metadata 130 may include attributes (e.g., grid point attributes) associated with the location and / or orientation of the 3D object in the frame 110. The metadata 130 is stored relative to (corresponding to) one of the compressed frames 140 of the video data 5.

[0057] The encoding module 120 may be configured to perform a series of encoding processes on the residual. For example, data corresponding to the residual may be transformed, quantized, and entropy encoded.

[0058] The transform residual may include converting data (e.g., pixel values) from the spatial domain into transform coefficients in the transform domain. The transform coefficients may correspond to a two-dimensional matrix of coefficients that is initially the same size as the original blocks and / or patches in the frame 110. In other words, there may be as many transform coefficients as data points (e.g., pixels) in the original blocks and / or patches in the frame 110. However, due to the transform, a portion of the transform coefficients may have a value equal to zero. Typically, transforms include Karhunen-Loève transform (KLT), discrete cosine transform (DCT), singular value decomposition transform (SVD), and asymmetric discrete sine transform (ADST).

[0059] Vector coordinates (e.g., representing transform coefficients) are usually given by floating point or double precision. However, such representations are usually more precise than actually needed. For example, the video data 5 may originate from a video capture device (e.g., a camera, a scanner, a computer-generated image program) with some measurement errors. Therefore, a relatively large number of lower bits may be noise. Quantization converts a given floating point number into an integer representation of b bits in length. Therefore, quantization can reduce the data in each transform coefficient. Quantization may involve mapping values ​​in a relatively large range to values ​​in a relatively small range, thereby reducing the amount of data required to represent the quantized transform coefficients. Quantization can convert transform coefficients into discrete quantum values, which are called quantized transform coefficients or quantization levels. For example, quantization can add zeros to the data associated with the transform coefficients. For example, a coding standard can define 128 quantization levels in a scalar quantization process.

[0060] The expected value of the information contained in the quantized transform coefficients (e.g., a data set of discrete quantum values). The more unique values ​​a data set contains, the higher the entropy. Repeated values ​​reduce entropy and achieve better compression. Therefore, the quantized transform coefficients can be entropy encoded. Entropy coding is performed using one of a group of techniques for compressing data sets based on data entropy. For example, entropy coding techniques can include Huffman coding, arithmetic coding, or the use of an asymmetric number system.

[0061] The entropy coded coefficients (for all pixels / blocks / patches in frame 110) are then output together with the information required to decode frame 110 (e.g., the prediction type used, motion vectors, and quantization values) as one of the compressed frames 140 and stored together with and / or in association with other compressed frames of video data 5.

[0062] Typically, the encoder 105 and the video storage 125 are elements in the cloud or the World Wide Web. For example, the encoder 105 can be one of a plurality of encoders implemented as computer hardware and / or computer software implemented in a cloud computing device, which is configured to compress video data (e.g., video data 5) using a compression scheme defined by one or more standards (e.g., H.264, H.265, HEVC, VP8, VP9, ​​VP10, AV1, etc.). For example, the video storage 125 can be implemented as at least one non-volatile memory, non-transitory computer-readable medium, etc. located in a cloud computing device (e.g., a streaming server). In at least one embodiment, the encoder 105 compresses the video data 5 into a video, and stores the compressed video data for future (e.g., later in time) playback of the video when streaming from the cloud computing device to a client device.

[0063] In this embodiment, the client device includes a decoder (e.g., decoder 145). In some embodiments, metadata 130 and / or stored 3D objects 135 can be communicated to the client device. Metadata 130 and / or stored 3D objects 135 can be communicated to the client device on demand and / or as an initialization process. Metadata 130 and / or stored 3D objects 135 can be compressed when stored in video storage 125 and / or compressed on demand before being communicated to the client device. In an example embodiment, the stored 3D objects 135 can be compressed using machine-trained generative modeling techniques to generate a reduced number of variables (referred to herein as a latent representation or reduced latent representation) associated with properties and positions of a mesh for the stored 3D object (as described in more detail below).

[0064] Therefore, if Figure 1As shown in , the decoder 145 includes a frame reconstruction module 150, a prediction module 155, and a decoding module 160. The decoding module 160 can be configured to perform the inverse operation of the encoding module 120. The decoding module 160 can receive a compressed frame 140 (e.g., representing a 3D movie selected by a user of a client device including the decoder 145 for streaming or downloading). The compressed frame 140 can be received one at a time, multiple frames or blocks of frames at a time, or received as a complete 3D movie. The decoder 145 can select one of the compressed frames 140, and the decoding module 160 can entropy decode, dequantize, and inverse transform the selected compressed frame.

[0065] In an example embodiment, the decoding module 160 uses the data elements within the selected compressed frame and decompresses the data by entropy decoding (e.g., using an inverse transform of Huffman coding, arithmetic coding, or asymmetric digital system coding) to produce a set of quantized transform coefficients. The quantized transform coefficients are dequantized, and the dequantized transform coefficients are inversely transformed (e.g., using an inverse transform of KLT, DCT, SVD, or ADST) to generate a derivative residual that may be the same (or approximately the same) as the residual generated by the prediction module 115.

[0066] The prediction module 155 may be configured to determine whether the selected compressed frame includes one of the stored 3D objects 135. In an example embodiment, the metadata 130 may be used to determine whether the selected compressed frame includes one of the stored 3D objects. For example, the prediction module 155 may query the metadata 130 in the selected compressed frame and determine that the selected compressed frame includes one of the stored 3D objects 135 if metadata is returned.

[0067] The prediction module 155 may be configured to identify the 3D object, the location (e.g., position) of the 3D object, and the orientation of the 3D object for the selected compressed frame based on the returned metadata. The prediction module 155 may be configured to select a stored 3D object (hereinafter referred to as a stored 3D object) from the stored 3D objects 135 and use the stored 3D object to decompress the selected frame.

[0068] In an example embodiment, a mesh corresponding to a stored 3D object may be translated and oriented based on the position (e.g., location) of the identified 3D object and the orientation of the identified 3D object. Points corresponding to the translated and oriented stored 3D object may then be matched with points of a corresponding (e.g., the same) 3D object located in a nearby (temporary), previously decoded key frame. The prediction module 155 may then use the matching points of the corresponding 3D object to select or predict pixels / blocks / patches in the key frame for regenerating (e.g., calculating) color values ​​and / or color properties of the translated and oriented stored 3D object based on the residuals at the identified positions in the selected frame.

[0069] The prediction module 155 may be further configured to regenerate (e.g., calculate) color values ​​and / or color attributes for the remainder of the selected frame based on the residual or remaining portion of the selected frame and the corresponding pixels / blocks / patches of the key frame. The frame reconstruction module 150 may be configured to reconstruct the selected frame based on the regenerated color values ​​and / or color attributes for the translated and oriented stored 3D object and the regenerated color values ​​and / or color attributes for the remainder of the selected frame. In an example embodiment, the frame reconstruction module 150 may be configured to splice the regenerated color values ​​and / or color attributes for the translated and oriented stored 3D object to the regenerated color values ​​and / or color attributes for the remainder of the selected frame based on the position of the identified 3D object.

[0070] Example implementations may include identifying two or more 3D objects, regenerating color values ​​and / or color attributes of each of the two or more 3D objects, and reconstructing a selected frame using each of the two or more 3D objects. The frame reconstruction module 150 may be configured to regenerate the video data 5 based on a plurality of reconstructed frames. The video data 5 (e.g., texture data and color data) may be rendered and color corrected for display on a display of a client device.

[0071] Compressing a video (or video frame) using an encoder (e.g., encoder 105) configured to use a color prediction scheme based on using 3D objects as geometric proxies may not result in the highest compression rate (e.g., smallest data size) for the video, video frame, or multiple video frames. Therefore, example embodiments may include compressing a video, video frame, or multiple video frames using two or more encoders, each capable of compressing video data using an identified color prediction scheme.

[0072] Figure 2 A block diagram illustrating a signal flow for storing compressed video (and / or compressed frames of video) according to an example embodiment. Figure 2As shown in , the first encoder 205 is an encoder (e.g., encoder 105) that uses a color prediction scheme based on using a 3D object as a geometric proxy. In addition, at least one second encoder 210-1 is an encoder that uses color prediction scheme 1, at least one second encoder 210-2 is an encoder that uses color prediction scheme 2, and at least one second encoder 210-i is an encoder that uses color prediction scheme i. Each of the first encoder 205 and at least one second encoder 210-1, 210-2, ..., 210-i can be configured to generate n frames with a total of x bits and communicate them to the compression size comparator 215. The color prediction schemes 1, 2, ..., i can be a default prediction scheme of the encoding standard, a configurable prediction scheme of the encoding standard, a custom prediction scheme based on time displacement, an alternative prediction scheme based on using a 3D object as a geometric proxy, etc.

[0073] The compressed size comparator 215 can be configured to select one of the outputs of the first encoder 205, the at least one second encoder 210-1, 210-2, ... or 210-i to save in the video storage 125 (e.g., for later streaming to a client device). In an example embodiment, the encoder output can be selected based on compression efficiency. For example, the compressed video (and / or compressed frame of the video) with the least number of bits (e.g., the smallest value of x) can be saved.

[0074] In another example embodiment, the color prediction scheme may be a preferred prediction scheme. In this embodiment, the compressed video (and / or compressed frames of the video) of the preferred color prediction scheme is saved unless some conditions exist. For example, the condition may be based on compression efficiency. Unless the output of one of the at least one second encoder 210-1, 210-2, ..., 210-i is at least a threshold percentage (e.g., 10%, 15%, 20%, etc.) more efficient than the output of the first encoder 205, the output of the first encoder 205 may be selected. If more than one of the at least one second encoder 210-1, 210-2, ..., 210-i is at least a threshold percentage (e.g., 10%, 15%, 20%, etc.) more efficient than the output of the first encoder 205, the highest efficiency (e.g., the minimum value of x) of the at least one second encoder 210-1, 210-2, ..., 210-i may be saved.

[0075] In an example implementation, Figure 2The signal stream for storing compressed video (and / or compressed frames of video) illustrated in FIG can be performed on frame-by-frame video and / or frame sets. For example, each frame of the video can be encoded as described above. Then, the compression size comparator 215 can select the encoder output based on efficiency or based on a conditional preferred color prediction scheme. In an example embodiment, for example, key frames can be used to select between frame-by-frame or frame set compression.

[0076] In an example implementation, one of the encoder outputs may always be saved. For example, the default color prediction scheme may be saved to ensure backward compatibility. Figure 2 In FIG. 2 , this embodiment is illustrated as a dashed line from at least one second encoder 210 - i.

[0077] Can be executed multiple times using different coding standards Figure 2 The signal flow for storing compressed video (and / or compressed frames of video) illustrated in FIG. For example, two or more of the coding standards such as H.264, H.265, HEVC, VP8, VP9, ​​VP10, AV1, etc. may be used to perform Figure 2 . Thus, a color prediction scheme based on using 3D objects as geometric proxies (e.g., encoder 105) may be implemented in two or more coding standards and on the same video (e.g., video data 5). Thus, video storage 125 may store multiple instances of a video (and / or compressed frames of a video), each instance having been compressed using a different standard and / or a different configuration of a standard. As a result, a streaming server may serve a video based on the capabilities of the requesting client device and / or the network through which the video is to be communicated.

[0078] Figure 3A FIG. 1 is a block diagram of an encoder prediction module according to an example implementation. Figure 3A As shown in , the prediction module 115 includes the frame 110 , a 3D object locator module 305 , a 3D object matching module 310 , a stored 3D object 135 , a stored 3D object translation module 315 , a key frame 320 , a block matching module 325 , and a residual generation module 330 .

[0079] In an example embodiment, the stored 3D objects 135 and the 3D objects in the frame 110 may be defined by a triangular mesh. A triangular mesh may be a collection of points connected by triangular faces. Each point may store various attributes. For example, the attributes may include the position, color, texture coordinates, etc. of each point. The attributes may include and / or indicate (e.g., multiple attributes may indicate) the orientation of the corresponding 3D object and / or the position of the corresponding 3D object in the frame 110 and / or image of the video.

[0080] The 3D object locator module 305 may be configured to identify and locate 3D objects (hereinafter referred to as 3D objects) in the frames 110 and the keyframes 320. For example, machine vision techniques, computer vision techniques, computer image recognition techniques, etc. may be used to identify and locate 3D objects in the frames 110 and the keyframes 320. Once the 3D objects have been identified, the 3D objects may be located using a coordinate system (e.g., 2D Cartesian, 3D Cartesian, polar coordinates, etc.) as an attribute of a point associated with a mesh of the identified 3D object.

[0081] In an example embodiment, computer image recognition techniques based on training (machine learning) convolutional neural networks using multiple known images can be used to identify 3D objects. For example, a block, multiple blocks, and / or patches are selected and / or identified from a selected frame. The trained convolutional neural network can operate on the selected block, multiple blocks, and / or patches. The results can be tested (e.g., error tests, loss tests, dispersion tests, etc.). If the test produces a value below a threshold (or alternatively, a value above a threshold depending on the type of test) threshold, the selected block, multiple blocks, and / or patch can be identified as a 3D object.

[0082] In an example embodiment, a video frame including a previously identified 3D object of interest may also include a tag (e.g., in a header, frame metadata, etc.) indicating that the previously identified 3D object of interest is included in the frame. The tag may include the identity and location of the 3D object (e.g., coordinate attributes of a point associated with a mesh). For example, a video may be generated using a computer generated image (CGI) tool (e.g., a computer animated film). Computer generated characters may be identified and labeled in each frame. In addition, a model (e.g., defined by a triangle mesh) for each identified 3D object of interest (e.g., a character in an animated film) may be stored as a stored 3D object 135.

[0083] In an example embodiment, the mesh properties of a 3D object may be sufficient to identify and locate the 3D object. A model including mesh properties for a plurality of generic 3D objects of interest may be stored in (or associated with) a 3D object locator module. The model may be standardized. For example, a model for a man, a woman, a teenager, a child, or more generally a human or a portion of a human (e.g., body, head, hand, etc.) may be stored. For example, a model for a dog, a cat, a deer, or more generally a quadruped or a portion of a quadruped (e.g., body, head, leg, etc.) may be stored. The properties of the model may then be used to search for 3D objects with similar properties in the frame.

[0084] The 3D object matching module 310 may be configured to match a 3D object (hereinafter, referred to as a 3D object) located in the frame 110 with one of the stored 3D objects 135. In an example embodiment, computer image recognition techniques are used by the 3D object locator module 305 to identify the 3D object. Identifying the 3D object may also include assigning a unique ID from an ID database to the 3D object. The unique ID may be used to search for the 3D object in the stored 3D objects 135. If the unique ID is found in the stored 3D objects 135, the corresponding one of the stored 3D objects matches the 3D object.

[0085] In an example embodiment, the tag of a 3D object found in a frame may be unique to the 3D object (e.g., there is a one-to-one relationship between the tag and the 3D object). The tag may be used to search for the 3D object in the stored 3D objects 135. If the tag is found in the stored 3D objects 135, a corresponding one of the stored 3D objects matches the 3D object.

[0086] In an example embodiment, one of the stored 3D objects 135 and a model having similar properties to the 3D object may be identified as a partial match. The stored 3D objects 135 may then be filtered based on the partial match. The 3D object may then be searched for in the stored 3D objects 135 using one or more properties or a combination of properties of the 3D object. If one or more properties or a combination of properties are found in the stored 3D object 135, then the corresponding one of the stored 3D objects is a match for the 3D object. The one or more properties or a combination of properties may uniquely identify the 3D object relative to the stored 3D object 135 with a relatively high degree of certainty. For example, the shape or relative position of a body part (e.g., face shape, nose shape, relative positions of eyes, nose and mouth, etc.), the color of a cartoon character's hair or skin, the shape and relative positioning of objects worn by the character (e.g., jewelry), the type of vehicle (e.g., car, tractor, etc.), etc. may uniquely identify the 3D object relative to the stored 3D object 135 (e.g., if the stored 3D object 135 corresponds to a set of 3D objects of interest for a video).

[0087] If the 3D object matching module 310 does not find a match for the 3D object among the stored 3D objects, the prediction module 115 may use standard prediction techniques defined by the coding standard. In other words, a color prediction scheme based on using the 3D object as a geometric proxy may not be used for the 3D object.

[0088] If the 3D object matching module 310 does not find a match for the 3D object in the stored 3D objects 135, the 3D object matching module 310 may be configured to add the 3D object to the stored 3D objects 135. For example, the 3D object matching module 310 may be configured to assign a unique ID or a unique label to the 3D object and define a model (e.g., a triangular mesh including points and corresponding attributes) for the 3D object. The model may then be stored as one of the stored 3D objects 135 and identified by the unique ID or unique label.

[0089] In an example embodiment, the stored 3D object 135 may have a predetermined mesh representation and / or data structure. Thus, the model may have a size (e.g., number of points, number of faces, number of vertices, etc.), number of attributes, type of attributes, etc., predefined based on the design of the stored 3D object 135. Furthermore, matching the 3D object with a 3D object in the stored 3D objects 135 may include redefining the 3D object based on the design of the stored 3D object 135 by the 3D object matching module 310 before searching for a match in the stored 3D objects 135.

[0090] The stored 3D object translation module 315 may be configured to translate (or transform) the matched one of the stored 3D objects 135 (hereinafter referred to as the stored 3D object). For example, the mesh corresponding to the stored 3D object may be translated and oriented to align with the orientation of the 3D object. Translation and orientation together may be referred to as transformation. The stored 3D object translation module 315 may be configured to transform the matched one of the stored 3D objects 135 associated with the frame 110 and the key frame 320. Thus, the stored 3D object translation module 315 may be configured to generate and transform a first 3D object proxy based on the stored 3D object for the 3D object associated with the frame 110. In addition, the stored 3D object translation module 315 may be configured to generate and transform a second 3D object proxy based on the stored 3D object for the 3D object associated with the key frame 320.

[0091] Transforming a 3D agent can include sending control points and values ​​for a Bezier, or NURBS, or subdivision surface by articulating mesh points (e.g., like a skeleton), generating a latent representation using an autoencoder (as described below), and locating values ​​associated with the latent representation. Each mesh can have a predetermined connectivity. The predetermined connectivity can allow direct correspondence between points of two meshes at different poses. Therefore, we can use the same texture parameterization for both meshes.

[0092] The stored 3D object translation module 315 may be configured to generate metadata 20. The metadata 20 includes information identifying the stored 3D object and information related to the translation and orientation of the stored 3D object. The information related to the translation and orientation of the stored 3D object may be used to perform the same translation and orientation of the stored 3D object in the future (e.g., by the decoder 145). In an example embodiment, metadata is generated and stored for both the first 3D object proxy and the second 3D object proxy.

[0093] The block matching module 325 may be configured to match points of the translated and oriented stored 3D object of the frame 110 corresponding to the translated and oriented stored 3D object of the key frame 320. In an example embodiment, the block matching module 325 may be configured to wrap a mesh (e.g., a 3D mesh) representing the translated and oriented stored 3D object with color and / or texture attributes. For example, the block matching module 325 may be configured to map color attributes from the 3D object identified in the frame 110 to the transformed first 3D object proxy. In addition, the block matching module 325 may be configured to map color attributes from the 3D object identified in the key frame 320 to the transformed second 3D object proxy. Mapping color attributes from the 3D object identified in frame 110 to the transformed first 3D object proxy may include converting the 3D object from a 3D space (e.g., XYZ space) to a 2D space (e.g., UV space) and / or converting the 3D object identified in frame 110 from a 2D space (e.g., UV space) to a 3D space (e.g., XYZ space).

[0094] Each mesh may have a predetermined connectivity. The predetermined connectivity may allow direct correspondence between points of two meshes in different poses. Therefore, we may use the same texture parameterization for both meshes. Therefore, mapping color attributes may include identifying pixels in a frame (e.g., frame 110 and / or key frame 320) based on pixel coordinates in the frame, and then setting the color attributes of the points in the mesh representation of the transformed 3D object proxy with the same coordinates to the same color value of the identified pixel. In an example embodiment, more than one frame may be used to generate pixel attributes. In addition, pixel attributes from a regenerated frame (e.g., from a regeneration loop in an encoder) may be used to generate pixel attributes.

[0095] In an example implementation, pixel attributes may be blended. Blending pixel attributes (or textures) may be done using a per-texel average of corresponding texels from those textures that have information about that texel, texels from different textures may have confidence values ​​(e.g., the temporal distance between the current frame and the frame from which that texture was generated), unobserved texels in a particular texture may also be predicted (lowering their confidence), for the current frame, using a new pose of the mesh and its texture to get a prediction for (a portion of) the current frame, etc.

[0096] In some cases, the frame 110 and / or key frame 320 may not have pixels corresponding to points on the mesh representation of the transformed 3D object proxy. Therefore, the mesh representation of the transformed 3D object proxy may retain the default color attributes of these points. Therefore, any residual color calculations may be based on the default color attributes.

[0097] The block matching module 325 can then use the matching points of the corresponding 3D object to select or predict pixels / blocks / patches in the key frame 320 for generating the residual 15 of the 3D object in the frame 110. In an example embodiment, the stored 3D object and / or 3D object proxy can be encoded using an autoencoder before translation and matching (described in more detail below). Encoding the stored 3D object or 3D object proxy converts the 3D object, the stored 3D object and / or the 3D object proxy into a potential representation. The potential representation includes fewer values ​​(e.g., points) than the mesh representing the 3D object, the stored 3D object and / or the 3D object proxy. Therefore, translating the 3D object, the stored 3D object and / or the 3D object proxy to the potential representation includes manipulating fewer points. In addition, mapping color properties includes mapping fewer points when encoding the 3D object, the stored 3D object and / or the 3D object proxy to the potential representation.

[0098] The residual generation module 330 may be configured to generate (or calculate) a residual 15 (e.g., a color displacement relative to the key frame 320) of the 3D object in the frame 110. For example, the residual generation module 330 may generate the residual 15 by subtracting the pixel attribute value for each point in the triangular mesh of the 3D object from the pixel attribute value for each matching point of the predicted pixel / block / patch in the key frame 320. In an example embodiment, the color attribute of a point in the mesh representation of the first 3D object proxy may be subtracted from the color attribute of a corresponding point (e.g., having the same point identity or being at the same position in the mesh sequence) in the mesh representation of the second 3D object proxy.

[0099] In an example embodiment, the encoder 105 includes a reconstruction path (not shown). The reconstruction path includes several components or software implementations that decode frames together using at least one inverse process of the above-mentioned encoding process. For example, the reconstruction path may include at least an inverse quantization process and an inverse transform process. When encoding the next consecutive frame, the reconstructed frame generated in the reconstruction path can be used to replace the key frame 320. Encoding the original frame will cause some losses. Therefore, the reconstructed frame generated in the reconstruction path includes some artifacts (e.g., errors) compared to the original frame. These artifacts may or may not be corrected in the prediction module 115. In some embodiments, the artifacts generated using the reconstructed frame can be corrected by the color correction module 635 described below.

[0100] Figure 3B A block diagram of another encoder prediction module according to an example embodiment is shown. Figure 3B As shown, the prediction module 115 includes the frame 110 , a 3D object locator module 305 , a 3D object matching module 310 , a stored 3D object 135 , a stored 3D object translation module 315 , a block matching module 325 , and a residual generation module 330 .

[0101] Figure 3B The implementation of the prediction module 115 shown is basically the same as Figure 3A The prediction module 115 shown is the same. However, Figure 3B The prediction module 115 shown in does not generate the residual 15 based on the key frame. Instead, Figure 3B The residual generation module 330 of the prediction module 115 shown in FIG. 1 generates a residual 15 using the 3D object and the translated and oriented stored 3D object.

[0102] In an example embodiment, the translated and oriented stored 3D object has a direct point-to-point relationship with the 3D object. In other words, a point in the triangular mesh defining the translated and oriented stored 3D object (e.g., with position attributes of x1, y1, z1) has a corresponding point in the triangular mesh defining the 3D object (e.g., with position attributes of x1, y1, z1). Therefore, the color attribute can be used to determine the relative color displacement between the 3D object and the stored 3D object. Therefore, the residual generation module 330 can be configured to generate (or calculate) the residual 15 (e.g., color displacement) of the 3D object in the frame 110 based on the stored object. For example, the residual generation module 330 can generate the residual 15 by subtracting the pixel attribute value for each point in the triangular mesh of the 3D object from the pixel attribute value of the corresponding point in the triangular mesh of the translated and oriented stored 3D object.

[0103] In an example embodiment, a residual may be generated using a translated 3D object proxy (e.g., the first 3D object proxy described above) based on a stored 3D object for a 3D object associated with frame 110 that has been mapped with the colors of frame 110 and a stored 3D object having default color attributes. Thus, color attributes of points in the mesh representation of the first 3D object proxy may be subtracted from color attributes of corresponding points in the stored mesh representation of the 3D object (e.g., having the same point identity or being at the same position in the mesh sequence).

[0104] Figure 4A A block diagram of a decoder prediction module according to an example implementation is shown. Figure 4A As shown, the prediction module 155 includes a frame 405 , a metadata module 410 , a 3D object matching module 415 , a stored 3D object 135 , a stored 3D object translation module 420 , a key frame 320 , a block matching module 325 , and a pixel regeneration module 425 .

[0105] Frame 405 is a compressed frame selected from compressed frames 140 as a frame to be decompressed. Key frame 320 is a decompressed key frame associated with (near, earlier in time, and / or later in time) frame 405. Frame 405 has been entropy decoded, inverse quantized, and inverse transformed by decoding module 160. Therefore, frame 405 may include a derivative residual that may be the same (or approximately the same) as a residual generated by prediction module 115.

[0106] The metadata module 410 may be configured to determine whether the frame 405 includes metadata and / or has associated metadata (e.g., metadata 130). The metadata module 410 may be configured to read the metadata from a header associated with the frame 405. The metadata may include metadata 20. The metadata may include data associated with at least one 3D object located in the frame 405 that has been predicted using one of the stored 3D objects 135. The metadata may include properties associated with the location and / or orientation of the 3D object in the frame 405 (e.g., grid point properties). The metadata may include information identifying the stored 3D object and information related to the translation and orientation of the stored 3D object. The information related to the translation and orientation of the stored 3D object may be used to perform the same translation and orientation of the stored 3D object as performed by the stored 3D object translation module 315.

[0107] The 3D object matching module 415 may be configured to match a 3D object (hereinafter, referred to as a 3D object) located in the frame 405 with one of the stored 3D objects 135. In an example embodiment, the metadata module 410 outputs a unique ID or tag that may be used to search for the 3D object in the stored 3D objects 135. If the unique ID or tag is found in the stored 3D objects 135, the corresponding one of the stored 3D objects 135 (hereinafter, referred to as a stored 3D object) is a match of the 3D object.

[0108] The stored 3D object translation module 420 may be configured to translate the stored 3D object. For example, before compressing the frame 405, when the 3D object is oriented in the frame 405, the mesh corresponding to the stored 3D object may be translated and oriented to align with the orientation of the 3D object. In other words, when the 3D object is oriented in the frame 110, the mesh corresponding to the stored 3D object may be translated and oriented to align with the orientation of the 3D object. The stored 3D object translation module 420 may be configured to translate the stored 3D object based on information related to the translation and orientation of the stored 3D object included in the metadata.

[0109] The stored 3D object translation module 420 may be configured to translate the matched one of the stored 3D objects 135 associated with the frame 405 and the key frame 320. Thus, the stored 3D object translation module 315 may be configured to generate a first 3D object proxy based on the stored 3D object for the 3D object associated with the frame 405. Furthermore, the stored 3D object translation module 315 may be configured to generate a second 3D object proxy based on the stored 3D object for the 3D object associated with the key frame 320. In an example embodiment, the second 3D object proxy may be generated once per key frame 320 and then used for each associated frame 405.

[0110] The block matching module 325 may be configured to match points of the translated and oriented stored 3D object of the frame 405 corresponding to the translated and oriented stored 3D object of the key frame 320. In an example embodiment, the block matching module 325 may be configured to map color attributes from the 3D object identified in the frame 110 to the transformed first 3D object proxy. In addition, the block matching module 325 may be configured to map color attributes from the 3D object identified in the key frame 320 to the transformed second 3D object proxy. Mapping the color attributes from the 3D object identified in the frame 110 to the transformed first 3D object proxy may include converting the 3D object from a 3D space (e.g., XYZ space) to a 2D space (e.g., UV space) and / or converting the 3D object identified in the frame 110 from a 2D space (e.g., UV space) to a 3D space (e.g., XYZ space).

[0111] In an example implementation, mapping color attributes may include identifying pixels in a frame (e.g., frame 405 and / or key frame 320) based on pixel coordinates in the frame, and then setting color attributes of points with the same coordinates in the mesh representation of the transformed 3D object proxy to the same color value of the identified pixels. In some cases, frame 405 and / or key frame 320 may not have pixels corresponding to points on the mesh representation of the transformed 3D object proxy. Therefore, the mesh representation of the transformed 3D object proxy may retain default color attributes for these points. Therefore, any residual color calculations may be based on the default colors.

[0112] The block matching module 325 may then use the matching points of the corresponding 3D object to select or predict pixels / blocks / patches in the key frame 320 for regenerating the pixels of the 3D object in the frame 405. The pixel regeneration module 425 may be configured to generate (or calculate) pixel values ​​for the 3D object 25 in the frame 405. For example, the pixel regeneration module 425 may generate pixel values ​​by adding pixel attribute values ​​for each point in the triangle mesh of the 3D object from the pixel attribute values ​​for each matching point for the predicted pixel / block / patch in the key frame 320 to the color values ​​and / or color attributes for the translated and oriented stored 3D object based on the residual at the identified position in the selected frame.

[0113] In an example embodiment, the 3D object, stored 3D object, and / or 3D object proxy has been encoded using an autoencoder. Therefore, before translation and matching, the 3D object, stored 3D object, and / or 3D object proxy may be decoded using an autoencoder (described in more detail below). Decoding the stored 3D object or 3D object proxy converts the latent representation of the 3D object, stored 3D object, and / or 3D object proxy into a regenerated mesh representation.

[0114] The pixel regeneration module 425 may be further configured to regenerate (eg, calculate) color values ​​and / or color attributes of the remaining portion of the selected frame based on the residual or remaining portion of the selected frame and corresponding pixels / blocks / patches of the key frame.

[0115] Figure 4B A block diagram of another decoder prediction module according to an example embodiment is shown. Figure 4B As shown, the prediction module 155 includes a frame 405 , a metadata module 410 , a 3D object matching module 415 , a stored 3D object 135 , a stored 3D object translation module 420 , a key frame 320 , a block matching module 325 , and a pixel regeneration module 425 .

[0116] Figure 4B The implementation of the prediction module 155 shown is essentially the same as Figure 4A The prediction module 155 shown is the same. However, Figure 4B The prediction module 155 shown in FIG. 1 does not regenerate the color values ​​of the 3D object based on the key frames. Instead, Figure 4B The pixel regeneration module 425 of the prediction module 155 shown in FIG. 4 regenerates the color value of the 3D object using the 3D object and the translated and oriented stored 3D object.

[0117] In an example embodiment, the translated and oriented stored 3D object has a direct point-to-point relationship with the 3D object. In other words, a point in the triangular mesh defining the translated and oriented stored 3D object (e.g., with position attributes of x1, y1, z1) has a corresponding point in the triangular mesh defining the 3D object (e.g., with position attributes of x1, y1, z1). Therefore, the color attributes of the translated and oriented stored 3D object can be used to determine the color value of the 3D object.

[0118] Therefore, the pixel regeneration module 425 can be configured to generate (or calculate) color values ​​for the 3D object in the frame 405 based on the stored object. For example, the pixel regeneration module 425 can regenerate the color value by adding the pixel value of each point in the triangular mesh of the 3D object read from the frame 405 to the color attribute value of the corresponding point in the triangular mesh of the stored 3D object that has been translated and oriented. The stored 3D object that has been translated and oriented with the calculated color attribute value is then output as the 3D object 25 that is the regenerated 3D object. As described above with reference to Figure 4A As described above, the remainder of frame 30 is regenerated.

[0119] As described above, each stored 3D object 135 can be defined by a triangular mesh. A triangular mesh can be a collection of points connected by triangular faces. Each point can store various attributes. For example, the attributes can include the position, color, texture coordinates, etc. of each point. Although a triangular mesh structure is referenced above, other polygons can also be used to define the stored 3D objects 135. In addition, the attributes of each point may include other attributes that may or may not be useful in the context of the present disclosure.

[0120] In an example embodiment, the stored 3D objects 135 may include 3D objects of interest for a CGI movie. Therefore, the stored 3D objects 135 may include 3D models or mesh data and corresponding attribute values ​​for a large number of 3D characters. Generally, 3D characters may include CGI actors (e.g., protagonists, supporting characters, and other characters), CGI pets, CGI creatures, CGI monsters, etc. As described above, each of these 3D characters may be a 3D object of interest stored as a 3D model or mesh data stored in the stored 3D objects 135. Each 3D character may include a unique ID or tag.

[0121] In some embodiments, a portion of a 3D object of interest may be stored as a 3D model or mesh data stored in the stored 3D object 135. Each of the portions of a 3D character may include a unique ID or tag. For example, at least one 3D character (e.g., a protagonist and a supporting character) may have an associated 3D model or mesh data representing at least one 3D character head or face, and another associated 3D model or mesh data representing at least one 3D character body (note that the head, face, or body may also be divided into smaller portions). By dividing the character into multiple portions, a relatively higher level of detail may be stored for more dynamic portions of the character (e.g., head, arms, and / or legs) compared to relatively less dynamic portions of the character (e.g., torso and / or shoulders).

[0122] In addition, the mesh data can be configured to fit a standard 3D model. For example, the first cubic model can be a standard size intended to fit a head model, while the second cubic model can be a standard size intended to fit a hand model. Other standard shaped 3D models may include spheres, rectangular prisms, rectangular prisms, pyramids, cylinders, etc. For example, an arm or leg may use a rectangular prism 3D model or a cylindrical 3D model. Standard 3D models can also set limits (maximum and / or minimum) on multiple points (e.g., vertices) used to define a 3D object.

[0123] Dividing the parts of the character and storing each part as a unique 3D object of interest in the stored 3D object 135 can cause the size of the stored 3D object 135 to grow exponentially. However, a 3D model or mesh data representing a part of a character can be used for many characters. For example, a torso model representing the torso of character A can also be used for character B. The stored 3D object 135 can also include metadata identifying the 3D model or mesh data representing a standard part of the character and some deformation information corresponding to the standard part for the character. Therefore, the 3D model or mesh data represents a standard part. For example, a 3D model representing the torso can be used as a 3D model to represent the torso of character A as a tall, thin man, and as a 3D model to represent the torso of character B as a short, fat man.

[0124] As the number of stored 3D objects 135 increases (e.g., for a full-length CGI animated film with many characters using a large number of 3D standard models), the amount of resources (e.g., memory) required to store the stored 3D objects 135 also increases. For example, a streaming server may store 100,000, 1 million, and millions of videos. As the demand for videos including 3D videos increases (e.g., in virtual reality, augmented reality, 3D CGI movies, etc.), the percentage of videos stored at the streaming server (usually stored as left-eye and right-eye 2D videos) will certainly increase. Therefore, when implementing the technology described herein, it is used for use with the growing number of videos. The number of stored 3D objects 135 is bound to increase, and therefore the amount of resources (e.g., memory) required to store the stored 3D objects 135 will also increase. In addition, during the streaming activity, it may require a large amount of bandwidth to transmit the stored 3D objects 135 from the streaming server to the client device. Therefore, for the streaming operation, it may become desirable to efficiently encode and decode the stored 3D objects 135.

[0125] Figure 5A FIG. 1 is a block diagram showing a signal flow for encoding a 3D object according to an example embodiment. Figure 5A As shown, the encoded 3D object may include at least the stored 3D object 135, a neural network encoder 505, and a potential representation 510 of the 3D object.

[0126] As described above, each stored 3D object 135 may be defined by a triangular mesh. A triangular mesh may be a collection of points or vertices connected by triangular faces. Each point may store various attributes. In an example embodiment, the number of points or vertices may be limited to a certain value.

[0127] The neural network encoder 505 may use generative modeling techniques to compress 3D objects. Example implementations of generative modeling techniques may include variational and / or convolutional autoencoders (VAEs), generative adversarial networks (GANs), and / or combined VAE-GANs. Although these generative modeling techniques are mentioned and / or discussed, example implementations are not limited thereto. Furthermore, although a neural network encoder is discussed as an implementation for compressing 3D objects, other implementations of encoding 3D objects (including triangular meshes) are also within the scope of the present disclosure.

[0128] The neural network encoder 505 can compress a 3D object defined by a mesh having a fixed number of vertices and fixed connectivity into a relatively small number of variables sometimes referred to as a latent space. For example, the VAE can be configured to learn a compact latent space for a 3D shape (e.g., a 3D shape on which the VAE has been trained). Both the neural network of the neural network encoder 505 and the neural network of the neural network decoder 515 can be machine-trained neural networks and / or machine-trained convolutional neural networks.

[0129] The neural network of the neural network encoder 505 may include coefficients associated with one or more convolutions and / or filters applied to a grid in order to encode the grid. Each convolution may have C filters, a K×K mask (K represents convolution), and a step factor. The coefficients may correspond to one or more of the C filters, the KxK mask, and the step factor. For example, the KxK mask may be a 3x3 mask. The 3x3 mask contains nine (9) variables for performing convolution on the grid. In other words, the 3x3 mask has nine (9) blocks, each containing variables. Each variable may be one of the coefficients.

[0130] The neural network of the neural network encoder 505 may include layers with different numbers of neurons. The KxK spatial range may include K columns and K (or L) rows. The KxK spatial range may be 2x2, 3x3, 4x4, 5x5, (KxL)2x4, etc. The convolution includes centering the KxK spatial range at a grid point and convolving all grid points in the spatial range, and generating a new value for the grid point based on all convolutions (e.g., sum) of all grid points in the spatial range. The spatial range is then moved to a new grid point based on a step size, and the convolution is repeated for the new grid point. The step size may be, for example, one (1) or two (2), where a step size of 1 moves to the next grid point and a step size of 2 skips a grid point.

[0131] The VAE may use the position coordinates for each point of the grid (e.g., a relatively large amount of data for most 3D shapes of any interest or with a visually significant amount of detail) as input to the neural network encoder 505, and generate a reduced (preferably relatively small) number of variables (e.g., from 64 to 128 variables) by convolving the neural network with the position coordinates for each point of the grid. The configuration of the C filters, KxK masks, and stride factors for the neural network may determine the number of variables in the latent space for the latent representation of the 3D object 510.

[0132] Figure 5B FIG. 1 is a block diagram showing a signal flow for decoding a 3D object according to an example embodiment. Figure 5B As shown, decoding a 3D object may include at least a potential representation 510 for the 3D object, a neural network decoder 515, and a stored 3D object 135. The potential representation 510 of the 3D object may be a potential representation 510 of the 3D object generated by encoding the stored 3D object 135 using a neural network encoder 505.

[0133] The neural network of the neural network decoder 515 may include coefficients associated with one or more convolutions and / or filters applied to a grid in order to encode the grid. Each convolution may have C filters, a K×K mask (K represents convolution), and a step factor. The coefficients may correspond to one or more of the C filters, the KxK mask, and the step factor. For example, the KxK mask may be a 3x3 mask. The 3x3 mask contains nine (9) variables for performing convolution on the grid. In other words, the 3x3 mask has nine (9) blocks, each containing variables. Each variable may be one of the coefficients.

[0134] The neural network of the neural network decoder 515 may include layers with different numbers of neurons. The KxK spatial range may include K columns and K (or L) rows. The KxK spatial range may be 2x2, 3x3, 4x4, 5x5, (KxL)2x4, etc. The convolution includes centering the KxK spatial range at a grid point and convolving all grid points in the spatial range, and generating a new value for the grid point based on all convolutions (e.g., sum) of all grid points in the spatial range. The spatial range is then moved to a new grid point based on upsampling, and the convolution is repeated for the new grid point. The upsampling factor may be, for example, one (1) or two (2), where a step factor of 1 moves to the next grid point and a step factor of 2 skips a grid point.

[0135] The convolution may include one or more zero-padding convolution operations and coefficient reorganization. In an example embodiment, the zero-padding includes padding zeros between non-zero pixels, and the coefficient reorganization may include convolution with a KxK mask rotated 180 degrees around the zero-padding pixels.

[0136] The VAE can use the variables in the latent space for the latent representation 510 of the 3D object as input to the neural network decoder 515 and regenerate the position coordinates for each point of the grid. The configuration of the C filters, KxK mask and step factor of the neural network can determine the number of grid points regenerated.

[0137] Thus, the neural network decoder 515 may be configured to use the variables for a corresponding one of the latent representations 510 for the 3D object to reproduce an approximation of the shape and / or model of the 3D object prior to compression of the 3D object by the neural network encoder 505. The neural network decoder 515 may be configured to generate an approximation of the 3D model corresponding to one of the stored 3D objects 135, including generating position coordinates of points in a mesh having the same number of vertices and connectivity as one of the stored 3D objects 135 prior to compression by the neural network encoder 505.

[0138] According to an example embodiment, each stored 3D object 135 may be defined by a grid having the same number of points, each point having a position coordinate. In addition, the VAE may include a neural network encoder 505 and a neural network decoder 515, each including a neural network having the same configuration of C filters, KxK masks, and step factors for the neural network. As a result, the number of variables in the latent space for each of the latent representations 510 for the 3D objects generated by the neural network encoder 505 is the same. In addition, the number of points in the grid regenerated by the neural network decoder 515 is the same.

[0139] In the above example implementation, the stored 3D object 135 may include a 3D object of interest of a CGI movie. Therefore, the stored 3D object 135 is likely to include 3D models or mesh data and corresponding attribute values ​​for a large number of 3D characters. As described above, the 3D model may represent the entire 3D object of interest and / or a portion of the 3D object of interest. A standard 3D model may be used to represent the entire 3D object of interest and / or a portion of the 3D object of interest.

[0140] In this example embodiment, the stored 3D objects 135 correspond to 3D objects of interest in the CGI movie (e.g., as video data 5), ​​and may not include any other 3D objects not included in the CGI movie. Typically, supervised machine learning methods learn one or more rules or functions to map between example inputs and desired outputs as predetermined by an operator. Supervised machine learning methods and semi-supervised learning methods can use labeled data sets. Therefore, machine training or training the neural network encoder 505 and the neural network decoder 515 can be accomplished using a supervised machine learning method or a semi-supervised learning method using the stored 3D objects 135 (e.g., each 3D object is labeled with a unique ID and / or label) as input data and comparison data.

[0141] Many training iterations used during the training process can produce an approximate logarithmic gain in the accuracy of the reconstructed mesh. Therefore, in a semi-supervised learning method, a threshold based on time, error (e.g., loss), and / or number of iterations can be used to stop further training. For example, a threshold can be set to a reconstruction error or loss based on the number of points and position coordinates of the reconstructed mesh. In an example embodiment, the reconstruction error can be a loss based on a shape reconstruction loss (Lr) and a regularized prior loss (Lp).

[0142] The shape reconstruction loss (Lr) can be based on obtaining a nearly identical transformation between the neural network encoder 505 and the neural network decoder 515. The shape reconstruction loss (Lr) can be based on obtaining a nearly identical transformation between the neural network encoder 505 and the neural network decoder 515. Figure 5B The stored 3D object 135 generated in Figure 5A The regularized prior loss (Lp) can be used on the latent variables based on Figure 5A The VAE loss or total loss is calculated based on the change of the vector or the previous iteration distribution of the stored 3D object 135, and the divergence algorithm is further based on the distribution of the latent variables. The VAE loss or total loss becomes L=Lr+λLp, where λ≥0, and controls the similarity of the change distribution to the previous change distribution.

[0143] In an example embodiment, the training process includes iterating through the following steps: Figure 5A The signal flow for encoding a 3D object shown in FIG. Figure 5B The signal flow for decoding 3D objects shown is used to train the neural network encoder 505 and the neural network decoder 515. Continuing with the above example, the stored 3D objects 135 correspond to the 3D objects of interest of the CGI movie. After each iteration (e.g., encoding and decoding each stored 3D object 135), Figure 5B The stored 3D object 135 (eg, as a reconstructed 3D object) generated in Figure 5AThe stored 3D object 135 is compared. In an example embodiment, the reconstruction error can be calculated based on a loss based on a shape reconstruction loss (Lr) and a regularized prior loss (Lp). If the reconstruction error is higher than a threshold (as described above), the variables used for the coefficients corresponding to one or more of the C filters, KxK masks, and step factors for at least one of the neural network encoder 505 and the neural network decoder 515 can be corrected. In addition, the variable λ associated with calculating the VAE loss can be corrected. In an example embodiment, the variables can be corrected based on the variable values ​​and results for the previous iteration and the variable values ​​and results for the current correction. After the variables are corrected, the next training iteration can be started.

[0144] In an example embodiment, streaming and / or downloading the video may include transmitting initialization data for use by the requesting client device (eg, for decompressing the compressed frames) prior to transmitting or streaming the compressed frames of the video. Fig. 6A A block diagram of the signal flow for streaming video and rendering the video on a client device is shown. Fig. 6A As shown, the streaming server 605-1 includes a video storage 125, an active streaming module 610, a streaming initialization module 615, and a transceiver 620. The video storage 125 includes compressed frames 140, metadata 130, and a potential representation 510 for a 3D object. The streaming server 605-1 can be communicatively coupled to a client device 650 via a transceiver 625. Fig. 6A As shown, client device 650 includes transceiver 625 , decoder 145 , renderer 630 , color correction module 635 , and display 640 .

[0145] The active streaming module 610 may be configured to stream the selected frames of the video to a requesting client device (e.g., client device 650). The active streaming module 610 may receive a request for a next frame from the client device 650. The active streaming module 610 may then select a next frame from the compressed frames 140 and select corresponding metadata from the metadata 130. In an example embodiment, the selected next frame may be a plurality of compressed frames, and the selected metadata may be a plurality of corresponding metadata elements. For example, the plurality of compressed frames may be a plurality of frames bounded by (and possibly including) a previous (e.g., temporally, before the frame) key frame and a future (e.g., temporally, after the frame) key frame.

[0146] The active streaming module 610 may then transmit the selected next frame (or frames) and the selected metadata (or metadata elements) to the transceiver 620. The transceiver 620 may then construct one or more data packets including the selected next frame and the selected metadata, assign the address of the client device 650 to the data packets, and transmit the data packets to the client device 650.

[0147] The streaming initialization module 615 may be configured to select data associated with a streaming video in response to a first request for a video by a client device 650. For example, a user of the client device 650 may download the video for future playback and / or begin streaming playback. The streaming initialization module 615 may select a stored 3D object 135 for the video as the set of potential representations 510 for 3D objects and select metadata corresponding to the video from the metadata 130. The streaming initialization module 615 may then transmit the selected stored 3D object 135 and the selected metadata to the transceiver 620. The transceiver 620 may then construct a data packet including the selected stored 3D object 135 and the selected metadata, assign an address of the client device 650 to the data packet, and transmit the data packet to the client device 650. In an example embodiment, the transceiver 620 may construct more than one data packet.

[0148] The client device 650 receives a data packet including the stored 3D object 135 and metadata via the transceiver 625. The transceiver 625 transmits the stored 3D object 135 and metadata to the decoder 145. The decoder 145 may be configured to store the stored 3D object 135 and metadata in association with the requested video. Then, the client device 650 receives a data packet including the selected next frame and the selected metadata via the transceiver 625. The selected next frame may be the first frame at the time of initial playback. The transceiver 625 transmits the selected next frame and the selected metadata to the decoder 145.

[0149] The decoder 145 then decodes (as described in more detail above) the next frame using the stored 3D object 135 and metadata associated with the next frame. In an example embodiment, the decoder 145 may be implemented as a graphics card and / or chip (e.g., an ASIC on a computer motherboard) including a graphics processing unit (GPU) or implemented in a graphics card and / or chip (e.g., an ASIC on a computer motherboard) including a graphics processing unit (GPU), configured to remove the burden from the central processing unit (CPU) of the device 650. The GPU may be configured to process large blocks of video data in parallel. The GPU may be configured to process (e.g., decompress) mesh data and generate pixel data from the mesh data. The decoder transmits texture data and color data to the renderer 630.

[0150] In 3D rendering systems, a Cartesian coordinate system is commonly used (x-(left / right), y-(up / down), and z-axis (near / far)). The Cartesian coordinate system provides an accurate mathematical method to position and represent objects in space. To simplify, the image of a frame can be thought of as the space where the object will be. Space can be oriented based on the camera position. For example, a camera can be placed at the origin and looking straight down the z-axis. Therefore, the translational motion (relative to the camera position) is z forward / backward, y up / down, and x left / right. The object is then projected into space based on its coordinates relative to the origin of space and any camera repositioning. Note that the object and / or the camera can move from frame to frame.

[0151] Renderer 630 can be configured to render texture data and color data for display. Rendering or drawing can be implemented in the graphics pipeline of the geometry stage and the rendering stage. The GPU can be configured to process the geometry stage and the rendering stage. The geometry stage performed by the CPU or GPU is configured to process all polygon activities and convert 3D space data into pixels. The geometry stage process can include but is not limited to scene (e.g., background) geometry generation, object motion based on camera motion, object motion, object transformation (e.g., rotation, translation and / or scaling), object visibility (e.g., occlusion and culling) and generation of polygons (e.g., triangles).

[0152] The rendering phase, performed by the 3D hardware accelerator of the GPU, is configured to manage memory and pixel activity, and process pixels to be drawn to the display 640. The rendering phase processes may include, but are not limited to, shading, texturing, depth buffering, and display.

[0153] Rendering texture data and color data for display may include using a 3D shader (e.g., a vertex shader, a geometry shader, etc.) to draw a mesh associated with a frame. The shader may be configured to generate primitives. The shader may be configured to transform the 3D position and texture of each vertex in the mesh into 2D coordinates (e.g., primitives) that appear on a display (e.g., display 640). Rendering texture data and color data for display may also include performing rasterization. Rasterization may include assigning pixel (e.g., color) values ​​to primitives based on the texture data and color data.

[0154] As described above, the geometry phase includes building a 3D grid in a 2D coordinate system, while the rendering phase includes adding color and texture to the grid. Therefore, the output of the decoder 145 may include object position data and color / texture data. Therefore, the compressed frame 140 may include compressed object position data and compressed color / texture data. The compressed object position data and compressed color / texture data may be based on displacement with the previous frame or the next frame, based on camera translation motion data, based on absolute data (e.g., x, y, z coordinates, RGB values ​​and / or the like) and / or combinations or variations thereof. Therefore, the encoder 105 may encode the scene of the frame to generate a compressed frame 140 including data having the compressed object position data and compressed color / texture data.

[0155] The data associated with the rendered next frame may be transmitted to the color correction module 635. The color correction module 635 may be configured to color correct the data associated with the rendered next frame. The color correction may include compensating for color differences between frames, compensating for color differences between multiple views of the same scene, correcting for object deformation (distortion), correcting for object boundary deformation, etc.

[0156] The color correction module 635 can transmit the color correction frame to the display 640. The display 640 can be configured to display the frame as a frame sequence representing the requested video. In an example embodiment, the display 640 includes and / or has an associated buffer and / or queue. The buffer and / or queue can be a first-in / first-out buffer and / or queue. Therefore, the color correction module 635 can transmit the color correction frame to the buffer and / or queue. The display 640 includes a component configured to select the next frame from the buffer and / or queue.

[0157] In an example implementation, streaming and / or downloading video may include transmitting data for use by a requesting client device (eg, for decompressing the compressed frames) in parallel (eg, substantially simultaneously) with transmitting or streaming compressed frames of the video. Figure 6B A block diagram of another signal flow for streaming video and rendering the video on a client device is shown. Figure 6B As shown, the streaming server 605 - 2 includes the video storage 125 , an active streaming module 610 , and a transceiver 620 .

[0158] exist Figure 6BIn the illustrated example implementation, the active streaming module 610 is further configured to select at least one 3D object of interest from the latent representations 510 for 3D objects to be used in compressing the selected next frame as at least one of the stored 3D objects 135. The active streaming module 610 transmits the at least one of the stored 3D objects 135 together with the selected next frame and the selected metadata to the transceiver 620. The transceiver 620 may then construct a data packet including at least one of the selected next frame, the selected metadata, and the stored 3D objects 135, assign an address of the client device 650 to the data packet, and transmit the data packet to the client device 650.

[0159] The client device 650 then receives a data packet including at least one of the selected next frame, the selected metadata, and the stored 3D object 135 via the transceiver 625. The transceiver 625 transmits at least one of the selected next frame, the selected metadata, and the stored 3D object 135 to the decoder 145. The decoder 145 may be further configured to add at least one of the stored 3D objects 135 to the stored 3D objects 135 associated with the requested video and / or use at least one of the stored 3D objects 135 to initialize the stored 3D objects 135 for the requested video.

[0160] Furthermore, although color prediction using the geometry proxy technique described herein may be most advantageous when encoding / decoding 3D objects in a video that have dynamic non-linear and / or random motion from frame to frame (sometimes referred to herein as dynamic 3D objects), other 3D objects or non-dynamic 3D objects may also be advantageously encoded / decoded using color prediction using the geometry proxy technique.

[0161] For example, a 3D object that appears to move from frame to frame due to camera or scene translation can be encoded / decoded using color prediction through a geometric proxy. In this example, a stationary 3D object may appear to be moving from frame to frame because the camera capturing the scene is moving (e.g., in a predictable manner and / or direction). For example, a 3D object that appears to move in a predictable manner within a scene can be encoded / decoded using color prediction through a geometric proxy. In this example, a 3D object (e.g., a vehicle, i.e., a train, car, or airplane) can move from frame to frame in a predictable manner (e.g., at a constant speed and / or direction). These objects are sometimes referred to herein as translational 3D objects. As another example, a 3D object that appears to not move within a scene can be encoded / decoded using color prediction through a geometric proxy. In this example, a stationary or fixed 3D object (e.g., the background of a scene, furniture at a fixed position within a scene, or an object moving slowly in the distance) may appear stationary from frame to frame (e.g., without any camera or scene translation). These objects are sometimes referred to herein as fixed or background 3D objects.

[0162] For a variety of reasons, other types of 3D objects exemplified above may be advantageously encoded / decoded using color prediction via geometry proxy techniques. For example (this is not intended to be an exhaustive list), at least one position (e.g., position in a frame) of a panning 3D object, a fixed 3D object, and / or a background 3D object may be transmitted from a streaming server (e.g., streaming servers 605-1, 605-2) to a client device (e.g., client device 650) to be used in a renderer and / or rendering operation; the number of frames between key frames may be increased due to the presence of a geometry proxy (e.g., stored 3D object 135); Z-order layering techniques may be used for dynamic 3D objects and non-dynamic 3D objects; lost (e.g., not retransmitted) frames may be recreated using temporally earlier and temporally later frames (e.g., stored in a queue before being rendered) and geometry proxies; 3D objects that appear and disappear between key frames may be encoded / decoded using a geometry proxy and / or out-of-frame background camera or scene translation may be encoded / decoded using a geometry proxy.

[0163] Thus, the metadata 20 described above may be used in the renderer 630 to more efficiently render the translational 3D objects, the fixed 3D objects, and / or the background 3D objects. In an example embodiment, the first object may be identified as a background 3D object. The metadata 20 may identify the first object as, for example, one of the stored 3D objects 135. The metadata 20 may also identify the origin coordinates (e.g., x0, y0, z0) of the first object (e.g., as the background of the frame) as a position attribute of one of the points in the grid representing the first object.

[0164] In addition, the at least one second object may be identified as a translating 3D object and / or a fixed 3D object. The metadata 20 may identify the at least one second object as, for example, one of the stored 3D objects 135. The metadata 20 may also identify the position (e.g., x) of the at least one second object. n ,y n ,z n ) is identified as a position attribute of one of the points in the grid representing the at least one second object. The identified position of the at least one second object may be relative to another object. The relative position of the at least one second object may vary from frame to frame (e.g., as a translating 3D object). The relative position of the at least one second object may be fixed from frame to frame (e.g., as a fixed 3D object).

[0165] For example, the identified position of the at least one second object can be relative to the identified origin coordinates for the background (e.g., the first object) of the frame. Using relative positions for the at least one second object can allow six degrees of freedom when positioning the at least one second object relative to the background of the frame. In other words, when positioning the at least one second object relative to the background of the frame, the at least one second object can have translational motion (e.g., forward / backward, up / down, and / or left / right) and rotational motion (e.g., pitch, roll, and / or yaw).

[0166] As described above, the first object may be identified as a background 3D object and may be one of the stored 3D objects 135. In addition, at least one second object may be identified as a translating 3D object and / or a fixed 3D object, which may also be one of the stored 3D objects 135. Thus, the first object and / or the at least one second object may be encoded by an autoencoder (implementing a neural network) into a latent representation of a corresponding stored 3D object 135 (as described in more detail above). The latent representation (e.g., as one or more latent representations 510 for the 3D object) and information (e.g., metadata) related to the structure of the neural network may be transmitted from the streaming servers 605-1, 605-2 to the client 650. The client 650 may reconstruct one or more corresponding stored 3D objects 135 (as discussed in more detail above) using a decoder of the autoencoder (implementing a neural network).

[0167] Figures 7 to 11 is a flow chart of a method according to an example embodiment. Figures 7 to 11 The steps described may be performed in conjunction with an apparatus (e.g., Fig.12 and 13 The device is implemented by software code stored in a memory associated with the device (e.g., at least one memory 1210, 1310) or executed by at least one processor associated with the device (e.g., at least one processor 1205, 1305).

[0168] However, alternative embodiments can be envisioned, such as systems embodied as dedicated processors. The dedicated processor can be a graphics processing unit (GPU). The GPU can be a component of a graphics card. The graphics card can also include video memory, a random access memory digital-to-analog converter (RAMDAC), and driver software. The video memory can be a frame buffer that stores digital data representing a scene of an image or frame. The RAMDAC can be configured to read the contents of the video memory, convert the contents to analog RGB signals, and then send the analog signals to a display or monitor. The driver software can be software code stored in the above-mentioned memory (e.g., at least one memory 1210, 1310). The software code can be configured to implement the steps described below (and / or the components, modules, and signal flows described above).

[0169] Although the steps described below are described as being performed by a processor and / or a dedicated processor, these steps are not necessarily performed by the same processor. In other words, at least one processor and / or at least one dedicated processor may perform the steps described below with reference to Figures 7 to 11 The steps described.

[0170] Figure 7 A block diagram of a method for compressing frames of a video according to at least one example embodiment is shown. Figure 7 As shown, in step S705, a file including multiple frames of a video is received. For example, the file can be saved or transmitted to a server (e.g., a streaming server). The file can include a video. The video can be a CGI 3D movie. The file can include multiple 3D objects of interest (e.g., characters in a 3D movie).

[0171] In an example embodiment, each of the multiple 3D objects of interest may be defined by a triangular mesh. A triangular mesh may be a collection of points connected by triangular faces. Each point may store various attributes. For example, the attributes may include the position, color, texture coordinates, etc. of each point. The mesh of each of the multiple 3D objects of interest may have the same number of points, each point having the same attributes. Therefore, when stored in a memory, the mesh of each of the multiple 3D objects of interest may be approximately the same size (e.g., number of bits).

[0172] In step S710, one of the plurality of frames is selected. For example, each of the plurality of frames may be a compression target. The plurality of frames may be compressed in time sequence. Therefore, in an initial step, the first frame in time is selected. Then the next frame may be selected sequentially.

[0173] In step S715, 3D objects are identified in the selected frames and key frames. The identified 3D objects may be dynamic 3D objects, non-dynamic 3D objects, fixed 3D objects, background 3D objects, etc. For example, machine vision, computer vision and / or computer image recognition technology may be used to identify and locate 3D objects.

[0174] In an example embodiment, computer image recognition techniques based on training (machine learning) a convolutional neural network using multiple known images can be used to identify 3D objects. For example, a block, multiple blocks, and / or patch are selected and / or identified from a selected frame. The trained convolutional neural network can operate on the selected block, multiple blocks, and / or patch. The results can be tested (e.g., error test, loss test, divergence test, etc.). If the test produces a value below a threshold (or alternatively, a value above a threshold depending on the type of test), the selected block, multiple blocks, and / or patch can be identified as a 3D object.

[0175] In an example embodiment, a frame of a video may include a tag indicating that a previously identified 3D object of interest is included in the frame. The tag may include the identity and location of the 3D object. For example, a video may be generated using a computer generated image (CGI) tool (e.g., a computer animated film). A computer generated character may be identified and labeled in each frame. In addition, a model of each of the identified 3D objects of interest (e.g., the identified character) may be stored as a stored 3D object 135.

[0176] In an example implementation, a 3D object may be defined by a triangular mesh. A triangular mesh may be a collection of points connected by triangular faces. Each point may store various attributes. For example, the attributes may include the position, color, texture coordinates, etc. of each point. The attributes may include and / or indicate (e.g., multiple attributes may indicate) the orientation of the corresponding 3D object and / or the position of the corresponding 3D object in the selected frame.

[0177] Thus, in an example embodiment, the mesh properties of a 3D object may be sufficient to identify and locate the 3D object. A model including mesh properties for a plurality of 3D objects of interest may be stored as a stored 3D object 135. The model may be standardized. For example, a model for a man, a woman, a teenager, a child, or more generally a human or a portion of a human (e.g., body, head, hand, etc.) may be stored as a stored 3D object 135. For example, a model for a dog, a cat, a deer, or more generally a quadruped or a portion of a quadruped (e.g., body, head, leg, etc.) may be stored as a stored 3D object 135. The properties of the model may then be used to search for 3D objects with similar properties in a frame.

[0178] In step S720, the position and orientation of the 3D object within the selected frame and keyframe are determined. For example, the 3D object can be defined by a grid having a plurality of points, each point having at least one attribute. The position of the 3D object can be the position of the 3D object within the frame. Therefore, the position of the 3D object within the frame can be based on a 3D Cartesian coordinate system for the frame. For example, at least one point of the grid can be located at an x, y, z position within the frame. The orientation of the 3D object can be based on the position coordinate attributes of each point in the grid used to define the 3D object. If the 3D object is a moving character, the orientation of the 3D object can be the posture of the character in the frame and / or keyframe.

[0179] In step S725, the 3D object is matched with the stored 3D objects. For example, the 3D object may be matched with one of the 3D objects of interest (e.g., the stored 3D objects 135). In an example embodiment, the identity of the 3D object, such as generated by computer image recognition techniques, may be used to search for the stored 3D objects 135. In an example embodiment, the label of the 3D object found in the frame may be matched with the label of one of the stored 3D objects 135. In an example embodiment, one of the stored 3D objects 135 and a model having similar properties to the 3D object may be identified as a match. The matched stored 3D object may be used as a geometric proxy.

[0180] In step S730, the stored 3D object is transformed based on the 3D object. For example, the stored 3D object may be deformed based on the 3D object. The stored 3D object may be resized based on the 3D object. The stored 3D object may be oriented based on the 3D object. The stored 3D object may be rotated based on the 3D object. The stored 3D object may be translated based on the 3D object. Transforming the stored 3D object may cause the stored 3D object (if rendered side by side with the 3D object on a display) to appear substantially similar to the 3D object (e.g., in terms of posture). Note that the stored 3D object may be defined by a different mesh and may have different color properties than the 3D object.

[0181] In an example embodiment, the mesh corresponding to the stored 3D object can be rotated and translated about and along the x-axis, y-axis, and z-axis to align with the orientation of the 3D object. The matched one of the stored 3D objects 135 associated with the frame and the keyframe can be transformed. Therefore, the stored 3D object can be used to generate a first 3D object proxy based on the stored 3D object for the 3D object associated with the frame. In addition, the stored 3D object can be used to generate a second 3D object proxy based on the stored 3D object for the 3D object associated with the keyframe. Therefore, the first 3D object proxy can be transformed based on the 3D object identified in the frame, and the second 3D object proxy can be transformed based on the 3D object identified in the keyframe.

[0182] In step S735, the selected frame is compressed using the transformed stored 3D object using a color prediction scheme. For example, points in a grid defining the transformed stored 3D object may be matched with points of a corresponding (e.g., the same) 3D object in a nearby (temporally) previously encoded keyframe. A prediction technique may then use the matching points of the corresponding 3D object to select or predict pixels / blocks / patches in the keyframe for computing a residual (e.g., a color displacement relative to the keyframe) of the 3D object in the selected frame.

[0183] In an example embodiment, the transformed first 3D object proxy may be wrapped (with color and / or texture) corresponding to the 3D object identified in the frame, and the transformed second 3D object proxy may be wrapped (with color and / or texture) corresponding to the 3D object identified in the key frame. Mapping color attributes from the 3D object identified in the frame 110 to the transformed first 3D object proxy may include converting the 3D object from 3D space (e.g., XYZ space) to 2D space (e.g., UV space) and / or converting the 3D object identified in the frame 110 from 2D space (e.g., UV space) to 3D space (e.g., XYZ space). A residual for the 3D object may be generated by subtracting color attributes of a point in the mesh representation of the first 3D object proxy from color attributes of a corresponding point in the mesh representation of the second 3D object proxy (e.g., having the same point identity or being at the same position in the mesh sequence).

[0184] In an example embodiment, two or more 3D objects are identified in a selected frame. Each of the identified 3D objects is predicted using the above-described prediction techniques. The remainder of the frame is generated using standard prediction techniques. Compressing the selected frame may include performing a series of encoding processes on the residual. For example, the residual may be transformed, quantized, and entropy encoded.

[0185] Additionally, the prediction scheme may generate metadata associated with the selected frame. The metadata may include data associated with at least one 3D object located in the selected frame that has been predicted using one of the 3D objects of interest (e.g., stored 3D object 135). The metadata may include properties (e.g., grid point properties) associated with the location and / or orientation of the 3D object in the selected frame and keyframe.

[0186] In step S740, the compressed selected frames are stored together with metadata identifying the 3D objects and the positions and orientations of the 3D objects. For example, the compressed selected frames and metadata may be stored in a memory associated with a streaming server. The compressed selected frames and metadata may be stored in association with a plurality of compressed frames corresponding to the video (or a portion thereof).

[0187] Figure 8 A block diagram of another method for compressing frames of a video according to at least one example embodiment is shown. Figure 8 As shown, in step S805, a file including multiple frames of a video is received. For example, the file can be saved or transmitted to a server (e.g., a streaming server). The file can include a video. The video can be a CGI 3D movie. The file can include multiple 3D objects of interest (e.g., characters in a 3D movie).

[0188] In an example embodiment, each of the multiple 3D objects of interest may be defined by a triangular mesh. A triangular mesh may be a collection of points connected by triangular faces. Each point may store various attributes. For example, the attributes may include the position, color, texture coordinates, etc. of each point. The mesh of each of the multiple 3D objects of interest may have the same number of points, each point having the same attributes. Therefore, when stored in a memory, the mesh for each of the multiple 3D objects of interest may have approximately the same size (e.g., number of bits).

[0189] In an example implementation, the compression may be performed (eg, using the above reference Figure 5AThe described techniques can be used to compress a mesh for each of a plurality of 3D objects of interest (e.g., using a neural network, a convolutional neural network, a VAE, etc.). For example, the mesh for each of the plurality of 3D objects of interest can be compressed using a generative modeling technique (e.g., using a neural network, a convolutional neural network, a VAE, etc.). The mesh for each of the plurality of 3D objects of interest can be compressed using a neural network encoder having a convolutional neural network having elements selected and trained based on a machine-trained generative modeling technique that is configured to generate a reduced number of variables associated with mesh properties and positions for each 3D object. The generated reduced number of variables associated with mesh properties and positions for the 3D objects are sometimes referred to as a compact latent representation for the 3D objects.

[0190] In step S810, one of the plurality of frames is selected. For example, each of the plurality of frames may be a compression target. The plurality of frames may be compressed in time sequence. Therefore, in an initial step, the first frame in time is selected. Then the next frame may be selected in sequence.

[0191] In step S815, 3D objects are identified in the selected frames and key frames. The identified 3D objects may be dynamic 3D objects, non-dynamic 3D objects, static 3D objects, background 3D objects, etc. For example, machine vision, computer vision and / or computer image recognition technology may be used to identify and locate 3D objects.

[0192] In an example embodiment, computer image recognition techniques based on training (machine learning) a convolutional neural network using multiple known images can be used to identify 3D objects. For example, a block, multiple blocks, and / or patch are selected and / or identified from a selected frame. The trained convolutional neural network can operate on the selected block, multiple blocks, and / or patch. The results can be tested (e.g., error test, loss test, divergence test, etc.). If the test produces a value below a threshold (or alternatively, a value above a threshold depending on the type of test), the selected block, multiple blocks, and / or patch can be identified as a 3D object.

[0193] In an example embodiment, a frame of a video may include a tag indicating that a previously identified 3D object of interest is included in the frame. The tag may include the identity and location of the 3D object. For example, a video may be generated using a computer generated image (CGI) tool (e.g., a computer animated film). A computer generated character may be identified and labeled in each frame. In addition, a model for each identified 3D object of interest (e.g., an identified character) may be stored as a stored 3D object 135.

[0194] In an example implementation, a 3D object may be defined by a triangular mesh. A triangular mesh may be a collection of points connected by triangular faces. Each point may store various attributes. For example, the attributes may include the position, color, texture coordinates, etc. of each point. The attributes may include and / or indicate (e.g., multiple attributes may indicate) the orientation of the corresponding 3D object and / or the position of the corresponding 3D object in the selected frame.

[0195] Thus, in an example embodiment, the mesh properties of a 3D object may be sufficient to identify and locate the 3D object. A model including mesh properties for a plurality of 3D objects of interest may be stored as a stored 3D object 135. The model may be standardized. For example, a model for a man, a woman, a teenager, a child, or more generally a human or a portion of a human (e.g., body, head, hand, etc.) may be stored as a stored 3D object 135. For example, a model for a dog, a cat, a deer, or more generally a quadruped or a portion of a quadruped (e.g., body, head, leg, etc.) may be stored as a stored 3D object 135. The properties of the model may then be used to search for 3D objects with similar properties in a frame.

[0196] In step S820, the position and orientation of the 3D object within the selected frame and keyframe are determined. For example, the 3D object can be defined by a grid having a plurality of points, each point having at least one attribute. The position of the 3D object can be the position of the 3D object within the frame. Therefore, the position of the 3D object within the frame can be based on a 3D Cartesian coordinate system for the frame. For example, at least one point of the grid can be located at an x, y, z position within the frame. The orientation of the 3D object can be based on the position coordinate attributes of each point in the grid used to define the 3D object. If the 3D object is a moving character, the orientation of the 3D object can be the posture of the character in the frame and / or keyframe.

[0197] In step S825, the 3D object is matched with a potential representation for the 3D object. For example, the 3D object may be matched with one of the 3D objects of interest (e.g., the stored 3D object 135). Then, as described above, the matched 3D object may be encoded as a potential representation for the 3D object. For example, the 3D object may be matched with one of the 3D objects of interest (e.g., the potential representations 510 for the stored 3D objects). In an example embodiment, the identity of the 3D object, such as generated by computer image recognition techniques, may be used to search for the stored potential representations 510 of the 3D object. In an example embodiment, the label of the 3D object found in the frame may be matched with the label of one of the stored potential representations 510 of the 3D object. In an example embodiment, a model having similar properties to the compressed mesh properties of the 3D object may be identified as a match.

[0198] In step S830, the potential representation is transformed based on the 3D object. For example, the stored 3D object may be deformed based on the 3D object. The stored 3D object may be resized based on the 3D object. The stored 3D object may be oriented based on the 3D object. The stored 3D object may be rotated based on the 3D object. The stored 3D object may be translated based on the 3D object. Transforming the stored 3D object may cause the stored 3D object (if rendered side by side with the 3D object on a display) to appear substantially similar to the 3D object. Note that the stored 3D object may be defined by a different mesh and may have different color properties than the 3D object.

[0199] In an example embodiment, the coordinates of each point of the potential representation of the 3D object can be rotated and translated about and along the x-axis, y-axis, and z-axis to align with the orientation of the 3D object. The potential representation of the matched 3D object associated with the frame and the keyframe can be transformed. Therefore, the potential representation of the 3D object can be generated as a first 3D object proxy based on the stored 3D object for the 3D object associated with the frame. In addition, the potential representation of the 3D object can be generated as a second 3D object proxy based on the stored 3D object for the 3D object associated with the keyframe. Therefore, the first 3D object proxy can be transformed based on the 3D object identified in the frame, and the second 3D object proxy can be transformed based on the 3D object identified in the keyframe.

[0200] In an example embodiment, the stored 3D object is stored as a compact latent representation based on the 3D object. Therefore, before transforming the stored 3D object, the compact latent representation based on the 3D object can be used to regenerate (e.g., using the above reference Figure 5B The described techniques can be used to decompress the stored 3D object. For example, the variables in the compact latent space for the stored 3D object can be input to a neural network decoder to regenerate the points of the grid defining the stored 3D object and the position coordinates for each point of the grid.

[0201] In step S835, the selected frame is compressed using a transformed stored 3D object having a compact latent representation for the 3D object with a color prediction scheme. As described above, the transformed stored 3D object having a compact latent representation for the 3D object can be a transformed regenerated stored 3D object. For example, points in a grid defining the transformed regenerated stored 3D object can be matched with points of a corresponding (e.g., the same) 3D object located in a nearby (in time), previously encoded key frame. A prediction technique can then use the matching points of the corresponding 3D object to select or predict pixels / blocks / patches in the key frame for calculating a residual (e.g., a color displacement relative to the key frame) of the 3D object in the selected frame.

[0202] In an example embodiment, the transformed first 3D object proxy may be wrapped (with color and / or texture) corresponding to the 3D object identified in the frame, and the transformed second 3D object proxy may be wrapped (with color and / or texture) corresponding to the 3D object identified in the key frame. Mapping color attributes from the 3D object identified in the frame 110 to the transformed first 3D object proxy may include converting the 3D object from a 3D space (e.g., XYZ space) to a 2D space (e.g., UV space) and / or converting the 3D object identified in the frame 110 from a 2D space (e.g., UV space) to a 3D space (e.g., XYZ space). In an example embodiment, a residual for the 3D object may be generated by subtracting color attributes of a point in the latent representation of the first 3D object proxy from color attributes of a corresponding point (e.g., having the same coordinates or being at the same position) in the latent representation of the second 3D object proxy.

[0203] In an example embodiment, a latent representation of a first 3D object proxy for a transformed wrapper and a latent representation of a second 3D object proxy for a transformed wrapper may be decoded. As described above, decoding the latent representation of the 3D object may regenerate a mesh representation including color attributes of the 3D object. A residual for the 3D object may be generated by subtracting the color attributes of a point in the regenerated mesh representation of the first 3D object proxy from the color attributes of a corresponding point (e.g., having the same point identity or being at the same position in a mesh sequence) in the regenerated mesh representation of the second 3D object proxy.

[0204] In an example embodiment, two or more 3D objects are identified in a selected frame. Each of the identified 3D objects is predicted using the above-described prediction techniques. The remainder of the frame is generated using standard prediction techniques. Compressing the selected frame may include performing a series of encoding processes on the residual. For example, the residual may be transformed, quantized, and entropy encoded.

[0205] Additionally, the prediction scheme may generate metadata associated with the selected frame. The metadata may include data associated with at least one 3D object located in the selected frame that has been predicted using one of the 3D objects of interest (e.g., stored 3D object 135). The metadata may include properties associated with the location and / or orientation of the 3D object in the selected frame and keyframe (e.g., grid point properties). The metadata may include information associated with a neural network for an autoencoder that is used to generate a latent representation of the 3D object (e.g., encoding) and to regenerate a grid representation including color properties of the 3D object (e.g., decoding).

[0206] In step S840, the compressed selected frames are stored together with metadata identifying the 3D object and the position and orientation of the 3D object. For example, the compressed selected frames and metadata may be stored in a memory associated with a streaming server. The compressed selected frames and metadata may be stored in association with a plurality of compressed frames corresponding to the video (or a portion thereof).

[0207] Fig. 9 A block diagram of a method for decompressing and rendering a frame of a video according to at least one example embodiment is shown. Fig. 9 As shown, in step S905, a data packet containing at least one compressed frame of a video is received. For example, a client device may request a next frame of a video from a streaming server. In response to the request, the streaming server may select a next frame (or multiple next frames), determine whether the selected next frame (or multiple next frames) has associated metadata, and select the associated metadata. The streaming server may generate a data packet (or multiple data packets) including the selected next frame (or multiple next frames) and the selected associated metadata, and transmit the data packet (or multiple data packets) to the requesting client device.

[0208] In step S910, at least one frame is selected for decompression. For example, the frame may be selected from the received data packet (or data packets). In step S915, it is determined whether the frame includes metadata. In response to determining that the frame includes metadata, processing continues to step S925. Otherwise, processing continues to step S920 and the selected frame is decoded by some other prediction scheme (e.g., a prediction scheme not based on using 3D objects as geometric proxies).

[0209] For example, when transmitting the data packets from the streaming server to the client, the header associated with the data packets for the selected frame can be configured to include metadata. Thus, the header associated with the data packets for the selected frame can be read (e.g., via the transceiver 625) and transmitted to a decoder (e.g., decoder 145) along with the selected frame. Thus, determining that the selected frame includes metadata can include determining that the metadata has been transmitted along with the selected frame. Alternatively, determining that the selected frame includes metadata can include determining that the metadata is stored in association with the selected frame (e.g., stored in a decoder).

[0210] In step S925, a 3D object is identified in the selected frame based on the metadata. For example, the metadata may include data associated with at least one 3D object (e.g., at least one of the stored 3D objects 135) located in the selected frame that has used the 3D object as a geometric proxy prediction. The metadata may include information identifying the 3D object used as a geometric proxy.

[0211] In step S930, based on the metadata, the position and orientation of the 3D object in the selected frames and keyframes are determined. For example, the metadata may include attributes (e.g., grid point attributes) associated with the positioning and / or orientation of the 3D object in the selected frames and keyframes.

[0212] In step S935, the 3D object is matched with the stored 3D objects. For example, the metadata may include information identifying that the 3D object has a corresponding stored 3D object. In an example embodiment, the metadata includes a unique ID or tag that can be used to search for the 3D object in the stored 3D objects 135. If the unique ID or tag is found in the stored 3D objects 135, the corresponding one of the stored 3D objects 135 is a match for the 3D object.

[0213] In step S940, the stored 3D object is transformed based on the metadata. For example, the metadata may include information identifying that the 3D object has a corresponding stored 3D object and information related to the transformation performed on the stored 3D object during the encoding process. The information related to the transformation performed on the stored 3D object during the encoding process can be used to perform the same transformation on the stored 3D object as performed on the stored 3D object to encode the selected frame.

[0214] The stored 3D object may be deformed based on information related to the transformation performed on the stored 3D object during the encoding process. The stored 3D object may be resized based on information related to the transformation performed on the stored 3D object during the encoding process. The stored 3D object may be oriented based on information related to the transformation performed on the stored 3D object during the encoding process. The stored 3D object may be rotated based on information related to the transformation performed on the stored 3D object during the encoding process. The stored 3D object may be translated based on information related to the transformation performed on the stored 3D object during the encoding process. Transforming the stored 3D object may cause the stored 3D object (if rendered side by side with the 3D object on a display) to appear substantially similar to the 3D object because the 3D object appeared in the selected frame before the selected frame was compressed.

[0215] In an example embodiment, the matched 3D objects in the stored 3D objects associated with the frame and the keyframe may be transformed. Thus, the stored 3D objects may be used to generate a first 3D object proxy based on the stored 3D objects for the 3D object associated with the frame. Furthermore, the stored 3D objects may be used to generate a second 3D object proxy based on the stored 3D objects for the 3D object associated with the keyframe. Thus, the first 3D object proxy may be transformed based on metadata for the 3D object identified in the frame, and the second 3D object proxy may be transformed based on metadata for the 3D object identified in the keyframe.

[0216] In an example embodiment, the first 3D object proxy and the second 3D object proxy are latent representations of the 3D object. Thus, the first 3D object proxy can be used to regenerate a mesh representation for the first 3D object proxy, and the second 3D object proxy can be used to regenerate a mesh representation for the second 3D object proxy. As described above, decoding the latent representation for the 3D object using an autoencoder can regenerate a mesh representation including color attributes for the 3D object. The autoencoder can use a neural network with a structure read from metadata.

[0217] In step S945, the selected frame is decompressed using the stored 3D object (as a geometric proxy) for the 3D object with a color prediction scheme. Initially, the selected frame may be entropy decoded, inverse quantized, and inverse transformed to generate a derivative residual that is the same (or approximately the same) as the residual generated by the encoder when encoding the selected frame and before transforming, quantizing, and entropy encoding the residual.

[0218] In an example embodiment, the prediction scheme includes matching points corresponding to the transformed (e.g., translated and oriented) stored 3D object with points of the key frame. The matching points are then used to select or predict pixels / blocks / patches in the key frame for reproducing the color values ​​of the pixels of the 3D object in the selected frame.

[0219] Regenerating the color value of the pixel may include adding the pixel attribute value for each point in the triangular mesh of the 3D object from the pixel attribute value for each matching point for the predicted pixel / block / patch in the key frame to the color value and / or color attribute for the stored 3D object translated and oriented based on the residual at the identified position in the selected frame. The color value and / or color attribute for the remaining portion of the selected frame may be regenerated based on the residual or remaining portion of the selected frame and the corresponding pixel / block / patch of the key frame.

[0220] In an example embodiment, the transformed first 3D object proxy may be wrapped (with color and / or texture) corresponding to the 3D object identified in the frame. In other words, the transformed first 3D object proxy may be wrapped with a residual corresponding to the 3D object identified in the frame. In addition, the transformed second 3D object proxy may be wrapped (with color and / or texture) corresponding to the 3D object identified in the key frame. The color value for the pixel of the 3D object may be regenerated by adding the color attribute of a point in the mesh representation of the first 3D object proxy to the color attribute of a corresponding point in the mesh representation of the second 3D object proxy (e.g., having the same point identification or being at the same position in the mesh sequence). Mapping the color attribute from the 3D object identified in the frame to the transformed first 3D object proxy may include converting the 3D object from a 3D space (e.g., XYZ space) to a 2D space (e.g., UV space) and / or converting the 3D object identified in the frame 110 from a 2D space (e.g., UV space) to a 3D space (e.g., XYZ space).

[0221] In an example embodiment, the first 3D object proxy and the second 3D object proxy are latent representations of the 3D object. Thus, the first 3D object proxy can be used to regenerate a mesh representation including color attributes for the first 3D object proxy, and the second 3D object proxy can be used to regenerate a mesh representation including color attributes for the second 3D object proxy. As described above, decoding the latent representation of the 3D object using an autoencoder can regenerate a mesh representation including color attributes for the 3D object. The autoencoder can use a neural network with a structure read from metadata.

[0222] In step S950, the transformed stored 3D object is stitched into the decompressed frame. The selected frame may be reconstructed based on the regenerated color values ​​and / or color attributes of the translated and oriented stored 3D object and the regenerated color values ​​and / or color attributes of the remaining portion of the selected frame. In an example embodiment, the selected frame may be reconstructed by stitching the regenerated color values ​​and / or color attributes of the translated and oriented stored 3D object into the regenerated color values ​​and / or color attributes of the remaining portion of the selected frame based on the identified position of the 3D object (e.g., as read from metadata).

[0223] In step S955, a frame including a 3D object is rendered. For example, regenerated texture data and regenerated color data may be rendered for display. Rendering texture data and color data for display may include using a 3D shader (e.g., a vertex shader, a geometry shader, etc.) to draw a mesh associated with a frame. The shader may be configured to generate primitives. The shader may be configured to transform the 3D position and texture of each vertex in the mesh into 2D coordinates (e.g., primitives) that appear on a display (e.g., display 640). Rendering texture data and color data for display may also include performing rasterization. Rasterization may include using assigned pixel (e.g., color) values ​​for primitives based on texture data and color data.

[0224] In step S960, the rendered frame is color corrected. For example, the color correction may include compensating for color difference between frames, compensating for color difference between multiple views of the same scene, correcting object deformation (distortion), correcting object boundary deformation, etc.

[0225] Example implementations may include identifying two or more 3D objects; reproducing color values ​​and / or color attributes of each of the two or more 3D objects; and reconstructing a selected frame using each of the two or more 3D objects. Video data 5 may be regenerated based on a plurality of reconstructed frames. Video data 5 (e.g., texture data and color data) may be rendered and color corrected for display on a display of a client device.

[0226] As described above, when implementing the techniques described herein (e.g., the above methods), for use with an increased number of videos. The number of stored 3D objects of interest (e.g., stored 3D objects 135) will necessarily increase, and the amount of resources (e.g., memory) required to store the 3D objects of interest will also increase. In addition, during streaming activities, transmitting the 3D objects of interest from the streaming server to the client device may require a large amount of bandwidth. Therefore, for streaming operations, it may become desirable to efficiently encode and decode the 3D objects of interest.

[0227] Fig.10A block diagram of a method for compressing a 3D object according to at least one example embodiment is shown. Fig.10 As shown, in step S1005, at least one 3D object of interest in the video is identified. For example, the object of interest may include 3D characters of a CGI movie, including CGI actors (e.g., protagonists, supporting roles, and extras), CGI pets, CGI creatures, CGI monsters, etc. The object of interest may include a vehicle (e.g., a train, a car, or an airplane) or an object that can move from frame to frame in a predictable manner (e.g., at a constant speed and / or direction). The object of interest may include a stationary or fixed 3D object (e.g., the background of a scene, furniture at a fixed position within a scene, or a slow-moving object in the distance) that may appear stationary from frame to frame (e.g., without any camera or scene translation).

[0228] The 3D objects of interest may be predetermined and stored in association with the video. The 3D objects of interest may be determined as needed (eg, when a key frame is selected, as part of an initialization operation) and stored in association with the video or added to previously stored 3D objects of interest.

[0229] In step S1010, the mesh properties (e.g., vertices and connectivity) and positions of the 3D objects of interest are determined. According to an example embodiment, each 3D object of interest may be defined by a mesh having the same number of points, each point having a position coordinate. Other properties of the points may be added as needed. Thus, the mesh properties for each 3D object of interest may include the same number of vertices with varying connectivity.

[0230] In step S1015, a machine-trained generative modeling technique is used to generate a reduced number of variables associated with the mesh properties and positions of the 3D object of interest. For example, a VAE can be used to generate the reduced number of variables. This reduced number of variables is sometimes referred to as a potential representation for the 3D object or a reduced potential representation for the 3D object. The VAE may include a neural network encoder and a neural network decoder, each of which includes a neural network having the same configuration of C filters, KxK masks, and step factors for the neural network. Therefore, the number of variables in the latent space for each potential representation of the 3D object of interest generated by the neural network encoder is the same for each 3D object of interest. In addition, the number of points in the mesh regenerated by the neural network decoder is the same for each 3D object of interest.

[0231] In an example implementation, the compression may be performed (eg, using the above reference Figure 5AThe described techniques compress) a mesh for each of a plurality of 3D objects of interest. For example, the mesh for each of the plurality of 3D objects of interest can be compressed using a generative modeling technique (e.g., using a neural network, a convolutional neural network, a VAE, etc.). The mesh for each of the plurality of 3D objects of interest can be compressed using a neural network encoder having a convolutional neural network having elements selected and trained based on a machine-trained generative modeling technique that is configured to generate a reduced number of variables associated with mesh properties and positions for each 3D object. The generated reduced number of variables associated with mesh properties and positions for the 3D objects is sometimes referred to as a compact latent representation for the 3D objects.

[0232] In step S1020, the variables are stored as compact decodable representations (or compact latent representations) of the 3D object of interest associated with the video. For example, the variables may be stored as latent representations for the 3D object 510. The latent representations for the 3D object 510 may be stored on a streaming server and / or a device including at least one encoder (e.g., encoder 105).

[0233] Fig.11 A block diagram of a method for decompressing a 3D object according to at least one example embodiment is shown. Fig.11 As shown, in step S1105, a compact decodable representation of a 3D object associated with a video is received. For example, variables stored as a potential representation of the 3D object 510 may be stored on a streaming server and / or a device including at least one encoder (e.g., encoder 105). During a video streaming operation, at least one set of variables stored as a potential representation of the 3D object may be received from a streaming server.

[0234] In step S1110, a machine-trained generative modeling technique may be used to generate mesh properties (e.g., vertices and connectivity) and positions of a 3D object. For example, the VAE may use variables in a latent space of a potential representation of a 3D object that may be used as input to a neural network decoder, and the VAE may regenerate the position coordinates of each point of the mesh. The configuration of the C filters, KxK masks, and step factors used for the neural network may determine the number of points of the regenerated mesh. When regenerating the points of the mesh, the neural network decoder may regenerate the mesh properties and positions of the 3D object.

[0235] In step S1115, the mesh properties, position, orientation and color properties of the 3D object associated with the video as the stored 3D object are stored. For example, when the 3D object is regenerated as a mesh including a collection of points connected by faces, the 3D object can be stored as the stored 3D object 135. Each regenerated point can store various properties. For example, the properties can include the position, color, texture coordinates, etc. of each point. Therefore, in a color prediction scheme based on using the 3D object as a geometric proxy as implemented in a 3D encoder and / or a 3D decoder, the regenerated 3D object included in the stored 3D object 135 can be used.

[0236] Fig.12 1 shows a video encoder system 1200 according to at least one example embodiment. Fig.12 As shown in FIG. 1 , the video encoder system 1200 includes at least one processor 1205, at least one memory 1210, a controller 1220, and a video encoder 105. The at least one processor 1205, at least one memory 1210, the controller 1220, the video encoder 105, and the video encoder 105 are communicatively coupled via a bus 1215.

[0237] exist Fig.12 In the example of , the video encoder system 1200 can be or include at least one computing device, and should be understood to represent substantially any computing device configured to perform the methods described herein. As such, the video encoder system 1200 can be understood to include various components that can be used to implement the techniques described herein or different or future versions thereof. For example, the video encoder system 1200 is illustrated as including at least one processor 1205, and at least one memory 1210 (e.g., a non-transitory computer-readable storage medium).

[0238] It is understood that the at least one processor 1205 can be used to execute instructions stored on the at least one memory 1210, thereby implementing the various features and functions described herein, or additional or alternative features and functions. Of course, the at least one processor 1205 and the at least one memory 1210 can be used for various other purposes. In particular, it is understood that the at least one memory 1210 can be understood as representing examples of various types of memory and related hardware and software, which can be used to implement any of the modules described herein.

[0239] At least one memory 1210 may be configured to store data and / or information associated with the video encoder system 1200. The at least one memory 1210 may be a shared resource. For example, the video encoder system 1200 may be an element of a larger system (e.g., a server, a personal computer, a mobile device, etc.). Thus, the at least one memory 1210 may be configured to store data and / or information associated with other elements within the larger system (e.g., image / video services, web browsing, or wired / wireless communications).

[0240] The controller 1220 may be configured to generate various control signals and communicate the control signals to various blocks in the video encoder system 1200. The controller 1220 may be configured to generate control signals to implement the above-described techniques. According to an example embodiment, the controller 1220 may be configured to control the video encoder 1225 to encode an image, an image sequence, a video frame, a video sequence, etc. For example, the controller 1220 may generate a control signal corresponding to a video quality.

[0241] The video encoder 105 may be configured to receive a video stream input 5 and output compressed (e.g., encoded) video bits 10. The video encoder 105 may convert the video stream input 5 into discrete video frames. The video stream input 5 may also be an image, and therefore, the compressed (e.g., encoded) video bits 10 may also be compressed image bits. The video encoder 105 may further convert each discrete video frame (or image) into a block matrix (hereinafter referred to as a block). For example, a video frame (or image) may be converted into a 16×16, 16×8, 8×8, 4×4, or 2×2 matrix of blocks having a plurality of pixels, respectively. Although five example matrices are listed, example embodiments are not limited thereto.

[0242] The compressed video bits 10 may represent the output of the video encoder system 1200. For example, the compressed video bits 10 may represent an encoded video frame (or encoded picture). For example, the compressed video bits 10 may be ready for transmission to a receiving device (not shown). For example, the video bits may be transmitted to a system transceiver (not shown) for transmission to a receiving device.

[0243] At least one processor 1205 can be configured to execute computer instructions associated with the controller 1220 and / or the video encoder 105. The at least one processor 1205 can be a shared resource. For example, the video encoder system 1200 can be an element of a larger system (e.g., a mobile device, a server, a streaming server, etc.). Thus, the at least one processor 1205 can be configured to execute computer instructions associated with other elements within the larger system (e.g., image / video services, web browsing, or wired / wireless communications).

[0244] In an example embodiment, the video encoder system 1200 may be implemented as a graphics card and / or chip (e.g., an ASIC on a computer motherboard) including a graphics processing unit (GPU) or implemented in a graphics card and / or chip, which is configured to remove the load from a central processing unit (CPU). At least one processor 1205 may be implemented as a GPU configured to process large blocks of video data in parallel. The GPU may be configured to process (e.g., compress) grid data and generate pixel data from the grid data. At least one memory 1210 may include video memory and driver software. The video memory may be a frame buffer that stores digital data representing a frame image or scene. The video memory may store digital data before and after GPU processing. The driver software may include a codec configured to decompress video data. The codec may include a color prediction scheme based on using a 3D object as a geometric proxy as described herein.

[0245] Fig.13 1300 is illustrated in accordance with at least one example embodiment. Fig.13 As shown in FIG. 1 , the video decoder system 1300 includes at least one processor 1305, at least one memory 1310, a controller 1320, and a video decoder 145. The at least one processor 1305, at least one memory 1310, the controller 1320, the video decoder 145, and the video decoder 145 are communicatively coupled via a bus 1315.

[0246] exist Fig.13 In the example of , video decoder system 1300 can be at least one computing device, and should be understood to represent virtually any computing device configured to perform the methods described herein. As such, video decoder system 1300 can be understood to include various components that can be used to implement the techniques described herein or different or future versions thereof. For example, video decoder system 1300 is illustrated as including at least one processor 1305, and at least one memory 1310 (e.g., a computer-readable storage medium).

[0247] Therefore, it is understood that the at least one processor 1305 can be used to execute instructions stored on the at least one memory 1310, thereby implementing the various features and functions described herein, or additional or alternative features and functions. Of course, the at least one processor 1305 and the at least one memory 1310 can be used for various other purposes. In particular, it is understood that the at least one memory 1310 can be understood to represent various types of memory and examples of related hardware and software that can be used to implement any of the modules described herein. According to an example embodiment, the video encoder system 1200 and the video decoder system 1300 can be included in the same larger system (e.g., a personal computer, a mobile device, etc.).

[0248] At least one memory 1310 may be configured to store data and / or information associated with the video decoder system 1300. The at least one memory 1310 may be a shared resource. For example, the video decoder system 1300 may be an element of a larger system (e.g., a personal computer, a mobile device, etc.). Thus, the at least one memory 1310 may be configured to store data and / or information associated with other elements within the larger system (e.g., web browsing or wireless communications).

[0249] The controller 1320 may be configured to generate various control signals and communicate the control signals to various blocks in the video decoder system 1300. The controller 1320 may be configured to generate control signals in order to implement the video encoding / decoding techniques described below. According to an example embodiment, the controller 1320 may be configured to control the video decoder 145 to decode video frames.

[0250] The video decoder 145 may be configured to receive input and output compressed (e.g., encoded) video bits 10 of a video stream 5. The video decoder 145 may convert discrete video frames of the compressed video bits 10 into the video stream 5. The compressed (e.g., encoded) video bits 10 may also be compressed image bits, and thus, the video stream 5 may also be an image.

[0251] At least one processor 1305 may be configured to execute computer instructions associated with controller 1320 and / or video decoder 145. At least one processor 1305 may be a shared resource. For example, video decoder system 1300 may be an element of a larger system (e.g., a personal computer, a mobile device, etc.). Thus, at least one processor 1305 may be configured to execute computer instructions associated with other elements within the larger system (e.g., web browsing or wireless communications).

[0252] In an example embodiment, the video decoder system 1300 may be implemented as or include a graphics card and / or chip (e.g., an ASIC on a computer motherboard) of a graphics processing unit (GPU) or therein, which is configured to remove the load from the central processing unit (CPU). At least one processor 1305 may be implemented as a GPU configured to process large blocks of video data in parallel. The GPU may be configured to process (e.g., decompress) grid data and generate pixel data from the grid data. At least one memory 1310 may include video memory and driver software. The video memory may be a frame buffer that stores digital data representing a frame image or scene. The video memory may store digital data before and after GPU processing. The driver software may include a codec configured to decompress video data. The codec may include a color prediction scheme based on using a 3D object as a geometric proxy as described herein.

[0253] Fig.14 Examples of computer device 1400 and mobile computer device 1450 that can be used with the techniques described herein are shown. Computing device 1400 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device 1450 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the implementation of the inventions described and / or claimed in this document.

[0254] The computing device 1400 includes a processor 1402, a memory 1404, a storage device 1406, a high-speed interface 1408 connected to the memory 1404 and a high-speed expansion port 1410, and a low-speed interface 1412 connected to a low-speed bus 1414 and the storage device 1406. Each of the components 1402, 1404, 1406, 1408, 1410, and 1412 is interconnected using various buses and can be mounted on a common motherboard or otherwise installed as appropriate. The processor 1402 can process instructions for execution within the computing device 1400, including instructions stored in the memory 1404 or on the storage device 1406 to display graphical information for a GUI on an external input or output device such as a display 1416 coupled to the high-speed interface 1408. In other embodiments, multiple processors and / or multiple buses can be used together with multiple memories and multiple types of memories as appropriate. Additionally, multiple computing devices 1400 may be connected, with each device providing portions of the necessary operations (eg, as a server bank, a group of blade servers, or a multi-processor system).

[0255] The memory 1404 stores information within the computing device 1400. In one implementation, the memory 1404 is one or more volatile memory units. In another implementation, the memory 1404 is one or more non-volatile memory units. The memory 1404 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0256] Storage device 1406 can provide mass storage for computing device 1400. In one embodiment, storage device 1406 can be or include computer readable media, such as a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid-state storage device or array of devices, including devices in a storage area network or other configurations. A computer program product can be tangibly embodied in an information carrier. A computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer readable medium or a machine readable medium, such as a memory 1404, a storage device 1406, or a memory on a processor 1402.

[0257] The high-speed controller 1408 manages bandwidth-intensive operations of the computing device 1400, while the low-speed controller 1412 manages less bandwidth-intensive operations. This allocation of functions is exemplary only. In one embodiment, the high-speed controller 1408 is coupled to the memory 1404, the display 1416 (e.g., through a graphics processor or accelerator), and to the high-speed expansion port 1410, which can accept various expansion cards (not shown). In this embodiment, the low-speed controller 1412 is coupled to the storage device 1406 and the low-speed expansion port 1414. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input or output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, for example, through a network adapter.

[0258] As shown in the figure, computing device 1400 can be implemented in many different forms. For example, it can be implemented as a standard server 1420, or multiple times in a group of such servers. It can also be implemented as part of a rack server system 1424. In addition, it can be implemented in a personal computer such as a laptop computer 1422. Alternatively, components from computing device 1400 can be combined with other components in a mobile device (not shown) such as device 1450. Each of such devices can contain one or more of computing devices 1400, 1450, and the entire system can be composed of multiple computing devices 1400, 1450 communicating with each other.

[0259] The computing device 1450 includes a processor 1452, a memory 1464, an input or output device such as a display 1454, a communication interface 1466, and a transceiver 1468, among other components. The device 1450 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components 1450, 1452, 1464, 1454, 1466, and 1468 are interconnected using various buses, and several components may be mounted on a common motherboard or otherwise as appropriate.

[0260] Processor 1452 can execute instructions within computing device 1450, including instructions stored in memory 1464. The processor can be implemented as a chipset of chips that include separate and multiple analog and digital processors. The processor can, for example, provide coordination for other components of device 1450, such as control of a user interface, applications executed by device 1450, and wireless communications performed by device 1450.

[0261] The processor 1452 can communicate with the user through a control interface 1458 and a display interface 1456 coupled to the display 1454. The display 1454 can be, for example, a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display or other appropriate display technology. The display interface 1456 may include appropriate circuits for driving the display 1454 to present graphics and other information to the user. The control interface 1458 may receive commands from the user and convert them for submission to the processor 1452. In addition, an external interface 1462 in communication with the processor 1452 may be provided to enable near-area communication of the device 1450 with other devices. The external interface 1462 may be provided, for example, for wired communication in some embodiments, or for wireless communication in other embodiments, and multiple interfaces may also be used.

[0262] The memory 1464 stores information within the computing device 1450. The memory 1464 may be implemented as one or more of one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 674 may also be provided and connected to the device 1450 via an expansion interface 1472, which may include, for example, a SIMM (single in-line memory module) card interface. Such an expansion memory 1474 may provide additional storage space for the device 1450, or may also store applications or other information for the device 1450. Specifically, the expansion memory 1474 may include instructions for executing or supplementing the above-mentioned processes, and may also include security information. Therefore, for example, the expansion memory 1474 may be provided as a security module for the device 1450, and may be programmed with instructions that allow the secure use of the device 1450. In addition, security applications and additional information may be provided via a SIMM card, such as placing identification information on a SIMM card in an unbreakable manner.

[0263] The memory may include, for example, flash memory and / or NVRAM memory, as discussed below. In one embodiment, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine readable medium, such as memory 1464, expansion memory 1474, or memory on processor 1452, which may be received, for example, via transceiver 1468 or external interface 1462.

[0264] Device 1450 can communicate wirelessly via communication interface 1466, which may include digital signal processing circuitry as necessary. Communication interface 1466 may provide for communication in various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS. Such communication may occur, for example, via radio frequency transceiver 1468. In addition, short-range communication may occur, such as using Bluetooth, WiFi or other such transceivers (not shown). In addition, GPS (global positioning system) receiver module 570 may provide additional navigation and location-related wireless data to device 1450, which may be used by applications running on device 1450 as appropriate.

[0265] Device 1450 may also communicate audibly using audio codec 1460, which may receive spoken information from a user and convert it into usable digital information. Audio codec 1460 may likewise generate audible sounds for the user, such as through a speaker—for example, in a headset of device 1450. Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on device 1450.

[0266] As shown in the figure, computing device 1450 can be implemented in many different forms. For example, it can be implemented as a cellular phone 1480. It can also be implemented as part of a smart phone 1482, a personal digital assistant, or other similar mobile device.

[0267] Various embodiments of the systems and techniques described herein may be implemented using digital electronic circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs executable and / or interpretable on a programmable system, the programmable system including at least one programmable processor, which may be dedicated or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to a storage system, at least one input device, and at least one output device. Various embodiments of the systems and techniques described herein may be implemented and / or generally referred to herein as circuits, modules, blocks, or systems that can be combined in software and hardware. For example, a module may include functions / behaviors / computer program instructions executed on a processor (e.g., a processor formed on a silicon substrate, a GaAs substrate, etc.) or some other programmable data processing device.

[0268] Some of the above example embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the operations as sequential processes, many operations may be performed in parallel, concurrently, or simultaneously. In addition, the order of the operations may be rearranged. These processes may be terminated when their operations are completed, but may also have additional steps not included in the figure. These processes may correspond to methods, functions, programs, subroutines, subprograms, etc.

[0269] The methods discussed above, some of which are illustrated by flowcharts, may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments for performing the necessary tasks may be stored in a machine or computer readable medium such as a storage medium. A processor may perform the necessary tasks.

[0270] The specific structural and functional details disclosed herein are merely representative for purposes of describing example embodiments. Example embodiments are embodied in many alternate forms, however, and should not be construed as limited to only the embodiments set forth herein.

[0271] It should be understood that, although the terms first, second, etc. can be used in this article to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first element can be called the second element, and similarly, the second element can be called the first element without departing from the scope of the exemplary embodiment. As used herein, the terms and / or include any and all combinations of one or more of the related enumerated items.

[0272] It should be understood that when an element is referred to as being connected or coupled to another element, it may be directly connected or coupled to the other element or there may be intervening elements. In contrast, when an element is referred to as being directly connected or directly coupled to another element, there are no intervening elements. Other words used to describe the relationship between elements (e.g., between versus directly between, adjacent versus directly adjacent, etc.) should be interpreted in a similar manner.

[0273] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, unless the context clearly indicates otherwise, the singular forms a, an and the are also intended to include plural forms. It should be further understood that the terms contain, include, include and / or include when used in this article, specifying the features, integers, steps, operations, elements and / or components of the statement, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups.

[0274] It should also be noted that, in some alternative implementations, the functions / behaviors indicated may not occur in the order indicated in the figures. For example, depending on the functionality or behavior involved, two figures shown in succession may actually be executed concurrently or may sometimes be executed in reverse order.

[0275] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the example embodiments belong. It should be further understood that, unless explicitly defined as such herein, terms (e.g., those defined in commonly used dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and should not be interpreted in an idealized or overly formal sense.

[0276] The various parts of the above-mentioned example embodiments and the corresponding detailed description are presented in terms of software or algorithms and symbolic representations of operations on data bits in computer memory. These descriptions and representations are descriptions and representations used by those of ordinary skill in the art to effectively convey the essence of their work to other persons of ordinary skill in the art. An algorithm (when the term is used here, and when it is used generally) is considered to be a self-consistent sequence of steps leading to a desired result. These steps are those steps that require physical manipulation of physical quantities. Typically, although not necessarily, these quantities take the form of optical, electrical or magnetic signals that can be stored, transferred, combined, compared and otherwise manipulated. Sometimes, mainly due to common reasons, it has proven to be convenient to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, etc.

[0277] In the above illustrative embodiments, references to behaviors and symbolic representations of operations (e.g., in the form of flowcharts) that can be implemented as program modules or functional processes include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types and can be described and / or implemented using existing hardware at existing structural elements. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application specific integrated circuits, field programmable gate arrays (FPGAs), computers, etc.

[0278] It should be remembered, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specifically stated, or as is apparent from the discussion, terms such as process or compute or calculate or determine display refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical, electronic quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission or display devices.

[0279] Note also that the software-implemented aspects of the example embodiments are typically encoded on some form of non-transitory program storage medium or implemented on some type of transmission medium. The program storage medium may be magnetic (e.g., a floppy disk or hard drive) or optical (e.g., a compact disk read-only memory or CD ROM), and may be read-only or random access. Similarly, the transmission medium may be a twisted pair, a coaxial cable, an optical fiber, or some other suitable transmission medium known in the art. The example embodiments are not limited by these aspects of any given implementation.

[0280] Finally, it should also be noted that although the appended claims set out particular combinations of features described herein, the scope of the present disclosure is not limited to the particular combinations thereafter claimed, but instead extends to include any combination of features or embodiments disclosed herein, regardless of whether the particular combination has been specifically enumerated in the appended claims at this time.

Claims

1. A method for video encoding, comprising: Receive video frames; identifying a three-dimensional 3D object in the frame; matching the 3D object with a stored 3D object; generating a first 3D object proxy based on the stored 3D object; transforming the first 3D object proxy based on the 3D object identified in the frame; compressing frames of the video using a color prediction scheme based on the transformed first 3D object proxy and a transformed second 3D object proxy generated based on the 3D object; as well as Compressed frames are stored with metadata identifying the 3D object, indicating a location of the 3D object in a frame of the video, and indicating an orientation of the 3D object in the frame of the video.

2. The method of claim 1 , wherein compressing frames of the video using the color prediction scheme based on the transformed first 3D object proxy and the transformed second 3D object proxy comprises: generating a second 3D object proxy based on the stored 3D object; Identifying the 3D object in a key frame of the video; transforming the second 3D object proxy based on the 3D object identified in the key frame; mapping a color attribute from the 3D object to a transformed first 3D object proxy; mapping a color attribute from the 3D object identified in the keyframe to a transformed second 3D object proxy; as well as A residual of the 3D object is generated based on the transformed color attribute of the first 3D object proxy and the transformed color attribute of the second 3D object proxy.

3. The method according to claim 1, wherein: Compressing a frame of the video using the color prediction scheme based on the transformed first 3D object proxy and the transformed second 3D object proxy comprises: generating a second 3D object proxy based on the stored 3D object; Identifying the 3D object in a key frame of the video; transforming the second 3D object proxy based on the 3D object identified in the key frame; mapping a color attribute from the 3D object to a transformed first 3D object proxy; and A residual of the 3D object is generated based on a color attribute of the transformed first 3D object proxy and a default color attribute of the transformed second 3D object proxy.

4. The method of claim 1 , wherein compressing frames of the video using the color prediction scheme based on the transformed first 3D object proxy and the transformed second 3D object proxy comprises: encoding the first 3D object proxy using an autoencoder, wherein transforming the first 3D object proxy comprises: transforming the encoded first 3D object proxy based on the 3D object identified in the frame; generating a second 3D object proxy based on the stored 3D object; encoding the second 3D object proxy using an autoencoder; Identifying the 3D object in a key frame of the video; a second 3D object proxy encoded based on the 3D object transform identified in the keyframe; mapping a color attribute from the 3D object to a transformed first 3D object proxy; mapping a color attribute from the 3D object identified in the keyframe to a transformed second 3D object proxy; and A residual of the 3D object is generated based on the transformed color attribute of the first 3D object proxy and the transformed color attribute of the second 3D object proxy.

5. The method of claim 1 , wherein compressing frames of the video using the color prediction scheme based on the transformed first 3D object proxy and the transformed second 3D object proxy comprises: encoding the first 3D object proxy using an autoencoder, wherein transforming the first 3D object proxy comprises: transforming the encoded first 3D object proxy based on the 3D object identified in the frame; generating a second 3D object proxy based on the stored 3D object; encoding the second 3D object proxy using an autoencoder; Identifying the 3D object in a key frame of the video; a second 3D object proxy encoded based on the 3D object transform identified in the keyframe; mapping a color attribute from the 3D object to a transformed first 3D object proxy; and A residual of the 3D object is generated based on a color attribute of the transformed first 3D object proxy and a default color attribute of the transformed second 3D object proxy.

6. The method according to claim 1, further comprising: Before storing the 3D object: identifying at least one 3D object of interest associated with the video; determining a plurality of mesh properties associated with the 3D object of interest; determining a location associated with the 3D object of interest; determining an orientation associated with the 3D object of interest; determining a plurality of color attributes associated with the 3D object of interest; and An autoencoder is used to reduce the number of variables associated with the mesh properties of the 3D object of interest. 7 . The method of claim 1 , wherein compressing the frames of the video comprises determining position coordinates of the 3D object relative to origin coordinates of background 3D objects in key frames.

8. The method according to claim 1, wherein: The stored 3D object includes a default color attribute, and The color prediction scheme uses the default color attributes.

9. The method according to claim 1, further comprising: identifying at least one 3D object of interest associated with the video; generating at least one stored 3D object based on the at least one 3D object of interest, each of the at least one stored 3D object being defined by a mesh including a set of points connected by faces, each point storing at least one attribute, the at least one attribute including a position coordinate of the corresponding point; and The at least one stored 3D object is stored in association with the video.

10. A method for video decoding, comprising: Receive video frames; identifying a three-dimensional 3D object in the frame; matching the 3D object with a stored 3D object; generating a first 3D object proxy based on the stored 3D object; transforming the first 3D object proxy based on the 3D object identified in the frame; decompressing a frame of the video using a color prediction scheme based on the transformed first 3D object proxy and a transformed second 3D object proxy generated based on the stored 3D object; as well as Rendering frames of the video.

11. The method of claim 10, wherein decompressing a frame of the video using the color prediction scheme based on the transformed first 3D object proxy and the transformed second 3D object proxy comprises: generating a second 3D object proxy based on the stored 3D object; Identifying the 3D object in a key frame of the video; transforming the second 3D object proxy based on the 3D object identified in the key frame; mapping a color attribute from the 3D object to a transformed first 3D object proxy; mapping a color attribute from the 3D object identified in the keyframe to a transformed second 3D object proxy; as well as A color attribute of the 3D object is generated based on the color attribute of the transformed first 3D object proxy and the color attribute of the transformed second 3D object proxy.

12. The method according to claim 10, wherein: Decompressing a frame of the video using the color prediction scheme based on the transformed first 3D object proxy and the transformed second 3D object proxy comprises: generating a second 3D object proxy based on the stored 3D object; Identifying the 3D object in a key frame of the video; transforming the second 3D object proxy based on the 3D object identified in the key frame; mapping a color attribute from the 3D object to a transformed first 3D object proxy; and A color attribute of the 3D object is generated based on the color attribute of the transformed first 3D object proxy and a default color attribute of the transformed second 3D object proxy.

13. The method of claim 10, wherein decompressing a frame of the video using the color prediction scheme based on the transformed first 3D object proxy and the transformed second 3D object proxy comprises: decoding the first 3D object proxy using an autoencoder; transforming the decoded first 3D object proxy based on metadata associated with the 3D object; generating a second 3D object proxy based on the stored 3D object; decoding the second 3D object proxy using an autoencoder; Identifying the 3D object in a key frame of the video; transforming the decoded second 3D object proxy based on metadata associated with the 3D object identified in the key frame; mapping a color attribute from the 3D object to the transformed first 3D object proxy; mapping a color attribute from the 3D object identified in the keyframe to the transformed second 3D object proxy; as well as A color attribute of the 3D object is generated based on the color attribute of the transformed first 3D object proxy and the color attribute of the transformed second 3D object proxy.

14. The method of claim 10, wherein decompressing a frame of the video using the color prediction scheme based on the transformed first 3D object proxy and the transformed second 3D object proxy comprises: decoding the first 3D object proxy using an autoencoder; transforming the decoded first 3D object proxy based on metadata associated with the 3D object; generating a second 3D object proxy based on the stored 3D object; decoding the second 3D object proxy using an autoencoder; Identifying the 3D object in a key frame of the video; transforming the decoded second 3D object proxy based on metadata associated with the 3D object identified in the key frame; mapping a color attribute from the 3D object to a transformed first 3D object proxy; as well as Color properties of the 3D object are generated based on the color properties of the transformed first 3D object proxy and default properties of the transformed second 3D object proxy.

15. The method according to claim 10, further comprising: receiving at least one potential representation of a 3D shape; Use machine-trained generative modeling techniques to: determining a plurality of mesh properties associated with the 3D shape; determining a position associated with the 3D shape; determining an orientation associated with the 3D shape; as well as determining a plurality of color attributes associated with the 3D shape, and The 3D shape is stored as a stored 3D object.

16. The method according to claim 10, wherein: Rendering a frame of the video includes: receiving position coordinates of the 3D object relative to the origin coordinates of the background 3D object in the key frame; and The 3D object is placed in the frame using the position coordinates.

17. The method according to claim 10, further comprising: receiving a neural network used by an encoder of an autoencoder to reduce a number of variables associated with mesh properties, position, orientation, and color properties of at least one 3D object of interest; regenerating points associated with a mesh of at least one 3D object of interest using the neural network in a decoder of the autoencoder, the regenerating the points comprising regenerating position attributes, orientation attributes, and color attributes; as well as The at least one 3D object of interest is stored as a stored 3D object.

18. A method for predicting color changes using a proxy: generating a first 3D object proxy based on the stored 3D object; generating a second 3D object proxy based on the stored 3D object; transforming the first 3D object proxy based on a 3D object identified in a frame of the video; transforming the second 3D object proxy based on the 3D object identified in a key frame of the video; mapping a color attribute from the 3D object identified in a frame of the video to a transformed first 3D object proxy; mapping a color attribute from the 3D object identified in the keyframe to a transformed second 3D object proxy; as well as Color data for the 3D object is generated based on the color attribute of the transformed first 3D object proxy and the color attribute of the transformed second 3D object proxy.

19. The method according to claim 18, further comprising: Before transforming the first 3D object proxy, encoding the first 3D object proxy using an autoencoder; as well as Prior to transforming the second 3D object proxy, the second 3D object proxy is encoded using the autoencoder.

20. The method of claim 18, further comprising: After transforming the first 3D object proxy, decoding the first 3D object proxy using an autoencoder; as well as After transforming the second 3D object proxy, the second 3D object proxy is decoded using the autoencoder.

21. The method of claim 18, wherein generating color data for the 3D object comprises subtracting color properties of the transformed first 3D object proxy from color properties of the transformed second 3D object proxy.

22. The method of claim 18, wherein generating color data for the 3D object comprises adding color properties of the transformed first 3D object proxy to color properties of the transformed second 3D object proxy.

Images