Four-dimensional (4D) video system capable of streaming

Abstract

The system comprises: a feature extraction network for generating two or more feature maps corresponding to two or more frames, each of the two or more frames being associated with a respective camera view of the video at a first timestamp; an encoder to generate a first encoded feature volume, where generating the first encoded feature volume comprises encoding a feature volume using a first encoder model, where the feature volume is generated based at least in part on two or more of the feature maps; and a server for providing the first encoded feature volume and a first decoder model for decoding the first encoded feature volume in response to a request for the video at the first timestamp.

Description

Technical Field

[0001] The present application generally relates to methods, systems and apparatus for four-dimensional (4D) video streaming. Background Art

[0002] More and more video and image content is being generated every day. User demand and usage of media has evolved to give users more capabilities to interact with video and image content. For example, some developments enable the creation of streamable, free-viewpoint videos, where the user is able to control the camera (e.g., viewing angle) of the streaming video.

[0003] Conventional techniques rely on an algorithm that uses a textured mesh to reconstruct the scene of the video, which results in time-consuming encoding and high bitrate consumption. Therefore, a more efficient 4D video streaming framework is provided. Summary of the invention

[0004] Tools and techniques for streamable 4D video are provided.

[0005] A method includes: acquiring two or more frames, each of the two or more frames being associated with a corresponding camera view of a video at a first timestamp. The method continues by generating two or more feature maps through a feature extraction network, each of the two or more feature maps corresponding to a corresponding frame of the two or more frames, and generating a first feature volume from the two or more feature maps. The method further includes generating a first encoded feature volume through an encoder, wherein generating the first encoded feature volume includes encoding the first feature volume using a first encoder model; and providing the first encoded feature volume and a first decoder model for decoding the first encoded feature volume through a server.

[0006] An apparatus includes a non-transitory computer-readable medium in communication with a processor, the non-transitory computer-readable medium including a set of instructions encoded thereon and executable by the processor. The set of instructions is executable by the processor to obtain two or more frames (each of the two or more frames is associated with a corresponding camera view of a video at a first timestamp), generate two or more feature maps through a feature extraction network (each of the two or more feature maps corresponds to a corresponding frame in the two or more frames), and generate a first feature volume from the two or more feature maps. The set of instructions is also executable by the processor to generate a first encoded feature volume through an encoder, wherein generating the first encoded feature volume includes encoding the first feature volume using a first encoder model; and providing the first encoded feature volume and a first decoder model for decoding the first encoded feature volume through a server.

[0007] A system comprises: a feature extraction network for generating one or more feature maps corresponding to two or more frames, each of the two or more frames being associated with a corresponding camera view of a video at a first timestamp; an encoder for generating a first encoded feature volume, wherein generating the first encoded feature volume comprises encoding the feature volume using a first encoder model, wherein the feature volume is generated at least in part based on two or more of the feature maps; and a server for providing the first encoded feature volume and a first decoder model for decoding the first encoded feature volume in response to a request for the video at the first timestamp.

[0008] These exemplary embodiments are mentioned not to limit or define the content of the present application, but to provide examples to aid in understanding the content of the present application. Other embodiments are discussed in the detailed description and further description is provided therein. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The nature and advantages of certain embodiments may be further understood by reference to the remainder of the specification and the accompanying drawings, wherein the same reference numerals are used to refer to similar components. In some cases, a sub-label is associated with a reference numeral to indicate one of multiple similar components. When a reference numeral is mentioned without specifying an existing sub-label, it is intended to refer to all of these multiple similar components.

[0010] Figure 1 is a schematic block diagram of a system for encoding and decoding streamable 4D video according to various embodiments;

[0011] Figure 2 is a functional block diagram of a streamable 4D video architecture according to various embodiments;

[0012] Figure 3 is a schematic block diagram of a streamable 4D video system according to various embodiments;

[0013] Figure 4 is a method flow chart of a streamable 4D video framework according to various embodiments;

[0014] Figure 5 is a schematic block diagram of a computer system for streamable 4D video according to various embodiments. DETAILED DESCRIPTION

[0015] Various embodiments provide tools and techniques for a streamable 4D video system.

[0016] In some embodiments, a method for streamable 4D video is provided. The method includes acquiring two or more frames, each of the two or more frames being associated with a corresponding camera view of the video at a first timestamp. The method continues to generate two or more feature maps through a feature extraction network, and generate a first feature volume from the two or more feature maps. The method further includes generating a first encoded feature volume through an encoder, wherein generating the first encoded feature volume includes encoding the first feature volume using a first encoder model; and providing the first encoded feature volume and a first decoder model for decoding the first encoded feature volume through a server.

[0017] In some examples, each feature map of the two or more feature maps corresponds to a corresponding frame of the two or more frames. In some examples, generating the encoded feature volume may further include: obtaining a second feature volume associated with a second timestamp before the first timestamp, connecting the second feature volume and the first feature volume; and encoding the first feature volume connected with the second feature volume. In some examples, the method may further include modifying at least a portion of the second feature volume based at least in part on the first feature volume. In some examples, the first decoder model may be at least a portion of an encoder-decoder model.

[0018] In a further example, the method may include providing, by the server, a second encoded feature volume and a second decoder model for decoding the second encoded feature volume. In some examples, the second decoder model may be different from the first decoder model. In some examples, the second decoder model may be at least a portion of an encoder-decoder model.

[0019] In some embodiments, a device for a streamable 4D video system is provided. The device includes a non-transitory computer-readable medium in communication with a processor, the non-transitory computer-readable medium having encoded thereon an instruction set executable by the processor to perform various functions. The instruction set is executable by the processor to obtain two or more frames (each of the two or more frames is associated with a corresponding camera view of a video at a first timestamp), generate two or more feature maps through a feature extraction network, and generate a first feature volume from the two or more feature maps. The instruction set is further executable by the processor to generate a first encoded feature volume through an encoder, wherein generating the first encoded feature volume includes encoding the first feature volume using a first encoder model; and providing the first encoded feature volume and a first decoder model for decoding the first encoded feature volume through a server.

[0020] In some examples, each feature map in the two or more feature maps corresponds to a corresponding frame in the two or more frames. In some examples, generating the encoded feature volume may further include obtaining a second feature volume associated with a second timestamp before the first timestamp, connecting the second feature volume and the first feature volume, and encoding the first feature volume connected with the second feature volume.

[0021] In some examples, the set of instructions is further executable by the processor to modify at least a portion of the second feature volume based at least in part on the first feature volume. In some examples, the first decoder model is at least a portion of an encoder-decoder model.

[0022] In some examples, the set of instructions is further executable by the processor to provide, by the server, a second encoded feature volume and a second decoder model for decoding the second encoded feature volume. In some examples, the second decoder model may be different from the first decoder model. In some examples, the second decoder model is at least a portion of an encoder-decoder model.

[0023] In a further embodiment, a system for streamable 4D video is provided. The system includes: a feature extraction network for generating two or more feature maps corresponding to two or more frames, each of the two or more frames being associated with a corresponding camera view of the video at a first timestamp; an encoder for generating a first encoded feature volume, wherein generating the first encoded feature volume includes encoding the feature volume using a first encoder model, wherein the feature volume is generated based at least in part on two or more of the feature maps; and a server for providing the first encoded feature volume and a first decoder model for decoding the first encoded feature volume in response to a request for the video at the first timestamp.

[0024] In some examples, the encoder may be further configured to obtain a second feature volume associated with a second time stamp before the first time stamp, connect the second feature volume and the first feature volume, and encode the first feature volume connected with the second feature volume. In some examples, the encoder may be further configured to modify at least a portion of the second feature volume based at least in part on the first feature volume.

[0025] In a further example, the server is configured to provide a second encoded feature volume and a second decoder model for decoding the second encoded feature volume, wherein the second decoder model is different from the first decoder model.

[0026] In the following description, for the purpose of explanation, many details are listed to provide a thorough understanding of the embodiments. However, it is obvious to those skilled in the art that other embodiments can be implemented without some of the details. A plurality of embodiments are described herein, and although different embodiments have different features, it should be understood that the features described in one embodiment can also be combined with other embodiments. Similarly, one or more features in any described embodiment should not be considered to be necessary for each embodiment, because other embodiments of the present application may omit these features.

[0027] When it is mentioned herein that an element is "connected" or "coupled" to another element, it should be understood that these elements can be directly connected to another element or there can be intermediate elements between the elements. Conversely, when an element is referred to as being "directly connected" or "directly coupled" to another element, it should be understood that there are no intermediate elements in the "direct" connection between the elements. However, the presence of a direct connection does not exclude other connections, in which there may be intermediate elements.

[0028] In addition, the terms left, right, front, back, up, down, forward, reverse, clockwise and counterclockwise are used for explanation purposes only and are not limited to any fixed direction or orientation. Instead, they are only used to indicate the relative position and / or direction between parts of objects and / or components.

[0029] In addition, for ease of description, the methods and processes described herein are described in a specific order. However, it should be understood that, unless otherwise specified in the context, intermediate processes may be performed before and / or after any part of the process, and various processes may be reordered, added, and / or omitted according to various embodiments.

[0030] Unless otherwise stated, all numbers used herein to indicate quantities, sizes, etc. are to be understood as being modified in all cases by the term "about". In this application, unless otherwise specifically stated, the use of the singular includes the plural; unless otherwise stated, the terms "and" and "or" refer to "and / or". In addition, the use of the term "including" and other forms, such as "includes" and "included", should be considered non-exclusive. At the same time, terms such as "element" or "component" include both elements and components that constitute one unit and elements and components that constitute multiple units, unless otherwise expressly provided.

[0031] As used herein, the use of the phrase "at least one of" preceding a list of items, with the terms "and" or "or" separating any one of the items, modifies the entire list, not each member of the list (i.e., each item). The phrase "at least one of" does not require that there be at least one of each item listed; rather, the phrase allows for the inclusion of at least one of any one item, and / or at least one of any combination of items. For example, the phrase "at least one of A, B, and C" or "at least one of A, B, or C" refers to only A, only B, or only C, respectively; and / or any combination of A, B, and C. Where the intended selection is "at least one of each of A, B, and C" or "at least one of A, at least one of B, and at least one of C," it will be explicitly described as such.

[0032] Traditional 4D video methods use textured grids, which results in time-consuming encoding steps and high transmission bit rates. Therefore, the following embodiments propose a more efficient and better performing 4D video encoding and transmission method. By utilizing feature volumes and neural networks, lower bit rates are used for transmission.

[0033] Figure 1 1 is a schematic block diagram of a system 100 for encoding and decoding a streamable 4D video according to various embodiments. The system 100 includes a feature extraction network 105 and an encoder-decoder network 130. The encoder-decoder network 130 includes an encoder 110 and a decoder 120. The encoder 110 includes one or more encoder blocks 115a-115n. The decoder 120 includes one or more decoder blocks 125a-125n. It should be noted that Figure 1 Components of the system 100 are schematically illustrated in FIG. 1 , and the components and other arrangements of the system 100 may be modified according to various embodiments.

[0034] In various embodiments, the feature extraction network 105 can be coupled to the encoder 110 of the encoder-decoder network 130. The encoder-decoder network 130 can be part of a transformer and / or CNN-based encoder-decoder architecture. Thus, the encoder-decoder network 130 can include hardware, software, or a combination of hardware and software for performing the encoding and decoding functions described herein.

[0035] The encoder 110 may include one or more encoder blocks 115a-115n. The encoder blocks in one or more encoder blocks 115a-115n may include various types of encoders. For example, in some embodiments, the encoder block may be an encoder of a transformer. Therefore, each encoder block in one or more encoder blocks 115a-115n may further include, for example, a multi-head attention block, a normalization layer, and / or a feed-forward network (FFN). In other examples, the encoder 110 may be an encoder-decoder architecture based on a convolutional neural network (CNN), such as ResNet, a recursive neural network (RNN), a region-based CNN (R-CNN), etc.

[0036] The encoder 110 may be further coupled to a decoder 120 of the encoder-decoder network 130. The decoder 120, like the encoder 110, may include one or more decoder blocks 125a. The decoder blocks in the one or more decoder blocks 125a-125n may include various types of decoders. For example, in some embodiments, the decoder block may be a decoder of a transformer. Therefore, each decoder block in the one or more decoder blocks 125a-125n may further include, for example, a multi-head attention block, a normalization layer, and / or a FFN. In other examples, the decoder 120, like the encoder 110, may be a decoder of an encoder-decoder architecture based on a CNN.

[0037] In various embodiments, the feature extraction network 105 and the encoder-decoder network 130 (e.g., encoder 110 and decoder 120) and / or components thereof may be implemented as hardware and / or software running on one or more computer systems. For example, the feature extraction network 105, encoder 110, decoder 120 and / or components thereof may be implemented as hardware logic (e.g., programmable logic such as a field programmable gate array (FPGA)) and / or a dedicated custom integrated circuit (e.g., an application specific integrated circuit (ASIC)).

[0038] In various examples, the feature extraction network 105 (often referred to as the backbone network) can be configured to generate two or more feature maps from input data, such as images (e.g., frames of a video). In various examples, two or more feature maps can be generated from two or more frames of a 4D video. As used herein, a 4D video can refer to a video in which a user can move a camera (e.g., a view) while the video is playing. Therefore, the three-dimensional (3D) scene of the 4D video frame can be a dynamic scene in which the user can freely manipulate the viewpoint of the video.

[0039] In some embodiments, each corresponding frame of the two or more frames may correspond to a corresponding camera (e.g., view). In a further example, each corresponding frame of the two or more frames may further correspond to the same timestamp of the video. The feature extraction network 105 may include various types of CNNs, such as ResNet, RNN, R-CNN, etc., for extracting and / or identifying features in the input data, and further generating corresponding feature maps from the corresponding frames.

[0040] In various embodiments, corresponding feature maps of two or more frames may be combined to form a feature volume. In some examples, generating a feature volume may include connecting two or more feature maps together. Thus, a feature volume may be considered as a feature map volume (e.g., a stack of feature maps). In some examples, a given feature volume may correspond to a corresponding timestamp of a 4D video. In other words, the feature volume of a given timestamp may include feature maps of at least two frames of the 4D video at the timestamp, or in some further examples, feature maps of all available frames. In some examples, feature volumes may be generated for all timestamps of a 4D video.

[0041] Thus, in various embodiments, the feature volume generated by the feature extraction network 105 may be provided as an input to the encoder-decoder network. In various embodiments, the encoder 110 may obtain the feature volume (one or more feature volumes of the 4D video) from the feature extraction network 105. In various embodiments, the encoder 110 may be used to encode each feature volume of the 4D video generated by the feature extraction network 105. For example, in some embodiments, the encoder 110 may apply an attention mechanism (e.g., self-attention) to each feature volume. In some examples, the attention weights may be used to filter useful features (e.g., relevant features) of the feature volume based on the query, keyword, and / or value vector. Thus, each feature volume may be encoded with attention weight information to generate an encoded feature volume. The encoded feature volume may be encoded using information about how a particular feature pays attention to all other features in each frame of the 4D video at a given timestamp. Each encoder block 115a-115n may pass the encoded feature volume to a subsequent encoder block in one or more encoder blocks 115a-115n for further encoding, wherein the first encoder block 115a receives an unencoded feature volume from the feature extraction network 105. Thus, in the above example, the encoder-decoder network 130 may be part of a transformer neural network.

[0042] In a further example, the encoder 110 may encode the feature volume using a CNN-based neural network. Thus, the encoder 110 may encode the feature volume using a CNN-based network. In some examples, the encoder 110 may be further configured to reduce the size of the feature volume (e.g., compress) by encoding the feature volume.

[0043] Once encoded, each encoded feature volume may be decoded by decoder 120 to generate a decoded feature volume, from which a frame corresponding to a camera (e.g., a viewpoint) selected by a user may be generated. In some examples, the feature volume may not include all viewpoints that a user may select. In other words, a user may select a camera that does not have an original frame in the 4D video. In various examples, through the decoding process, a frame corresponding to a camera selected by the user may be generated.

[0044] As previously described with respect to encoder 110, decoder 120 may include a transformer decoder. Alternatively, decoder 120 may be a CNN-based decoder. Unlike encoder 110, decoder 120 takes an encoded feature volume as its input. In some examples, decoder 120 may also take a user-selected camera position (e.g., a viewpoint) as input. Thus, decoder 120 may have a similar (or identical) structure to encoder 110. However, unlike encoder 110, decoder 120 may accept different inputs, such as different attention weights (e.g., queries, keywords, and / or value inputs). Thus, in various examples, the output of decoder 120 may be a predicted frame (e.g., an image) generated from the decoded feature volume for a selected camera position. In some examples, the predicted frame may be generated based on a view synthesis algorithm, such as a volume-based implicit representation and / or image-based rendering (e.g., light field rendering).

[0045] Figure 2 is a functional block diagram of a streamable 4D video architecture 200 according to various embodiments. The architecture 200 includes a feature volume 205, an encoder 210, a decoder 215, a camera view 220, a loss function 225, and a ground truth image 230. It should be noted that Figure 2 The various components of the architecture 200 are schematically illustrated in FIG. 2 . The various components and other layouts of the architecture 200 may be modified according to different embodiments.

[0046] In various embodiments, architecture 200 illustrates the Figure 1The functionality of the encoder-decoder network described in, and the training of the encoder-decoder network. As previously described, the feature volume 205 can be provided to the encoder 210. For example, the feature volume 205 can be generated by a feature extraction network and generated by combining two or more feature maps. Each feature map of the feature volume can correspond to a corresponding frame of the 4D video at a given timestamp, wherein each frame further corresponds to a corresponding camera view (e.g., a viewpoint).

[0047] Therefore, the feature volume 205 can be provided to the encoder 210 for encoding. In some examples, the encoder 210 can encode the feature volume 205 through a self-attention mechanism. In other examples, the encoder 210 can be a CNN-based encoder for encoding the feature volume 205.

[0048] The encoded feature volume from the encoder 210 may be used as an input to the decoder 215 along with the camera view 220. In some examples, the camera view 220 may be a user-selected camera view. For training purposes, in some examples, the user-selected camera view may be selected to coincide with a frame of an existing ground truth image 230. As previously described, in some examples, the decoder 215 may further include an attention mechanism, wherein the selected camera view may be used as a query for the decoder 215. Thus, in various embodiments, the encoder 210 and the decoder 215 may be part of a transformer-based network. Alternatively, the encoder 210 and the decoder 215 may be part of an encoder-decoder network based on a CNN.

[0049] The output of decoder 215 may itself be an encoded version of the encoded feature volume. In some examples, this is an image predicted from camera view 220 (e.g., the viewpoint of a selected camera). The output of decoder 215 may be input to loss function 225 along with the corresponding ground truth image 230 for calculating loss function 225. Architecture 200 may be guided by loss function 225, which may be a contrast loss in various examples. In some embodiments, contrast loss employs a similarity matrix of the predicted image and the ground truth image 230.

[0050] In various embodiments, the loss function 225 may be optimized to adjust the encoder 210 and / or decoder 215 models. For example, optimization of the loss function may include adjusting one or more parameters to minimize the loss and / or maximize the similarity between the predicted image (e.g., the output of the decoder 215) and the ground truth image 230. For example, optimizing the parameters of the feature extraction network, the encoder 210, or the decoder 215 may include modifying filters, masks, and / or weights used in feature extraction, layer normalization, and / or FFN.

[0051] Figure 33 is a schematic block diagram of a streamable 4D video system 300 according to various embodiments. The system 300 includes an encoder 305, a server 310, a network 315, and one or more client devices 320a-320n. It should be noted that Figure 3 Various components of system 300 are schematically shown in FIG. 3 , and various components and other arrangements of system 300 may be modified according to various embodiments.

[0052] In various examples, encoder 305 may be an encoder of an encoder-decoder network as described above. Encoder 305 may be configured to provide coded feature volumes (more specifically, all coded feature volumes of 4D video) to server 310. In some embodiments, encoder 305 may include one or more encoders of different types. For example, in some embodiments, multiple encoders of different types may be used to encode groups of feature volumes of 4D video. In some examples, a first encoder of a first type may be used to encode a first group of coded feature volumes, a second encoder of a second type may be used to encode a second group of coded feature volumes, and so on. In some examples, if present, the first group of coded feature volumes, the second group of feature volumes, and the coded feature volumes of other groups may overlap (e.g., cover the same timestamp range). In other examples, different groups of coded feature volumes may cover different timestamps that do not overlap (e.g., each timestamp is encoded by a single type of encoder).

[0053] For example, in some embodiments, the encoder 305 can utilize feature volumes from a frame window (e.g., a sliding window of timestamps). In some examples, when processing frame i+I, feature volumes that have been encoded for a previous frame (e.g., frame i) can be utilized. Thus, as described below, the feature volume of frame i and the feature volume of frame i+I can be concatenated and used to encode and render frame i+I through a decoding process.

[0054] Although only one previous frame is used in this example, in other examples, multiple previous frames may be used (e.g., concatenated) with the feature volume of the current frame. In further examples, the encoding of the current frame may be improved using "future" frames. For example, after processing frame i+I (e.g., generating a feature volume), the feature volume of frame i may be timed (e.g., modified) based on the feature volume of frame i+I.

[0055] In various examples, server 310 may be further configured to obtain a corresponding neural network model (interchangeably referred to as a decoder model) of a decoder for decoding each corresponding encoded feature volume. In some examples, server 310 may obtain the decoder model from encoder 305 or an encoder-decoder network. For example, if a first feature volume is encoded using a first encoder-decoder network, a first decoder corresponding to encoder 305 may be provided (e.g., transmitted or otherwise provided) to server 310 for decoding the first feature volume. Thus, the first decoder may be a decoder corresponding to the encoder-decoder network used to encode the corresponding feature volume.

[0056] In various examples, the server 310 may be used to provide 4D video content that can be streamed. Accordingly, the server 310 may include a server in a content distribution network or other streaming media server, which is configured to provide 4D video content that can be streamed to one or more client devices 320a-320n via a network 315. Therefore, in various embodiments, the server 310 is used to provide a coded feature volume and a corresponding decoder to one or more client devices 320a-320n. In some embodiments, the server 310 may be used to receive a request for 4D video content at a specific timestamp from a client device in one or more client devices 320a-320n. For example, a first client device 320a may request a 4D video starting from a first timestamp. The server 310 may be used to transmit a first coded feature volume (e.g., a first coded feature volume associated with a first timestamp) of a 4D video at a first timestamp to the client device 320a and a first decoder for decoding the first coded feature volume. Therefore, as previously described, the first decoder corresponds to a first encoder for encoding a first feature volume in an encoder-decoder network. The server 310 may transmit in real time for each subsequent timestamp a subsequent encoded feature volume and a corresponding decoder so that the requested 4D video content may be decoded on the requesting client device (e.g., the first client device 320a). In this way, the 4D video content may be streamed to the client devices 320a-320n, while the encoded feature volume is decoded on the corresponding client devices 320a-320n.

[0057] In various embodiments, the client device 320a can therefore provide the decoder with a corresponding user-specified camera position (e.g., viewpoint) and decode the feature volume to generate an image (e.g., frame) at the user-specified camera position. In addition, each client device 320a-320n can each request 4D video content at a different timestamp (or completely different 4D video content). Therefore, the server 310 can be configured to support multiple 4D video streams on each client device 320a-320n. For example, a first client device 320a can request a 4D video at a first timestamp, and a second client device 320b can request a 4D video at a second timestamp. In a certain example, the first timestamp and the second timestamp can be different. Therefore, the server 310 can provide a first encoded feature volume (e.g., associated with the first timestamp) of the first timestamp to the first client device 320a, and provide a second encoded feature volume of the second timestamp to the second client device.

[0058] The server 310 may continue to transmit subsequent encoded feature volumes (e.g., corresponding to subsequent timestamps) and corresponding decoders to decode each corresponding encoded feature volume, so that the feature volume can be decoded in real time at the client devices 320a-320n. When a new request with a different timestamp is received, the server 310 provides the encoded feature volume corresponding to the requested timestamp and the decoder corresponding to the encoder used to encode the corresponding encoded feature volume, so as to decode the corresponding encoded feature volume.

[0059] Figure 4 4 is a flow chart of a method 400 for a streamable 4D video framework according to various embodiments. The method 400 includes acquiring two or more frames of a 4D video at step 405. As previously described, the two or more frames may be frames of the same timestamp of the 4D video, but are different camera views (e.g., viewpoints). At step 410, the method 400 continues to generate corresponding feature maps for the two or more frames. As previously described, a feature extraction network may be used to generate two or more corresponding feature maps. The feature extraction network may include various types of CNNs, such as ResNet, RNN, R-CNN, etc., for extracting and / or identifying features in two or more frames, and further generating corresponding feature maps from the corresponding frames.

[0060] At step 415, method 400 continues to generate feature volumes from corresponding feature maps. In various embodiments, as previously described, corresponding feature maps of two or more frames may be combined together to form a feature volume. In some examples, generating a feature volume may include connecting two or more feature maps together. Thus, a feature volume may be considered as a feature map volume (e.g., a stack of feature maps). In some examples, a given feature volume may correspond to a corresponding timestamp of a 4D video. In other words, the feature volume for a given timestamp may include feature maps of at least two frames of the 4D video at the timestamp, or in some further examples, feature maps of all available frames. In further embodiments, feature volumes may be generated for all timestamps of a 4D video.

[0061] In step 420, method 400 continues to encode the feature volume. In various embodiments, the feature volume generated by the feature extraction network 105 may be provided as an input to the encoder-decoder network. In various examples, the encoder may obtain the feature volume (one or more feature volumes of the 4D video) from the feature extraction network. The encoder may be used to encode each feature volume of the 4D video generated by the feature extraction network. For example, in some embodiments, the encoder may apply an attention mechanism (e.g., self-attention). Therefore, each feature volume can be encoded with attention weight information to generate an encoded feature volume. In other examples, the encoder may be a CNN-based encoder as described above.

[0062] In some embodiments, encoder 305 can utilize feature volumes from a window of frames (e.g., a sliding window of timestamps). In some examples, when processing frame i+I, feature volumes that have been encoded for the previous frame (e.g., frame i) can be utilized. Thus, the feature volume of frame i and the feature volume of frame i+I can be concatenated and used for encoding.

[0063] Although only one previous frame is used in this example, in other examples, multiple previous frames can be used (e.g., concatenated) with the feature volume of the current frame. In further examples, the encoding of the current frame can be improved using "future" frames. For example, after processing frame i+I (e.g., generating a feature volume), the feature volume of frame i can be timed (e.g., modified) based on the feature volume of frame i+I.

[0064] At step 425, method 400 includes receiving a request for a 4D video from a client device. As previously described, a 4D video is a video in which a user can dynamically interact with a 3D scene by controlling a camera position. Thus, the 3D scene of a 4D video frame can be a dynamic scene in which a user can freely manipulate the viewpoint of the video. In some embodiments, the request for a 4D video may include a timestamp of the requested 4D video.

[0065] Accordingly, at step 430, method 400 may continue by providing an encoded feature volume and a corresponding decoder model, the decoder model being used to decode the encoded feature volume. As previously described, in various examples, a corresponding neural network model may be provided for a decoder (e.g., a decoder model) for decoding each corresponding encoded feature volume and provided together with the encoded feature volume. Thus, the first decoder may be a decoder corresponding to an encoder-decoder network for encoding the corresponding feature volume. In this way, streamable 4D video content may be provided. In some examples, subsequent encoded feature volumes and corresponding decoders may be further provided after receiving an initial request for a 4D video. As previously described, by providing the encoded feature volume and the decoder model to the client device, the client device may locally decode the encoded feature volume to generate frames of a user-selected camera view (e.g., a viewpoint), including viewpoints that may not have corresponding ground truth frames in the 4D video. For example, in some embodiments, the above-described view synthesis algorithm may be used to generate a predicted frame (e.g., an image).

[0066] Figure 5 is a schematic block diagram of a computer system 500 for streamable 4D video according to various embodiments. Figure 5 A schematic illustration of one embodiment of a computer system 500 is provided, such as the systems 100, 300 and architecture 200 or components thereof, which can perform the methods provided by various other embodiments described herein. It should be noted that Figure 5 Only a general description of the various components is provided, one or more of which may be used as appropriate. Figure 5 It generally describes how the various system components can be implemented in a separate or integrated manner.

[0067] The computer system 500 includes a plurality of hardware elements that may be electrically coupled (or otherwise in communication, as appropriate) via a bus 505. The hardware elements may include one or more processors 510, including but not limited to one or more general-purpose processors and / or one or more special-purpose processors (such as microprocessors, digital signal processing chips, graphics acceleration processors, and microcontrollers); one or more input devices 515, including but not limited to a mouse, a keyboard, one or more sensors, and / or similar devices; and one or more output devices 520, including but not limited to a display device and / or similar devices.

[0068] The computer system 500 may further include (and / or communicate with) one or more storage devices 525, which may include, but are not limited to, local and / or network accessible storage, and / or may include, but are not limited to, disk drives, drive arrays, optical storage devices, solid-state storage devices, such as random access memory ("RAM") and / or read-only memory ("ROM"), which may be programmable, flash-updated, and / or the like. Such storage devices may be used to implement any suitable data storage, including, but not limited to, various file systems, database structures, and / or the like.

[0069] The computer system 500 may also include a communication subsystem 530, which may include but is not limited to a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device and / or a chipset (e.g., Bluetooth TM Devices, 802.11 devices, WiFi devices, WiMax devices, WWAN devices, Z-Wave devices, ZigBee devices, cellular communication facilities, etc.), and / or low-power wireless devices. The communication subsystem 530 can allow data to be exchanged with other computer or hardware systems, between data centers or between different cloud platforms, and / or with any other devices described herein through a network (such as the network described below, just to name an example). In many embodiments, the computer system 500 also includes a working memory 535, which can include RAM or ROM devices as described above.

[0070] The computer system 500 may also include software elements, shown as currently located in the working memory 535, including an operating system 540, device drivers, executable libraries, and / or other code, such as one or more application programs 545, which may include computer programs provided by various embodiments, and / or may be designed to implement the methods provided by other embodiments described herein, and / or configure the system. By way of example only, one or more programs described in the methods discussed above may be implemented as code and / or instructions executable by a computer (and / or a processor in a computer); in one aspect, such code and / or instructions may be used to configure and / or adjust a general purpose computer (or other device) to perform one or more operations according to the described methods.

[0071] The set of these instructions and / or codes may be encoded and / or stored on a non-transitory computer-readable storage medium, such as the storage device 525 described above. In some cases, the storage medium may be built into a computer system, such as the system 500. In other embodiments, the storage medium may be separate from the computer system (i.e., removable media, such as a compact disc, etc.), and / or provided in an installation package, so that the storage medium can use the instructions / codes stored therein to program, configure, and / or adapt a general-purpose computer. These instructions may take the form of executable code, which is executed by the computer system 500, and / or may take the form of source code and / or installable code, which is converted into the form of executable code after being compiled and / or installed on the computer system 500 (e.g., using any of a variety of general-purpose compilers, installers, compression / decompression tools).

[0072] It will be apparent to those skilled in the art that a great many adjustments can be made based on specific requirements. For example, custom hardware (such as programmable logic controllers, single board computers, FPGAs, ASICs, and SoCs) may also be used, and / or specific elements may be implemented in hardware, software (including portable software such as applets, etc.), or a combination of both. In addition, other computing devices may be connected, such as network input / output devices.

[0073] In one aspect, some embodiments may employ a computer or hardware system (e.g., computer system 500) to perform methods consistent with various embodiments of the present application. According to one set of embodiments, some or all of the procedures of these methods are performed by computer system 500 in response to processor 510 executing one or more instruction sequences stored in working memory 535 (these instruction sequences may be integrated into operating system 540 and / or other code, such as application program 545). These instructions may be read into working memory 535 from another computer-readable medium (e.g., one or more storage devices 525), and by way of example only, execution of the instruction sequences contained in working memory 535 may cause processor 510 to perform one or more procedures of the methods described herein.

[0074] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any medium that participates in providing data that causes a machine to operate in a particular manner. In one embodiment implemented using computer system 500, various computer-readable media may participate in providing instructions / codes for execution to processor 510 and / or for storing and / or carrying such instructions / codes (e.g., as signals). In many embodiments, a computer-readable medium is a non-temporary, physical, and / or tangible storage medium. In some embodiments, a computer-readable medium may have a variety of forms, including but not limited to non-volatile media, volatile media, or similar media. For example, non-volatile media include optical and / or magnetic disks, such as storage device 525. Volatile media include but are not limited to dynamic memory, such as working memory 535. In some alternative embodiments, a computer-readable medium may take the form of a transmission medium, including but not limited to coaxial cables, copper wires, and optical fibers, including wires that make up bus 505, and various components of communication subsystem 530 (and / or the medium by which communication subsystem 530 communicates with other devices). In another set of alternative embodiments, transmission media may also take the form of waves (including but not limited to radio waves, acoustic waves and / or light waves, such as those generated during radio waves and infrared data communications).

[0075] Common forms of physical and / or tangible computer readable media include, for example, floppy disks, diskettes, hard disks, magnetic tapes or any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media with patterns of holes, RAM, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cassettes, carrier waves as described below, or any other medium from which a computer can read instructions and / or code.

[0076] Various forms of computer-readable media may be used to transmit one or more sequences of instructions to the processor 510 for execution. By way of example only, the instructions may initially be carried on a magnetic disk and / or optical disk of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions as signals over a transmission medium to be received and / or executed by the computer system 500. According to various embodiments of the present application, these signals may be in the form of electromagnetic signals, acoustic wave signals, optical wave signals, and / or similar signals, which are all examples of carrier waves on which instructions may be encoded.

[0077] The communication subsystem 530 (and / or its components) typically receives the signal, and the bus 505 may then transfer the signal (and / or the data, instructions, etc. carried by the signal) to the working memory 535, from which the processor 510 retrieves and executes the instructions. The instructions received by the working memory 535 may optionally be stored on the storage device 525 before or after execution by the processor 510.

[0078] Although some features and aspects have been described with respect to the embodiments, those skilled in the art will recognize that many modifications may be made. For example, the methods and processes described herein may be implemented using hardware components, software components, and / or any combination thereof. In addition, although the various methods and processes described herein may be described with respect to specific structural and / or functional components for ease of description, the methods provided by the various embodiments are not limited to any specific structural and / or functional architecture, but may be implemented on any suitable hardware, firmware, and / or software configuration. Similarly, although some functions are attributed to one or more system components, unless the context otherwise dictates, these functions may be distributed across various other system components according to multiple embodiments.

[0079] In addition, although the steps of the methods and processes described herein are described in a particular order for ease of description, various steps may be reordered, added, and / or omitted according to various embodiments unless the context otherwise requires. In addition, the steps described for a method or process may be incorporated into other described methods or processes; similarly, system components described according to a particular structural architecture and / or for a system may be organized into alternative structural architectures and / or incorporated into other described systems. Therefore, although various embodiments are described as having or not having some features for ease of description and illustration of some aspects of these embodiments, various components and / or functions described herein for specific embodiments may be replaced, added, and / or reduced in other described embodiments unless the context otherwise requires. Therefore, although several embodiments are described above, it will be understood that the present application is intended to cover all modifications and equivalents within the scope of the claims.

Claims

1. A method, include: acquiring two or more frames, each of the two or more frames being associated with a corresponding camera view of the video at a first timestamp; generating two or more feature maps through a feature extraction network; generating a first feature volume from two or more of the feature maps; generating, by an encoder, a first encoded feature volume, wherein generating the first encoded feature volume comprises encoding the first feature volume using a first encoder model; And providing, by a server, the first encoded feature volume and a first decoder model for decoding the first encoded feature volume.

2. The method according to claim 1, It is characterized in that Each of the two or more feature maps corresponds to a corresponding frame of the two or more frames.

3. The method according to claim 1, It is characterized in that Generating the encoded feature volume further includes: Acquire a second feature volume associated with a second timestamp before the first timestamp; The second feature volume is connected to the first feature volume; and the first feature volume connected to the second feature volume is encoded.

4. The method according to claim 3, further comprising: include: Based at least in part on the first characteristic volume, at least a portion of the second characteristic volume is modified.

5. The method according to claim 1, It is characterized in that The first decoder model is at least a part of an encoder-decoder model.

6. The method according to claim 1, further comprising: include: A second encoding feature volume and a second decoder model for decoding the second encoding feature volume are provided by the server.

7. The method according to claim 6, Features: The second decoder model is different from the first decoder model.

8. The method according to claim 7, Features: The second decoder model is at least a part of an encoder-decoder model.

9. A non-transitory computer-readable medium in communication with a processor, the non-transitory computer-readable medium comprising a set of instructions encoded thereon executable by the processor to perform: Acquire two or more frames, each of the two or more frames being associated with a corresponding camera view of the video at a first timestamp; Generate two or more feature maps through a feature extraction network; generating a first feature volume from two or more of the feature maps; A first encoded feature volume is generated by an encoder, wherein: Generating the first encoded feature volume includes encoding the first feature volume using a first encoder model; And providing, by a server, the first encoded feature volume and a first decoder model for decoding the first encoded feature volume.

10. The non-transitory computer readable medium of claim 9, It is characterized in that Each of the two or more feature maps corresponds to a corresponding frame of the two or more frames.

11. The non-transitory computer readable medium of claim 9, It is characterized in that The generating of the encoding feature volume further comprises: Acquire a second feature volume associated with a second timestamp before the first timestamp; The second feature volume is connected to the first feature volume; and the first feature volume connected to the second feature volume is encoded.

12. The non-transitory computer readable medium of claim 11, It is characterized in that The set of instructions is also executable by the processor to modify at least a portion of the second characteristic volume based at least in part on the first characteristic volume.

13. The non-transitory computer readable medium of claim 9, It is characterized in that The first decoder model is at least a part of an encoder-decoder model.

14. The non-transitory computer readable medium of claim 9, It is characterized in that The set of instructions is also executable by the processor to: provide, by the server, a second encoded feature volume and a second decoder model for decoding the second encoded feature volume.

15. The non-transitory computer readable medium of claim 14, It is characterized in that The second decoder model is different from the first decoder model.

16. The non-transitory computer readable medium of claim 15, It is characterized in that The second decoder model includes at least a portion of an encoder-decoder model.

17. A system, include: a feature extraction network for generating two or more feature maps corresponding to two or more frames, each of the two or more frames being associated with a corresponding camera view of the video at a first timestamp; an encoder for generating a first encoded feature volume, wherein generating the first encoded feature volume comprises encoding a feature volume using a first encoder model, wherein the feature volume is generated based at least in part on two or more of the feature maps; and A server for providing the first encoding feature volume and a first decoder model for decoding the first encoding feature volume in response to a request for the video at the first timestamp.

18. A system according to claim 17, It is characterized in that The encoder is further used to: obtain a second feature volume associated with a second timestamp before the first timestamp; connecting the second characteristic volume and the first characteristic volume; And encoding the first feature volume connected to the second feature volume.

19. The system of claim 18, It is characterized in that The encoder is also configured to modify at least a portion of the second characteristic volume based at least in part on the first characteristic volume.

20. The system according to claim 17, It is characterized in that The server is further configured to: provide a second encoded feature volume and a second decoder model for decoding the second encoded feature volume; Wherein, the second decoder model is different from the first decoder model.

Images