Three-dimensional video highlights from camera sources

Abstract

According to an aspect, a method includes generating a three-dimensional (3D) video segment from two-dimensional (2D) video content captured by a camera system, including obtaining 3D movement data of an object detected in the 2D video content from a 3D pose estimation engine, and generating an animated object based on the 3D movement data, the movement of the animated object corresponds to the movement of the object in the 2D video content. The method includes generating 3D video content from the 3D video segment and transmitting the 3D video content to a user device for display.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 367,525, filed on July 1, 2022, entitled “THREE DIMENSIONAL (3D) HIGHLIGHTS,” the disclosure of which is incorporated herein by reference in its entirety. Background Art

[0002] Three-dimensional (3D) production can be the process of creating and producing content in three dimensions (e.g., height, width, and depth), and can encompass various techniques and technologies for capturing, generating, manipulating, and / or presenting content in a 3D format. In 3D production, content can be created or captured in a way that simulates depth perception, thereby providing a more immersive and realistic experience. Conventional 3D production can involve many technical processes that can be time consuming and can use relatively large amounts of computing resources (e.g., memory, central processing unit (CPU), and / or graphics processing unit (GPU) power). Summary of the invention

[0003] In some aspects, the technology described herein relates to a method that includes: generating a three-dimensional (3D) video clip from two-dimensional (2D) video content captured by a camera system, including: obtaining 3D movement data of an object detected in the 2D video content from a 3D pose estimation engine; and generating an animated object based on the 3D movement data so that movement of the animated object corresponds to movement of the object in the 2D video content; generating 3D video content from the 3D video clip; and transmitting the 3D video content to a user device for display.

[0004] In some aspects, the technology described herein relates to a non-transitory computer-readable medium storing executable instructions that cause at least one processor to perform operations including: transmitting a three-dimensional (3D) viewing request over a network to a video manager executable by at least one server computer, the 3D viewing request being a request to view two-dimensional (2D) content captured by a camera system in a 3D format; receiving 3D video content from the video manager over the network, the 3D video content being generated from a 3D video clip generated by the video manager using the 2D video content; and initiating display of the 3D video content on an interface of a user device, the 3D video content including an animated object whose movement corresponds to movement of an object in the 2D video content.

[0005] In some aspects, the technology described herein relates to an apparatus that includes: at least one processor; and a non-transitory computer-readable medium storing executable instructions, wherein the executable instructions cause the at least one processor to: receive a three-dimensional (3D) viewing request from a user device to view two-dimensional (2D) video content in 3D format; retrieve a 3D video clip corresponding to the 2D video content from a video database, the 3D video clip including an animated object, the movement of the animated object corresponding to the movement of an object in the 2D video content; generate 3D video content from the 3D video clip; and transmit the 3D video content to the user device for display.

[0006] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1A A system for generating three-dimensional (3D) video clips from two-dimensional (2D) video content according to an aspect is shown.

[0008] Figure 1B An example of a video manager for identifying 2D video segments from television content according to an aspect is shown.

[0009] Figure 1C An example of an object tracker for generating 3D motion data according to an aspect is shown.

[0010] Figure 1D An example of 3D movement data of multiple key points according to an aspect is shown.

[0011] Figure 1E An object tracker according to another aspect is shown.

[0012] Figure 1F An example of a 3D object engine configured to generate animated objects using 3D movement data is shown according to an aspect.

[0013] Figure 1G An example of a 3D object engine selecting a 3D object model from an object model database according to an aspect is shown.

[0014] Figure 1H An example of a 3D object engine that generates a 3D object model from 2D video content according to an aspect is shown.

[0015] Fig. 1I An interface depicting example user controls for controlling playback of a 3D video clip is shown according to an aspect.

[0016] Figure 1JExample communications between a video manager and a user device according to an aspect are shown.

[0017] Figure 1K An example of a video manager for generating metadata associated with 3D video clips and searching for 3D video clips in response to a search query is shown according to an aspect.

[0018] Figure 1L An example of sharing customized 3D video content with another user device according to an aspect is shown.

[0019] Figure 1M An example of a system having a video manager executing on one or more server computers according to an aspect is shown.

[0020] Figure 1N An example of a system having a video manager portion executing on one or more server computers and a video manager portion executing on a user device according to an aspect is shown.

[0021] Figures 2A to 2M

[0013] An example interface for displaying a 3D video clip with user controls for adjusting playback of the 3D video clip is shown in accordance with various aspects.

[0022] Figure 3A and Figure 3B An example interface for displaying a 3D video clip is shown according to an aspect.

[0023] FIG. 4A to FIG. 4C An example display of 2D video content and 3D video content according to an aspect is shown.

[0024] FIG. 5A to FIG. 5C Example 3D video content depicting animated objects that move in a manner corresponding to objects in 2D video content is shown according to an aspect.

[0025] Figure 6 A flow chart depicting example operation of a system for generating 3D video clips according to an aspect is shown.

[0026] Figure 7 A flow chart depicting example operation of a system for receiving and displaying 3D video clips according to an aspect is shown.

[0027] Figure 8 A flow chart depicting example operation of a system for retrieving and distributing 3D video clips according to an aspect is shown. DETAILED DESCRIPTION

[0028] The present disclosure is related to a system that generates three-dimensional (3D) video clips (e.g., 3D model data) from two-dimensional (2D) video content (e.g., sports footage), and can render 3D video content in near real time, which is transmitted to and displayed on the interface of an application executable by a user device. The 3D video content includes images of scenes captured in 2D video content (e.g., basketball players dunking, football teams scoring goals, etc.) replayed in 3D format. In some examples, the 3D video content includes virtual reality (VR) content configured to be displayed on a 3D display (e.g., VR wearable device). In some examples, the 3D video content includes augmented reality (AR) content configured to be displayed on a 2D display. The system includes a video manager that performs technical processes to improve the speed and / or accuracy of 3D production while reducing the amount of computing resources (e.g., memory, central processing unit (CPU) power, graphics processing unit (GPU) power, etc.) used to enable the display of 3D video content on a 2D display or a 3D display.

[0029] For example, the video manager can quickly and accurately generate 3D video clips (e.g., 3D video highlights) from 2D video content by obtaining 3D movement data of objects detected in the 2D video content from a 3D pose estimation engine, wherein the 3D movement data includes the positioning and orientation of objects in 3D space that capture the movement of the objects in 3D space. The video manager generates animated objects based on the 3D movement data, wherein the movement of the animated objects corresponds to the movement of the objects in the 2D video content. For example, the real movement captured by the 3D movement data is applied to a computer-generated object (e.g., a 3D object model), which converts the static model into a dynamic model that moves in the same / similar manner as the object in the 2D video content. The system can achieve real-time rendering of 3D video clips on a user device by using a streaming engine (e.g., a cloud-based 3D pixel streaming engine), which generates (e.g., renders) 3D video content from the 3D video clips and regenerates (e.g., re-renders) the 3D video content according to any user selection, which can minimize, reduce, or eliminate the need for the user device to have a dedicated GPU device.

[0030] The video manager includes a 3D video generator configured to generate 3D video clips (e.g., 3D model data) from 2D video content. The 2D video content can be any type of 2D video content captured by the camera system. In some examples, the 2D video content is a portion of a video file or video content from a video file. In some examples, the 2D video content includes video highlights from a sporting event. However, it is worth noting that the systems discussed herein are not limited to sports highlights, and the techniques discussed herein can be used for any type of 2D video content that has been captured from a camera system.

[0031] In some examples, a user may use their user device to view 2D video content and may initiate a 3D viewing request to view the 2D video content in 3D format. For example, the interface may include a UI element that, when selected, causes the 3D viewing request to be transmitted to the video manager. In response to the 3D viewing request, the 3D video generator may generate a 3D video clip (e.g., generate the 3D video clip on the fly or in near real time). In some examples, in response to the 3D viewing request, the 3D video generator may obtain the 3D video clip from a video database (e.g., a previously generated 3D video clip).

[0032] In some examples, the 2D video content includes television content (e.g., live television content), and the 3D video generator can generate one or more 3D video clips from one or more 2D video clips while the live television content is being broadcast. In some examples, the 3D video generator identifies that the 2D video content includes a key event (e.g., an athlete makes a key performance such as a goal), selects the 2D video clip that includes the key event, and generates the 3D video clip from the 2D video clip. In some examples, the 3D video generator can receive the 2D video clip from an external service during the live broadcast process, wherein the external service includes logic for selecting the 2D video clip that includes the key event.

[0033] The 3D video generator can use a machine learning (ML) model to obtain 3D movement data (e.g., 3D pose data, 3D subject positioning data) of a relevant object (or objects) detected in the 2D video content. In some examples, the 3D video generator can obtain 3D movement data from an existing (known) ML pose estimation model (e.g., a 3D human pose estimation model and / or a 3D object (e.g., ball) tracking model). The 3D movement data may include positioning and rotation information describing the movement and orientation of an object in 3D space. In some examples, the 3D video generator includes an ML human pose estimation model that is configured to detect the movement and orientation of a pose of a human object in 3D space (e.g., represented by key points such as ankles, shoulders, necks, hands, etc.). In some examples, the 3D video generator includes an object detection and tracking model that is configured to detect the movement and orientation of a non-human object (e.g., a ball) in 2D space or 3D space. In some examples, the object detection and tracking model can provide 2D positioning of non-human objects. In some examples, the object detection and tracking model may provide a height from the ground relative to the player at each frame.

[0034] The 3D video generator can use the 3D movement data to generate an animated object and include the animated object in a 3D video clip. In some examples, the 3D video generator can generate an animated object by applying the 3D movement data to a 3D object model that can represent an object in some examples. In some examples, the 3D video generator can select an existing 3D object model corresponding to the detected object from an object model database (e.g., a model list), or can use one or known mesh generation techniques to generate the 3D object model itself using 2D video content. The operations performed by the 3D video generator can enable the animated object depicted in the 3D video content to have a smooth and realistic movement that reflects the movement of the object in the 2D video content. For example, if the object is a basketball player, the 3D video generator can generate 3D movement data about the movement of the player in 3D space from the 2D video content. In some examples, a 3D generator may obtain (or generate) a 3D model object representing a basketball player (which may be a specific known basketball player or a general basketball player) and apply 3D movement data to the 3D object model to generate an animated object, wherein the 3D object model is animated according to the movement of the player.

[0035] The system may include a streaming engine configured to generate 3D video content from a 3D video clip. In some examples, the streaming engine is a 3D pixel streaming engine. The streaming engine can implement near real-time streaming and rendering of 3D graphics and interactive content over a network, which may include one or more rendering operations, such as geometry processing, shading and material calculations, camera and viewpoint calculations, frame synthesis, video encoding, and / or network transmission. For example, the streaming engine includes one or more GPUs configured to perform rendering operations on 3D video clips to generate images (e.g., video frames) of 3D video content. The streaming engine can transmit the 3D video content to a user device for display. In some examples, since the rendering operation is performed by the streaming engine, the 3D video content is transmitted to the user without the user device having special GPU hardware. The streaming engine can receive information indicating one or more user selections of user controls for adjusting and / or customizing the playback of the 3D video clip, and can regenerate the 3D video content from the 3D video clip according to the user selection.

[0036] The interface may include one or more user controls that enable a user to modify playback of the 3D video clip, such as adjusting the viewing angle (e.g., selecting a quarterback's point of view (POV), selecting a ball's POV, etc.), adding virtual content (e.g., animation effects, statistics, graphics, etc.), adjusting the viewing speed (e.g., slowing down or speeding up), modifying animated objects) (e.g., selecting a different model, modifying one or more properties of an object model). In some examples, a user may use the user controls to create customized 3D video content and use the sharing controls to share the customized 3D video content with one or more other devices. In some examples, the interface includes a search field that receives a search query from a user, and the interface may display search results identifying one or more 3D video clips in response to the search query. For example, a user may retrieve 3D highlights for a certain game, athlete, team, type of movement, etc. These and other features are further explained with reference to the accompanying drawings.

[0037] Figures 1A to 1N A system 100 for generating and distributing 3D video segments 122 to user devices 152 according to one aspect is shown. The system 100 includes a video manager 102 configured to use one or more machine learning models 105 to generate 3D video segments 122 from 2D video content 134 and enable 3D video content 121 generated from the 3D video segments 122 to be displayed in an interface 140 on a display 123 of the user device 152. The video manager 102 can implement technical features that can increase the speed and / or accuracy of 3D production while reducing the amount of computing resources (e.g., memory, central processing unit (CPU) resources, graphics processing unit (GPU) resources) used to enable display of the 3D video segments 122 on the display 123, which can be a 2D display or a 3D display.

[0038] For example, the video manager 102 can quickly and accurately generate a 3D video clip 122 (e.g., a 3D video highlight) from the 2D video content 134 by obtaining 3D motion data 110 of an object 108 detected in the 2D video content 134 from the 3D pose estimation engine 188, wherein the 3D motion data 110 includes a position and orientation of the object 108 in the 3D space that captures the movement of the object in the 3D space. The video manager 102 generates an animated object 114 based on the 3D motion data 110, wherein the movement of the animated object 114 corresponds to the movement of the object 108 in the 2D video content 134. For example, the real movement captured by the 3D motion data 110 is applied to a computer-generated object (e.g., a 3D object model 116), which converts the static model into a dynamic model that moves in the same / similar manner as the object 108 in the 2D video content 134. The system 100 can enable real-time rendering of 3D video fragments 122 on the user device 152 by using a streaming engine 120 (e.g., a cloud-based 3D pixel streaming engine) that generates (e.g., renders) 3D video content 121 from the 3D video fragments 122 and regenerates (e.g., re-renders) the 3D video content 121 based on any user selections 171, which can minimize, reduce, or eliminate the need for the user device 152 to have a dedicated GPU device.

[0039] The 3D video clip 122 includes 3D model data 125, which includes one or more animated objects 114 that move in the 3D scene in a manner corresponding to the movement of the object 108 in the 2D scene of the 2D video content 134. In some examples, the object 108 can be a structure of the object 108 (e.g., a human body or a body of a non-human object such as a ball or other movable object). The 3D video clip 122 can include other model data that defines one or more static structures and / or the environment of the 3D scene. In some examples, one or more machine learning models 105 (e.g., 3D pose estimation engine 188) can detect 3D movement data 110 about the movement and orientation of the object 108 in the 3D space from the 2D positioning of the object 108 in the frame of the 2D video content 134. The video manager 102 can generate the animated object 114 based on the 3D movement data 110. The animated object 114 can move in the 3D space in a manner corresponding to the movement of the object 108 in the 2D video content 134. In some examples, video manager 102 may apply 3D movement data 110 to 3D object model 116 , which creates animated object 114 that moves in the 3D scene in a manner that corresponds to the movement of object 108 in the 2D scene.

[0040] For example, the video manager 102 may detect 3D motion data 110 about the movement of the object 108 from the 2D video content 134. In some examples, the 3D motion data 110 is 3D gesture data over time. The 3D motion data 110 includes positioning and rotation information describing the movement and orientation of the object 108 in 3D space. In some examples, the 3D motion data 110 includes 3D positioning coordinates (e.g., x, y, and z values) and rotation information (e.g., Euler angles, quaternions, and / or rotation matrices) of one or more key points 170 on the object 108 over a period of time. The key points 170 may represent different parts of the structure of the object 108, and the 3D motion data 110 may capture the movement and orientation of the key points 170 over time. The video manager 102 may generate an animated object 114 by applying the 3D motion data 110 to a 3D object model 116 representing the object 108. In some examples, 3D object model 116 is selected from object model database 180, and 3D movement data 110 is applied to the selected 3D object model 116. In some examples, 3D object model 116 is generated from 2D video content 134 using one or more known mesh generation techniques. Animated object 114 may move in the 3D scene in a manner that corresponds to the movement of object 108 in the 2D scene.

[0041] 3D video clips 122 may be 3D video highlights (e.g., 3D sports highlights). However, the techniques discussed herein may be applied to any type of underlying video content having one or more objects 108 moving in 2D video content 134. Video manager 102 may convert 2D video highlights to 3D video highlights so that users may replay video highlights in 3D format to enable users to view highlights from different angles and / or speeds and / or change aspects of 3D video clips 122 such as adding graphics, statistics, and / or animation effects to highlights, customizing animated objects 114 (e.g., changing the clothing, style, or other characteristics of athletes, etc.), and / or customizing other parts of 3D scenes. For example, users may visually explore 3D video clips 122 based on user selections 171 received via one or more user controls 142 on interface 140. Users may be able to view highlights from different camera perspectives and / or speeds, and create customized 3D video content 121a (with graphics and / or animation effects) that may be shared with other users.

[0042] The video manager 102 includes a 3D video generator 104 configured to receive 2D video content 134 and generate a 3D video clip 122 based on the 2D video content 134. The 2D video content 134 includes video data (and, in some examples, audio data) captured from a camera system 132. The camera system 132 may include one or more camera devices configured to capture video in two dimensions, thereby representing a scene as a flat image or a sequence of frames having a height and a width. Each frame in the 2D video content 134 includes a 2D array of pixels, where each pixel includes color and brightness information. In some examples, the camera system 132 includes an audio system having one or more microphones configured to capture audio data. In some examples, the camera system 132 does not include a specialized capture device (e.g., a stereo camera or depth sensing technology) for obtaining additional depth information for a 3D experience. In some examples, the 2D video content 134 includes television content 157 (e.g., live television content).

[0043] 2D video content 134 is displayed on display 123 of user device 152. In some examples, display 123 is a 2D display. In some examples, display 123 is a 3D display, and 2D video content 134 is displayed in a 3D display in a 2D format. In some examples, while camera system 132 generates 2D video content 134, 2D video content 134 is streamed to user device 152 via a media platform (e.g., a streaming platform). In some examples, 2D video content 134 is stored on a remote server computer and streamed from the remote server computer to user device 152. In some examples, 2D video content 134 is stored on user device 152.

[0044] The application 146 executable by the user device 152 can receive the 2D video content 134 and display the 2D video content 134 in the display 123. The application 146 can be a native application executable by the operating system 186 of the user device 152. In some examples, the application 146 is a streaming application. In some examples, the application 146 is a video sharing application. In some examples, the application 146 is a browser application. The browser application can render a web page (or execute an application) in a browser tab, which streams the 2D video content 134 from a streaming platform (also referred to as a media platform or a media provider) to the user device 152.

[0045] 3D video generator 104 may obtain 2D video content 134 from a streaming platform that distributes and / or stores 2D video content 134 captured from camera system 132. In some examples, 3D video generator 104 may obtain 2D video content 134 from a streaming platform while camera system 132 is generating 2D video content 134. In some examples, 3D video generator 104 may receive 2D video content 134 from a remote server computer that stores 2D video content 134. In some examples, 2D video content 134 is associated with a resource identifier (e.g., a uniform resource locator (URL)) that identifies the location of 2D video content 134. In some examples, 3D video generator 104 may retrieve 2D video content 134 using the resource identifier. In some examples, 3D video generator 104 may receive the resource identifier and / or 2D video content 134 from user device 152.

[0046] In some examples, a user can view 2D video content 134 in interface 140 of application 146. Interface 140 may include UI element 107 configured to enable the user to view 2D video content 134 in 3D format. In some examples, UI element 107 is a UI control that enables the user to view 2D highlights in 3D format. In some examples, selection of UI element 107 causes 3D viewing request 184 to be generated and transmitted to video manager 102. 3D viewing request 184 may include information identifying 2D video content 134. In some examples, 3D viewing request 184 includes a resource identifier of 2D video content 134. In some examples, 3D viewing request 184 may include 2D video content 134.

[0047] In some examples, application 146 may generate and transmit 3D viewing request 184. In some examples, user device 152 includes 3D client manager 155 configured to generate and transmit 3D viewing request 184. In some examples, 3D client manager 155 is a separate program from application 146. In some examples, 3D client manager 155 is an operating system program. In some examples, 3D client manager 155 is a component (e.g., a subcomponent) of a browser application. 3D client manager 155 and application 146 may communicate with each other via an application programming interface (API) or an inter-process communication (IPC) link. In some examples, 3D client manager 155 is configured to receive an indication of a user selection 171 of user control 142, which may cause 3D client manager 155 to generate and transmit 3D viewing request 184. In some examples, 3D client manager 155 is included as part of application 146.

[0048] In response to the 3D viewing request 184, the 3D video generator 104 obtains the 2D video content 134 and generates the 3D video fragment 122. In some examples, the 3D video generator 104 can generate a single 3D video fragment 122 from the 2D video content 134. In some examples, the 3D video generator 104 can generate multiple 3D video fragments 122 from the 2D video content 134. In some examples, the 3D video generator 104 can retrieve the 2D video content 134 using a resource identifier of the 2D video content 134. In some examples, the 3D video generator 104 can obtain the 2D video content 134 from the 3D viewing request 184. In some examples, the video manager 102 can use information from the 3D viewing request 184 to determine whether the 3D video fragment 122 has been generated for the 2D video content 134. For example, the video manager 102 may query (e.g., using the resource identifier, another identifier that may uniquely identify the 2D video content 134, and / or other information included in the 3D viewing request 184) a video database 196 that stores a plurality of 3D video fragments 122. If the video manager 102 determines that the 3D video fragment 122 has not been generated from the 2D video content 134, the video manager 102 may cause the 3D video generator 104 to generate the 3D video fragment 122.

[0049] In some examples, 3D video generator 104 may generate one or more 3D video clips 122 from 2D video content 134 without prompting from a user. Figure 1B As shown, the video manager 102 may include a segment selector 128 configured to identify one or more 2D video segments 134a from the 2D video content 134. The 2D video segment 134a may be an example of the 2D video content 134. In some examples, the 2D video segment 134a is a shorter video clip (e.g., a highlight) from a longer video content (e.g., the 2D video content 134). The 2D video content 134 may include television content 157. The television content 157 may be a program associated with a television channel. The program may be a sports program. In some examples, the media platform (or media provider) is configured to stream the television content 157 over the network 150 and / or broadcast the television content 157 using radio waves. In some examples, the television content 157 includes live television content. The live television content may be digital data that is streamed or broadcast when the 2D video content 134 is captured by the camera system 132.

[0050] The segment selector 128 detects that a portion of the television content 157 includes the key event 127, and identifies the 2D video segment 134a from the portion of the television content 157. For example, the segment selector 128 may receive the television content 157 when the television content 157 is captured by the camera system 132 and streamed / broadcasted to the audience. The segment selector 128 may detect whether the 2D video content 134 includes the key event 127, and if so, may identify the 2D video segment 134a. The key event 127 may be a specific sports action or event (e.g., a goal, a foul, a pass, etc.). In response to detecting the key event 127 in the 2D video content 134, the segment selector 128 may determine the beginning and end of the scene and select the portion for the 2D video segment 134a. The 2D video segment 134a may be a clip or portion of the television content 157 that contains a highlight (e.g., the key event 127). The 2D video segment 134a may include a key performance from a sports event. The 2D video segment 134a may be a video clip of a basketball player dunking a ball or a soccer player scoring a goal.

[0051] Segment selector 128 executes one or more existing event detection algorithms to analyze video footage of television content 157 to determine whether 2D video content 134 depicts key event 127 , and if so, may identify 2D video segment 134a as a highlight of 2D video content 134 .

[0052] The segment selector 128 may include an event recognition model (e.g., a convolutional neural network (CNN) or a recurrent neural network (RRN)) configured to (e.g., trained to) recognize key events 127 in one or more types of sporting events. For basketball, the segment selector 128 detects a key event 127 when a player scores a basketball, performs a certain action like a dunk, assists, steals, etc. For football, the segment selector 128 may detect a key event 127 when a player catches the ball for more than a threshold number of yards, scores a touchdown, etc. In some examples, the segment selector 128 includes a motion analysis model that is configured to analyze the motion pattern of a detected object (e.g., an object 108 detected by the object tracker 106) to detect key events 127 (e.g., key moves) in the 2D video content 134. Techniques such as optical flow that estimates the motion of pixels between consecutive frames can be used to calculate the magnitude and direction of the motion of each object 108. Changes in motion, sudden accelerations, or specific movement patterns may indicate a key event 127. In some examples, segment selector 128 includes a rule-based model that analyzes 2D video content 134 to apply specific rules or criteria to detect key events 127. For example, in basketball, detecting a slam dunk or a three-point shot may be based on height, ball trajectory, and shot location. By defining rules or thresholds based on sports rules, segment selector 128 can detect key events 127 in 2D video content 134.

[0053] like Figure 1B As shown, the segment selector 128 may identify a 2D video segment 134a-1 and a 2D video segment 134a-2 from the 2D video content 134. Then, the 3D video generator 104 may generate a 3D video segment 122-1 corresponding to the 2D video segment 134a-1 and a 3D video segment 122-2 corresponding to the 2D video segment 134a-2. Figure 1G Two 3D video segments 122 are depicted in FIG. 1 , but the 3D video generator 104 may generate any number of 3D video segments 122 from the television content 157 , which may depend on the number of key events 127 detected in the 2D video content 134 .

[0054] Reference again Figure 1A, the 3D video generator 104 may include one or more object trackers 106 configured to detect one or more types of objects 108 in the 2D video content 134 and generate 3D motion data 110 for the objects 108 detected in the 2D video content 134. In some examples, the object trackers 106 include separate object trackers 106 configured to detect a certain type (or classification) of objects 108 and generate 3D motion data 110 for the objects 108 of that type, wherein the 3D motion data 110 may be different across different types of objects 108. In some examples, a single object tracker 106 may detect multiple different types of objects 108 and generate 3D motion data 110 for different types of objects 108. In some examples, a first object tracker is configured to generate 3D motion data 110 for one or more types of objects 108 (e.g., players), and a second object tracker is configured to generate 2D motion data for one or more types of objects 108 (e.g., balls).

[0055] In some examples, the object tracker 106 includes a 3D pose estimation engine 188 for human or non-human pose detection. In some examples, the 3D pose estimation engine 188 includes an existing 3D pose estimation model (e.g., Embody model, PyMAF model). The 3D movement data 110 may include positioning and rotation information describing the movement and orientation of the object 108 in 3D space. In some examples, the 3D pose estimation engine 188 includes an ML human pose estimation model, which is configured to detect the movement and orientation of the pose of a human object in 3D space (e.g., represented by key points 170 such as ankles, shoulders, necks, hands, etc.). In some examples, the object tracker 106 includes an object detection and tracking model, which is configured to detect the movement and orientation of a non-human object (e.g., a ball) in 2D space or 3D space. In some examples, the object detection and tracking model can provide 2D positioning of a non-human object. In some examples, the object detection and tracking model can provide a ground height relative to the athlete at each frame.

[0056] like Figure 1C As shown, 2D video content 134 may visually depict object 108-1 and object 108-2 in a frame of 2D video content 134. In some examples, object 108-1 is a person. In some examples, object 108-2 is a non-human object such as a ball. Figure 1CTwo objects 108 are depicted in 2D video content 134, but object tracker 106 may detect and generate 3D movement data 110 for any number (or type) of objects 108 in 2D video content 134, including one object 108 or any number of objects 108 greater than two. Objects 108 may represent various physical objects, such as players, a ball, and / or other physical elements depicted in 2D video content 134.

[0057] In some examples, object tracker 106 may detect that object 108-1 is a first type of object 108 (e.g., object 108-1 has a human body), and may generate 3D motion data 110 from 3D video content 134. In some examples, object tracker 106 may further detect a subtype of the first type, such as whether the person is an athlete or a non-athlete (e.g., a coach or referee, a training staff member, etc.), whether the person is a specific type of athlete (e.g., a guard, center, forward, receiver, quarterback), and / or whether the person is a known entity (e.g., a specific person, such as Khris Middleton, Jrue Holiday, Jordan Love, etc.). In some examples, object tracker 106 (or, in some examples, another object tracker) may detect that object 108-2 is a second type of object 108 (e.g., a ball), and may generate 3D motion data 110 from 2D video content 134. In some examples, object tracker 106 (or other object trackers) may detect other types of objects 108 and generate 3D motion data 110 from 2D video content 134 .

[0058] The object tracker 106 includes a 3D pose estimation engine 188 configured to generate 3D movement data 110 of an object 108 (e.g., object 108-1, object 108-2) detected in the 2D video content 134. The 3D pose estimation engine 188 may include one or more machine learning (ML) models 105. In some examples, the 3D movement data 110 is a 3D pose of the object 108 over time. The 3D movement data 110 includes positioning and rotation information describing the movement and orientation of the object 108 in 3D space. In some examples, the 3D movement data 110 includes 3D positioning coordinates (e.g., x, y, and z values) and rotation information (e.g., Euler angles, quaternions, and / or rotation matrices) of at least one of a plurality of key points 170 over a period of time. The key points 170 may represent different parts of the structure of the object 108.

[0059] like Figure 1DAs shown, the object 108 can be defined by one or a set of key points 170, whose 3D position and orientation are estimated by the 3D pose estimation engine 188. The key points 170 can be associated with different parts of the object 108 (e.g., for a human body, these parts can be ankles, hips, head, shoulders, elbows, etc.). The key points 170 can include key points 170-1, key points 170-2, and key points 170-3 to key points 170-N. It is worth noting that the 3D pose estimation engine 188 can track a single key point 170 (e.g., a center point on the object 108) or multiple key points 170, such as any number greater than or equal to two.

[0060] The 3D pose estimation engine 188 generates the 3D movement data 110 by estimating the 3D position of the key point 170 from the 2D position of the key point 170 in the 2D video content 134. The key point 170 may include a part that forms a pose. In some examples, for a human body, the key point 170 may include the head, neck, right shoulder, left shoulder, right elbow, left elbow, right wrist, left wrist, right hip, left hip, right knee, left knee, right corner, and / or left ankle. In some examples, the key point 170 includes a nose, a left eye, and / or a right eye. The 3D position of the key point 170 may refer to the 3D spatial positioning of the key point 170 in the 3D space (e.g., positioning coordinates (e.g., X, Y, Z values)), and in some examples, involves rotation information about the orientation of the key point 170 in the 3D space. In some examples, the rotation information may include Euler angles, quaternions, and / or rotation matrices.

[0061] In some examples, the 3D pose estimation engine 188 estimates the 2D pose of an object 108 (e.g., a person) in the input data (e.g., 2D video content 134). 2D pose estimation involves detecting and locating key points 170 (e.g., key body joints) in each frame of the input data. This can be implemented using various techniques such as convolutional neural networks (CNNs) or pose estimation algorithms based on graphical models. In some examples, the 3D pose estimation engine 188 can use the 2D pose to estimate the depth of the key points 170, which can include triangulating the 2D positioning of the key points 170 from multiple camera views or inferring depth information using a depth map. Once the 2D positioning and depth information (if available) are obtained, the 3D pose estimation engine 188 can perform 3D pose estimation. There are different means for 3D pose estimation, including model-based methods, direct regression methods, and learning-based methods.

[0062] In some examples, the 3D movement data 110 generated by the 3D pose estimation engine 188 may not include the positioning of the object 108 relative to static structures in the 2D video content 134 (e.g., the positioning of a player on a court or field). Figure 1EAs shown, the object tracker 106 may include a relative position detector 172 configured to generate relative position information 129 regarding the position of the object 108 (e.g., one or more key points 170 of the object 108) relative to one or more static structures 108a representing physical structures (e.g., a court, a field, a goal, etc.) in the 2D video content 134. In some examples, the relative position detector 172 may control one or more aspects of the 3D pose estimation engine 188 to calculate the 3D motion data 110 at one or more times when the one or more key points 170 of the object 108 contact portions of the static structure 108a, or select portions of the 3D motion data 110 at one or more times when the one or more key points 170 of the object 108 contact portions of the static structure 108a.

[0063] Static structure 108a represents a physical space having a surface defined by a width (in direction A1) and a length (in direction A2), and in some examples, a height defined in a direction normal to the surface (direction A3 depicted as extending into and out of the page). Relative positioning information 129 may include the 3D positioning of object 108-1 relative to another object, namely static structure 108a.

[0064] The relative positioning detector 172 can detect the static structure 108a from the 2D video content 134, and can calculate the positioning (e.g., 3D positioning) of the object 108 relative to the static structure 108a from the 2D video content 134 over time. In some examples, the relative positioning detector 172 can use inverse kinematics (IK) techniques to calculate the positioning of the object relative to the static structure 108a from the video frames of the 2D video content 134. In some examples, the relative positioning detector 172 can use IK techniques to calculate the positioning of the object relative to the static structure 108a at one or more key events (e.g., the positioning of each foot on the court when the player's feet touch the ground in the 2D video content 134, the start and / or end time and position of any jumps, when the ball changes hands and is released). In some examples, the relative positioning detector 172 can use a camera calibration tool and camera parameters from the camera system 132 to calculate the positioning of the object 108 relative to the static structure 108a from the video frames of the 2D video content 134.

[0065] Reference again Figure 1A, 3D video generator 104 includes a 3D object engine 112 configured to generate an animated object 114 using 3D movement data 110. In some examples, 3D object engine 112 may apply 3D movement data 110 to a 3D object model 116 representing a 3D version of object 108. In some examples, 3D object model 116 does not represent object 108, but rather represents a user of user device 152. In some examples, 3D object model 116 may be any type of computer-generated object configured to be enhanced with 3D movement data 110. In some examples, if object 108 is a basketball player, 3D object model 116 is a model representation of the basketball player.

[0066] The 3D object model 116 may include information about the geometry, topology, and appearance of the 3D object. The 3D object model 116 may define the shape and structure of the 3D object, and may include information about the vertices, edges, and / or faces that form the surface of the object. The 3D object model 116 may include information about the connectivity and relationships between the geometric elements of the model, and may define how the vertices, edges, and faces are connected to form the structure of the object. The 3D object model 116 may include texture coordinates that define how to map a texture or image to the surface of the model, and may provide correspondence between points on the 3D surface and pixels in a 2D texture image. In some examples, the 3D object model 116 may include information about normals (e.g., vectors perpendicular to the surface at each vertex or face), which determine the orientation and direction of the surface, thereby indicating how light interacts with the surface during shading calculations. The 3D object model 116 may include information about material properties that describe the visual appearance and characteristics of the surface of the 3D object, and may include information such as color, reflectivity, transparency, glossiness, and other parameters that affect how the surface interacts with light.

[0067] In some examples, the 3D object model 116 is initially configured as a static model. However, when the 3D object engine 112 applies the 3D motion data 110 from the object tracker 106, the static model is transformed into an animated object 114, thereby generating a dynamic object. In some examples, the animated object 114 may be referred to as an animated mesh or an animated rig. Applying the 3D motion data 110 to the 3D object model 116 may include adding the 3D motion data 110 to the 3D object model 116 to generate the animated object 114, wherein the 3D object model 116 is configured to move in a manner indicated by the 3D motion data 110.

[0068] In other words, Figure 1FAs shown, animated object 114 is defined by 3D object model 116, which is enhanced with 3D movement data 110 generated by 3D pose estimation engine 188. Animated object 114 is configured to have a 3D pose in a video frame of 3D video content 121 that corresponds to a pose in a video frame of 2D video content 134, thereby reproducing the movement of the object in 3D. Animated object 114 may also include any of the information discussed with reference to 3D object model 116.

[0069] In some examples, 3D object engine 112 selects 3D object model 116 to represent object 108 detected in 2D video content 134. Figure 1G As shown, the 3D object engine 112 may include an object model selector 178. For an object 108-1 detected in the 2D video content 134, the object model selector 178 may select a 3D object model 116-1 representing the object 108 from an object model database 180 storing a plurality of 3D object models 116. The object model database 180 may be referred to as a 3D object model inventory. The 3D object models 116 stored in the object model database 180 may be different versions of the 3D model. The 3D object models 116 stored in the object model database 180 may include a 3D object model 116-1 and a 3D object model 116-2. The 3D object model 116-2 may have at least one characteristic different from the 3D object model 116-1. However, the 3D object models 116 stored in the object model database 180 may include any number of 3D models (e.g., ranging from tens, to hundreds, to thousands). In some examples, object model selector 178 may select one of 3D object models 116 stored in object model database 180 based on object data 174 associated with object 108 - 1 .

[0070] In some examples, object data 174 includes information detected by object tracker 106, such as classification type, for example, whether object 108 is an athlete or non-athlete, whether it is a specific type of athlete and / or whether it is a specific known athlete. In some examples, if object data 174 indicates that object 108 is an athlete, object model selector 178 can obtain object model 116-1 representing a general athlete. In some examples, if object data 174 indicates that object 108 is a specific type of athlete (e.g., a receiver, running back, center, or guide), object model selector 178 can obtain object model 116-1 representing the specific type of athlete. In some examples, if object data 174 indicates that object 108 is a specific known athlete (e.g., Kris Middleton), object model selector 178 can obtain object model 116-1 representing the specific known athlete.

[0071] In some examples, 3D object engine 112 generates 3D object model 116 using video footage of object 108 in 2D video content 134. For example, Fig. 1I As shown, the 3D object engine 112 may include a mesh generator 182 configured to receive the 2D video content 134 including the object 108 and generate a 3D object model 116 to have characteristics that mimic the characteristics of the object in the 2D video content 134. The mesh generator 182 may use one or more existing 3D reconstruction techniques to create the 3D object model 116 from the 2D video content 134. In some examples, the 3D object model 116 includes a 3D mesh representing the structure of the object 108.

[0072] In some examples, the mesh generator 182 can obtain or determine intrinsic parameters of the camera system 132, such as focal length, principal point, and lens distortion. This information can be used to accurately project 3D geometry into 2D image coordinates. In order to reconstruct the 3D geometry, the mesh generator 182 can extract features or key points from the 2D video frames. These features can be points, edges, corners, or other characteristic elements. Feature tracking algorithms are used to match corresponding features across different frames, allowing them to be tracked over time. In some examples, the mesh generator 182 may include one or more motion recovery structure (SfM) techniques. SfM is a technique for estimating camera poses from a collection of 2D images and reconstructing the 3D structure of a scene. It uses the tracked features and camera calibration information to determine the camera position and orientation at different times. The SfM algorithm estimates the 3D structure by triangulating corresponding feature points from multiple camera viewpoints.

[0073] Once the initial sparse 3D structure is estimated, the mesh generator 182 can use dense reconstruction techniques to refine the initial sparse 3D structure. Dense reconstruction can reconstruct the 3D geometry of the object 108 with a higher level of detail by estimating the depth value of the pixels (e.g., each pixel) in the 2D video frame. Dense reconstruction can be performed using techniques such as depth mapping, stereo matching, or motion recovery structure refinement. After obtaining a dense point cloud representing the 3D structure of the object, the mesh generator 182 can generate a 3D mesh representing the surface of the object. Surface reconstruction techniques such as Delaunay triangulation or Poisson surface reconstruction can be applied to convert the point cloud into a mesh representation. These algorithms create a collection of connected triangles that approximate the surface geometry of the object. Depending on the quality and accuracy of the initial mesh, additional refinement steps may be required. Mesh refinement techniques such as smoothing, regularization, or texture mapping can be applied to improve the visual quality of the mesh, remove noise, and align it more closely with the real geometry of the object.

[0074] Reference again Figure 1A, the 3D video clip 122 includes animated objects 114 (e.g., defined by a 3D object model 116 augmented with 3D movement data 110) of one or more objects 108 detected in the 2D video content 134. In some examples, the 3D video clip 122 includes an animated object 114 of a single object 108 (e.g., a single athlete) detected in the 2D video content 134. In some examples, the 3D video clip 122 includes multiple animated objects 114 of multiple objects 108 (e.g., multiple athletes) detected in the 2D video content 134. The 3D video clip 122 may include other computer-generated model objects and content representing a 3D scene. For example, if the 3D scene is related to a basketball event, the 3D video generator 104 can generate the 3D video clip 122 to include other computer-generated model objects (e.g., static and / or animated models) that represent other objects in the 3D scene, such as a court (e.g., an outdoor basketball court, a basketball within an arena within an area), a basketball hoop, fans, lighting and shading indicating the time of day (e.g., night or day), etc.

[0075] The video manager 102 system may include a streaming engine 120 configured to generate 3D video content 121 from 3D video clips 122. In some examples, the streaming engine 120 renders video frames from the 3D video clips 122, wherein the 3D video content 121 includes rendered video frames. In some examples, the streaming engine 120 encodes and compresses the rendered video frames, wherein the 3D video content 121 is a video stream having encoded and compressed video frames.

[0076] The streaming engine 120 can enable near real-time streaming and rendering of 3D graphics and interactive content over the network 150, which can include one or more rendering operations, such as geometry processing, shading and material calculations, camera and viewpoint calculations, frame synthesis, video encoding, and / or network transmission. For example, the streaming engine 120 includes one or more GPUs 126 configured to perform rendering operations on 3D video segments 122 to generate images (e.g., video frames) of 3D video content 121. The streaming engine 120 can transmit the 3D video content 121 to the user device 152 for display. In some examples, because the rendering operations are performed by the streaming engine 120, the 3D video content 121 is transmitted to the user without requiring the user device to have special GPU hardware. In some examples, the 3D video segment 122 is a video stream.

[0077] In some examples, application 146 receives 3D video content 121, decodes 3D video content 121, and displays 3D video content 121 in interface 140 of application 146. In some examples, 3D client manager 155 receives 3D video content 121, decodes 3D video content 121, and provides decoded (rendered) video frames to application 146 for display on interface 140. In some examples, 3D client manager 155 is a program configured to operate as an intermediary between application 146 and video manager 102. In this manner, 3D client manager 155 can perform operations associated with communicating with video manager 102 and decoding 3D video content 121, which can improve performance of application 146.

[0078] The streaming engine 120 may receive information indicating one or more user selections 171 of one or more user controls 140 for adjusting and / or customizing playback of the 3D video clip 122, and the streaming engine 120 may regenerate the 3D video content 121 from the 3D video clip 122 according to the user selections 171. The streaming engine 120 may transmit the 3D video content 121-1 to the user device 152 for display in the interface 140. The interface 140 may include one or more user controls 142 that enable a user to view, modify, and share playback of the 3D video clip 122. In response to the user selections 171 of the user controls 142, the streaming engine 120 may receive information regarding the user selections 171, generate the 3D video content 121-2 from the 3D video clip 122 according to the user selections 171, and transmit the 3D video content 121-2 to the user device 152 for display in the interface 140.

[0079] like Fig. 1I and Figure 1J As shown, the user controls 142 may include one or more custom controls 142a that may modify the playback of the 3D video clip 122. The custom controls 142a may include a viewing angle control 131 configured to enable a user to adjust the viewing angle 111 of the 3D video clip 122. The viewing angle control 131 may provide a plurality of viewing angles 111 including a viewing angle 111a and a viewing angle 111b. Although Fig. 1I 1 , but it is noteworthy that the 3D video clip 122 may be replayed from many viewing angles 111 , including all viewing angles 111 . In some examples, the viewing angles 111 include a predefined set of viewing angles 111 .

[0080] Custom controls 142a may include virtual content controls 135 configured to enable a user to add virtual content 124 to the playback of 3D video clip 122. For example, virtual content controls 135 may enable a user to add one or more graphics 113 to 3D video clip 122. Graphics 113 may include statistics and other graphics associated with the 3D scene (e.g., how fast a player is running, how high a player jumps, the number of yards passed, etc.). In some examples, virtual content controls 135 may enable a user to add animation effects 115 to the 3D scene.

[0081] The customization controls 142a may include an object customization control 137 configured to enable a user to modify one or more properties 117 of the animated object 114. For example, a user may use the object customization control 137 to change certain aspects of the 3D object model 116 (e.g., change the clothing or other attributes of the underlying avatar) or use a different 3D object model 116 (e.g., use a different avatar). The customization controls 142a may include a viewing speed control 139 configured to enable a user to adjust the viewing speed 119 of the 3D video clip 122. The viewing speed control 139 may provide a plurality of viewing speeds 119 including a viewing angle 119a and a viewing angle 119b. Although Fig. 1I 1 , but it is noted that the 3D video clip 122 may be replayed from any number of viewing speeds 119 . In some examples, the viewing speeds 119 include a predefined set of viewing speeds 119 .

[0082] In some examples, interface 140 includes a search field 147 that enables a user to submit a search query 149 from the user, and in response to submission of search query 149, interface 140 may display search results 159 that identify one or more 3D video clips 122 that are responsive to search query 149. For example, a user may retrieve 3D highlights for a certain game, player, team, type of movement, etc.

[0083] For example, Figure 1KAs shown, the video manager 102 may include a metadata generator 173. The metadata generator 173 may receive content data 165 about the 2D video content 134 and generate (and / or associate) metadata 175 with the 3D video fragment 122. The metadata generator 173 may generate metadata 175 about the 2D video content 134 and store the metadata 175 within the 3D video fragment 122 in the video database 196. In some examples, the metadata generator 173 may generate the metadata 175 when the 3D video generator 104 generates the 3D video fragment 122. In some examples, the metadata generator 173 may also associate the 3D video fragment 122 with a corresponding 2D video fragment 134a used to generate the 3D video fragment 122. In some examples, the 3D video fragment 122 includes an identifier (e.g., a resource identifier or other identifier) ​​that identifies the location of the 2D video fragment 134a used to generate the 3D video fragment 122. In some examples, 2D video fragment 134a includes an identifier (eg, a resource identifier or other identifier) ​​that identifies the location of the corresponding 3D video fragment 122 .

[0084] The content data 165 may include information associated with the 2D video content 134, such as a title, location, time, team, key players, a description of the underlying event, and / or any metadata included as part of the 2D video content 134. In some examples, the content data 165 may include any information detected from the object tracker 106. The metadata 175 may include event data 179 about the time, location, duration, team, record, key players, and / or any information about the underlying event. The metadata 175 may include object data 181 about the object 108 detected by the object tracker 106. For example, the object data 181 may identify a particular player or information about the player. In some examples, the metadata 175 includes movement type data 183 that classifies a particular movement of the object 108. For example, the object tracker 106 may be able to classify certain player movements as belonging to known sports movements (e.g., a 360-degree dunk or a windmill dunk, etc.).

[0085] like Figure 1KAs shown, the video database 196 may store the 3D video segment 122-1, and the video database 196 may store the metadata 175 associated with the 3D video segment 122-1 generated by the metadata generator 173. In some examples, the video database 196 also associates the 2D video segment 134a-1 with the 3D video segment 122-1, wherein the 2D video segment 134a-1 is used to generate the 3D video segment 122-1. The video database 196 may store the 3D video segment 122-2, and the video database 196 may store the metadata 175 associated with the 3D video segment 122-2 generated by the metadata generator 173. In some examples, the video database 196 also associates the 2D video segment 134a-2 with the 3D video segment 122-2, wherein the 2D video segment 134a-2 is used to generate the 3D video segment 122-2.

[0086] The video manager 102 may include a video search engine 194. The video search engine 194 may receive a search query 149 and terms included in the search query 149 from the user device 152 to search the metadata 175 to identify one or more 3D video clips 122 in the video database 196 that are responsive to the search query 149. The video search engine 194 may provide search results 159 that identify the 3D video clips 122 that are responsive to the search query 149. In some examples, the search results 159 may also identify the 2D video clips 134a for the corresponding 3D video clips 122 included in the search results 159. In this manner, the user may view the original 2D footage and explore the highlights in 3D format.

[0087] In some examples, reference Fig. 1I and 1L, the interface 140 may include a sharing control 145 configured to enable a user to share the customized 3D video content 121a with one or more other users (e.g., user device 152b). The user may use the user controls 142 to modify aspects of the 3D video clip 122 to create the customized 3D video content 121a, and use the sharing controls 145 to transmit the customized 3D video content 121a to the user device 152b. In some examples, the user customized 3D video content 121a is stored at the video manager 102, and the user may create a message with a resource identifier (e.g., a selectable URL link) identifying the location of the customized 3D video content 121a, and the message may be transmitted to the user device 152b. In some examples, the user device 152 may share the customized 3D video content 121a with other users by posting the message identifying the customized 3D video content 121a to the social platform 198. In some examples, user device 152 is associated with user account 149a of social platform 198, and user device 152b is associated with user account 149b of social platform 198. In some examples, the user may discover a message with customized 3D video content 121a on social platform 198 using user device 152b.

[0088] refer to Figure 1M In some examples, video manager 102 may be executed by one or more server computers 160. Server computers 160 may include one or more processors 161 and one or more memory devices 163.

[0089] like Figure 1M As shown, user device 152 may include application 146 and 3D client manager 155. User device 152 may include one or more processors 101 and one or more memory devices 103. In some examples, 3D client manager 155 is configured to communicate with application 146 to detect user selection 171 of one or more user controls 142, and transmit information about user selection 171 to video manager 102. 3D client manager 155 may receive 3D video content 121 from video manager 102, decode 3D video content 121, and provide 3D video content 121 to application 146. In some examples, 3D client manager 155 is a component that is part of application 146. In some examples, 3D client manager 155 is a separate component from application 146. In some examples, 3D client manager 155 is a component of operating system 186 or a browser application.

[0090] like Figure 1NAs shown, in some examples, video manager portion 102-1 may be executed by server computer 160 and video manager portion 102-2 may be executed by user device 152. For example, some of the operations of video manager 102 may be performed by server computer 160, and some of the operations of video manager 102 may be performed by user device 152. In some examples, video manager portion 102-1 includes 3D video generator 104, and video manager portion 102-2 includes streaming engine 120. In some examples, video manager portion 102-1 includes object tracker 106 and 3D object engine 112. In some examples, video manager portion 102-1 includes object tracker 106, and video manager portion 102-2 includes 3D object engine 112.

[0091] User device 152 may be any type of computing device including one or more processors 101, one or more memory devices 103, a display 123, and an operating system 186 configured to execute (or assist in executing) one or more applications (including application 146). In some examples, display 123 is a 2D display. In some examples, display 123 is a 3D display. In some examples, operating system 186 is application 146. In some examples, application 146 is an application executable by operating system 186. In some examples, user device 152 is a laptop computer. In some examples, user device 152 is a desktop computer. In some examples, user device 152 is a tablet computer. In some examples, user device 152 is a smart phone. In some examples, user device 152 is a wearable device. In some examples, user device 152 is a virtual reality (VR) device (which may include a headset). In some examples, user device 152 is an augmented reality (AR) device. In some examples, display 123 is a display of user device 152.

[0092] Processor 101 may be formed in a substrate that is configured to execute one or more machine-executable instructions or one or more pieces of software, firmware, or a combination thereof. Processor 101 may be semiconductor-based—that is, the processor may include semiconductor material that can execute digital logic. Memory device 103 may include main memory that stores information in a format that processor 101 can read and / or execute. Memory device 103 may store applications 146, operating system 186, and / or 3D client manager 155, which, when executed by processor 101, perform certain operations discussed herein. In some examples, memory device 103 includes a non-transitory computer-readable medium that includes executable instructions that cause at least one processor (e.g., processor 101) to perform operations.

[0093] Server computer 160 can be a computing device in the form of a variety of different devices (e.g., a standard server, a group of such servers, or a rack server system). In some examples, server computer 160 can be a single system that shares components such as a processor and memory. In some examples, server computer 160 can be multiple systems that do not share a processor and memory. Network 150 may include the Internet and / or other types of data networks, such as a local area network (LAN), a wide area network (WAN), a cellular network, a satellite network, or other types of data networks. Network 150 may also include any number of computing devices (e.g., computers, servers, routers, network switches, etc.) configured to receive and / or send data within network 150. Network 150 may further include any number of hardwired and / or wireless connections.

[0094] The video manager 102 (or portions thereof) may be executed by one or more server computers 160. The server computers 160 may include one or more processors 161 formed in a substrate, an operating system (not shown), and one or more memory devices 163. The memory device 163 may represent any kind (or multiple kinds) of memory (e.g., RAM, flash memory, cache, disk, tape, etc.). In some examples (not shown), the memory device may include external storage, for example, a memory physically remote from the server computer 160 but accessible by the server computer. The processor 161 may be formed in a substrate that is configured to execute one or more machine-executable instructions or one or more pieces of software, firmware, or a combination thereof. The processor 161 may be semiconductor-based—that is, the processor may include semiconductor material that can execute digital logic. The memory device 163 may store information in a format that is readable and / or executable by the processor 161. The memory device 163 may store the video manager 102 (or portions thereof) that, when executed by the processor 161, performs certain operations discussed herein. In some examples, memory device 163 includes non-transitory computer-readable media including executable instructions that cause at least one processor (eg, processor 161 ) to perform operations.

[0095] Figures 2A to 2M An example of one or more interfaces 240 of an application displaying a 3D video clip and user controls for controlling the 3D video clip is shown according to an aspect.

[0096] like Figure 2A As shown, interface 240 may display 2D video clip 234. 2D video clip 234 may be a highlight from a football program. In some examples, the football program is still live (e.g., Figures 1A to 1N2D video clip 234 may be associated with a previous key performance in the football program. 2D video clip 234 may be one of a plurality of key performances in the football program. 2D video clip 234 may be Figures 1A to 1N For example, segment selector 128 may have detected key event 127 in the 2D footage and selected 2D video segment 234 as a highlight to be included in a key performance of the football program. In some examples, in response to 2D video segment 234 being detected as a highlight, Figures 1A to 1N The 3D video generator 104 can generate a 3D video clip. The 2D video clip 234 includes a plurality of objects 208. The objects 208 are the body structures of a football player.

[0097] like Figure 2A As shown, the interface 240 includes a UI element 207a that, when selected, enables the user to view the 2D video clip 234 in 3D format. The UI element 207a may be Figures 1A to 1N 107a. The user can select UI element 207a at any time during the display of 2D video clip 234 (or after 2D video clip 234 has ended). In some examples, in response to selection of UI element 207a, such as Figure 2B As shown, interface 240 may display UI information element 213 (e.g., “Swipe to Rotate”) and UI information element 215 (“Pinch to Zoom”) regarding user selections that may adjust the view of the 3D video clip. Interface 240 may also display UI element 207b (“Explore in 3D”) that, when selected, enables a 3D viewing request (e.g., Figures 1A to 1N The 3D viewing request 184) is transmitted to the streaming engine 120 (e.g., Figures 1A to 1N In some examples, selection of UI element 207a causes a 3D viewing request to be transmitted to the streaming engine.

[0098] like Figure 2C As shown, the interface 240 displays 3D video content 221a. The 3D video content 221a includes a plurality of animated objects 214 that move in a manner corresponding to the movement of the object 208 in the 2D video clip 234. The animated objects 214 may include players and a ball. Each animated object 214 may be Figures 1A to 1N In response to the 3D viewing request, the streaming engine may obtain a 3D video segment corresponding to the 2D video segment 234 (eg, Figures 1A to 1NThe streaming engine may generate 3D video content 221a from the 3D video fragment and transmit the 3D video content 221a to the user device for display on the interface 240. In some examples, for the initial view, the streaming engine may select the default settings of the user control 242 and generate the 3D video content 221a from the 3D video fragment according to the default settings.

[0099] The interface 240 includes a plurality of user controls 242 for adjusting the playback of the 3D video clip. The user controls 242 include a viewing angle control 231 configured to enable the user to adjust the viewing angle and a viewing speed control 239 configured to enable the user to adjust the viewing speed of the playback. The user controls 242 may include a statistics setting control 235a, which, when enabled, inserts statistics (including graphics) into the display of the 3D video clip. The user controls 242 may include an effect setting control 235b, which, when enabled, inserts one or more animation effects into the display of the 3D video clip. The user controls 242 may include a sharing control 245, which, when selected, provides one or more interfaces to enable the user to share the 3D video content with other users.

[0100] In response to selection of the viewing angle control 231, as Figure 2D As shown, the interface 240 displays a menu with a plurality of viewing angles 211. The viewing angles 211 may include a viewing angle 211a (drone camera), a viewing angle 211b (bird's eye view), a viewing angle 211c (in the game), a viewing angle 211d (quarterback viewpoint), a viewing angle 211e (becoming the ball), and a viewing angle 211f (on the sideline).

[0101] In response to the selection of the viewing angle 211d, the application may transmit information about the user selection (e.g., the viewing angle 211d) to the streaming engine. The streaming engine may generate 3D video content 221b from the 3D video clip according to the user selection and transmit the 3D video content 221b to the application for display on the interface 240, such as Figure 2E 3D video content 221b depicts highlights from the quarterback's point of view.

[0102] In response to the selection of the viewing angle 211e, the application may transmit information about the user selection (e.g., the viewing angle 211e) to the streaming engine. The streaming engine may generate 3D video content 221c from the 3D video clip according to the user selection and transmit the 3D video content 221c to the application for display on the interface 240, such as Figure 2F3D video content 221c depicts highlights from the ball's viewpoint. In some examples, the user can adjust the viewing speed of the playback. For example, refer to Figure 2F , the user uses viewing speed control 239 to adjust the viewing speed to slow motion speed to view the 3D video clip from the ball's perspective and in slow motion. In some examples, the application can transmit information about the user selection (e.g., slow motion speed), and the streaming engine can adjust the playback to the slow motion speed. In some examples, the speed of the playback is controlled by the application.

[0103] In response to the selection of the statistics setting control 235a, the application can transmit information about the user selection (e.g., the statistics setting control 235a) to the streaming engine. The streaming engine can generate 3D video content 221d from the 3D video clip according to the user selection and transmit the 3D video content 221d to the application for display on the interface 240, such as Figure 2G and Figure 2H 3D video content 221d depicts animated object 214, with virtual content 224 (eg, graphics and / or statistics about the performance) added to the replay.

[0104] In response to selection of the effect setting control 235b, as Fig.2I As shown, the interface 240 displays a menu with multiple effect options 237. The effect options 237 may include effect option 237a (celebration), effect option 237b (eight-bit mode), and effect option 237c (retro mode). In response to the selection of effect option 237a, the application may transmit information about the user selection (e.g., effect option 237a) to the streaming engine. The streaming engine may generate 3D video content 221e from the 3D video clip according to the user selection, and transmit the 3D video content 221e to the application for display on the interface 240, such as Figure 2E 3D video content 221e depicts animated object 214, where virtual content 224 is added to the 3D scene.

[0105] In response to selection of the share control 245, the application may display a clip creator interface 271 that enables the user to create a video clip 269 that includes a portion of the 3D video content 221e. The video clip 269 having the portion of the 3D video content 221e may be shared with one or more other users. The video clip 269 may be Figures 1A to 1NThe example of customized 3D video content 121a of the present invention. The clip creator interface 271 may include one or more movable elements 275 for enabling the user to define the beginning and end of the video clip and a progress indicator 273 showing the time positioning of the currently displayed video frame. The user can use the progress indicator 273 to move back and forth in the video clip. After the user has completed the editing of the video clip, the user can select the sharing element 245a, and the application can display a sharing interface 299 with multiple sharing options for sharing the video clip 269. The user can download the video clip 269, post the video clip 269 to one or more social or media platforms for other users to find, transfer the video clip 269 via email or direct message, and / or upload it to an online storage system. In some examples, if the user has posted the video clip 269 to a social or media platform, other users can find a message identifying the video clip 269 and select the video clip 269 to watch the user's customized 3D highlight video.

[0106] Figure 3A and Figure 3B An example of an interface 340 of an application is shown. The interface 340 may display highlights of a basketball game. For example, the interface 340 may display a 2D video clip 334 of a basketball performance. The interface 340 may display a UI element 307 that, when selected, enables the 2D video clip 334 to be viewed in 3D format.

[0107] In some examples, in response to selection of UI element 307, the application (or a 3D client manager in communication with the application (e.g., Figures 1A to 1N The 3D client manager 155)) can send a request to the video manager (e.g., Figures 1A to 1N The video manager 102 transmits a 3D viewing request (eg, Figures 1A to 1N In some examples, the video manager is configured to generate a 3D video fragment from a 2D video fragment 334 (e.g., Figures 1A to 1N In some examples, the video manager can retrieve the already created 3D video clips from the video database. In response to the 3D viewing request, such as Figure 3B As shown, interface 340 can display 3D video content 321. 3D video content 321 includes animated objects 314 that move in a manner corresponding to the movement of the players in 2D video clip 334 (e.g., Figures 1A to 1N Animated object 114).

[0108] FIG. 4A to FIG. 4CA comparative view of display 401 of 2D video content 434 with object 408 (eg, a basketball player) and display 403 of 3D video content 421 with animated object 414 in timeline 405 is shown. Figure 4A Frames of 2D video content 434 and corresponding frames of 3D video content 421 at a point in timeline 405 are shown. Figure 4B Frames of 2D video content 434 and corresponding frames of 3D video content 421 at a subsequent point in timeline 405 are shown. Figure 4C Frames of 2D video content 434 and corresponding frames of 3D video content 421 at subsequent points in timeline 405 are shown. FIG. 4A to FIG. 4C As shown, animated object 414 may move in a manner corresponding to the movement of object 408 .

[0109] FIG. 5A to FIG. 5C An example of 3D video content 521 created from 2D video content 534 is shown, where animated object 514 in 3D video content 521 may move in a manner corresponding to the movement of object 508 in 2D video content 534 . Figure 5A Video frame 535 from 2D video content 534 is shown. 2D video content 534 includes object 508 (eg, a basketball player performing a slam dunk). Figure 5B 3D video content 521 is shown in which animated object 514 is configured to move in a manner corresponding to the movement of object 508 in 2D video content 521. 3D video content 521 also includes other computer-generated content, such as virtual object 540 (e.g., a basketball backboard and basket), virtual object 516 (e.g., a ball), virtual object 530 (e.g., a court), and virtual object 520 (e.g., background graphics). Animated object 514 can be an avatar of a user that has been enhanced with 3D movement data so that animated object 514 can move in the same / similar manner as a basketball player in 3D video content 521. In some examples, one or more aspects of animated object 514 can be modified by the user (e.g., changing clothing, adding accessories, etc.). Reference Figure 5C , a user may add one or more animation effects 515 to the animated object 514 and / or the 3D scene.

[0110] Figure 6 600 is a flowchart depicting an example operation of a system for generating and distributing 3D video clips. Flowchart 600 may depict the operation of a computer-implemented method. Figures 1A to 1N 100, but the flowchart 600 can be applied to any of the implementations discussed herein. Figure 6The flowchart 600 of FIG. 600 shows the operations in sequential order, but it should be understood that this is merely an example and additional or alternative operations may be included. Further, Figure 6 The operations and related operations may be performed in an order different from that shown, or in parallel or in an overlapping manner.

[0111] Operation 602 includes obtaining two-dimensional (2D) video content 134 captured by camera system 132. Operation 604 includes generating three-dimensional (3D) video clip 122 from 2D video content 134, 3D video clip 122 including animated object 114 configured to move in a manner corresponding to movement of object 108 in 2D video content 134. In some examples, the generating includes obtaining 3D movement data 110 of object 108 detected in 2D video content 134 from 3D pose estimation engine 188, and generating animated object 114 based on 3D movement data 110 such that movement of animated object 114 corresponds to movement of object 108 in 2D video content 134. Operation 606 includes generating 3D video content 121 from 3D video clip 122. Operation 608 includes transmitting 3D video content 121 to user device 152 for display.

[0112] Figure 7 700 is a flowchart depicting an example operation of a system for rendering a 3D video clip. Flowchart 700 may depict the operation of a computer-implemented method. Figures 1A to 1N 100, but the flowchart 600 can be applied to any of the implementations discussed herein. Figure 7 Flowchart 700 shows operations in sequential order, but it should be understood that this is merely an example and additional or alternative operations may be included. Figure 7 The operations and related operations may be performed in an order different from that shown, or in a parallel or overlapping manner.

[0113] Operation 702 includes transmitting, over a network 150, a three-dimensional (3D) viewing request 184 to a video manager 102 executable by at least one server computer 160, the 3D viewing request 184 being a request to view two-dimensional (2D) video content 134 captured by a camera system 132 in a 3D format. Operation 704 includes receiving, over the network 150, from the video manager 102 3D video content 121 generated from a 3D video clip 122 generated by the video manager 102 using the 2D video content 134. Operation 706 includes initiating display of the 3D video content 121 on an interface 140 of a user device 152, the 3D video content 121 including an animated object 114 that moves in a manner corresponding to the movement of an object 108 in the 2D video content 134.

[0114] Figure 8 800 is a flowchart depicting an example operation of a system for retrieving and distributing 3D video clips. Flowchart 800 may depict the operation of a computer-implemented method. Figures 1A to 1N 100, but the flowchart 800 can be applied to any of the implementations discussed herein. Figure 8 Flowchart 800 shows operations in sequential order, but it should be understood that this is merely an example and additional or alternative operations may be included. Further, Figure 8 The operations and related operations may be performed in an order different from that shown, or in parallel or in an overlapping manner.

[0115] Operation 802 includes receiving a three-dimensional (3D) viewing request 184 from a user device 152 to view two-dimensional (2D) video content 134 in 3D format. Operation 804 includes retrieving a 3D video clip 122 corresponding to the 2D video content 134 from a video database 196, the 3D video clip 122 including an animated object 114 whose movement corresponds to the movement of an object 108 in the 2D video content 134. Operation 806 includes generating 3D video content 121 from the 3D video clip 122. Operation 808 includes transmitting the 3D video content 121 to the user device 152 for display.

[0116] Clause 1. A method comprising: generating a three-dimensional (3D) video clip from two-dimensional (2D) video content captured by a camera system, comprising: obtaining 3D movement data of an object detected in the 2D video content from a 3D pose estimation engine; and generating an animated object based on the 3D movement data so that movement of the animated object corresponds to movement of the object in the 2D video content; generating 3D video content from the 3D video clip; and transmitting the 3D video content to a user device for display.

[0117] Clause 2. The method of Clause 1, further comprising: receiving a 3D viewing request from a user device to view at least a portion of the 2D video content in 3D format; and generating 3D video content from the 3D video clip in response to the 3D viewing request.

[0118] Clause 3. The method as described in Clause 1 further includes: storing the 3D video clip in a video database; receiving a 3D viewing request from a user device to view at least a portion of the 2D video content in 3D format; and in response to the 3D viewing request: retrieving the 3D video clip from the video database; and generating 3D video content from the 3D video clip.

[0119] Clause 4. The method of any one of clauses 1 to 3, wherein the 2D video content comprises television content, further comprising: detecting that a portion of the television content comprises a key event; identifying a 2D video segment from the portion of the television content; and generating a 3D video segment from the 2D video segment.

[0120] Clause 5. A method as described in any of Clauses 1 to 4, wherein the 3D video content is first 3D video content, the method further comprising: receiving information indicating a user selection for adjusting playback of a 3D video segment from a user device; generating second 3D video content based on the 3D video segment and the user selection; and transmitting the second 3D video content to the user device for display.

[0121] Clause 6. The method of Clause 5, wherein the user selection comprises at least one of an adjustment to a viewing angle or an adjustment to a characteristic of an animated object.

[0122] Clause 7. The method of clause 5 or 6, wherein the user selection comprises selection of virtual content, the virtual content comprising at least one of statistical data or animation effects.

[0123] Clause 8. The method of any of Clauses 1 to 7, wherein generating the 3D movement data comprises estimating the 3D position of at least one of the plurality of key points of the object from the 2D position of the object in a plurality of frames of the 2D video content.

[0124] Clause 9. The method of any one of clauses 1 to 8, further comprising: generating an animated object by applying the 3D movement data to a 3D object model representing an object detected in the 2D video content.

[0125] Clause 10. The method of Clause 9, further comprising: selecting the 3D object model from an object model database based on object data associated with the object; or generating the 3D object model based on 2D video content.

[0126] Clause 11. The method of any of clauses 1 to 10, wherein the 3D movement data comprises position and rotation information describing the movement and orientation of the object in 3D space.

[0127] Clause 12. A non-transitory computer-readable medium storing executable instructions that cause at least one processor to perform operations comprising: transmitting a three-dimensional (3D) viewing request to a video manager executable by at least one server computer over a network, the 3D viewing request being a request to view two-dimensional (2D) content captured by a camera system in a 3D format; receiving 3D video content from the video manager over the network, the 3D video content being generated from a 3D video clip generated by the video manager using the 2D video content; and initiating display of the 3D video content on an interface of a user device, the 3D video content including an animated object whose movement corresponds to movement of an object in the 2D video content.

[0128] Clause 13. The non-transitory computer-readable medium of clause 12, wherein the 3D video content is first 3D video content, the interface includes a user control configured to enable a user to adjust playback of the 3D video clip, wherein the operation further includes: transmitting information indicating a user selection of the user control over a network; and receiving second 3D video content having an image according to the user selection over the network.

[0129] Clause 14. The non-transitory computer-readable medium of Clause 13, wherein the user controls include a viewing angle control configured to enable a user to adjust a viewing angle of the 3D video clip.

[0130] Clause 15. The non-transitory computer-readable medium of Clause 13 or 14, wherein the user controls include a virtual content control configured to enable a user to add virtual content to the playback of the 3D video clip.

[0131] Clause 16. The non-transitory computer-readable medium of any of Clauses 13 to 15, wherein the user control comprises an object customization control configured to enable a user to modify a property of the animated object.

[0132] Clause 17. A non-transitory computer-readable medium as described in any of Clauses 12 to 16, wherein the 3D video clip is a first 3D video clip, wherein the interface includes a search field configured to enable a user to submit a search query, wherein the operation further includes: in response to submission of the search query, receiving at least one search result identifying a second 3D video clip responsive to the search query.

[0133] Clause 18. The non-transitory computer-readable medium of any of Clauses 12 to 17, wherein the user device is a first user device, and the interface comprises a sharing control configured to enable the user to share the customized 3D video content with a second user device.

[0134] Clause 19. A device comprising: at least one processor; and a non-transitory computer-readable medium storing executable instructions, wherein the executable instructions cause the at least one processor to: receive a 3D viewing request from a user device to view two-dimensional (2D) video content in a three-dimensional (3D) format; retrieve a 3D video clip corresponding to the 2D video content from a video database, the 3D video clip including an animated object, the movement of the animated object corresponding to the movement of an object in the 2D video content; generate 3D video content from the 3D video clip; and transmit the 3D video content to the user device for display.

[0135] Clause 20. An apparatus as described in Clause 19, wherein the 3D video content is first 3D video content, and wherein the executable instructions include instructions that cause at least one processor to perform the following operations: receive information indicating a user selection for adjusting playback of a 3D video segment from a user device; generate second 3D video content based on the 3D video segment and the user selection; and transmit the second 3D video content to the user device for display.

[0136] Clause 21. The device of Clause 20, wherein the user selection comprises at least one of an adjustment to a viewing angle or an adjustment to a characteristic of an animated object.

[0137] Various implementations of the systems and techniques described herein may be implemented in digital electronic circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor that may be special-purpose or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0138] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium," "computer-readable medium," refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0139] To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, verbal, or tactile input.

[0140] The systems and techniques described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0141] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on the respective computers and placing each other in a client-server relationship.

[0142] In this specification and the appended claims, the singular forms "a", "an", and "the" include plural referents unless the context clearly indicates otherwise. Further, conjunctions such as "and", "or", and "and / or" are inclusive unless the context clearly dictates otherwise. For example, "A and / or B" includes only A, only B, and A and B. Further, the connecting lines or connectors shown in the various figures presented are intended to represent example functional relationships and / or physical or logical couplings between various elements. In actual devices, there may be many alternative or additional functional relationships, physical connections, or logical connections. In addition, unless an element is specifically described as "essential" or "critical", no item or component is essential for the practice of the implementation disclosed herein.

[0143] Terms such as, but not limited to, approximately, substantially, substantially, etc. are used herein to indicate that exact values ​​or ranges are not required and need not be specified. As used herein, the terms discussed above have their ready and immediate meanings to those of ordinary skill in the art.

[0144] Furthermore, the use of terms such as upper, lower, top, bottom, side, end, front, rear, etc. herein are all used with reference to the orientation currently considered or shown. If these terms are considered relative to another orientation, it should be understood that such terms must be modified accordingly.

[0145] Further, in this specification and the appended claims, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. Moreover, conjunctions such as "and", "or", and "and / or" are inclusive unless the context clearly dictates otherwise. For example, "A and / or B" includes only A, only B, and A and B.

[0146] Although certain example methods, devices, and articles have been described herein, the scope of coverage of this patent is not limited thereto. It should be understood that the terms used herein are for the purpose of describing specific aspects and are not intended to be limiting. On the contrary, this patent covers all methods, devices, and articles that fairly fall within the scope of the claims of this patent.

Claims

1. A method comprising: Generating a three-dimensional 3D video clip from two-dimensional 2D video content captured by a camera system, including: obtaining 3D movement data of objects detected in the 2D video content from a 3D pose estimation engine; and generating an animated object based on the 3D movement data such that movement of the animated object corresponds to movement of the object in the 2D video content; generating 3D video content from the 3D video clip; and The 3D video content is transmitted to a user device for display.

2. The method of claim 1, further comprising: receiving a 3D viewing request from the user device to view at least a portion of the 2D video content in 3D format; as well as In response to the 3D viewing request, the 3D video content is generated from the 3D video fragment.

3. The method of claim 1, further comprising: storing the 3D video clip in a video database; receiving a 3D viewing request from the user device to view at least a portion of the 2D video content in 3D format; as well as In response to the 3D viewing request: Retrieving the 3D video clip from the video database; and The 3D video content is generated from the 3D video fragment.

4. The method according to any one of claims 1 to 3, wherein the 2D video content comprises television content, further comprising: detecting that a portion of the television content includes a key event; identifying a 2D video segment from the portion of the television content; as well as The 3D video fragment is generated from the 2D video fragment.

5. The method according to any one of claims 1 to 4, wherein the 3D video content is a first 3D video content, the method further comprising: receiving, from the user device, information indicating a user selection for adjusting playback of the 3D video segment; generating second 3D video content based on the 3D video clip and the user selection; as well as The second 3D video content is transmitted to the user device for display.

6. The method of claim 5, wherein the user selection comprises at least one of an adjustment to a viewing angle or an adjustment to a characteristic of the animated object.

7. The method of claim 5 or 6, wherein the user selection comprises selection of virtual content, the virtual content comprising at least one of statistical data or animation effects.

8. The method of any one of claims 1 to 7, wherein generating the 3D movement data comprises estimating the 3D position of at least one of a plurality of key points of the object from the 2D position of the object in a plurality of frames of 2D video content.

9. The method according to any one of claims 1 to 8, further comprising: The animated object is generated by applying the 3D movement data to a 3D object model representing the object detected in the 2D video content.

10. The method of claim 9, further comprising: selecting the 3D object model from an object model database based on object data associated with the object; or The 3D object model is generated based on the 2D video content.

11. The method of any one of claims 1 to 10, wherein the 3D movement data comprises positioning and rotation information describing the movement and orientation of the object in 3D space.

12. A non-transitory computer-readable medium storing executable instructions that cause at least one processor to perform operations comprising: transmitting, over a network, to a video manager executable by at least one server computer, a three-dimensional (3D) viewing request, the 3D viewing request being a request to view two-dimensional (2D) video content captured by the camera system in a 3D format; receiving 3D video content from the video manager through the network, the 3D video content generated from a 3D video fragment generated by the video manager using the 2D video content; as well as Display of the 3D video content is initiated on an interface of a user device, the 3D video content including an animated object, the movement of the animated object corresponding to the movement of an object in the 2D video content.

13. The non-transitory computer-readable medium of claim 12, wherein the 3D video content is a first 3D video content, the interface comprises a user control configured to enable a user to adjust playback of the 3D video clip, wherein the operations further comprise: transmitting, over the network, information indicative of a user selection of the user control; as well as A second 3D video content having an image selected by the user is received through the network.

14. The non-transitory computer-readable medium of claim 13, wherein the user controls include a viewing angle control configured to enable the user to adjust a viewing angle of the 3D video clip.

15. The non-transitory computer-readable medium of claim 13 or 14, wherein the user controls include a virtual content control configured to enable the user to add virtual content to the playback of the 3D video clip.

16. The non-transitory computer-readable medium of any one of claims 13 to 15, wherein the user control comprises an object customization control configured to enable the user to modify characteristics of the animated object.

17. The non-transitory computer-readable medium of any one of claims 12 to 16, wherein the 3D video clip is a first 3D video clip, wherein the interface comprises a search field configured to enable a user to submit a search query, wherein the operations further comprise: In response to submission of the search query, at least one search result identifying a second 3D video clip responsive to the search query is received.

18. The non-transitory computer-readable medium of any one of claims 12 to 17, wherein the user device is a first user device, and the interface comprises a sharing control configured to enable a user to share customized 3D video content with a second user device.

19. A device comprising: at least one processor; as well as a non-transitory computer-readable medium storing executable instructions that cause the at least one processor to: receiving a 3D viewing request from a user device to view two-dimensional (2D) video content in a three-dimensional (3D) format; Retrieving from a video database a 3D video clip corresponding to the 2D video content, the 3D video clip comprising an animated object, wherein movement of the animated object corresponds to movement of an object in the 2D video content; generating 3D video content from the 3D video clip; as well as The 3D video content is transmitted to the user device for display.

20. The apparatus of claim 19, wherein the 3D video content is a first 3D video content, wherein the executable instructions include instructions that cause the at least one processor to: receiving, from the user device, information indicating a user selection for adjusting playback of the 3D video segment; generating second 3D video content based on the 3D video clip and the user selection; and The second 3D video content is transmitted to the user device for display.

21. The apparatus of claim 20, wherein the user selection comprises at least one of an adjustment to a viewing angle or an adjustment to a characteristic of the animated object.

Images