Apparatuses, methods and non-transitory computer-readable storage mediums for video stitching

Abstract

A system for providing panoramic video includes at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the system to receive video frames from at least two cameras, receive pose data of a user, generate a pose prediction for the user based on the pose data, the video frames, and network data, generate a quality level indicator, and generate a panoramic video by combining the video frames based on the pose prediction and the quality level indicator. The video frames are configured to be combined together at an overlap portion of adjacent video frames. The quality level indicator includes a quality level for each overlap portion of adjacent video frames based on the pose prediction.

Description

APPARATUSES, METHODS AND NON-TRANSITORY COMPUTER-READABLE STORAGE MEDIUMS FOR VIDEO STITCHINGTECHNICAL FIELD

[0001] One or more example embodiments relate to methods, apparatuses, and / or non-transitory computer-readable storage mediums for stitching of video frames together based on a pose prediction.BACKGROUND

[0002] Creation of panoramic or 360 degree video requires more than one video frame to be stitched together. To generate real time panoramic or 360 degree video, a pose of a user may be considered.SUMMARY

[0003] The scope of protection sought for various example embodiments is set out by the independent claims. The example embodiments and / or features, if any, described in this specification that do not fall under the scope of the independent claims are to be interpreted as examples useful for understanding various embodiments.

[0004] At least one example embodiment describes a system for providing panoramic video. The system may include at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the system to receive video frames from at least two cameras, receive pose data of a user, generate a pose prediction for the user based on the pose data of the user, the video frames, and network data, generate a quality level indicator including a quality level for each overlap portion of adjacent video frames based on the pose prediction, and generate a panoramic video by combining the video frames based on the pose prediction and the quality level indicator. The video frames may be configured to be combined together at an overlap portion of adjacent video frames. The network data may include a network latency.

[0005] In at least one example embodiment, the at least one memory may store instructions that, whenexecuted by the processor, cause the system to monitor the network latency and a pose prediction error to generate a quality measurement of the panoramic video.

[0006] In at least one example embodiment, the at least one memory may store instructions that, when executed by the processor, cause the system to generate the pose prediction with a machine learning model using reinforcement learning. In at least one example embodiment, the at least one memory may store instructions that, when executed by the processor, cause the system to train the machine learning model based on the pose prediction and a latency during the combining of the video frames. The latency may include a network latency and a stitching procedure latency.

[0007] In at least one example embodiment, the at least one memory may store instructions that, when executed by the processor, cause the system to obtain the pose data from at least one internal camera of a headset worn by the user. In at least one example embodiment, the at least one memory may store instructions that, when executed by the processor, cause the system to output the panoramic video via the headset worn by the user.

[0008] In at least one example embodiment, the at least one memory may store instructions that, when executed by the processor, cause the system to determine a stitching latency. The stitching latency may be a latency between the video frames received from the at least two cameras and the video frames of the panoramic video being stitched.

[0009] In at least one example embodiment, combining the video frames based on the pose prediction and the quality level indicator may include combining a first set of adjacent video frames at a first quality level of the quality level indicator and combining a second set of adjacent video frames at a second quality level of the quality level indicator. The first quality level may be greater than the second quality level. In at least one example embodiment, a region that is stitched together of the first set of adjacent video frames may align with the pose prediction for the user.

[0010] In at least one example embodiment, combining the video frames together may include stitching the overlap portions of the video frames together.

[0011] In at least one example embodiment, the video frames, the pose data, and the network data maybe continuously received and the pose prediction, quality level indicator, and panoramic video may be continuously generated based on the video frames, the pose data, and the network data.

[0012] Also described herein is a method for providing panoramic video. The method may include receiving video frames from at least two cameras, receiving pose data of a user, generating a pose prediction for the user based on the pose data of the user, the video frames, and network data, generating a quality level indicator including a quality level for each overlap portion of adjacent video frames based on the pose prediction, and generating a panoramic video by combining the video frames based on the pose prediction and the quality level indicator. The video frames may be configured to be combined together at an overlap portion of adjacent video frames. The network data may include a network latency.

[0013] In at least one example embodiment, the method may further include monitoring the network latency and a pose prediction error to generate a quality measurement of the panoramic video.

[0014] In at least one example embodiment, the method may further include generating the pose prediction with a machine learning model using reinforcement learning. In at least one example embodiment, the method may further include training the machine learning model based on the pose prediction and a latency during the combining of the video frames. The latency may include a network latency and a stitching procedure latency.

[0015] In at least one example embodiment, the method may further include obtaining the pose data from at least one internal camera of a headset worn by the user. In at least one example embodiment, the method may further include outputting the panoramic video via the headset worn by the user.

[0016] In at least one example embodiment, the method may further include determining a stitching latency. The stitching latency may be a latency between the video frames received from the at least two cameras and the video frames of the panoramic video being stitched.

[0017] In at least one example embodiment, combining the video frames based on the pose prediction and the quality level indicator may include combining a first set of adjacent video frames at a first quality level of the quality level indicator and combining a second set of adjacent video frames at a second quality level of the quality level indicator. The first quality level may be greater than the second quality level. In atleast one example embodiment, a region that is stitched together of the first set of adjacent video frames may align with the pose prediction for the user.

[0018] In at least one example embodiment, combining the video frames together may include stitching the overlap portions of the video frames together.

[0019] In at least one example embodiment, the video frames, the pose data, and the network data may be continuously received and the pose prediction, quality level indicator, and panoramic video may be continuously generated based on the video frames, the pose data, and the network data.

[0020] Also described herein is a non-transitory computer-readable storage medium storing computerexecutable instruction that, when executed by the at least one processor of a system, may cause the system to perform a method for providing panoramic video. The method may include receiving video frames from at least two cameras, receiving pose data of a user, generating a pose prediction for the user based on the pose data of the user, the video frames, and network data, generating a quality level indicator including a quality level for each overlap portion of adjacent video frames based on the pose prediction, and generating a panoramic video by combining the video frames based on the pose prediction and the quality level indicator. The video frames may be configured to be combined together at an overlap portion of adjacent video frames. The network data may include a network latency.

[0021] Also described herein is a device for providing panoramic video. The device may include means for receiving video frames from at least two cameras, receiving pose data of a user, generating a pose prediction for the user based on the pose data of the user, the video frames, and network data, generating a quality level indicator including a quality level for each overlap portion of adjacent video frames based on the pose prediction, and generating a panoramic video by combining the video frames based on the pose prediction and the quality level indicator. The video frames may be configured to be combined together at an overlap portion of adjacent video frames. The network data may include a network latency.BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Example embodiments will become more fully understood from the detailed description given herein below and the accompanying drawings, wherein like elements are represented by like reference numerals, which are given by way of illustration only and thus are not limiting of this disclosure.

[0023] FIG. 1 illustrates an edge device that may implement methods of video stitching according to example embodiments.

[0024] FIG. 2 illustrates an example embodiment of a system for video stitching.

[0025] FIG. 3 illustrates another example embodiment of a system for video stitching.

[0026] FIG. 4 illustrates an example embodiment of a prediction module of the system of FIG. 2.

[0027] FIG. 5 illustrates an example embodiment of a stitcher module of the system of FIG. 2.

[0028] FIG. 6 is a flow chart illustrating a method of stitching video frames, according to example embodiments.

[0029] FIG. 7 is a flow chart illustrating a method of generating a panoramic video of step 610 of FIG. 6, according to example embodiments.

[0030] It should be noted that these figures are intended to illustrate the general characteristics of methods, structure and / or materials utilized in certain example embodiments and to supplement the written description provided below. These drawings are not, however, to scale and may not precisely reflect the precise structural or performance characteristics of any given embodiment, and should not be interpreted as defining or limiting the range of values or properties encompassed by example embodiments. The use of similar or identical reference numbers in the various drawings is intended to indicate the presence of a similar or identical element or feature.DETAILED DESCRIPTION

[0031] Various example embodiments will now be described more fully with reference to the accompanying drawings in which some example embodiments are shown.

[0032] Detailed illustrative embodiments are disclosed herein. However, specific structural andfunctional details disclosed herein are merely representative for purposes of describing example embodiments. The example embodiments may, however, be embodied in many alternate forms and should not be construed as limited to only the embodiments set forth herein.

[0033] It should be understood that there is no intent to limit example embodiments to the particular forms disclosed. On the contrary, example embodiments are to cover all modifications, equivalents, and alternatives falling within the scope of this disclosure. Like numbers refer to like elements throughout the description of the figures.

[0034] While one or more example embodiments may be described from the perspective of a function or network element such as an edge device, a virtual reality (VR) headset, or a cloud device, etc., it should be understood that one or more example embodiments discussed herein may be performed by one or more processors (or processing circuitry) at the applicable device, apparatus, or system. For example, according to one or more example embodiments, at least one memory may store instructions that, when executed by one or more processors, cause the device, the edge device, the VR headset, the cloud device, or the like, to perform the operations discussed herein.

[0035] As discussed herein, the term "mechanism," in addition to its plain and ordinary meaning, may refer to methods, apparatuses and / or non-transitory computer readable storage mediums where applicable.

[0036] As discussed herein, the terminology “one or more” and “at least one” may be used interchangeably.

[0037] It will be appreciated that a number of example embodiments may be used in combination.

[0038] One or more example embodiments provide mechanisms to facilitate creation of panoramic and / or 360 degree video via video stitching. To create the panoramic and / or 360 degree video, more than one video frame is stitched together based on at least a pose of a user and a network latency. The systems and methods described herein may make use of all the pose and the latency in order to generate a video that has different quality levels at different portions to create a real time video with higher quality where a user is focused and a lower quality where the user is not focused.

[0039] FIG. 1 illustrates an example embodiment of an edge device 100 that may implement the VR headsets and / or cloud devices described herein.

[0040] As shown, the edge device 100 includes: a processor 110, a memory 120 connected to the processor 110, and various communication interfaces 130 connected to the processor 110. The various communication interfaces 130 may constitute a transceiver for transmitting / receiving data from / to other network elements (e.g., devices, gateways, etc.). As will be appreciated, depending on the implementation of the edge device 100, the edge device 100 may include many more components than those shown in FIG. 1. However, it is not necessary that all of these generally conventional components be shown in order to disclose the illustrative example embodiment. For example purposes, the example embodiment shown in FIG. 1 will be discussed with regard to a processor 110. However, it should be understood that the edge device 100 shown in FIG. 1 may include one or more processors or other processing circuitry, such as one or more Application Specific Integrated Circuits (ASICs).

[0041] The memory 120 may be a computer readable storage medium that generally includes a random access memory (RAM), read only memory (ROM), and / or a permanent mass storage device, such as a disk drive. The memory 120 may also store an operating system and any other routines / modules / applications for providing the functionalities of the edge device 100 (including UPF, CPF, MPF, etc.) to be executed by the processor 110. These software components may also be loaded from a separate computer readable storage medium into the memory 120 using a drive mechanism (not shown). Such separate computer readable storage medium may include a disc, tape, DVD / CD-ROM drive, memory card, or other like computer readable storage medium (not shown). In some example embodiments, software components may be loaded into the memory 120 via one of the various communication interfaces 130, rather than via a computer readable storage medium.

[0042] The processor 110 or other processing circuitry may be configured to carry out instructions of a computer program by performing the arithmetical, logical, and input / output operations of the system. Instructions may be provided to the processor 110 by the memory 120.

[0043] The various communication interfaces 130 may be wired and may include components that interface the processor 110 with the other input / output components. As will be understood, the various communication interfaces 130 and programs stored in the memory 120 to set forth the special purpose functionalities of the system 100 will vary depending on the implementation of the device and / or gateway.

[0044] The various communication interfaces 130 may also include one or more user input devices (e.g., a keyboard, a keypad, a mouse, or the like) and user output devices (e.g., a display, a speaker, or the like).

[0045] The processor 110, the memory 120, and the various communication interfaces 130 may implement the elements described herein of the VR headset and cloud devices.

[0046] FIG. 2 illustrates an example embodiment of a system 200 for video stitching. The system may include a prediction module or circuit 202 and a stitcher module or circuit 204. The prediction module 202 and the stitcher module 204 may both receive one or more video frames 206. The one or more video frames 206 may be received from one or more cameras or other image capture devices. For example, a headset 208 may include more than one camera. In at least one example embodiment, the headset 208 may be a VR headset or a Head Mounted Display (HMD). Each camera may simultaneously produce a video frame that may be input as the one or more video frames 206 into the prediction module 202 and the stitcher module 204.

[0047] The prediction module 202 may also receive pose data from the headset 208. The pose data may include head tracking data and / or eye tracking data. The head tracking data may indicate a position of a user’s head relative to the space that the user is in. The eye tracking data may indicate a direction of focus of the user’s eyes within the space. The eye tracking data of the pose data may be obtained from one or more internal cameras of the headset 208 that are generally used to track eye movement of a user. In at least one example embodiment, there may be a calibration process which requires a user to move his or her eyes to follow a dot. Once this calibration process is complete, the headset 208 may be able to track and / or follow movement of the user’s eyes. The steps of obtaining the head tracking data and the eye tracking data may be well-known in the field of virtual reality involving HMDs, and thus, a detailed discussion of these steps is omitted for the sake of brevity.

[0048] The prediction module 202 may also receive one or more latency values. In at least one example embodiment, the prediction module 202 may receive a compositing latency 210 and a network latency 212. The compositing latency 210 may be a latency to form a stitched frame. The network latency 212 may be latency associated with each frame as the frames arrive at the prediction module 202.

[0049] The prediction module 202 may also receive a prediction error 214. The prediction error 214 may be a difference between a pose received at a first time and a pose received at a second time. For example, if a pose at time tniis Tni, and a pose at time tn2 is T^, then the prediction error may be: Errni= Tni- T^.

[0050] The prediction module 202 may be configured to output a pose prediction 216 and a stitching quality indication 218 based on the one or more video frames 206, the pose from the headset 208, the compositing latency 210, the network latency 212, and the prediction error 214. The pose prediction 216 may be a prediction of where a user of the headset 208 may direct his or her focus at a next instant given the input data at a first instant. The stitching quality indication 218 may be a quality level indicator such as an array or another data structure that may include a quality level of each of the areas of the video frames to be stitched together. For example, the stitching quality indication 218 may indicate that a first set of adjacent video frames should be stitched together at a first quality level and that a second set of adjacent video frames should be stitched together at a second quality level. In at least one example embodiment, the first quality level may be different than the second quality level. Additional details of generating the pose prediction 216 and the stitching quality indication 218 will be discussed with respect to FIG. 4.

[0051] The stitcher module 204 may receive the pose prediction 216 and the stitching quality indication 218 from the prediction module 202 as well as the one or more video frames 206 from the one or more cameras. The stitcher module 204 may be configured to stitch the one or more video frames 206 together based on the pose prediction 216 and the stitching quality indication 218. The stitcher module 204 may output a stitched video that may be a panoramic video and / or a 360 degree video based on the one or more video frames 206. Additional details for generating a stitched video, according to example embodiments, will be discussed with respect to FIG. 5.

[0052] FIG. 3 illustrates another example embodiment of the system 200. In at least one example embodiment, a multicast approach may be used to send the one or more video frames 206 to the prediction module 202 and the stitcher module 204. Using a multicast approach may allow the one or more video frames 206 to be fed in parallel to the prediction module 202 and the stitcher module 204 which enables parallel resources for transmitting, receiving, and processing the data from the one or more video frames 206. In at least one example embodiment, the prediction module 202 and the stitcher module 204 may be using separate communication socket and / or graphical processing unit (GPU) resources. Using a multicast approach may allow the stitching process to be performed in close to real-time to provide a relatively high-quality video experience to the user. This multicast approach may also prevent additional delay from being introduced because it may avoid generating additional traffic between the prediction module 202 and the stitcher module 204.

[0053] FIG. 4 illustrates an example embodiment of the prediction module 202. The prediction module 202 may utilize machine learning to generate the pose prediction 216 and the stitching quality indication 218. In at least one example embodiment, the machine learning algorithm may utilize Reinforcement Learning (RL). RL is a type of machine learning algorithm that focuses on teaching agents how to make decisions by allowing them to interact with an environment and receiving feedback in the form of rewards or penalties. The goal of a system that utilizes RL is to maximize an accumulated reward over time by learning to select a best action or actions based on a current state. In at least one example embodiment, transfer learning, Generative Adversarial Networks (GAN), and / or deep neural networks may also be used to generate the pose prediction.

[0054] RL as utilized in the prediction module may have a goal of keeping a pose prediction within an area of a 360 equirectangular projection of a frame. For example, the 360 equirectangular projection of a frame may be a vertical rectangle defined by a field of view (FOV) of a HMD such as the headset 208. The vertical rectangle may include one or more stitching areas or may exclude the one or more stitching areas.

[0055] A reward that is accumulated via the RL function of the prediction module 202 may be based on the accuracy of the pose prediction that is determined by the prediction module 202. The reward may also be based on latency such as the compositing latency 210 and the network latency 212. For example, as the compositing latency 210 and the network latency 212 are reduced and the pose prediction accuracy is increased, the reward may increase.

[0056] The prediction module 202 may include an agent 402 that may be the machine learning algorithm such as the RL function as described above. The agent 402 may interact with an environment 404. The environment 404 may be the real-world scenario that the user is within as well as movement of the head and the eyes of the user. The agent 402 may receive at least the one or more video frames 206 and the pose from the headset 208 as well as data from the environment 404 and may calculate the reward and use the reward calculations for future pose prediction calculations. The agent 402 may be configured to receive the one or more video frames 206 and the pose and may generate a pose prediction. As described above, the pose may include head tracking data and / or eye tracking data. For example, if a user’s head is at a first location and the eyes are looking in a first direction, based on additional pose data of other times or instances, the agent 402 may generate a pose prediction for where the user’s head and eyes will be oriented at a time following the time of the received one or more video frames 206 and the pose. The agent 402 may additionally be configured to modify the way that the stitching process is performed by the stitching module 204.

[0057] The agent 402 may also be configured to generate the stitching quality indication 218 based on the pose prediction 218. For example, there may be a first video frame and a second video frame that may be stitched together by the stitcher module 204 from the one or more video frames 206. The first video frame and the second video frame may include a first stitched area that may be an area of overlap between the first video frame and the second video frame that may be combined to generate the panoramic or 360 degree video that is output by the stitcher module 204. If the pose prediction generated by the prediction module 202 indicates that a pose of the user will place the first stitched area within a FOV of the user, then the stitching quality indication for the first stitched area may be of a high quality. If the pose predictiongenerated by the prediction module 202 indicates that a pose of the user will place the first stitched area outside of the FOV of the user, then the stitching quality indication for the first stitched area may be of a lower quality. For example, the pose prediction may indicate that the FOV of the user may be entirely within the first video frame, excluding the first stitched area. If the user will not be focused on the first stitched area, the first video frame and the second video frame may be stitched together with a lower quality. This may allow the panoramic or 360 degree video to be created in as close to real time as possible, which may provide an enhanced experience for the user.

[0058] As another example, if the one or more video frames 206 includes a first video frame, a second video frame, and a third video frame, there may be a first stitched area that may be an area of overlap between the first video frame and the second video frame and there may be a second stitched area that may be an area of overlap between the second video frame and the third video frame. If the system 200 is configured to generate a 360 degree video, then there may additionally be a third stitched area that may be an area of overlap between the third video frame and the first video frame. As described above, the prediction module 202 may generate a pose prediction and may determine whether the pose prediction places the user’s FOV within any of the first stitched area, the second stitched area, or the third stitched area. If the pose prediction indicates that the user’s FOV is within one of these areas, then the prediction module 202 may generate instructions indicating that the area including the user’s FOV be stitched together at a higher quality than the other stitched areas. The indication for each of the stitched areas may be the stitching quality indication 218.

[0059] In at least one example embodiment, the agent 402 may generate the pose prediction 216 and the stitching quality indication 218 to be sent to the stitching module 204. The agent 402 may continuously receive pose data from the headset 208 and one or more video frames at an instant in time following the first received one or more video frames. The agent 402 may also continuously receive the compositing latency 210, the network latency 212, and the prediction error 214. The agent 402 may be configured to learn and improve the pose prediction 216 and / or the stitching quality indication 218 based on the reward as described above when the agent 402 is an RL system. In at least one example embodiment, the machinelearning algorithm such as the RL function may be pre-trained with videos of an environment in order to generate a neural network that may be able to generate a pose prediction. Additionally, the machine learning algorithm such as the RL function may not receive training data and may instead be configured to learn adaptively and iteratively. For example, the RL function may utilize patterns such as reward shaping, sparse and dense rewards, intrinsic motivation, and / or curiosity -driven models. These RL function patterns may be well-known in the field of machine learning, and thus, a detailed discussion of these patterns is omitted for the sake of brevity.

[0060] While described in the context of a machine learning algorithm that utilizes RL, the prediction module 202 may utilize other known machine learning algorithms to generate the pose prediction 216 and the stitching quality indication 218 based on the one or more video frames 206, the pose from the VR headset 208, the compositing latency 210, the network latency 212, and the prediction error 214.

[0061] FIG. 5 is an example embodiment of the stitcher module 204. The stitcher module 204 may include a registration module 502 and a compositor module 504. The registration module 502 may take the one or more video frames 206 as inputs and may perform a calibration process to obtain registration data 506. In particular, at a first instance, the one or more video frames 206 may be the first video frames input into the registration module 502. The registration module 502 may then perform an initial calibration to obtain first registration data. After the first registration data is obtained, the registration module 502 may be executed asynchronously in parallel with the compositor module to obtain updated registration data to be used within the stitcher module 204.

[0062] The registration module 502 may perform a series of steps such as a feature matching, pairwise matching, homography estimation, camera adjusting, warping masks, exposure compensation, and seam finding to obtain the registration data. These steps may be well-known in the field of video stitching, and thus, a detailed discussion of these steps is omitted for the sake of brevity. For example, OpenCV stitching modules may perform the above-recited steps for video stitching. The registration data 506 may include at least a camera intrinsic matrix and an extrinsic matrix. In at least one example embodiment, the camera intrinsic matrix may be a 3x4 matrix that includes five parameters to correct the real camera view so thatthe real camera view may be treated as an ideal pinhole camera. The extrinsic matrix may be a 4x4 matrix that may compensate for the rotation and translation of the cameras in the real world. Thus, it may be possible to convert the 3D camera coordinates into 2D image coordinates (a.k.a., pixels).

[0063] The registration data 506 may be input into the compositor module 504.

[0064] In at least one example embodiment, along with the registration data 506, the compositor module 504 receives the one or more video frames 206, the pose prediction 216, and the stitching quality indication 218. The pose prediction 216 and the stitching quality indication 218 may be input into a configurator 508 of the compositor module 504. The compositor module 504 may be configured to perform a series of steps such as warping images and masks, exposure compensation, warping conversion, dilate, resize, blender preparation, and blender feed. Then, a full blender process occurs where the one or more video frames 206 are stitched together to create a stitched frame 510. The stitched frame 510 may be the panoramic or 360 degree video. These steps may be well-known in the field of video stitching, and thus, a detailed discussion of these steps is omitted for the sake of brevity. For example, OpenCV stitching modules may perform the above-recited steps for video stitching.

[0065] In at least one example embodiment, the configurator 508 is configured to interact with each of the steps of the compositor module 504 such that the compositor module 504 utilizes the pose prediction 216 and the stitching quality indication 218 to generate the stitched frame 510. For example, as described above, if the one or more video frames 206 includes a first video frame, a second video frame, and a third video frame, there may be a first stitched area that may be an area of overlap between the first video frame and the second video frame and there may be a second stitched area that may be an area of overlap between the second video frame and the third video frame. The prediction module 202 may generate the stitching quality indication 218 that may indicate a first quality level for the first stitched area and a second quality level for the second and third stitched areas. The stitching quality indication 218 and the pose prediction 216 may be input into the configurator 508 of the stitcher module 204 to generate a video that include a first quality level at the first stitched area and a second quality level at both the second stitched area and the third stitched area.

[0066] In at least one example embodiment, each of the one or more video frames 206 may include a timebased identifier such that only video frames taken at the same time are transformed into the stitched frame 510.

[0067] The stitcher module 204 may also be configured to determine a stitching latency 512. The stitching latency 512 may be determined by calculating a difference in the one or more video frames 206 that are being received from the headset 208 and the stitched frame 510 that is being output.

[0068] In at least one example embodiment, the stitcher module 204 may be implemented as a plugin that may be located in a device relatively close to the camera, in the edge, or in the headset 208 or a personal computer (PC) attached to the headset 208. For example, the stitcher module 204 may be a gstreamer plugin in at least one example embodiment. If the stitcher module 204 is in a device relatively close to the camera, then the stitcher module 204 may be a board with GPU encoding capabilities. If the stitcher module 204 is in the edge, it may be within an edge device such as inside a multi-access edge computing device (MEC) in a 6G network. The stitcher module 204 is not limited in location to the above noted locations, but may be located such that it is in communication with at least the headset 208 and the prediction module 202.

[0069] Similarly, the prediction module 202 may be located in a device relatively close to the camera, in the edge, or in the headset 208 or a PC attached to the headset 208. If either the stitcher module 204 or the prediction module 202 is located in the edge, the other module may be located in the headset 208 or a PC attached to the headset 208. In at least one example embodiment, both the prediction module 202 and the stitcher module 204 may be located in an edge device which may allow for service for multiple users and may reduce the complexity of the headset 208. Alternatively, if both the prediction module 202 and the stitcher module 204 are located in the headset 208, latency may be reduced, but the headset 208 may require specialized hardware to properly function.

[0070] FIG. 6 is a flow chart illustrating a method of stitching video frames together. For example purposes, the method shown in FIG. 6 will be discussed with regard to the system 200 shown in FIGs. 2-5.

[0071] At step 602, the system 200 may receive the one or more video frames 206 from at least two cameras of the headset 208. The one or more video frames 206 may be input into both the prediction module202 and the stitcher module 204 as described above.

[0072] At step 604, the system 200 may receive pose data from a user. The pose data may include head tracking data and / or eye tracking data. The pose data may be input into the prediction module 202 of the system 200 as described above.

[0073] At step 606, the prediction module 202 may generate the pose prediction 216 for the user based on the pose data and the one or more video frames 206.

[0074] At step 608, the prediction module 202 may generate a quality level indicator such as the stitching quality indication 218 based on the pose prediction 216. As described above, the stitching quality indication 218 may include a quality indicator for each stitching location between adjacent video frames of the one or more video frames 206.

[0075] At step 610, the stitcher module 204 may generate a stitched video such as a panoramic video or a 360 degree video based on the pose prediction 216, the stitching quality indication 218 and the one or more video frames 206. A method for generating a panoramic video at step 610, according to example embodiments, is described in more detail with regard to FIG. 7.

[0076] FIG. 7 is a flow chart illustrating an example embodiment of step 610 of the method 600 for stitching video frames together. For example purposes, the method shown in FIG. 7 will be discussed with regard to the system 200 shown in FIGs. 2-5.

[0077] At step 702, a first set of adjacent video frames are combined at a first quality level of the stitching quality indication 218 by the stitcher module 204. In particular, the first set of adjacent video frames may include a first video frame and a second video frame. The first video frame and the second video frame may include a portion that may be overlapping to be stitched together by the stitching module 204.

[0078] At step 704, a second set of adjacent video frames are combined at a second quality level of the stitching quality indication 218 by the stitcher module 204. In particular, the second set of adjacent video frames may include the second video frame and a third video frame. The second video frame and the third video frame may include a portion that may be overlapping to be stitched together by the stitchingmodule 204.

[0079] In at least one example embodiment, the first quality level may be different from the second quality level. For example, if the pose prediction indicates that a user’s FOV will include the portion of the first set of adjacent video frames that are stitched together, the first quality level may be higher than the second quality level. Similarly, if the pose prediction indicates that a user’s FOV will include the portion of the second set of adjacent video frames that are stitched together, the second quality level may be higher than the first quality level.

[0080] At step 706, the stitcher module 204 may combine the first set of adjacent video frames that have been stitched together and the second set of video frames that have been stitched together to create a panoramic video or a 360 degree video that is output to the user via the headset 208.

[0081] The systems, apparatuses, and methods described herein provide a high quality video experience. In at least one example embodiment, a panoramic or 360 degree video may be generated in close to realtime and may be output via a headset to a user based on a pose prediction generated from pose data of the user. The embodiments described herein provide a dynamic and adaptable video experience that is capable of changing with changing conditions and data received by the system 200.

[0082] Although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and similarly, a second element could be termed a first element, without departing from the scope of this disclosure. As used herein, the term "and / or," includes any and all combinations of one or more of the associated listed items.

[0083] When an element is referred to as being "connected," or "coupled," to another element, it can be directly connected or coupled to the other element or intervening elements may be present. By contrast, when an element is referred to as being "directly connected," or "directly coupled," to another element, there are no intervening elements present. Other words used to describe the relationship between elements should be interpreted in a like fashion (e.g., "between," versus "directly between," "adjacent," versus "directly adjacent," etc.).

[0084] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a," "an," and "the," are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises," "comprising," "includes," and / or "including," when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0085] It should also be noted that in some alternative implementations, the functions / acts noted may occur out of the order noted in the figures. For example, two figures shown in succession may in fact be executed substantially concurrently or may sometimes be executed in the reverse order, depending upon the functionality / acts involved.

[0086] Specific details are provided in the following description to provide a thorough understanding of example embodiments. However, it will be understood by one of ordinary skill in the art that example embodiments may be practiced without these specific details. For example, systems may be shown in block diagrams so as not to obscure the example embodiments in unnecessary detail. In other instances, well- known processes, structures and techniques may be shown without unnecessary detail in order to avoid obscuring example embodiments.

[0087] As discussed herein, illustrative embodiments will be described with reference to acts and symbolic representations of operations (e.g., in the form of flow charts, flow diagrams, data flow diagrams, structure diagrams, block diagrams, etc.) that may be implemented as program modules or functional processes include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types and may be implemented using existing hardware at, for example, existing network nodes, BNGs, servers, ANs, CPEs, routers, or other network elements and / or hardware. Such existing hardware may be processing or control circuitry such as, but not limited to, one or more processors, one or more Central Processing Units (CPUs), one or more controllers, one or more arithmetic logic units (ALUs), one or more digital signal processors (DSPs), one or more microcomputers, one ormore field programmable gate arrays (FPGAs), one or more System-on-Chips (SoCs), one or more programmable logic units (PLUs), one or more microprocessors, one or more Application Specific Integrated Circuits (ASICs), or any other device or devices capable of responding to and executing instructions in a defined manner.

[0088] Although a flow chart may describe the operations as a sequential process, many of the operations may be performed in parallel, concurrently or simultaneously. In addition, the order of the operations may be re-arranged. A process may be terminated when its operations are completed, but may also have additional steps not included in the figure. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the main function.

[0089] As disclosed herein, the term "storage medium," "computer readable storage medium" or "non- transitory computer readable storage medium” may represent one or more devices for storing data, including read only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage mediums, optical storage mediums, flash memory devices and / or other tangible machine-readable mediums for storing information. The term "computer-readable medium" may include, but is not limited to, portable or fixed storage devices, optical storage devices, and various other mediums capable of storing, containing or carrying instruction(s) and / or data.

[0090] Furthermore, example embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine or computer readable medium such as a computer readable storage medium. When implemented in software, a processor or processors will perform the necessary tasks. For example, as mentioned above, according to one or more example embodiments, at least one memory may include or store computer program code, and the at least one memory and the computer program code may be configured to, with at least one processor, cause a network element or network device to perform the necessary tasks. Additionally, the processor, memory and example algorithms, encoded ascomputer program code, serve as means for providing or causing performance of operations discussed herein.

[0091] A code segment of computer program code may represent a procedure, function, subprogram, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable technique including memory sharing, message passing, token passing, network transmission, etc.

[0092] The terms “including” and / or “having,” as used herein, are defined as comprising (i.e., open language). The term “coupled,” as used herein, is defined as connected, although not necessarily directly, and not necessarily mechanically. Terminology derived from the word “indicating” (e.g., “indicates” and “indication”) is intended to encompass all the various techniques available for communicating or referencing the object / information being indicated. Some, but not all, examples of techniques available for communicating or referencing the object / information being indicated include the conveyance of the object / information being indicated, the conveyance of an identifier of the object / information being indicated, the conveyance of information used to generate the object / information being indicated, the conveyance of some part or portion of the object / information being indicated, the conveyance of some derivation of the object / information being indicated, and the conveyance of some symbol representing the object / information being indicated.

[0093] According to example embodiments, network functions, edge devices, headsets, or other network elements, may be (or include) hardware, firmware, hardware executing software or any combination thereof. Such hardware may include processing or control circuitry such as, but not limited to, one or more processors, one or more CPUs, one or more controllers, one or more ALUs, one or more DSPs, one or more microcomputers, one or more FPGAs, one or more SoCs, one or more PLUs, one or more microprocessors, one or more ASICs, or any other device or devices capable of responding to and executing instructions in a defined manner.

[0094] Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments of the invention. However, the benefits, advantages, solutions to problems, and any element(s) that may cause or result in such benefits, advantages, or solutions, or cause such benefits, advantages, or solutions to become more pronounced are not to be construed as a critical, required, or essential feature or element of any or all the claims.

Claims

WHAT IS CLAIMED IS:

1. A system for providing panoramic video, the system comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the system to receive video frames from at least two cameras, the video frames configured to be combined together at an overlap portion of adjacent video frames, receive pose data of a user, generate a pose prediction for the user based on the pose data of the user, the video frames, and network data, the network data including a network latency, generate a quality level indicator including a quality level for each overlap portion of adjacent video frames based on the pose prediction, and generate a panoramic video by combining the video frames based on the pose prediction and the quality level indicator.

2. The system of claim 1, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the system to: monitor the network latency and a pose prediction error to generate a quality measurement of the panoramic video.

3. The system of any one of the preceding claims, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the system to: generate the pose prediction with a machine learning model using reinforcement learning.

4. The system of claim 3, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the system to: train the machine learning model based on the pose prediction and a latency during the combining of the video frames, the latency including a network latency and a stitching procedure latency.

5. The system of any one of the preceding claims, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the system to: obtain the pose data from at least one internal camera of a headset worn by the user.

6. The system of claim 5, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the system to: output the panoramic video via the headset worn by the user.

7. The system of any one of the preceding claims, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the system to: determine a stitching latency, the stitching latency being a latency between the video frames received from the at least two cameras and the video frames of the panoramic video being stitched.

8. The system of any one of the preceding claims, wherein combining the video frames based on the pose prediction and the quality level indicator includes combining a first set of adjacent video frames at a first quality level of the quality level indicator and combining a second set of adjacent video frames at a second quality level of the quality level indicator, the first quality level being greater that the second quality level.

9. The system of claim 8, wherein a region that is stitched together of the first set of adjacent video frames aligns with the pose prediction for the user.

10. The system of any one of the preceding claims, wherein combining the video frames together includes stitching the overlap portions of the video frames together.

11. The system of any one of the preceding claims, wherein the video frames, the pose data, and the network data are continuously received and the pose prediction, quality level indicator, and panoramic video are continuously generated based on the video frames, the pose data, and the network data.

12. A method for providing panoramic video, the method comprising: receiving video frames from at least two cameras, the video frames configured to be combined together at an overlap portion of adjacent video frames, receiving pose data of a user, generating a pose prediction for the user based on the pose data of the user, the video frames, and network data, the network data including a network latency, generating a quality level indicator including a quality level for each overlap portion of adjacent video frames based on the pose prediction, and generating a panoramic video by combining the video frames based on the pose prediction and the quality level indicator.

13. The method of claim 12, further comprising: monitoring the network latency and a pose prediction error to generate a quality measurement of the panoramic video.

14. The method of claim 12 or claim 13, further comprising: generating the pose prediction with a machine learning model using reinforcement learning.

15. The method of claim 14, further comprising: training the machine learning model based on the pose prediction and a latency during the combining of the video frames, the latency including a network latency and a stitching procedure latency.

16. The method of any one of claims 12-15, further comprising: obtaining the pose data from at least one internal camera of a headset worn by the user.

17. The method of claim 16, further comprising: outputting the panoramic video via the headset worn by the user.

18. The method of any one of claims 12-17, further comprising: determining a stitching latency, the stitching latency being a latency between the video frames received from the at least two cameras and the video frames of the panoramic video being stitched.

19. The method of any one of claims 12-18, wherein combining the video frames based on the pose prediction and the quality level indicator includes combining a first set of adjacent video frames at a first quality level of the quality level indicator and combining a second set of adjacent video frames at a second quality level of the quality level indicator, the first quality level being greater that the second quality level.

20. The method of claim 19, wherein a region that is stitched together of the first set of adjacent video frames aligns with the pose prediction for the user.

21. The method of any one of claims 12-20, wherein combining the video frames together includes stitching the overlap portions of the video frames together.

22. The method of any one of claims 12-21, wherein the video frames, the pose data, and the network data are continuously received and the pose prediction, quality level indicator, and panoramic video are continuously generated based on the video frames, the pose data, and the network data.

Images