CONTENT DISTRIBUTION METHOD, IMAGE CAPTURING AND PROCESSING SYSTEM, Playback SYSTEM, METHOD OF OPERATING THE Playback system, AND COMPUTER READABLE MEDIUM

By capturing images at high resolution but low frame rate in virtual reality systems and generating high frame rate data using interpolation technology, the image transmission efficiency and cost issues are solved, and the quality of the 3D virtual reality experience is improved.

CN119031109APending Publication Date: 2024-11-26NEVERMIND CAPITAL LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411157134.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2019-04-24
Filing Date
2020-04-24
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The prior art is difficult to efficiently capture and transmit high-resolution images, especially in 3D virtual reality systems, resulting in poor image playback experience, high cost and serious bandwidth limitations.

Method used

Using images to capture high resolution but operate the camera at a lower frame rate, generate image data at a high frame rate through interpolation, and select interpolated segments based on motion and lens field of view information before transmission, reducing the transmitted interpolated frame data , effectively utilize bandwidth.

Benefits of technology

Reduces the risk of motion artifacts in virtual reality systems, reduces content processing and bandwidth complexity, improves image transmission efficiency, and supports a high-quality 3D virtual reality experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119031109A_ABST
    Figure CN119031109A_ABST
Patent Text Reader

Abstract

The invention relates to a content distribution method, an image capturing and processing system, a playback system, a method of operating the playback system, and a computer readable medium. Methods and apparatus for capturing, communicating and using image data to support a virtual reality experience are described. An image, such as a frame, is captured at a high resolution but a lower frame rate than the frame rate used for playback. An interpolation is applied to the captured frame to generate an interpolated frame. The captured frame is communicated to the playback device along with the interpolated frame information. The combination of the captured frame and the interpolated frame corresponds to a second frame playback rate that is higher than the image capture rate. The camera operates at a high image resolution but the frame rate is slower than images captured at a lower resolution with the same camera. Interpolation is performed prior to delivery to the user device, where the segment to be interpolated is selected based on motion and / or lens FOV information. Relatively small amounts of interpolated frame data are communicated for effective bandwidth usage as compared to capturing frame data.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Related patent applications

[0002] This application is a divisional application of the invention patent application with international application number PCT / US2020 / 029958, international application date April 24, 2020, date of entry into the Chinese national stage December 23, 2021, Chinese national application number 202080046243.2, and invention name “Content distribution method, image capture and processing system, playback system, method for operating a playback system, and computer-readable medium”.

[0003] This application claims the benefit of the filing date of U.S. Provisional Patent Application Serial No. 62 / 838,300, filed on April 24, 2019, which is hereby expressly incorporated by reference in its entirety. Technical Field

[0004] The present invention relates to methods and apparatus for encoding and delivering images with motion for a virtual reality system (e.g., a system supporting playback on a virtual reality device, e.g., which can render 2D or 3D image content). Background Art

[0005] High quality image capture may be important for practical virtual reality applications. Unfortunately, capturing high resolution images at high frame rates may involve the use of extremely expensive high speed high resolution cameras, which may not be practical due to cost reasons for many applications.

[0006] While capturing high-resolution images to support virtual reality is desirable, delivering large numbers of high-resolution images in bulk can present communications issues given the bandwidth constraints of the communications networks available to transmit the images to playback devices.

[0007] The problem of image capture and data transfer constraints becomes a particularly severe problem for non-stereo systems in the context of 3D video and 3D virtual reality systems, where separate left-eye and right-eye image data often need to be transferred.

[0008] In the case of 3D virtual reality systems, the difference between the left eye image content and the right eye image content is what typically conveys to a human observer a sense of depth and the 3D nature of the viewed object. Therefore, for an actual depth experience, it is typically important to capture and preserve image detail.

[0009] In view of the foregoing, it should be appreciated that there is a need for methods and apparatus for supporting the use of high resolution images for virtual reality applications. It would be desirable if the image capture rate did not need to match the image playback rate to allow the use of cameras that may not support the desired image playback rate when capturing images at a high level of detail. In addition, there is a need for methods and apparatus that can support playback and sufficient detail information to allow users of the playback system to enjoy the experience. It is desirable that at least some of the methods and / or apparatus address the problem of delivering image content to playback devices with limited bandwidth for delivering such content, and it is desirable that at least some of the methods are usable in situations where the content may include objects in motion (such as may be expected in a sporting event where a ball or other object may be moving). All embodiments need not address all of the above issues, and embodiments and features that address one or more of the issues discussed are useful and desirable. Summary of the invention

[0010] The present invention describes methods and apparatus for capturing, delivering, and using image data to support virtual reality experiences. Images, such as frames, are captured at a high resolution but at a frame rate lower than the frame rate to be supported for playback. Interpolation is applied to the captured frames. The captured frames are delivered to a playback device along with the interpolated frame information. The combination of the captured frames and the interpolated frames corresponds to a second frame playback rate that is higher than the image capture rate. By operating a camera at a high image resolution but at a slower frame rate than the same camera can capture images at a lower resolution, details are preserved. Interpolation is performed prior to delivery to a user device, where the segments to be interpolated are selected based on motion and / or lens field of view (FOV) information. A relatively small amount of interpolated frame data is delivered compared to the captured frame data to efficiently use bandwidth.

[0011] Although various features are discussed in the Summary of the Invention, all embodiments need not include all of the features discussed in the Summary of the Invention. Therefore, the discussion or reference to one or more features in the Summary of the Invention is not intended to imply that the one or more features discussed are required or essential in all embodiments.

[0012] The described method and apparatus have several advantages over other methods. The method addresses and avoids some of the more severe motion artifacts that may occur when content is limited to updating at the capture frame rate when playing immersive content on certain HMDs (head mounted displays). By understanding the nature of the artifacts, for example, they are created due to the motion and / or shape of the portion of the lens used to capture the image portion, the method addresses reducing the risk of such artifacts by selectively updating high motion segments and / or segments corresponding to portions of the camera lens that are subject to high distortion, without the need to configure multiple VR (virtual reality) systems.

[0013] The method also reduces the complexity of processing and bandwidth required to deliver content compared to other systems, with fewer artifact issues than some known systems because the method efficiently utilizes the limited bandwidth available to deliver image data to the playback device. One or more aspects of the method may be implemented on an embedded device at the last point in the content delivery pipeline.

[0014] An exemplary content distribution method relates to a content distribution method, which includes: storing images captured at a first frame rate; performing interpolation to generate interpolated frame data to support a second frame rate higher than the first frame rate; and transmitting the captured frame data and the interpolated frame data to at least one playback device.

[0015] The invention also relates to various playback system features, methods, and implementations.

[0016] An exemplary method implementation relates to a method of operating a playback system, the method comprising: receiving capture frame data and interpolated frame data; recovering a capture frame from the received capture frame data; generating one or more interpolated frames from the received interpolated frame data; rendering a video sequence including one or more capture frames and at least one interpolated frame; and outputting one or more rendered images to a display device.

[0017] Many variations of the methods and apparatus described above are possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 An exemplary system that may be used to capture content, stream content, and output content to one or more users, implemented in accordance with some embodiments of the present invention is shown.

[0019] Figure 2 An exemplary content delivery system having encoding capabilities that may be used to encode and stream content in accordance with features of the present invention is shown.

[0020] Figure 3 shows a method for receiving, decoding and displaying Figure 2 An exemplary content playback system for content streamed by the system.

[0021] Figure 4 A camera rig is shown that includes multiple camera pairs for capturing left-eye and right-eye images corresponding to different 120-degree sectors of a 360-degree field of view, and one or more cameras pointed skyward to capture a view of the sky.

[0022] Figure 5It shows how 5 different environment meshes corresponding to different camera views can be combined to create a complete spherical view / environment to which a background image can be applied as part of the playback operation.

[0023] Figure 6 Shown is the complete assembly of 5 meshes to create a spherical simulation environment.

[0024] Figure 7 An environment mesh model corresponding to a sector of a camera rig is shown, wherein one of the images is applied (eg, projected) onto the environment mesh to generate a background image.

[0025] Figure 8 The application of images captured by the cameras corresponding to each sector as well as the sky and ground cameras of the camera rig is shown to simulate a complete 3D environment in the form of a sphere, which can be used as a background for applicable foreground objects.

[0026] Fig. 9 A method of processing, capturing image content according to the present invention is shown, which can be and sometimes is performed by Figure 1 The content delivery system shown and / or Figure 1 The content delivery system shown is implemented using image processing, calibration and encoding equipment.

[0027] Fig.10 is a diagram showing how the transmission frame rate generated by the content delivery system exceeds the capture frame rate in various embodiments of the present invention.

[0028] Fig.11 An exemplary overall transmission frame sequence is shown, where CF begins to indicate capture frames that are delivered, for example, as part of a base video layer, and IF is used to indicate interpolated frame information that is delivered as an enhancement layer in some embodiments, where each set of interpolated frame information includes one or more segments generated, for example, by using motion interpolation.

[0029] Fig.12 A captured frame used as a base layer frame is shown, where the circle corresponds to a portion of the frame that includes image content corresponding to a fisheye lens used to capture the image in some embodiments. Edge portions outside the circle are not of interest because they will not be used as texture and correspond to areas outside the captured scene area of ​​interest.

[0030] Fig.13 It is shown how information for reconstructing a specific segment is sent as interpolated frame data.

[0031] Fig.14 It is shown how the velocity vector and various rates and positions relative to the fisheye lens capture area can be considered to determine the frame rate that should be supported through interpolation.

[0032] Fig.15 is a chart showing the various interpolated frame times and which segments will be interpolated and delivered based on the selected frame rate to be supported by the particular segment.

[0033] Fig.16 is a flow chart of an exemplary playback method according to an exemplary embodiment. DETAILED DESCRIPTION

[0034] Figure 1 An exemplary system 100 implemented according to some embodiments of the present invention is shown. System 100 supports content delivery (e.g., imaging content delivery) to one or more client devices (e.g., playback devices / content players) located at client locations. System 100 includes an exemplary image capture device 102, a content delivery system 104, a communication network 105, and multiple client locations 106, ..., 110. Image capture device 102 supports the capture of stereoscopic images. Image capture device 102 captures and processes imaging content according to features of the present invention. Communication network 105 can be, for example, a hybrid fiber coaxial (HFC) network, a satellite network, and / or the Internet.

[0035] The content delivery system 104 includes an image processing, calibration and encoding apparatus 112 and a content delivery device 114, such as a streaming server 114. The image processing, calibration and encoding apparatus 112 is responsible for performing a variety of functions, including performing camera calibration based on one or more target images and / or grid patterns captured during the camera calibration process, generating a distortion correction or compensation grid that a playback device can use to compensate for distortion introduced by the calibration camera, processing, such as cropping and encoding, the captured images, and providing calibration and / or environmental information to the content delivery device 114, which can be provided to the playback device and used in the rendering / image playback process. The content delivery device 114 can be implemented as a server, as will be discussed below, and the delivery device responds to requests for content using image calibration information, optional environmental information, and one or more images captured by the camera equipment 102 that can be used to simulate a 3D environment. The streaming of images and / or content can be, and sometimes is, a function of feedback information, such as a viewer's head position and / or a user selection of a location at an event corresponding to the camera equipment 102, which camera equipment will be the source of the image. For example, a user may select or switch between images from a camera rig positioned at a centerline to a camera rig positioned at a live target, and the simulated 3D environment and streaming images are changed to those corresponding to the camera rig selected by the user. Figure 11, multiple camera rigs may be present in the system and located at different physical locations in a sporting or other event, with the user being able to switch between the different locations and the user selection being passed from the playback device 122 to the content server 114. Although separate devices 112, 114 are shown in the image processing and content delivery system 104, it should be understood that the system may be implemented as a single device including separate hardware for performing the various functions, or with different functions controlled by different software or hardware modules but implemented in or on a single processor.

[0036] The encoding device 112 can and in some embodiments does include one or more encoders for encoding image data according to the present invention. Encoders can be used in parallel to encode different parts of a scene and / or to encode a given part of a scene to generate encoded versions with different data rates. When real-time or near real-time streaming is to be supported, using multiple encoders in parallel may be particularly useful.

[0037] The content streaming device 114 is configured to stream, for example, transmit encoded content to deliver encoded image content to one or more client devices, for example, via the communication network 105. Via the network 105, the content delivery system 104 can send information and / or exchange information with devices located at the client locations 106, 110, as shown, represented by a link 120 that passes through the communication network 105.

[0038] Although the encoding device 112 and the content delivery server 114 Figure 1 In the example, they are shown as separate physical devices, but in some embodiments, they are implemented as a single device that encodes content and streams the content. The encoding process can be a 3D (e.g., stereo) image encoding process, in which information corresponding to the left eye view and the right eye view of the scene portion is encoded and included in the encoded image data, so that 3D image viewing can be supported. The specific encoding method used is not critical to the present application, and a wide range of encoders can be used as or implement the encoding device 112.

[0039] Each customer site 106, 110 may include multiple playback systems, such as devices / players, such as devices that can decode and play back / display imaged content streamed by content streaming device 114. Customer site 1 106 includes playback system 101, which includes a decoding device / playback device 122 coupled to a display device 124. Customer site N 110 includes playback system 111, which includes a decoding device / playback device 126 coupled to a display device 128. In some embodiments, display devices 124, 128 are head-mounted stereoscopic display devices. In various embodiments, playback system 101 is a head-mounted system supported by a strap worn around a user's head. Thus, in some embodiments, customer site 1 106 includes playback system 1 101 including a decoding device / playback apparatus 122 coupled to a display 124 (e.g., a head-mounted stereoscopic display), and customer site N 110 includes playback system N 111 including a decoding device / playback apparatus 126 coupled to a display 128 (e.g., a head-mounted stereoscopic display).

[0040] In various embodiments, the decoding devices 122, 126 present the imaged content on the corresponding display devices 124, 128. The decoding devices / players 122, 126 may be devices capable of decoding imaged content received from the content delivery system 104, generating imaged content using the decoded content, and rendering the imaged content (e.g., 3D image content) on the display devices 124, 128. Any of the decoding devices / playback devices 122, 126 may be used as Figure 3 The decoding device / playback device 800 shown. System / playback device (such as Figure 3 The system / playback device shown) can be used as either of the decoding devices / playback devices 122, 126.

[0041] Figure 3 An exemplary content delivery system 700 having encoding capabilities that may be used to encode and stream content in accordance with features of the present invention is shown.

[0042] The system can be used to perform object detection, encoding, storage and transmission and / or content output according to features of the present invention. The content delivery system 700 can be used as Figure 1 System 104. Although Figure 3 The system shown is for encoding, processing, and streaming of content, but it should be understood that the system 700 may also include the ability to decode and display processed and / or encoded image data to an operator, for example.

[0043] System 700 includes display 702, input device 704, input / output (I / O) interface 706, processor 708, network interface 710, and memory 712. The various components of system 700 are coupled together via bus 709, which allows data to be transferred between the components of system 700.

[0044] The memory 712 includes various modules, such as routines, which when executed by the processor 708 control the system 700 to implement the partitioning, encoding, storage and streaming / transmission and / or output operations according to the present invention.

[0045] The memory 712 includes various modules, such as routines, which control the computer system 700 to implement the immersive stereoscopic video acquisition, encoding, storage and transmission and / or output method according to the present invention when executed by the processor 707. The memory 712 includes a control routine 714, a partitioning module 706, an encoder 718, a detection module 719, a streaming controller 720, a received input image 732 (e.g., a 360-degree stereoscopic video of a scene), an encoded scene portion 734, timing information 736, an environmental mesh model 738, a UV map 740, and a plurality of correction grid information sets, including a first correction grid information 742, a second correction grid information 744, a third correction grid information 746, a fourth correction grid information 748, a fifth correction grid information 750, and a sixth correction grid information 752. In some embodiments, the module is implemented as a software module. In other embodiments, the module is implemented in hardware, for example, as a single circuit, wherein each module is implemented as a circuit for performing the function corresponding to the module. In other embodiments, the module is implemented using a combination of software and hardware. The memory 712 also includes a frame buffer 715 for storing captured images (e.g., left-eye and right-eye images captured by the left and right cameras of the camera pair) and / or interpolated frame data representing interpolated frames generated by interpolating a sequence of buffered frames between captured frames. In some embodiments, the segment definition and position change information is stored in a portion 717 of the memory 712. The definition information can, and sometimes does, define a set of blocks corresponding to an object that is interpreted as an image segment.

[0046] As will be discussed below, an interpolated frame can and sometimes does include one or more interpolated segments that are combined with data from a previous frame or a subsequent frame to form a complete interpolated frame. In various embodiments, an interpolated frame is represented using a portion of the data used to represent a captured frame, for example, 1 / 20 or less, in some cases 1 / 100, the amount of data used to represent an intra-encoded captured frame. Therefore, when transmitting encoded frames and interpolated frames, relatively less data can be used to transmit the interpolated frame. In some cases, the decoder of the playback device 122 uses the image content from the previous frame to perform a default filling operation to fill in the portion of the interpolated frame that is not transmitted to the playback device. The segments of the interpolated frame generated by the interpolation can be transmitted to the playback device 122 as inter-encoded data or as intra-encoded data using motion vectors depending on the embodiment.

[0047] The control routines 714 include device control routines and communication routines to control the operation of the system 700. The partitioning module 716 is configured to partition the received stereoscopic 360 degree version of the scene into N scene portions according to features of the present invention.

[0048] Encoder 718 may include and in some embodiments does include multiple encoders configured to encode received image content (e.g., 360-degree version of the scene and / or one or more scene parts) according to the features of the present invention. In some embodiments, the encoder includes multiple encoders, each of which is configured to encode stereoscopic scenes and / or partitioned scene parts to support a given bit rate stream. Therefore, in some embodiments, multiple encoders can be used to encode each scene part to support multiple different bit rate streams for each scene. The output of encoder 718 is a coded scene part 734 stored in a memory for streaming to a client device (e.g., a playback device). The coded content can be streamed to one or more different devices via network interface 710.

[0049] The detection module 719 is configured to detect a network-controlled switch of streaming content from a current camera pair (e.g., a first stereo camera pair) to another camera pair (e.g., a second stereo camera pair or a third stereo camera pair). That is, the detection module 719 detects whether the system 700 has switched from a streaming content stream generated using images captured by a given stereo camera pair (e.g., a first stereo camera pair) to a streaming content stream generated using images captured by another camera pair. In some embodiments, the detection module is also configured to detect a user-controlled change from receiving a first content stream including content from a first stereo camera pair to receiving a second content stream including content from a second stereo camera pair, for example, detecting a signal from a user playback device indicating that the playback device is attached to a content stream different from the content to which it was previously attached. The streaming controller 720 is configured to control the streaming of the encoded content to deliver the encoded image content to one or more client devices, for example, via the communication network 105.

[0050] The streaming controller 720 includes a request processing module 722, a data rate determination module 724, a current head position determination module 726, a selection module 728, and a streaming control module 730. The request processing module 722 is configured to process the received request for imaging the content from the client playback device. In various embodiments, the request for content is received via a receiver in the network interface 710. In some embodiments, the request for content includes information indicating the identity of the requested playback device. In some embodiments, the request for content may include the data rate supported by the client playback device, the current head position of the user, such as the position of a head-mounted display. The request processing module 722 processes the received request and provides the retrieved information to other elements of the streaming controller 720 to take further action. Although the request for content may include data rate information and current head position information, in various embodiments, the data rate supported by the playback device may be determined based on network testing and other network information exchanges between the system 700 and the playback device.

[0051] The data rate determination module 724 is configured to determine available data rates that can be used to stream the imaged content to the client device, e.g., due to support for multiple encoded scene portions, the content delivery system 700 may support streaming of content to the client device at multiple data rates. The data rate determination module 724 is further configured to determine the data rates supported by the playback device requesting the content from the system 700. In some embodiments, the data rate determination module 724 is configured to determine the available data rates for delivery of the image content based on network measurements.

[0052] The current head position determination module 726 is configured to determine the current viewing angle and / or current head position of the user based on information received from the playback device, such as the position of a head mounted display. In some embodiments, the playback device periodically sends the current head position information to the system 700, where the current head position determination module 726 receives and processes the information to determine the current viewing angle and / or current head position.

[0053] The selection module 728 is configured to determine which parts of the 360-degree scene to stream to the playback device based on the user's current viewing angle / head position information. The selection module 728 is also configured to select an encoded version of the determined scene portion to support streaming of the content based on the available data rate.

[0054] The streaming control module 730 is configured to control the streaming of image content (e.g., multiple parts of a 360-degree stereoscopic scene) at various supported data rates according to the features of the present invention. In some embodiments, the streaming control module 730 is configured to control the stream N parts of the 360-degree stereoscopic scene of the playback device of the request content to initialize the scene memory in the playback device. In various embodiments, the streaming control module 730 is configured to, for example, periodically send the selected coded version of the determined scene part at a determined rate. In some embodiments, the streaming control module 730 is further configured to send a 360-degree scene update to the playback device according to a time interval (e.g., once per minute). In some embodiments, sending a 360-degree scene update includes sending N scene parts or NX scene parts of a complete 360-degree stereoscopic scene, where N is the total number of parts that the complete 360-degree stereoscopic scene has been divided into, and X represents the selected scene part that has been recently sent to the playback device. In some embodiments, the streaming control module 730 waits a predetermined time after initially sending the N scene portions for initialization before sending the 360-degree scene update. In some embodiments, timing information for controlling the sending of the 360-degree scene updates is included in the timing information 736. In some embodiments, the streaming control module 730 is further configured to identify a scene portion that has not been transmitted to the playback device during the refresh interval; and transmit an updated version of the identified scene portion that has not been transmitted to the playback device during the refresh interval.

[0055] In various embodiments, the streaming control module 730 is configured to periodically deliver at least a sufficient number of the N portions to the playback device to allow the playback device to fully refresh the 360 ​​degree version of the scene at least once during each refresh cycle.

[0056] In some embodiments, the streaming controller 720 is configured to control the system 700 to transmit, for example, a stereoscopic content stream (e.g., the encoded content stream 734) via a transmitter in the network interface 710, the stereoscopic content stream including images from one or more cameras (e.g., cameras of a stereoscopic camera pair, such as Figure 4 In some embodiments, the streaming controller 720 is configured to control the system 700 to transmit an environment mesh model 738 to be used for rendering the image content to one or more playback devices. In some embodiments, the streaming controller 720 is further configured to transmit to the playback device a first UV map to be used to map a portion of the image captured by the first stereo camera pair to a portion of the environment mesh model as part of an image rendering operation.

[0057] In various embodiments, the streaming controller 720 is further configured to provide (e.g., transmitted via a transmitter in the network interface 710) one or more sets of correction grid information to the playback device, such as first correction grid information, second correction grid information, third correction grid information, fourth correction grid information, fifth correction grid information, and sixth correction grid information. In some embodiments, the first correction grid information is used to render image content captured by the first camera of the first stereo camera pair, the second correction grid information is used to render image content captured by the second camera of the first stereo camera pair, the third correction grid information is used to render image content captured by the first camera of the second stereo camera pair, the fourth correction grid information is used to render image content captured by the second camera of the second stereo camera pair, the fifth correction grid information is used to render image content captured by the first camera of the third stereo camera pair, and the sixth correction grid information is used to render image content captured by the second camera of the third stereo camera pair. In some embodiments, the streaming controller 720 is further configured to indicate to the playback device, for example, by sending a control signal, that the third correction grid information and the fourth correction grid information should be used when content captured by the second stereo camera pair is streamed to the playback device instead of content from the first stereo camera pair. In some embodiments, the streaming controller 720 is further configured to indicate to the playback device that third correction grid information and fourth correction grid information should be used in response to the detection module 719 detecting i) a network-controlled switch of streaming content from the first stereo camera pair to the second stereo camera pair, or ii) a user-controlled change from receiving a first content stream including content from the first stereo camera pair to receiving a second content stream including encoded content from the second stereo camera pair.

[0058] The memory 712 also includes an environment mesh model 738, a UV map 740, and a correction grid information set, the correction grid information set including first correction grid information 742, second correction grid information 744, third correction grid information 746, fourth correction grid information 748, fifth correction grid information 750, and sixth correction grid information 752. The system provides the environment mesh model 738 to one or more playback devices for rendering image content. The UV map 740 includes at least a first UV map, which is used to map a portion of the image captured by the first stereo camera pair to a portion of the environment mesh model 738 as part of an image rendering operation. The first correction grid information 742 includes information generated based on a measurement of one or more optical properties of the first lens of the first camera of the first stereo camera pair, and the second correction grid includes information generated based on a measurement of one or more optical properties of the second lens of the second camera of the first stereo camera pair. In some embodiments, the first stereo camera pair and the second stereo camera pair correspond to a forward viewing direction, but correspond to different locations at an area or event location where content is being captured for streaming.

[0059] In some embodiments, the processor 708 is configured to perform various functions corresponding to the steps discussed in flowcharts 600 and / or 2300. In some embodiments, the processor uses routines and information stored in memory to perform various functions and control system 700 to operate according to the method of the present invention. In one embodiment, the processor 708 is configured to control the system to provide a first correction grid information and a second correction grid information to a playback device, the first correction grid information being used to render the image content captured by the first camera, and the second correction grid information being used to render the image content captured by the second camera. In some embodiments, the first stereo camera pair corresponds to a first direction, and the processor is further configured to control system 700 to transmit a stereo content stream, the stereo content stream comprising an encoded image generated by the image content captured by the first camera and the second camera. In some embodiments, the processor 708 is further configured to transmit an environmental mesh model to be used for rendering image content to the playback device. In some embodiments, the processor 708 is further configured to transmit to the playback device a first UV map that will be used to map a portion of the image captured by the first stereo camera pair to a portion of the environmental mesh model as part of an image rendering operation. In some embodiments, the processor 708 is further configured to control the system 700 to provide a third correction grid information and a fourth correction grid information to the playback device, the third correction grid information being used to render the image content captured by the first camera of the second stereo camera pair, and the fourth correction grid information being used to render the image content captured by the second camera of the second stereo camera pair. In some embodiments, the processor 708 is further configured to control the system 700 to indicate to the playback device (e.g., transmitted via the network interface 710) that the third correction grid information and the fourth correction grid information should be used when the content captured by the second camera pair is streamed to the playback device instead of the content from the first camera pair. In some embodiments, the processor 708 is further configured to control the system 700 to indicate to the playback device that the third correction grid information and the fourth correction grid information should be used in response to the system detecting: i) a network-controlled switch of streaming content from the first stereo camera pair to the second stereo camera pair, or ii) a user-controlled change from receiving a first content stream including content from the first stereo camera pair to receiving a second content stream including encoded content from the second stereo camera pair. In some embodiments, the processor 708 is further configured to control the system 700 so that the system provides fifth correction grid information and sixth correction grid information to the playback device, wherein the fifth correction grid information is used to render image content captured by the first camera of the third stereo camera pair, and the sixth correction grid information is used to render image content captured by the second camera of the third stereo camera pair.

[0060] Figure 32 shows a playback system 300 implemented according to an exemplary embodiment of the present invention. The playback system 300 is, for example, Figure 1 The exemplary playback system 300 includes a computer system / playback device 800 coupled to a display 805 (e.g., a head-mounted stereoscopic display). The computer system / playback device 800 implemented in accordance with the present invention may be used to receive, decode, store, and display content from a content delivery system (such as a head-mounted stereoscopic display). Figure 1 and Figure 2 The playback device may be used with a 3D head mounted display such as an OCULUS RIFTTM VR (virtual reality) headset, which may be a head mounted display 805. The device 800 includes the ability to decode the received encoded image data and generate 3D image content for display to the customer. In some embodiments, the playback device is located at a customer premises location such as a residence or office, but may also be located at the image capture site. The device 800 may perform signal reception, decoding, display, and / or other operations according to the present invention.

[0061] The device 800 includes a display 802, a display device interface 803, an input device 804, a microphone (mic) 807, an input / output (I / O) interface 806, a processor 808, a network interface 810, and a memory 812. The various components of the playback device 800 are coupled together via a bus 809, which allows data to be passed between the components of the system 800. Although in some embodiments, the display 802 is included as an optional element as shown using a dashed box, in some embodiments, an external display device 805 (e.g., a head-mounted stereoscopic display device) can be coupled to the playback device via the display device interface 803.

[0062] Via I / O interface 806, system 800 can be coupled to external devices to exchange signals and / or information with other devices. In some embodiments, via I / O interface 806, system 800 can receive information and / or images from external devices and output information and / or images to external devices. In some embodiments, via interface 806, system 800 can be coupled to an external controller, such as a handheld controller.

[0063] The processor 808 (e.g., a CPU) executes the routines 814 and modules in the memory 812 and uses the stored information to control the system 800 to operate in accordance with the present invention. The processor 808 is responsible for controlling the overall general operation of the system 800. In various embodiments, the processor 808 is configured to perform the functions that have been discussed as being performed by the playback system 800.

[0064] Via the network interface 810, the system 800 transmits signals and / or information (e.g., including coded images and / or video content corresponding to the scene) to various external devices and / or receives signals and / or information from various external devices through a communication network (e.g., such as the communication network 105). In some embodiments, the system receives one or more content streams including coded images captured by one or more different cameras from the content delivery system 700 via the network interface 810. The received content stream may be stored as received coded data, such as the coded image 824. In some embodiments, the interface 810 is configured to receive a first coded image including image content captured by a first camera and a second coded image corresponding to a second camera. The network interface 810 includes a receiver and a transmitter, via which receiving and transmitting operations are performed. In some embodiments, the interface 810 is configured to receive correction grid information corresponding to a plurality of different cameras, the correction grid information including first correction grid information 842, second correction grid information 844, third correction grid information 846, fourth correction grid information 848, fifth correction grid information 850, and sixth correction grid information 852, which are then stored in the memory 812. In addition, in some embodiments, via the interface 810, the system receives one or more masks 832, environment mesh models 838, UV maps 840, which are then stored in the memory 812.

[0065] The memory 812 includes various modules, such as routines, which control the playback device 800 to decode and output operations according to the present invention when executed by the processor 808. The memory 812 includes a control routine 814, a request to a content generation module 816, a head position and / or viewing angle determination module 818, a decoder module 820, a stereo image rendering engine 822 also referred to as a 3D image generation module, a determination module, and data / information including received encoded image content 824, decoded image content 826, a 360-degree decoded scene buffer 828, generated stereo content 830, a mask 832, an environment mesh model 838, a UV map 840, and a plurality of received correction grid information sets, the plurality of received correction grid information sets including a first correction grid information 842, a second correction grid information 844, a third correction grid information 846, a fourth correction grid information 848, a fifth correction grid information 850, and a sixth correction grid information 852.

[0066] The control routines 814 include device control routines and communication routines to control the operation of the device 800. The request generation module 816 is configured to generate a request for content to be sent to the content delivery system to provide the content. In various embodiments, the request for content is sent via the network interface 810. The head position and / or perspective determination module 818 is configured to determine the user's current perspective and / or current head position, such as the position of a head mounted display, and report the determined position and / or perspective information to the content delivery system 700. In some embodiments, the playback device 800 periodically sends the current head position information to the system 700.

[0067] Decoder module 820 is configured to decode the encoded image content 824 received from content delivery system 700 to generate decoded image data, such as decoded image 826. Decoded image data 826 may include decoded stereoscopic scene and / or decoded scene part. In some embodiments, decoder 820 is configured to decode the first encoded image to generate the first decoded image, and decode the received second encoded image to generate the second decoded image. The decoded first image and the second image are included in the stored decoded image 826.

[0068] The 3D image rendering engine 822 performs rendering operations (e.g., using content and information received and / or stored in the memory 812, such as the decoded image 826, the environment mesh model 838, the UV map 840, the mask 832, and the mesh correction information) and generates a 3D image for display to a user on the display 802 and / or the display device 805 according to the features of the present invention. The generated stereoscopic image content 830 is the output of the 3D image generation engine 822. In various embodiments, the rendering engine 822 is configured to perform a first rendering operation using the first correction information 842, the first decoded image, and the environment mesh model 838 to generate a first image for display. In various embodiments, the rendering engine 822 is further configured to perform a second rendering operation using the second correction information 844, the second decoded image, and the environment mesh model 838 to generate a second image for display. In some such embodiments, the rendering engine 822 is further configured to perform the first rendering operation and the second rendering operation using the first UV map (included in the received UV map 840). When performing a first rendering operation to compensate for distortion introduced into the first image by the lens of the first camera, the first correction information provides information about corrections to be made to the node positions in the first UV map, and when performing a second rendering operation to compensate for distortion introduced into the second image by the lens of the second camera, the second correction information provides information about corrections to be made to the node positions in the first UV map. In some embodiments, the rendering engine 822 is further configured to use a first mask (included in mask 832) to determine how the portion of the first image is combined with the portion of the first image corresponding to a different field of view when applying the portion of the first image to the surface of the environment mesh model as part of the first rendering operation. In some embodiments, the rendering engine 822 is further configured to use the first mask to determine how the portion of the second image is combined with the portion of the second image corresponding to a different field of view when applying the portion of the second image to the surface of the environment mesh model as part of the second rendering operation. The generated stereoscopic image content 830 includes a first image and a second image (e.g., corresponding to a left eye view and a right eye view) generated as a result of the first rendering operation and the second rendering operation. In some embodiments, the portion of the first image corresponding to the different fields of view corresponds to the sky or ground field of view. In some embodiments, the first image is a left eye image corresponding to the forward field of view, and the first image corresponding to the different fields of view is a left eye image captured by a third camera corresponding to a side field of view adjacent to the forward field of view. In some embodiments, the second image is a right eye image corresponding to the forward field of view, and wherein the second image corresponding to the different fields of view is a right eye image captured by a fourth camera corresponding to a side field of view adjacent to the forward field of view. Therefore, the rendering engine 822 renders the 3D image content 830 to the display.In some embodiments, an operator of playback device 800 may control one or more parameters via input device 804 and / or a selection operation to be performed (eg, selecting to display a 3D scene).

[0069] The network interface 810 allows the playback device to receive content and / or transfer information from the streaming device 114, such as the viewing head position and / or position (camera equipment) selection indicating the selection of a specific viewing position at the event. In some embodiments, the decoder 820 is implemented as a module. In such embodiments, when executed, the decoder module 820 causes the received image to be decoded, and the 3D image rendering engine 822 causes the image to be further processed according to the present invention and optionally spliced ​​together as part of the rendering process.

[0070] In some embodiments, the interface 810 is further configured to receive additional grid correction information corresponding to multiple different cameras, for example, third grid correction information, fourth grid correction information, fifth grid correction information, and sixth grid correction information. In some embodiments, the rendering engine 822 is also configured to use the grid correction information corresponding to the fourth camera (e.g., fourth grid correction information 848) when rendering an image corresponding to the fourth camera, the fourth camera being one of the multiple different cameras. The determination module 823 is configured to determine which grid correction information the rendering engine 822 is to use when performing a rendering operation based on which camera captured image content is used for the rendering operation or based on an instruction from the server, the instruction indicating which grid correction information should be used when rendering an image corresponding to the received content stream. In some embodiments, the determination module 823 may be implemented as part of the rendering engine 822.

[0071] In some embodiments, Figure 2 The memory 712 and Figure 3 The modules and / or elements shown in the memory 812 of the embodiment are implemented as software modules. In other embodiments, the modules and / or elements, although shown as included in the memory, are implemented in hardware as, for example, separate circuits, wherein each element is implemented as a circuit for performing the function corresponding to the element. In other embodiments, the modules and / or elements are implemented using a combination of software and hardware.

[0072] Although in Figure 2 and Figure 3800, and in some embodiments, are fully implemented in hardware within the processor, such as as a separate circuit of the corresponding device, such as in the case of the content delivery system, in the processor 708, and in the case of the playback system 800, in the processor 808. In other embodiments, some elements are implemented in the corresponding processors 708 and 808, for example, as circuits, while other elements are implemented and coupled to the processors as circuits outside the processors. It should be understood that the integration level of the module on the processor and / or the integration level of some modules outside the processor can be one of the design choices. Alternatively, rather than being implemented as a circuit, all or some elements can be implemented in software and stored in the memory, wherein the software module controls the operation of the corresponding systems 700 and 800 to implement the functions corresponding to the modules when the modules are executed by their corresponding processors (e.g., processors 708 and 808). In other embodiments, various elements are implemented as a combination of hardware and software, for example, wherein a circuit outside the processor provides an input to the processor, and then the input operates under software control to perform a part of the functions of the module.

[0073] Although in Figure 2 and Figure 3 In each of the embodiments of the present invention, a single processor, such as a computer, is shown, but it should be understood that each of the processors 708 and 808 can be implemented as one or more processors, such as computers. When one or more elements in the memories 712 and 812 are implemented as software modules, the modules include code that, when executed by a processor of the corresponding system (e.g., processors 708 and 808), configures the processor to implement the functions corresponding to the module. Figure 7 and Figure 8 In an embodiment in which the various modules shown are stored in a memory, the memory is a computer program product including a computer-readable medium including codes for causing at least one computer (e.g., a processor) to implement the functions corresponding to the modules, e.g., separate codes for each module.

[0074] Modules that are completely hardware-based or completely software-based may be used. However, it should be understood that any combination of software and hardware (e.g., circuit-implemented modules) may be used to implement these functions. It should be understood that Figure 2 The modules shown control and / or configure the system 700 or elements therein, such as the processor 708, to perform the functions of the corresponding steps of the method of the present invention, such as those shown and / or described in the flowchart. Similarly, Figure 3 The illustrated modules control and / or configure the system 800 or elements therein, such as the processor 808, to perform the functions of corresponding steps of the methods of the present invention, such as those shown and / or described in the flowcharts.

[0075] To facilitate understanding of the image capture process, we will now refer to Figure 4 The exemplary camera equipment shown. The camera equipment 1300 can be used as Figure 1 The system is equipped with 102, and includes a plurality of stereo camera pairs, each corresponding to a different one of the three sectors. A first stereo camera pair 1301 includes a left eye camera 1302 (e.g., a first camera) and a right camera 1304 (e.g., a second camera), the left eye camera and the right camera being intended to capture images corresponding to those images that would be seen by the left eye and right eye of a person positioned at the position of the first camera pair. A second stereo camera pair 1305 corresponds to the second sector and includes a left camera 1306 and a right camera 1308, while a third stereo camera pair 1309 corresponds to the third sector and includes a left camera 1310 and a right camera 1312. Each camera is mounted in a fixed position in a support structure 1318. A camera 1314 facing upward is also included. In Figure 4 A downward facing camera that is not visible in the apparatus may be included below camera 1314. A stereo camera pair is used in some embodiments to capture paired upward and downward images, whereas in other embodiments a single upward camera and a single downward camera are used. In other embodiments, a downward image is captured prior to equipment placement and used as a static ground image for the duration of the event. Given that the ground view tends not to change significantly during an event, this approach is often satisfactory for many applications. The output of the cameras of equipment 1300 is captured and processed.

[0076] When using Figure 4 When a camera rig is used, each sector corresponds to a known 120 degree viewing area relative to the camera rig position, where the captured images from different sector pairs are stitched together based on the known images mapped to the simulated 3D environment. Although a 120 degree portion of each image captured by a sector camera is typically used, the camera captures a wider image corresponding to an approximately 180 degree viewing area. Therefore, the captured images can be subjected to masking in a playback device as part of the 3D environment simulation. Figure 5 14 is a composite diagram 1400 showing how a 3D spherical environment can be simulated using portions of an environment grid corresponding to different pairs of cameras of the rig 102. Note that one grid portion is shown for each of the sectors of the rig 102, with the sky grid being used relative to the top camera view and the ground grid being used for ground images captured by the downward facing camera. While the masks for the top and bottom images are circular in nature, the mask applied to the sector images is truncated to reflect that the top and bottom portions of the scene area will be provided by the top and bottom cameras, respectively.

[0077] When combined, the overall meshes corresponding to different cameras produce a spherical mesh, such as Figure 6 Note that the grid is shown for a monocular image, but in the case of capturing a stereoscopic image pair, it is used for both the left-eye image and the right-eye image.

[0078] Figure 5 Grid and mask information of the type shown can be, and sometimes is, communicated to the playback device. The information communicated will vary depending on the rig configuration. For example, if a greater number of sectors are used, the mask corresponding to each of the sectors will correspond to a viewing area of ​​less than 120 degrees where more than 3 environment grids are required to cover the diameter of the sphere.

[0079] Environmental mapping information is shown as optionally transmitted to the playback device in step 1132. It should be understood that environmental mapping information is optional, because without transmitting such information, the environment can be assumed to be a default size range. In the case where multiple different default size spheres are supported, an indication of what size sphere to use can and sometimes is transmitted to the playback device.

[0080] Image capture operations may be performed continuously during the event, particularly with respect to each of the three sectors that may be captured by camera equipment 102 .

[0081] Note that although multiple camera views corresponding to different sectors are captured, the image capture rate need not be the same for all sectors. For example, a front-facing sector corresponding to, for example, the main play field may capture images at the fast frame rate of cameras corresponding to other sectors and / or top (sky) and bottom (ground) views.

[0082] Figure 7 Mapping of the portion of the image corresponding to the first sector to a corresponding 120 degree portion of a sphere representing the 3D viewing environment is shown.

[0083] The images corresponding to different portions of the 360 ​​degree environment are combined to the extent necessary to provide a continuous viewing area to the viewer, e.g., depending on head position. For example, if the viewer is viewing the intersection of two 120 degree sectors, the portions of the images corresponding to each sector will be stitched together and presented to the viewer based on the known angle and position of each image in the overall 3D environment being simulated. The stitching and generation of images will be performed for each of the left eye view and the right eye view so that, in the case of a stereoscopic implementation, two separate images are generated, one for each eye.

[0084] Figure 8It shows how multiple decoded, rectified, and cropped images may be and sometimes are mapped and stitched together to create a 360 degree viewing environment that may be used as a background to which a foreground image of an object represented by point cloud data may be applied.

[0085] Fig. 9 Methods of capturing, processing, and delivering captured image content and interpolated content are shown. Fig. 9 The method shown may be implemented by the stereoscopic image capture system 102 working in conjunction with the content delivery system 104. Figure 1 1 is shown as a combination of separate image processing device 112 and content delivery device 114, but in some embodiments, these components are implemented as a single device, such as Figure 2 The content delivery system 700 shown may be, and is sometimes used as, Figure 1 The content delivery system 104 of the embodiment of the present invention is provided. In some embodiments, the processor 708 in the content delivery system communicates with the cameras of the camera pair (e.g., the left eye camera 1302 and the right eye camera 1304) and controls the cameras of one or more camera pairs to capture images of the environment at a first frame rate and provide the captured images to the content delivery system 104 for processing. In some embodiments, high-resolution images are captured at a first frame rate, but the image content is provided to the playback devices 122, 126 to support a second, higher frame rate, where interpolation is performed by the content delivery system to generate image data for supporting the higher frame rate. This allows the camera to operate in a high-resolution operating mode in which a high level of detail is captured, but if the camera instead captures a lower resolution image, it may be at a slower frame rate. The capture of high-resolution content provides details that allow a high-quality virtual reality experience, while the interpolation facilitates a frame rate that allows an actual motion experience when supporting 3D.

[0086] In some embodiments, using Figure 1 The system shown is implemented Fig. 9 The method 900 is shown, wherein Figure 2 The content delivery system 700 is Figure 1 104, and wherein the image is captured by a camera of camera equipment 102, which is sometimes used Figure 4 The apparatus 1300 is implemented, for example, under the control of a content delivery system and / or a system operator.

[0087] In a start step 902, the method 900 begins with the components of the system 100 being powered on, e.g., the content delivery system 104 and the camera equipment 102 being powered on and beginning operation. Operation proceeds from the start step 902 to an image capture step 904, wherein the left camera 1302 and the right camera 1304 of the camera pair of the camera equipment 102 are used to capture images, e.g., left eye images and right eye images, at a first image capture rate and a first resolution, respectively. In some embodiments, the first image resolution is a first maximum resolution supported by the cameras 1302, 1304, and the first image (e.g., frame) capture rate is lower than the second image (e.g., frame) capture supported when the first camera and the second camera operate at a lower resolution. Thus, in at least some embodiments, the cameras are operated to maximize the level of detail captured, although the trade-off is that the image capture rate is lower than the image capture rate that can be obtained using a lower image resolution.

[0088] In step 908, the image 908 captured by the camera in step 902 is stored in a frame buffer 715, which can be, and sometimes is, included in the content delivery system's memory 712. Capturing and storing can be, and sometimes is, performed on an ongoing basis while capturing and processing images of an ongoing event, sporting event.

[0089] The high-resolution image captured at the first frame rate and available for processing operations proceeds to step 910, in which motion analysis is performed on the frames (e.g., captured frames in a video sequence (e.g., left-eye frames and / or right-eye frame sequences)) to detect motion and identify moving segments. In at least one such embodiment, a block is a rectangular portion of a frame, and a segment is a group of one or more moving blocks. A segment can correspond to, and sometimes does correspond to, an object, such as a ball that changes position over time in the environment of the captured environment, such as a stadium or sports field.

[0090] The motion analysis performed in step 910 allows identification of image (e.g., frame) segments that may, and in some cases will, adversely affect the 3D playback experience if they are not updated at a rate greater than the first frame rate during playback. The processing in step 910 and other steps of flowchart 900 can be, and in some embodiments are, implemented by and / or under the control of the processor 708 of the content delivery system, which controls components of the system 100 to implement Fig. 9 The steps of method 900 are shown.

[0091] In some embodiments, step 910 includes one, more than one, or all of steps 912, 914, and 916. In step 912, the captured images of the video sequence are analyzed to identify matching groups of blocks, such as pixel blocks corresponding to objects that appear in consecutive frames of the image sequence (e.g., a sequence of left-eye and / or right-eye image frames). Then in step 914, the identified matching groups of blocks are defined as fragments. In some such embodiments, each fragment is a group of physically adjacent blocks, such as a group of blocks corresponding to physical objects moving in the environment of the captured image. In at least some embodiments, information about the position and size of the defined fragment (e.g., an object image) is stored in step 914, wherein the position information is stored on a per-frame basis. In step 916, the position change of the fragment of the captured frame from one frame time to the next frame time is determined, and the position change information is stored on a per-fragment (e.g., object) basis. In some embodiments, the change information is stored as a motion vector indicating how the fragment moves from the previous frame (e.g., the immediately previous frame in the frame sequence). By analyzing the change in position of segments corresponding to objects in the frames, such as a ball, over time, the movement of the segments from one frame time to the next can be determined, as well as the position of the ball in the environment and relative to the camera lens used to capture images of the environment. In step 916, the segment definitions and position change information are stored in portion 717 of memory 712 in some embodiments. The motion vectors determined for one or more segments in step 916 can be used and sometimes used to generate interpolated frames.

[0092] Where image segments have been defined and motion vectors have been generated for the segments indicating motion of the segments between captured frames, operation proceeds from step 910 to step 918 .

[0093] In step 918, the motion vector corresponding to the segment is considered, and the segment velocity of the segment is determined. It is also determined which portion of the camera's fisheye lens that the segment corresponds to that was used to capture the segment. It may be important to determine the portion of the fisheye lens that was used to capture the image segment and how much the segment will move because some portions of the fisheye lens are more curved than others. Image segments that may be captured by the same portion of the lens over an extended period of time may suffer less lens-induced distortion between frame times than segments that will move more in terms of the curvature of the lens area used to capture the image of the segment. Thus, step 918 allows the effects of motion to be considered in conjunction with the effects of lens curvature when determining at what rate the segment should be presented to the user and therefore what interpolation rate (if any) should be supported for a particular segment.

[0094] With the effect of the fisheye lens FOV already accounted for and quantified in terms of segment motion relative to the lens FOV, the operation proceeds from step 918 to step 920. In step 920, information defining each captured frame is stored, wherein the information includes segment information as well as image pixel values. Also stored as part of the frame information are pixel values ​​representing the captured image. The frame information and data may be stored in the segment portion of the frame buffer 715 and / or memory 717.

[0095] Operation proceeds from step 920 to step 922. In step 922, a frame rate for at least some image segments is determined based on the amount of motion of one or more segments. In some embodiments, the frame rate determination is performed on a per-segment basis. In some cases, step 922 includes step 923, which involves performing a segment frame rate determination based not only on the amount or rate of motion of the segment but also on the position of the segment in the field of view within the time period corresponding to the frame rate determination. In some cases, different frame rates are determined for segments of the same size corresponding to different portions of the field of view. In some embodiments, a segment that changes position relative to the camera field of view corresponding to a more curved lens portion is assigned a higher frame rate than a segment that changes position in a FOV portion corresponding to a less curved lens portion. In this way, in at least some embodiments, the effect of lens appearance distortion can be reduced by supporting a higher frame rate for segments that are more affected by the combination of motion and lens shape than for segments that are less affected by the combination of motion and lens shape. In at least some cases, the frame rate of segments that are subject to high motion is selected to be greater than the actual captured image frame rate. Segments that are not experiencing motion or are experiencing a motion rate below a threshold (e.g., a predetermined or variable segment rate motion threshold used to control whether interpolation is performed for the segment) are determined to have a selected frame rate equal to the image capture rate (e.g., a first frame rate) and, therefore, will not be interpolated. The interpolation segment motion threshold can, and sometimes is, varies depending on the amount of data available for delivering the data to the playback device (e.g., the data rate). Thus, for a first, low level data rate for delivering data to the playback device, in some embodiments, a first, higher threshold, and, therefore, a higher motion rate, is required for segment interpolation to be achieved, while for a second, higher level data rate for delivering data to the playback device, in some embodiments, a second, lower threshold, and, therefore, a lower motion rate, is required for segment interpolation to be achieved.

[0096] Where a frame rate has been determined for the identified image segments in steps 922 and / or 923, the operation proceeds to step 924. In step 924, segments are interpolated between capture frames to generate a set of interpolated segments for each non-capture frame for which frame information is to be communicated in the transport stream, wherein different numbers of interpolated segments are generated and communicated for at least some different interpolated (e.g., non-captured) frames.

[0097] By making decisions about which segments to interpolate before transmission to the playback device, information that is not usually available to the playback device, such as information about the shape of the camera lens used to capture the image and where the segments fall relative to the camera lens field of view (FOV), can be used to make decisions about the frame rate to support and which segments to interpolate for a particular frame time. In addition, relatively powerful processing resources are usually available in the distribution system compared to the playback device. This is because the distribution system can perform interpolation and encoding of content for multiple devices, allowing the use of relatively powerful and expensive processors or processor groups to support distribution. Given that playback devices are usually owned by individual customers, their cost and corresponding processing power are usually much smaller than that of the distribution system. In fact, in many cases, the playback device can find normal decoding and display of left-eye image content and right-eye image content to support 3D virtual reality experience computing classification, even if interpolation between captured frames is not supported. Therefore, for various reasons, it is beneficial to shift interpolation-related decisions and processing to the distribution side rather than the playback side. Furthermore, by using interpolation in accordance with the present invention, the distribution system may deliver frames at a higher frame rate than the image capture rate, thereby allowing for a more pleasing and realistic viewing experience than would be possible if the resulting image capture were reduced and detail necessary for an actual 3D experience was lost or changes between frames due to motion were excessively large.

[0098] In step 924, the segments are interpolated to support the frame rate selected for the segments in step 922. Thus, the segments for which different frame rates are selected (i.e., determined) in step 922 and determined (i.e., selected) in step 920 will be interpolated at their different corresponding frame rates. The segments for which the image capture rate is selected will not be subject to interpolation. The interpolated image data is used as interpolated frame information. The interpolated frame information is typically much smaller, for example, less than 1 / 20 of the amount of data used to represent the captured image frame, or less than 1 / 200 in some cases. Thus, in at least some cases, the interpolated frame information can be delivered to the playback device without a significant impact on the total amount of data transmitted. The interpolated frame information may include interpolated segments delivered as intra-frame coded image data and / or inter-frame coded image data. The interpolated frame information generated in step 924 may and sometimes also include padding information that provides instructions to the playback device, where missing frame information is obtained and / or how to fill a portion of the interpolated frame, which may have gaps due to the motion of the segment relative to the previous frame without the padding operation. In various embodiments, fill image data is obtained from a previous frame or a subsequent frame in some embodiments. The fill portion may be a portion of the environment that is not moving but is occluded by a moving object. Such fill data may be obtained from a frame that is one or more capture frame times away from the interpolated frame time.

[0099] Where interpolated segments and / or other interpolated frame data have been generated in step 924, operation proceeds to an encoding step 925, in embodiments where encoding of the frames is performed, or directly to a delivery step 926. In step 926, the captured frame data and interpolated frame data are delivered to at least one playback device, e.g., after being stored in the memory 700 of the content delivery system.

[0100] In step 925, when used, the captured frame and / or interpolated frame is encoded. MPEG or other image encoding content can be used to encode the captured image frame. In some embodiments, the left eye image data and the right eye image data corresponding to the frame time are merged into a single frame for encoding purposes, while in other embodiments, the left eye image data and the right eye image data are encoded as separate image streams and / or interlaced image streams.

[0101] Intra-frame and / or intra-coding techniques can be, and are sometimes, used to encode captured frames and interpolated frames.

[0102] It should be understood that the capture frame and the interpolated frame can be delivered to the playback device in an encoded or unencoded form. In most embodiments, the encoding of the captured image frame that can be considered as the base view layer is implemented together with the encoding of the interpolated frame. In step 925, the encoding of the frame information to be stored and transmitted to the playback device is implemented. Then in step 926, the frame data is passed to the playback device. In some embodiments, step 926 includes steps 927 and 928. The capture frame data (e.g., the capture frame) is passed to the playback device in, for example, an encoded form when encoding is used, and is passed to the playback device in an unencoded form when the encoding step 925 is not used. In some embodiments, the capture frame forms the base video layer delivered in step 927. In step 928, the interpolated frame information is delivered to the playback device. The interpolated frame information may include an encoded interpolated frame that conveys an interpolated fragment and / or fill information that can be used to construct a complete interpolated frame. In some cases, a motion vector included in the interpolated frame information indicates where in a captured frame, and in some cases another interpolated frame, segment content to be included in the interpolated frame may be found, as well as the distance the content will be moved from its original position in the source frame to form a portion of the interpolated frame. Thus, a motion vector corresponding to a segment of an interpolated frame can, and sometimes does, indicate the source of the segment image content to be included in an interpolated frame generated using the segment motion vector, as well as the position in the frame where the segment will be located.

[0103] In other embodiments, the interpolated frame information is communicated as a fragment motion vector and / or other information about how to construct the remainder of the interpolated frame from one or more other frames, for example, a previous or subsequent frame from which fill content is obtained and placed at a location in the generated interpolated frame, where the location of placement is specified by information in the interpolated frame information.

[0104] While the various figures illustrate a transmission order that matches the expected frame display order, it should be understood that the actual transmission order may vary as a result of buffering and reordering of transmission frames implemented by both the content delivery system and the playback system. In such cases, if the content delivery system reorders the captured frames and interpolated frames for transmission, the playback device will transition and restore the frames to the proper display order prior to display.

[0105] Although flowchart 900 ends with step 928, it should be understood that the capture, processing, and transmission of content can be performed on an ongoing basis or over a period of time corresponding to an event, for example. Thus, the steps of method 900 are repeatedly performed as new content (e.g., images of an environment) is captured, processed, and stored for future transmission or delivery to one or more playback systems.

[0106] It should be understood that according to Fig. 9A method comprising capturing content of images (e.g., frames) and interpolated frame data (e.g., interpolated frames) that, in some embodiments, is delivered to one or more playback systems 101, 111 for decoding and display, for example as part of a virtual reality experience where separate left-eye and right-eye images are presented to a user to provide a 3D experience. Fig.10 is a diagram 1000 illustrating how a transmission frame rate (TFR) 1002 generated by the content delivery system 104 exceeds a capture frame rate (CFR) in various embodiments of the invention.

[0107] Fig.11 An exemplary overall transmission frame sequence 1100 is shown, where CF is used to indicate a capture frame delivered, for example, as part of a base video layer, and IF is used to indicate interpolated frame information delivered as an enhancement layer in some embodiments, where each set of interpolated frame information includes one or more segments generated, for example, by using motion interpolation. Line 1106 is used to indicate a transmission frame sequence number. An exemplary frame group 1102 includes a first capture frame, which is capture frame 1 (CF1 1108), followed by multiple interpolated frames 1104 (interpolated frame 1 (IF1) 1110, interpolated frame 2 (IF2) 1112, interpolated frame 2 (IF3) 1114...), followed by a second capture frame (CF2) 1116. CF1 1008 corresponds to transmission frame number 1; IF1 1110 corresponds to transmission frame number 2; IF2 1112 corresponds to transmission frame number 2; IF3 1114 corresponds to transmission frame number 4, and CF2 1116 corresponds to transmission frame number Y, for example, where Y = frame 1104 + the number of interpolated frames in the 2 group.

[0108] Fig.12 FIG. 1200 shows captured frames (capture frame 1 (CF1) 1202, capture frame 2 (CF2) 1204, capture frame 3 (CF3) 1206, ..., capture frame N (CFN) 1208) used as base layer frames with circles (1203, 1205, 1207, ..., 1209) corresponding to frames (CF1 1202, CF2 1204, CF3, 1206, ..., CFN) respectively. 1208), which includes image content corresponding to the fisheye lens used to capture the image, and edge portions ((1210, 1212, 1214, 1215), (1218, 1220, 1222, 1224), (1226, 1228, 1230, 1232), …, (1234, 1236, 1238, 1240)) respectively outside the circle (1203, 1205, 1207, …, 1209) are not of interest in some embodiments because they will not be used as texture and correspond to areas outside the captured scene area of ​​interest.

[0109] Fig.13 Included is Figure 13000, which shows how information used to reconstruct a particular fragment can be sent as interpolated frame data. Note that information for different fragments can be transmitted at different frame rates, with updated fragment information for some interpolated frame times and some fragments of a frame set being omitted. When determining the frame rate for a single fragment, the position of the fragment relative to the fisheye lens that captured the image and the direction of motion of objects in the fragment (as shown by the directional arrows) can be and sometimes are considered. The fragment information is sent as supplemental information and is sometimes referred to as side information because the fragment information is information sent in addition to the base video frame layer information. In some embodiments, the fragment information for the interpolated frame is generated by using motion interpolation prior to encoding and transmission to a playback device.

[0110] Figure 13000 includes a set of frames, including capture frame 1 (CF1) 13002, six interpolated frames (IF1 13004, IF2 13006, IF3 13008, IF4 13010, IF5 13012, IF6 13014) and capture frame 2 (CF1) 13016. Five exemplary segments of interest (segment 1 (S1) 13050, segment 1 (S2) 13052, segment 3 (S3) 13054, segment 4 (S4) 13056, segment 5 (S5) 13058) are identified within the image capture area 13003 of capture frame 1 (CF1) 13002. The direction of motion of the objects in each segment (S1 13050, S2 13052, S3 13054, S4 13056, S5 13058) is represented by directional arrows (13051, 13053, 13055, 13057, 13059), respectively.

[0111] Five exemplary segments (Segment 1 (S1) 13050', Segment 1 (S2) 13052', Segment 3 (S3) 13054', Segment 4 (S4) 13056', Segment 5 (S5) 13058') are identified within the image capture area 13017 of capture frame 2 (CF2) 13016. Exemplary segment 1 (S1) 13050' of CF2 13016 includes the same object as segment 1 (S1) 13050 of CF1 13002. Exemplary segment 2 (S2) 13052' of CF2 13016 includes the same object as segment 2 (S2) 13052 of CF1 13002. Exemplary segment 3 (S3) 13054' of CF2 13016 includes the same object as segment 3 (S3) 13054 of CF1 13002. The exemplary segment 4 (S4) 13056' of CF2 13016 includes the same objects as the segment 4 (S4) 13056 of CF1 13002. The exemplary segment 5 (S5) 13058' of CF2 13016 includes the same objects as the segment 5 (S5) 13058 of CF1 13002.

[0112] The direction of motion of the objects in each segment (S1 13050', S2 13052', S3 13054', S4 13056', S5 13058') is indicated by direction arrows (13051', 13053', 13055', 13057', 13059'), respectively.

[0113] Fig.14 FIG. 14000 is an illustration showing how the velocity vector and various rates and positions relative to the fisheye lens capture area (13003) may be considered to determine the frame rate that should be supported through interpolation. Fig.14 Identified motion segments (S1 13050, S2 13052, S3 13054, S4 13056, S5 13058) are shown with corresponding determined velocity vectors (13051, 13053, 13055, 13057, 13059), respectively. The selected rates (R1 14002, R2 14004, R3 14006, R4 14008, R5 14010) for interpolation are based on the determined velocity vectors (13051, 13053, 13055, 13057, 13059), respectively, and on the positions of the segments (S1 13050, S2 13052, S3 13054, S4 13056, S5 13058) in the field of view 13003 of the capture frame 13002, respectively.

[0114] Fig.151 is a diagram 1500 showing various interpolated frame times and which segments will be interpolated and delivered based on the selected frame rate to be supported for a particular segment. Note that for different segments, different frame rates will be supported, where Y indicates that information for that segment will be included and N indicates that for a given frame time listed in the top row, interpolated information will not be generated and delivered as side information. Note that a number subscript is used after IF to indicate the interpolated frame to which the frame information relates. For example, IF1 corresponds to interpolated frame 1 and IF2 corresponds to interpolated frame 2.

[0115] The first column 1502 includes information identifying the motion segment of each row of the table. The second column 1504 includes information identifying the selected rate for interpolation of each motion segment. The third column 1508 includes information indicating whether interpolation information of the motion segment is to be generated, included, and transmitted for the interpolated frame IF1. The fourth column 1510 includes information indicating whether interpolation information of the motion segment is to be generated, included, and transmitted for the interpolated frame IF2. The fifth column 1512 includes information indicating whether interpolation information of the motion segment is to be generated, included, and transmitted for the interpolated frame IF3. The fifth column 1514 includes information indicating whether interpolation information of the motion segment is to be generated, included, and transmitted for the interpolated frame IF5. The sixth column 1516 includes information indicating whether interpolation information of the motion segment is to be generated, included, and transmitted for the interpolated frame IF6.

[0116] Row 1518 includes information identifying that for motion segment S1, the selected rate is R1, and that interpolation information for motion segment S1 is to be generated, included, and transmitted for each of interpolated frames IF1, IF2, IF3, IF4, IF5, and IF6.

[0117] Row 1520 includes information identifying that for motion segment S2, the selected rate is R2, and that interpolation information for motion segment S2 is to be generated, included, and transmitted for each of interpolated frames IF2, IF3, IF4, and IF5, but not for interpolated frames IF1 and IF6.

[0118] Row 1522 includes information identifying that for motion segment S3, the selected rate is R3, and that interpolation information for motion segment S3 is to be generated, included, and transmitted for each of interpolated frames IF3, IF4, and IF5, but not for interpolated frames IF1, IF2, and IF6.

[0119] Row 1524 includes information identifying that for motion segment S4, the selected rate is R4, and that interpolation information for motion segment S4 is to be generated, included, and transmitted for each of interpolated frames IF3, IF4 but not for interpolated frames IF1, IF2, IF5, and IF6.

[0120] Line 1526 includes information identifying that for motion segment S5, the selected rate is R5, and that interpolation information for motion segment S4 is to be generated, included, and transmitted for interpolated frame IF3 but not for interpolated frames IF1, IF2, IF4, IF5, and IF6.

[0121] A numbered list of exemplary embodiments is shown below. The numbers in each list are used to refer to the embodiment included in the list using the number.

[0122] First list of numbering method implementation schemes

[0123] Method Implementation 1 A content distribution method, the method comprising: storing (908) an image (751) captured at a first frame rate, performing interpolation (924) to generate interpolated frame data to support a second frame rate higher than the first frame rate; and transmitting (926) the captured frame data and the interpolated frame data to at least one playback device.

[0124] Method implementation 2 The method according to method implementation 1, wherein the first frame rate is the highest frame rate supported by the camera used to capture images when the camera operates at a maximum image capture resolution.

[0125] Method implementation 3 According to the method described in method implementation 1, it also includes: operating the camera of the first camera in the camera pair to capture the image at a first rate.

[0126] Method Implementation 4 The method according to Method Implementation 3, wherein the first camera supports a higher frame rate than the first frame rate when capturing images at a lower resolution.

[0127] Method implementation 5 is a method according to method implementation 1, wherein transmitting the captured frame data includes transmitting the captured frame at a first data rate corresponding to the image capture rate.

[0128] Method implementation 6 is a method according to method implementation 5, wherein the combination of the captured frame data and the interpolated frame data corresponds to a second frame rate that the playback device will use to display the image.

[0129] Method implementation 7 The method according to method implementation 6, wherein the second frame rate is a stereoscopic frame rate for displaying left-eye images and right-eye images to a user of the playback device to support a 3D viewing experience.

[0130] Method Embodiment 7A The method according to method embodiment 7, wherein the interpolated frame data conveys an interpolated frame that adds a supplement to the captured frame, thereby increasing the frame rate from the captured frame rate to a second frame rate.

[0131] Method implementation 8 The method according to method implementation 1 also includes: performing (910) motion analysis on blocks of capture frames to detect motion and identify moving segments; for at least a first segment, selecting (922) a frame rate based on the amount of motion of the first segment from a current capture frame and one or more other capture frames (for example, at least the next capture frame); and wherein performing interpolation (924) to generate interpolated frame data includes interpolating the segments between capture frames to generate interpolated frame information.

[0132] Method implementation 9 is a method according to method implementation 8, wherein the at least one fragment is an image fragment of a moving object.

[0133] Method Embodiment 9A The method according to Method Embodiment 8, wherein the moving object is a ball.

[0134] Method implementation 10 is a method according to method implementation 8, wherein interpolating the segment includes interpolating a version of the first segment to support the selected frame rate.

[0135] Method implementation 11 According to the method described in method implementation 8, it also includes: delivering (926) the captured frame to the playback device as part of the base video layer; and delivering (928) information corresponding to one or more interpolated fragments corresponding to the interpolated frame time as part of the additional video information.

[0136] Method Embodiment 11A The method of Method Embodiment 8, wherein the base video layer includes 3D video information in the form of a plurality of left-eye capture frames and right-eye capture frames.

[0137] Method Embodiment 11B The method of method embodiment 11A, wherein delivering (926) the captured frame data includes delivering a plurality of captured frames in encoded form in the base video layer, the video base layer having a first frame rate.

[0138] Method Embodiment 11C The method of Method Embodiment 11B, wherein the 3D video information includes left eye and right eye information corresponding to each frame time of a base video layer.

[0139] First numbered list of system implementation plans

[0140] System implementation scheme 1 An image capture and processing system includes: at least a first camera (1302) for capturing (904) images at a first frame rate; and a content delivery system (700) including: a memory (712); and a processor (708) coupled to the memory, the processor being configured to control the content delivery system to: perform interpolation (924) on captured frames to generate interpolated frame data to support a second frame rate higher than the first frame rate; and pass (926) the captured frame data and the interpolated frame data to at least one playback device.

[0141] System Implementation 2 The image capture and processing system according to System Implementation 1, wherein the first frame rate is the highest frame rate supported by the camera used to capture images when the camera operates at a maximum image capture resolution.

[0142] System implementation 3 is an image capture and processing system according to system implementation 1, wherein capturing images by the first camera at a first frame rate is performed by the first camera of the camera pair.

[0143] System Implementation 4 An image capture and processing system according to System Implementation 3, wherein the first camera supports a higher frame rate than the first frame rate when capturing images at a lower resolution.

[0144] System Implementation 5 The image capture and processing system according to System Implementation 1, wherein as part of delivering captured frame data, the processor controls the content delivery system to: deliver captured frames at a first data rate corresponding to the image capture rate.

[0145] System Implementation 6 The image capture and processing system of System Implementation 5, wherein the combination of captured frame data and interpolated frame data corresponds to a second frame rate used by the playback device to display images.

[0146] System Implementation 7 An image capture and processing system according to System Implementation 6, wherein the second frame rate is a stereoscopic frame rate for displaying left eye images and right eye images to a user of a playback device to support a 3D viewing experience.

[0147] System Implementation 7A An image capture and processing system according to System Implementation 7, wherein the interpolated frame data transmits an interpolated frame that adds a supplement to the captured frame, thereby increasing the frame rate from the captured frame rate to a second frame rate.

[0148] System implementation 8 The image capture and processing system according to system implementation 1, wherein the processor (708) is further configured to control the content delivery system: perform (910) motion analysis on blocks of captured frames to detect motion and identify moving segments; for at least a first segment, select (922) a frame rate based on an amount of motion of the first segment from a current captured frame and one or more other captured frames (e.g., at least a next captured frame); and wherein performing interpolation (924) to generate interpolated frame data includes interpolating segments between captured frames to generate interpolated frame information.

[0149] System implementation 9 is an image capture and processing system according to system implementation 8, wherein at least one of the fragments is an image fragment of a moving object.

[0150] System Implementation 9A comprises an image capture and processing system according to System Implementation 8, wherein the moving object is a ball.

[0151] System implementation 10 comprises an image capture and processing system according to system implementation 8, wherein the interpolated segment includes interpolating a version of the first segment to support the selected frame rate.

[0152] System Implementation 11 The image capture and processing system of system implementation 8, wherein the processor (708) is further configured to control the content delivery system as part of delivering (926) captured frame data and interpolated frame data to at least one playback device: delivering (926) the captured frame to the playback device as part of a base video layer; and delivering (928) information corresponding to one or more interpolated segments corresponding to the interpolated frame time as part of the additional video information.

[0153] System Embodiment 11A The image capture and processing system according to System Embodiment 8, wherein the base video layer includes 3D video information in the form of multiple left-eye capture frames and right-eye capture frames.

[0154] System implementation 11B is an image capture and processing system according to system implementation 11A, wherein delivering (926) captured frame data includes delivering multiple captured frames in encoded form in the base video layer, and the video base layer has a first frame rate.

[0155] System Embodiment 11C An image capture and processing system according to System Embodiment 11B, wherein the 3D video information includes left eye information and right eye information corresponding to each frame time of a base video layer.

[0156] Computer readable first numbered list

[0157] Media Implementation Plan

[0158] Computer-readable medium implementation scheme 1 A non-transitory computer-readable medium comprising computer-executable instructions, which, when executed by a processor (706) of a content delivery system (700), causes the processor to control the content delivery system (700) to: access an image (751) stored in a memory (712) of the content delivery system (700), the image having been captured by a camera (1302) at a first frame rate; perform interpolation (924) to generate interpolated frame data to support a second frame rate higher than the first frame rate; and transmit (926) the captured frame data and the interpolated frame data to at least one playback device.

[0159] Second list of numbering method implementation schemes

[0160] Method implementation 1. A method of operating a playback system (300), the method comprising: receiving (1603) captured frame data and interpolated frame data; recovering (1607) a captured frame from the received captured frame data; generating (1610) one or more interpolated frames from the received interpolated frame data; rendering (1618) a video sequence including one or more captured frames and at least one interpolated frame; and

[0161] One or more rendered images are output (1620) to a display device (702).

[0162] Method implementation 2. A method according to method implementation 1, wherein the captured frame corresponds to a first frame rate; and the generated video sequence including the captured frame and one or more interpolated frames has a second frame rate higher than the first frame rate.

[0163] Method implementation 3. A method according to method implementation 2, wherein outputting (1620) one or more rendered images to a display device (702) includes outputting a portion of the rendered image at a second frame rate.

[0164] Method implementation 4. A method according to implementation 3, wherein the second frame rate matches a display refresh rate supported by the display device (702); and wherein the display device is a head-mounted display device.

[0165] Method implementation 5. A method according to method implementation 1, wherein the second frame rate is a frame rate that is higher than an image capture rate used to capture images received by the playback system.

[0166] Method Implementation 6. A method according to Implementation 5, wherein the playback system does not interpolate between received frames to increase the frame output rate beyond the received frame rate.

[0167] Method implementation 7. A method according to method implementation 2, wherein generating (1610) one or more interpolated frames from the received interpolated frame data includes: using (1612) a motion vector to generate a portion of a first interpolated frame from a first captured frame; and using (1614) content from one or more previous frames to fill in areas of the first interpolated frame for which interpolated fragment data is not provided.

[0168] Second numbered list of system implementation plans

[0169] System implementation 1. A playback system (300), the method comprising: a display device (805); a network interface (810), which receives (1603) capture frame data and interpolated frame data; and a processor (808), which is configured to control the playback system to: recover (1607) a capture frame from the received capture frame data; generate (1610) one or more interpolated frames from the received interpolated frame data; render (1618) a video sequence including one or more capture frames and at least one interpolated frame; and output (1620) one or more rendered images to the display device (702).

[0170] System Implementation 2. A playback system (300) according to system implementation 1, wherein the captured frames correspond to a first frame rate; and wherein the generated video sequence includes captured frames and one or more interpolated frames and has a second frame rate higher than the first frame rate.

[0171] System Implementation 3. A playback system (300) according to System Implementation 2, wherein outputting (1620) one or more rendered images to a display device (702) includes outputting a portion of the rendered images at a second frame rate.

[0172] System Implementation 4. A playback system (300) according to System Implementation 3, wherein the second frame rate matches a display refresh rate supported by the display device (702); and wherein the display device is a head-mounted display device.

[0173] System Implementation 5. A playback system (300) according to system implementation 1, wherein the second frame rate is a frame rate that is higher than an image capture rate used to capture images received by the playback system.

[0174] System Embodiment 6. The playback system (300) of System Embodiment 5, wherein the playback system does not interpolate between received frames to increase the frame output rate beyond the received frame rate.

[0175] System Embodiment 7. The playback system (300) of System Embodiment 2, wherein as part of generating (1610) one or more interpolated frames from the received interpolated frame data, the process is configured to: generate a portion of a first interpolated frame from a first captured frame using (1612) motion vectors; and fill in areas of the first interpolated frame for which interpolated segment data is not provided using (1614) content from one or more previous frames.

[0176] Second number list of computers

[0177] Readable medium implementation

[0178] Computer-readable medium implementation scheme 1. A non-transitory computer-readable medium, comprising computer-executable instructions (814), which, when executed by a processor (808) of a content playback system (300), causes the processor to control a content delivery system (700) to: receive (1603) capture frame data and interpolated frame data; recover (1607) a capture frame from the received capture frame data; generate (1610) one or more interpolated frames from the received interpolated frame data; render (1618) a video sequence comprising one or more capture frames and at least one interpolated frame; and output (1620) one or more rendered images to a display device (702).

[0179] One goal of an immersive VR experience is to deliver a realistic experience. To achieve this goal, the content consumed by users should adhere to very high-quality specifications. In particular, high-action content such as sports is extremely sensitive to motion fidelity. Any issues regarding motion capture, processing, transmission, playback, and rendering must be addressed in an appropriate manner. Various factors such as content capture systems, encoding and playback, and display refresh frame rates affect motion fidelity. In particular, under certain conditions, when the capture frame rate and rendering frame rate are lower than the device / HMD display refresh rate, high-resolution fast-motion immersive content results in motion blur, strobing, and short-cut artifacts.

[0180] Various embodiments are directed to methods and / or apparatus that take into account one, more, or all of the above factors and support dynamic processing of content to achieve good fidelity of content delivered and played back to a user of a virtual reality device. Various features are directed to motion segmentation, depth analysis, and estimation performed on captured images, and taking into account the amount of motion in each time segment, e.g., a set of 3D image content corresponding to a time segment, and assigning a frame rate to provide sensing of smooth motion given the amount of motion of an object detected in a captured frame, the amount of motion corresponding to a time segment including multiple frames (e.g., 2, 15, 30 or more sequentially captured frames).

[0181] In various embodiments, the use of fisheye lens based image capture is contemplated. The motion trajectory of an object or portion of content is considered relative to the curvature of the fisheye lens. The mapping function considers the rate of change of the real world display size and the movement perceived by the fisheye lens in the 3D spherical domain for capturing, for example, an image of a scene (such as a scene of a sporting event in progress).

[0182] In a 3D specific implementation, the method processes and operates on stereo images (e.g., paired left and right eye images). Taking into account the stereo properties and image capture of both the left and right eyes, selecting a desired frame rate different from the capture rate based on motion and stereo related issues can result in reduced computation, bandwidth, and processing compared to a system that captures, encodes, and transmits images at a fixed rate regardless of motion or 3D issues. In various embodiments, image capture, processing, encoding, and communication are performed in real time, for example, while the motion or other event that is capturing the image is still ongoing.

[0183] In various embodiments, images are captured at a rate lower than the rate at which the playback device generates and displays images to the user of the playback device. The playback device can be and sometimes is a virtual reality system including a processor and a head-mounted display, which in some embodiments can display different image content to the left and right eyes of the user. In at least some embodiments, the captured image content is analyzed and a determination is made about the frame rate used to deliver the frame to the playback device. In many cases, the determined frame rate intended to give the user a sense of smooth motion is typically higher than the image capture rate. Interpolation of the captured image is used to generate a complete set of frames corresponding to the determined frame rate. The rate at which a specific segment (sometimes corresponding to a moving object) is delivered to the playback device is selected based on the amount of motion exhibited by the segment / object and / or the portion of the fisheye lens of the camera device used to capture the segment. In various embodiments, segments that are subject to high motion / change rates are encoded and delivered at a different rate from relatively static segments.

[0184] In various embodiments, images are captured, for example, at a high resolution but at an image capture rate that is lower than the frame playback rate achieved by a virtual reality playback device. Frames are interpolated between captured frames to account for the motion of objects. The interpolation process is able to use the entire set of captured image content because the interpolation process is performed prior to lossy encoding and transmission to the playback device. Therefore, the system that processes the captured images has more information available to it than a playback device that typically receives compressed image data and images that have been degraded by compression. It should be understood that the system that generates content to be delivered to the playback device may also have one or more processors that can perform interpolation operations prior to encoding. The encoding may take into account the differences between the encoded set of frames to be delivered to the playback device and the movement or change from one frame to the next.

[0185] In some systems using the present invention, a fisheye lens is used to capture image content to be delivered to a playback device. Fisheye lenses tend to distort the image being captured. In some, but not necessarily all, embodiments, for encoding purposes, for example, when deciding which portions of a captured frame or a frame generated by interpolation should be delivered to a playback device, the position of the motion relative to which portion of the fisheye lens is used to capture image content corresponding to the motion is considered.

[0186] Fig.16 A playback method 1600 is shown, which can be, and in some embodiments is, performed by Figure 1 In some embodiments, when the system 300 is used to Figure 1 In the system, the method is Figure 3 The playback system 300 is shown implemented.

[0187] The method 1600 begins at start step 1602, where the playback system 300 is powered on and the processor 808 begins controlling the playback system 300 under the control of the routine 814 stored in the memory 800, which controls the playback system to implement Fig.16 method.

[0188] Operation proceeds from start step 1602 to receive frame information step 1603, where a capture frame and interpolated frame data, such as an interpolated frame, are received. In some embodiments, step 1603 includes step 1604 of receiving a base layer frame (e.g., an encoded capture frame corresponding to a first frame rate), and step 1606 of receiving interpolated frame data (e.g., data representing the interpolated frame). Capture frame and interpolated frame data have been previously discussed with respect to content delivery system 104, which provides such data to the playback system, and thus will not be described in detail.

[0189] In the event that capture frame data and interpolated frame data have been received, in the event that the received capture frame is encoded, the operation proceeds from step 1603 to step 1608. In step 1608, the received capture frame (e.g., base layer frame) is decoded to produce an unencoded capture frame before the operation proceeds to step 1610. If the capture frame is received in an unencoded form in step 1603, the decoding step 1608 is skipped and the operation proceeds directly from step 1603 to step 1610.

[0190] In step 1610, one or more interpolated frames are generated from the received interpolated frame data. In some embodiments, step 1610 includes steps 1612 and 1614. In step 1612, motion vectors and / or intra-frame coding segments are used to generate, for example, portions of interpolated frames that are different from one or more previous frames or subsequent frames in position or content. In step 1614, one or more regions of interpolated frames for which interpolated segment data is not provided are filled using content from one or more previous frames or subsequent frames. At the end of step 1610, the playback device has both captured frames and interpolated frames, which can be used to generate a video frame sequence with a second frame rate higher than the image capture rate. The second frame rate can and sometimes is equal to the display rate and / or reference rate achieved by the playback device. Therefore, by using interpolated frames, the refresh rate that can be supported in the playback device can be higher than the image capture rate, while avoiding the need for the playback device to interpolate frames to support a higher playback rate. Therefore, in some embodiments, the playback device supports playback or frame refresh rates higher than the image capture rate without performing interpolation between frames to support the playback rate. This is because the content delivery system performs any required frame interpolation to generate interpolated frames before the frames are provided to the playback device.

[0191] Operation proceeds from step 1610 to step 1610, where a sequence of frames is generated by, for example, spacing interpolated frames from captured frames in an intended display sequence. In some embodiments supporting stereoscopic images, separate left-eye image sequences and right-eye image sequences are generated in step 1616, each image sequence having a frame rate corresponding to a second frame rate (i.e., a supported frame rate).

[0192] Operation proceeds from step 1616 to step 1618, wherein images of the one or more video sequences generated in step 1616 are rendered, such as by rendering the left eye image and the right eye image by applying the images as textures to a mesh model.

[0193] Operation proceeds from step 1618 to step 1620, where at least a portion of the rendered frame is output to a display. The size of the portion of the rendered frame (e.g., image) output depends on the size of the display device used and / or the user's field of view. As part of step 1620, in a stereoscopic implementation, different images are displayed to the user's left and right eyes, for example, using different portions of a head-mounted display.

[0194] Operations are shown proceeding from step 1620 back to step 1603 to indicate that the process may be repeated over time, with the playback device receiving, processing, and displaying sets of images corresponding to different portions of the video sequence at different points in time.

[0195] Although in some embodiments, the display outputs images at a second frame rate, the method does not preclude the playback device from performing further interpolation to further increase the refresh rate supported by the playback device. However, significantly, at least some frame interpolation is performed outside the playback device to support a higher frame rate than the capture frame rate, thereby eliminating the need for the playback device to perform interpolation to achieve a second frame rate that is higher than the frame capture rate. Because the frames are captured and delivered at a high level of detail, a lower cost camera than might be required in the case of a camera that can support the second frame rate can be used to achieve an actual 3D experience, where the images are captured using the same level of detail.

[0196] In some embodiments, motion segmentation is used to segment captured images, such as frames, for processing by a process that, in some embodiments, includes encoding for delivery purposes. In some embodiments, during the encoding process or as a separate pre-encoding process, the image is segmented or processed on a block basis, where an image including a group of blocks (e.g., rectangular portions of a frame) is analyzed to detect objects that move (i.e., are in motion) from one frame to the next. In some embodiments, the system analyzes the temporal motion of multiple blocks, for example in a variable-sized look-ahead buffer that includes captured image content (e.g., blocks of previously captured frames). The data is organized to form, i.e., identify meaningful segments of motion in the scene, e.g., a group of blocks that move as a unit from one frame to another, as perceived from observing multiple frames. The group of blocks may correspond to, and sometimes will correspond to, a moving object such as a ball or other object that can move, while other objects in the scene area, such as a playing field for a sporting event, remain fixed. In some embodiments, identified image segments that are subject to motion are pruned based on spatial texture information, taking into account the angular velocity of objects when captured in fisheye lens-based video, i.e., some image segments that are subject to motion are excluded from further consideration to obtain final motion segments to be considered for additional processing, and then the group of segments considered for further processing are sorted in some but not all embodiments in order of relative complexity, where the complexity can and sometimes does depend on the level of detail and / or the number of different colors in each segment, where segments with higher levels of detail and more colors or brightness levels are considered to be more complex than segments with less detail, fewer colors, and / or a lower number of brightness levels.

[0197] In some embodiments, depth analysis is used to determine depth, for example, the distance of the object in the identified segment that is subject to motion. Therefore, in at least some embodiments that support 3D content, for example, when different left-eye images and right-eye images are presented to the user during playback, the depth of the object and therefore the depth associated with the image segment representing such an object are considered. Consider the motion in 2D left-eye and right-eye video segments (e.g., images), and consider its impact on the perception of objects in 3D (e.g., stereoscopic domain) when deciding the frequency at which frames should be encoded and delivered to support the smooth motion of 3D objects. Considering the 3D effect and depth based on the distance from the camera (e.g., left-eye camera and right-eye camera), the left-eye image and the right-eye image are captured respectively, and in some embodiments, it is promoted and used to determine the appropriate motion interpolation rate based on the distance of the segment from the camera origin. Therefore, in many cases, the object distance may and sometimes does affect the rate at which the object will be interpolated at a higher rate of mutual relationship, thereby providing a better 3D effect. The interpolation rate associated with the segment corresponding to a single object can be and sometimes is adjusted based on the display refresh rate of one or more playback devices supported or used. It should be understood that because different objects are at different distances, different interpolation rates can and sometimes are selected for different objects due to differences in depth. In some embodiments, objects that are closer to the camera and therefore will be perceived as closer to the user during playback are given a higher priority for interpolation than objects that are farther away (e.g., objects at greater depth). This is in part because objects that are close tend to be more distinct, and users expect such objects to have a higher level of detail than objects that are farther away.

[0198] In some embodiments, motion estimation and representation are performed on a per-segment basis, where a segment may correspond to an object or set of objects of an image that may and sometimes does move in position over time, for example, where a segment changes position from frame to frame. The change in position may be and sometimes is due to the movement of an object such as a ball that moves corresponding to the segment, while the camera and other objects remain fixed. In some embodiments, the frame rate of each segment is calculated based on a single segment based on spatiotemporal data and depth data associated with the segment. In some embodiments, motion interpolation is performed separately for each segment, and motion vectors and optimal block information for each segment are calculated. Motion vectors and other information (e.g., fill information for portions of an image that cannot be accurately represented by a motion vector) are transmitted as side information and / or metadata and may be embedded in the stream along with information representing the captured frame. Therefore, portions of frames determined to correspond to some segments may be refreshed more frequently by the transmitted data than other portions of the captured frame. If the content is delivered to multiple sets / versions of display devices such as head-mounted displays (HMDs), there may be multiple versions of motion segment side information based on device specifications.

[0199] It should be understood that various features relate to image capture, processing, and transmitting content in a manner that allows the limited data transmission capacity of a communication channel to be effectively used to a user playback device. By transmitting information corresponding to some fragments rather than entire frames in at least some frame times, and wherein the fragments are selected based on which part of the motion and / or fisheye lens is used to capture the portion of the image corresponding to the fragment, the updating of the fragments can occur at a rate higher than the rate of content updating of the entire frame, wherein the updating of the fragments promotes a satisfactory 3D playback experience. During playback, the received fragments corresponding to the motion can be decoded and displayed, thereby allowing the detected motion portion of the scene area to be updated more frequently at the playback device than other parts of the environment corresponding to the captured image of the scene area. The entire frame at a frame rate lower than the desired playback frame rate can and sometimes is transmitted as part of the video base stream. In at least some such embodiments, information corresponding to one or more fragments is transmitted as enhanced information, which provides information corresponding to one or more fragments of a time period (e.g., frame time) occurring between frames of the base video stream. The enhancement information can be, and is sometimes, combined with the base stream information at a playback device to generate a video output stream having left-eye and right-eye images that has a higher frame rate than the base video stream.

[0200] Client device playback is supported by various methods and apparatus. The playback process on the client device includes an initial step in which an underlying video stream (e.g., one or more captured frames) is decoded, followed by extraction of side information conveyed using the underlying stream. The side information allows some segments of the displayed image to be updated at a faster rate than other segments of the displayed image. Interpolated blocks in frames with side information are reconstructed from one or more reference pictures (e.g., captured images) using motion compensation so that the playback frame rate matches the supported display refresh rate of the playback device, which in some but not necessarily all embodiments exceeds the frame rate of the captured image.

[0201] In some embodiments, when generating an interpolated frame, in order to fill the gaps of pixels where gaps have been created due to motion and no motion information is provided to fill the gaps from another portion of the image, the playback device fills the gaps with content from a juxtaposed block that is positionally attached to where the gap occurs. In the absence of an indication of motion, the motion-free portion of the interpolated frame is generated by acquiring content from a previous or subsequent frame, where the content is taken from locations in the previous or subsequent frame corresponding to where one or more motion-free blocks occur. For example, content from a block of another frame having the same x, y coordinates as a block that does not specify motion estimation content is in some cases acquired from the temporally closest reference frame to the generated interpolated frame. In the case of motion-free blocks with drastic brightness and / or color changes, a more optimized bidirectional prediction / average can be and is sometimes explicitly signaled in the information passed to the playback device, in which case the brightness and / or chrominance may depend on the content of multiple frames.

[0202] Movement :

[0203] In some embodiments, motion compensated interpolation involves motion estimation to produce satisfactory or optimal motion vectors for each variable-sized block shape of an object or segment. For pixels located outside the selected motion segment in the frame, a co-located block from a reference picture is used by the playback device at the interpolated frame time. In some embodiments, the rule for specifying a reference for a "no motion" pixel / segment is implicitly calculated based on the nearest temporal neighborhood. In the case where bidirectional prediction / averaging is preferred or optimal for "no motion" pixels, it is sent as part of the interpolated frame data, which is transmitted in the form of side information in addition to the encoded frame of the base video layer that typically includes the captured frame. In some embodiments, the motion vector of the selected motion segment is explicitly specified in the transmitted side information in addition to the base layer video data and the source block coordinates to form a complete motion segment.

[0204] Various feature structures can be used to provide a general solution for motion fidelity that can be scaled and customized to any number of unique HMDs and their variants. It can also be extended to AR / MR / XR applications.

[0205] The features of this approach fit naturally with any field-of-view based streaming and display approach, where only the relevant motion segments and associated side information can be streamed along with the FoV.

[0206] Although the steps are shown in an exemplary order, it should be understood that in many cases, the order of the steps can be changed without adversely affecting the operation. Therefore, unless the exemplary step order is required for proper operation, the step order is considered to be exemplary rather than limiting.

[0207] Some embodiments relate to a non-transitory computer-readable medium that embodies a set of software instructions, such as computer-executable instructions, for controlling a computer or other device to encode and compress stereoscopic video. Other embodiments relate to a computer-readable medium that embodies a set of software instructions, such as computer-executable instructions, for controlling a computer or other device to decode and decompress the video on the player end. Although encoding and compression are mentioned as possible separate operations, it should be understood that encoding can be used to perform compression, and therefore encoding can include compression in some cases. Similarly, decoding can involve decompression.

[0208] The techniques of various embodiments may be implemented using software, hardware, and / or a combination of software and hardware. Various embodiments relate to apparatus, e.g., image data processing systems. Various embodiments also relate to methods, e.g., methods of processing image data. Various embodiments also relate to non-transitory machines such as computer-readable media, e.g., ROM, RAM, CD, hard disk, etc., which include machine-readable instructions for controlling a machine to implement one or more steps of a method.

[0209] Various features of the present invention are implemented using modules. Such modules can be implemented as software modules, and in some embodiments are implemented as software modules. In other embodiments, modules are implemented in hardware. In other embodiments, modules are implemented using a combination of software and hardware. In some embodiments, modules are implemented as separate circuits, wherein each module is implemented as a circuit for executing the function corresponding to the module. Various embodiments are envisioned, including some embodiments in which different modules are implemented in different ways, such as some in hardware, some in software, and some using a combination of hardware and software. It should also be noted that some of the routines and / or subroutines, or steps performed by such routines, can be implemented in dedicated hardware, which is in contrast to software executed on a general-purpose processor. Such embodiments are still within the scope of the present invention. Many of the above methods or method steps can be implemented using machine executable instructions (such as software), which are included in machine-readable media (such as storage devices, such as RAM, floppy disks, etc.) to control machines (e.g., general-purpose computers with or without additional hardware) to implement all or part of the above methods. Thus, the present invention relates, among other things, to a machine-readable medium comprising machine-executable instructions for causing a machine (eg, a processor and associated hardware) to perform one or more steps of one or more of the methods described above.

[0210] Numerous additional variations of the methods and apparatus of the various embodiments described above will be apparent to those skilled in the art from the above description. Such variations should be considered within the scope.

Claims

1. A method, comprising: Acquire a first stereoscopic frame pair from a stereoscopic frame set having a first frame rate; obtaining motion vector data indicating motion of a portion of the first stereoscopic frame pair; obtaining depth data for the portion of the first stereoscopic frame pair; generating one or more additional frame pairs using the first stereoscopic frame pair, the motion vector data, and the depth data; as well as A video sequence is generated including the set of frames and the one or more additional frame pairs, wherein the video sequence is associated with a second frame rate greater than the first frame rate. 2 . The method of claim 1 , wherein the video sequence comprises the first stereoscopic frame pair followed by the one or more additional frame pairs.

3. The method of claim 1 , wherein generating the one or more additional frame pairs comprises: The area of ​​the one or more additional frame pairs is populated using content from one or more previous frames of the stereoscopic frame set. The method of claim 1 , wherein the motion vector data comprises one or more 3D motion vectors.

5. The method of claim 1, wherein the second frame rate corresponds to a display refresh rate supported by a display device. The method of claim 5 , wherein the display device is included in a head mounted device. The method of claim 6 , wherein the display device supports a 3D viewing experience.

8. A non-transitory computer readable medium comprising computer readable code executable by one or more processors, the computer readable code being configured to: Acquire a first stereoscopic frame pair from a stereoscopic frame set having a first frame rate; obtaining motion vector data indicating motion of a portion of the first stereoscopic frame pair; obtaining depth data for the portion of the first stereoscopic frame pair; generating one or more additional frame pairs using the first stereoscopic frame pair, the motion vector data, and the depth data; as well as A video sequence is generated including the set of frames and the one or more additional frame pairs, wherein the video sequence is associated with a second frame rate greater than the first frame rate.

9. The non-transitory computer-readable medium of claim 8, wherein the video sequence comprises the first stereoscopic frame pair followed by the one or more additional frame pairs.

10. The non-transitory computer readable medium of claim 8, wherein the computer readable code for generating the one or more additional frame pairs comprises computer readable code for: The area of ​​the one or more additional frame pairs is populated using content from one or more previous frames of the stereoscopic frame set.

11. The non-transitory computer readable medium of claim 8, wherein the motion vector data comprises one or more 3D motion vectors.

12. The non-transitory computer readable medium of claim 8, wherein the second frame rate corresponds to a display refresh rate supported by a display device.

13. The non-transitory computer-readable medium of claim 12, wherein the display device is included in a head mounted device.

14. The non-transitory computer readable medium of claim 13, wherein the display device supports a 3D viewing experience.

15. A system comprising: one or more processors; as well as One or more computer readable media comprising computer readable code executable by the one or more processors, the computer readable code for: Acquire a first stereoscopic frame pair from a stereoscopic frame set having a first frame rate; obtaining motion vector data indicating motion of a portion of the first stereoscopic frame pair; obtaining depth data for the portion of the first stereoscopic frame pair; generating one or more additional frame pairs using the first stereoscopic frame pair, the motion vector data, and the depth data; as well as A video sequence is generated including the set of frames and the one or more additional frame pairs, wherein the video sequence is associated with a second frame rate greater than the first frame rate.

16. The system of claim 15, wherein the video sequence comprises the first stereoscopic frame pair followed by the one or more additional frame pairs.

17. The system of claim 15, wherein the computer readable code for generating the one or more additional frame pairs comprises computer readable code for: The area of ​​the one or more additional frame pairs is populated using content from one or more previous frames of the stereoscopic frame set.

18. The system of claim 15, wherein the motion vector data comprises one or more 3D motion vectors.

19. The system of claim 15, wherein the second frame rate corresponds to a display refresh rate supported by a display device.

20. The system of claim 19, wherein the display device is included in a head mounted device.