Multi-tier concave centrally streaming

Abstract

A non-transitory computer readable medium may include instructions that, when executed by at least one computer processor, cause the at least one computer processor to obtain an initial image frame of a first resolution. The initial frame may include a first number of pixels. The at least one computer processor may be caused to generate a reduced image frame of the second resolution from the initial image frame. The at least one computer processor may be caused to obtain one or more subframes of the initial image frame. Each of the subframes may include a portion of the number of pixels. The at least one computer processor may be caused to transmit the reduced image frame and at least one of the subframes to a playback device. The reduced image frames and subframes can be combined to form a target frame that includes subframes of different resolutions.

Description

Background Art

[0001] Foveal streaming can be used in rendering operations including two-dimensional (2D) media and / or three-dimensional (3D) media. In applications involving foveal streaming, a foveal near-eye display can be used to present high-resolution images along the direction of the user's eye gaze as well as low-resolution peripheral images. Foveal rendering is a rendering technique that uses an eye tracker integrated with a foveal near-eye display to reduce the rendering workload by greatly reducing the image quality in peripheral vision (outside the area gazed by the fovea). Existing foveal streaming techniques cannot consistently maintain high resolution in an area of ​​interest that follows the user's eye gaze direction. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Figure 1 An example of a content delivery system providing multi-layer foveated streaming is shown in accordance with one or more embodiments.

[0003] Figure 2 An example of a playback system for rendering multi-layer foveated streaming according to one or more embodiments is shown.

[0004] Figure 3 An example technique for providing and rendering portions of a multi-layer foveated stream in accordance with one or more embodiments is shown in flowchart form.

[0005] Figure 4 A technique for providing multi-layer foveated streaming for playback according to one or more embodiments is shown in flow chart form.

[0006] Figure 5 A technique for creating a combined image frame based on multi-layer foveal flow in accordance with one or more embodiments is shown in flow chart form.

[0007] Figure 6 A simplified system diagram of a multi-layer streaming system according to one or more embodiments is shown in block diagram form.

[0008] Figure 7 A simplified system diagram of a multifunction electronic device according to one or more embodiments is shown in block diagram form. DETAILED DESCRIPTION

[0009] The present disclosure relates to systems, methods, and computer-readable media configured to combine multiple layers of foveated media in streaming applications. In some embodiments, the multi-layer foveated streaming technology disclosed herein consistently maintains high quality in an area of ​​interest that follows a user's head posture or a user's eye gaze direction. In one or more embodiments, the multi-layer foveated streaming technology can be performed by at least one content delivery system and at least one playback system. The content delivery system is configured to provide encoded subframes to the playback system, and the encoded subframes can be combined to consistently generate high-quality (e.g., high-resolution) images. The content delivery system can control the size of the encoded subframes based on current bandwidth information for communication between the content delivery system and the playback system or based on the processing power of the playback system (e.g., based on the ability of the decoding circuit at the playback system to decode the encoded subframes, which can vary according to the size and resolution of the encoded subframes).

[0010] In some embodiments, the content delivery system can obtain an initial image frame for the foveal stream. The content delivery system can generate a coded comparator image subframe and a coded reduced image frame based on the initial image frame. The coded comparator image subframe can be a subframe based on the initial image frame, and the subframe based on the initial image frame includes at least one subframe representing the difference between the original subframe of the initial image frame and the rescaled subframe. The coded reduced image frame is a reduced version of the initial image frame. The content delivery system can be configured to provide the coded comparator image subframe and the coded reduced image frame to a playback device for rendering.

[0011] In some embodiments, the playback system obtains the encoded comparator image subframes and the encoded reduced image frame from the content delivery system. In addition, the playback system may receive head posture / attention information corresponding to the user. The playback system may decode at least one of the received subframes corresponding to the fovea in the initial image frame. In this way, the initial image frame may be a fovea image frame. Based on the head posture / attention information, the fovea may be identified as including a region of interest. Once a subframe with a region of interest is selected, the subframe and the encoded reduced image frame may be decoded and combined into a combined image frame for playback.

[0012] In the above-mentioned multi-layer fovea streaming technology, the function of the content delivery system emulation playback system is to preemptively identify the possible quality loss of the region of interest in the fovea stream. Based on the bandwidth availability or processing power of the playback system, coded image frames and subframes of different sizes can be provided to the playback system. These sizes can be automatically selected to change the streaming time between the content delivery system and the playback system. As mentioned above, the subframe for the region of interest can be selected based on the head posture / attention information of the user, which allows the playback device to reduce power consumption by processing the region of interest with only reduced image frames to generate high-quality combined image frames for playback. In some embodiments, the playback system is configured to apply edge smoothing (edge ​​smoothing) to improve the quality across the combined image frame boundary. The playback system can also be configured to use high dynamic range (HDR) capabilities to render the combined image frame.

[0013] The content delivery system and the playback system can be configured according to a power target. The power target can be a threshold value of power consumption that the content delivery system and / or the playback system are configured to maintain. In this regard, the multi-layer foveated streaming technology can be implemented to meet the power target by reducing the power consumption in the content delivery system, the playback system, or both. The reduction in power consumption in these systems can include reducing the processing time and / or processing power in one or more operations. For example, in order to reduce the power consumption at the content delivery system, the content delivery system can increase the scaling of the reduced image frame to provide a relatively small encoded reduced image frame to the playback system.

[0014] Here, the terms "high" and "low" are relative values ​​corresponding to the quality (e.g., resolution allocation and / or dynamic range) of individual frames. For example, a frame may be considered to include a low resolution as long as the frame includes a resolution allocation that is lower than the resolution allocation of the current resolution allocation being encoded or the resolution allocation of other frames in the same media stream. In some embodiments, "low resolution" or "low frame rate" may indicate a relative resolution or frame rate relative to other frames in the combined media stream (which are considered "high resolution" or "high frame rate" frames).

[0015] In the following description, for the purpose of explanation, a lot of specific details are set forth in order to provide a thorough understanding of the disclosed concept. As part of this specification, some drawings of the present disclosure represent structures and devices in the form of block diagrams to avoid blurring the novel aspects of the disclosed embodiments. In this context, it should be understood that references to numbered drawing elements (e.g., 100) without associated identifiers refer to all instances of drawing elements with identifiers (e.g., 100a and 100b). In addition, as part of this specification, some drawings of the present disclosure can be provided in the form of flow charts. The blocks in any particular flow chart can be presented in a specific order. However, it should be understood that the specific flow of any flow chart is only used to illustrate an embodiment. In other embodiments, any of the various components depicted in the flow chart can be deleted, or the components can be executed in different orders, or even simultaneously. In addition, other embodiments may include additional steps that are not shown as a part of the flow chart. The language used in this disclosure is mainly selected for readability and guidance purposes, and it may not be selected to depict or limit the disclosed subject matter. Reference in this disclosure to "one embodiment" or "an embodiment" is meant to include a particular feature, structure, or characteristic described in connection with that embodiment in at least one embodiment, and multiple references to "one embodiment" or "an embodiment" should not be construed as necessarily all referring to the same or different embodiments.

[0016] It will be appreciated that in the development of any actual implementation (as in any development project), many decisions must be made to achieve the developer's specific goals (e.g., to meet system and business-related constraints), and that these goals will vary from one implementation to another. It will also be appreciated that such a development effort may be complex and time-consuming, but will nevertheless be a routine undertaking for those of ordinary skill in the art who benefit from the image capture disclosed herein.

[0017] For purposes of this disclosure, the term "camera system" refers to one or more lens assemblies and one or more sensor elements and other circuitry for capturing images. For purposes of this disclosure, a "camera" may include more than one camera system, such as a stereo camera system, a multi-camera system, or a camera system capable of sensing the depth of a captured scene.

[0018] The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical objects, such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0019] refer to Figure 1, the example diagram illustrates a content delivery system 100 according to one or more embodiments. The content delivery system 100 may obtain an initial image frame 130 for a foveal stream. The content delivery system 100 may generate an encoded comparator image subframe and an encoded reduced image frame based on the initial image frame 130. The encoded comparator image subframe may be a subframe representing the initial image frame 130, which includes at least one subframe representing the difference between the original subframes 131 to 139 and the rescaled subframes 161 to 169 of the initial image frame 130. The encoded reduced image frame is a reduced version of the initial image frame 130. The content delivery system 100 may be configured to provide the encoded comparator image subframe and the encoded reduced image frame to the playback device 200 for rendering. In some embodiments, providing the encoded comparator image subframe and the encoded reduced image frame to the playback device 200 at the same time prevents decoding of duplicate information before rendering. That is, the playback device 200 does not receive duplicate information in the two streams. Reference will be made to Figure 2 The playback system 200 is described in detail.

[0020] In some embodiments, the content delivery system 100 is shown as obtaining an initial image frame 130 having a plurality of sub-frames 131 to 139. The initial image frame 130 includes a width 110A and a height 120A. In one or more embodiments, each of the plurality of sub-frames 131 to 139 includes a plurality of pixels and / or pixel blocks. Figure 1 In the example of , the scaler 105A is used to reduce the initial image frame 130 to create a reduced image frame 140. The reduced image frame 140 may have a width 110B and a height 120B, which are reduced to the width 110A and the height 120A respectively by a reduction factor. The reduction factor may be the same or different for the width and the height. The scaler 105A may be configured to reduce the initial image frame 130 by a predetermined reduction factor or by a dynamic reduction factor.

[0021] The reduced image frame 140 may be encoded using the encoder 115A. The encoder 115 may be configured to compress the image content of the reduced image frame 140 into an encoded reduced image frame. The content delivery system 100 may provide the encoded reduced image frame directly to the playback system 200. In some embodiments, the reduced image frame 140 is reduced by a common factor "N" in each direction to reduce the gradient difference between the initial image frame 130 and the enlarged image frame 160.

[0022] In one or more embodiments, the content delivery system 100 may also provide the encoded reduced image frame to its own decoder 150. The decoder 150 may be configured to decode the encoded reduced image frame in a manner that mimics the decoding capabilities of the playback system 200. Once the encoded reduced image frame is decoded, the content delivery system 100 may use the scaler 105B to generate an enlarged image frame 160. The enlarged image frame 160 may have a width 110A and a height 120A of the initial image frame 130. Therefore, the enlarged image frame 160 may include a plurality of subframes 161 to 169 that are very similar to the subframes 131 to 139 of the initial image frame 130.

[0023] In some embodiments, at the content comparator 170, the fovea region in the subframes 131 to 139 of the initial image frame 130 includes a target region (e.g., the fovea), which is compared with the corresponding region in the subframes 161 to 169 of the enlarged image frame 160 to generate one or more comparator image subframes. In this case, the comparator image subframe is a subframe that represents the difference between the target region and the corresponding region. In other embodiments, at the content comparator 170, the subframes 131 to 139 of the initial image frame 130 are compared with the subframes 161 to 169 of the enlarged image frame 160 to generate the comparator image subframe. The comparator image subframe is a subframe that represents the difference between the two groups of subframes. In other embodiments, the encoded comparator image subframe may include all subframes that represent the difference between the subframes 131 to 139 and the subframes 161 to 169. In all embodiments, the comparator image subframe can be provided to the encoder 115B. Once the comparator image sub-frames are encoded, these encoded comparator image sub-frames may be provided to the playback system 200 as encoded comparator image sub-frames.

[0024] The initial image frame 130 may be a frame among a plurality of frames in the foveated stream. In this way, the content delivery system 100 is configured to simultaneously provide (e.g., send, transmit) the encoded comparator image subframe and the encoded reduced image frame to the playback device 200, or include information associating the encoded comparator image subframe with the corresponding encoded reduced frame.

[0025] The content delivery system 100 may be configured to encode multiple frames and subframes to meet predefined or dynamic power consumption targets. Changes in power consumption targets may directly affect the scaling provided by the sealer 105A. For example, when the power consumption target is higher, the size of the reduced image frame 140 may be larger.

[0026] In some embodiments, the sealer 105A and the sealer 105B may be the same sealer 105 configured to perform two distinct and different scaling operations. In addition, the encoder 115A and the encoder 115B may be the same encoder 115 configured to perform multiple encoding operations based on the received input. For example, the encoder 115 may perform one type of encoding operation based on whether the input is a portion of a separate sub-frame or an entire reduced image frame 140. The encoder 115 and the decoder 150 may be configured to perform encoding / decoding according to the High Efficiency Video Coding (HEVC) standard to produce a compressed video bitstream (e.g., a sequence of decoded frames).

[0027] refer to Figure 2 , an example diagram shows a playback system 200 according to one or more embodiments. The playback system 200 can be obtained from reference Figure 1 The described content delivery system 100 obtains the encoded comparator image subframes and the encoded scaled image frame. In addition, the playback system 200 may receive head pose / gaze information 210 corresponding to the user. The playback system 200 may decode at least one of the received subframes that corresponds to the fovea in the initial image frame 130. A region of interest (ROI) 230 may be identified based on the head pose / gaze information 210 and the fovea. Once a subframe having the ROI 230 is selected, the subframe and the encoded scaled image frame may be decoded and combined into a high quality combined image frame 260 for playback.

[0028] exist Figure 2 , the playback system 200 is shown as obtaining an encoded comparator image subframe from the content delivery system 100. The encoded comparator image subframe may be provided to the playback system 200 via a plurality of streams. The playback system 200 may include a subframe selector 220 configured to obtain head pose / gaze information 210 and select one of the streams corresponding to the subframe including the fovea based on the head pose / gaze information 210. The subframe selector 220 determines whether the selected stream includes the subframe or the portion of the subframe including the fovea. At this point, the decoder 205A may decode the selected stream of the encoded comparator image subframe. The decoded version of the selected stream is then provided to a scaler, rotator, color (SRC) converter 215A to obtain the ROI 230. The SRC converter 215A enables the playback system 200 to implement a human perception based color image enhancement algorithm that provides color constancy, a spatial modulation engine, and dynamic range compression (DRC) to the ROI 230. The SRC converter 215A may be configured to identify image enhancements that may be used to achieve color and brightness reproduction as well as DRC and any additional rendering parameters (eg, resolution) for the ROI 230. Once the ROI 230 is obtained, the ROI 230 is provided to the controller 240.

[0029] exist Figure 2 , the playback system 200 is shown as obtaining an encoded reduced image frame from the content delivery system 100. The encoded reduced image frame may be provided to the playback system 200 via a single stream. The playback system 200 may include a decoder 205B configured to decode the encoded reduced image frame. The decoded version of the encoded reduced image frame is then provided to the SRC converter 215B to obtain a decoded reduced image frame 250. The SRC converter 215B may be configured to identify image enhancements that can be used to achieve color and brightness reproduction as well as DRC and any additional rendering parameters (e.g., resolution) for the decoded reduced image frame 250. Once the decoded reduced image frame 250 is obtained, the decoded reduced image frame 250 is provided to the controller 240.

[0030] The controller 240 may be one or more processors configured to obtain and combine the ROI 230 with the decoded reduced image frame 250. In the combination, the controller 240 generates a combined image frame 260 for playback. The combined image frame 260 for playback produces an image frame including high-level details in the fovea. The combined image frame 260 for playback may be a target frame provided to a rendering module for magnification and rendering prior to playback.

[0031] The playback system 100 may be configured to decode and combine multiple frames and subframes to meet a predefined or dynamic power consumption target. Changes in the power consumption target may directly affect the decoding provided by the decoder 205. For example, when the power consumption target is higher, the ROI 230 may be larger in size or have a greater resolution.

[0032] In some embodiments, decoder 205A and decoder 205B may be the same decoder 205 configured to perform multiple decoding operations based on received input. For example, decoder 205 may perform a type of decoding operation based on whether the input is a portion of a separate subframe or an entire encoded reduced image frame. Similar to encoder 115 and decoder 150, decoder 205 may be configured to perform encoding / decoding according to the HEVC standard to produce a compressed video bitstream (e.g., a sequence of decoded frames). SRC converter 215A and SRC converter 215B may be the same SRC converter 215 configured to identify spatial transformation parameters associated with ROI 230 and decoded reduced image frame 250.

[0033] Figure 3 Is caused by Figure 2 The playback system 200 communicates Figure 1A flowchart of a multi-layer foveated streaming process performed by a content delivery system 100 of the present invention. Specifically, the flowchart shows a process for the following operations: obtaining an initial image frame 130 having a first quality (e.g., resolution or dynamic range); generating a reduced image frame 140 from the initial image frame 130 at a second quality; obtaining one or more subframes of the initial image frame 130; and sending the reduced image frame and at least one of the one or more subframes. The content delivery system 100 may include an electronic device or system configured to generate multiple layers of foveated streaming content. The playback system 200 may be an electronic device or system that receives layers of foveated streaming content and combines the layers into a target frame including subframes of different qualities. In the target frame, the area representing the fovea may include a high quality, while the area outside the fovea and the foveal area may include a low quality.

[0034] In one or more embodiments, to reduce processing for decoding encoded frames / subframes at the playback system 200, the content delivery system 100 provides a lower quality first stream comprising encoded downscaled image frames and a higher quality second stream comprising encoded comparator image subframes.

[0035] In the multi-layer foveal streaming technology, the content delivery system 100 simulates the function of the playback system 200 to proactively identify the possible loss of pixel quality in the fovea in the foveal stream. Based on bandwidth availability, different sizes of encoded image frames and subframes can be provided to the playback system 200. These sizes can be automatically selected to change the streaming time between the content delivery system 100 and the playback system 200. As described above, the subframes for the foveal area can be selected based on the user's head posture / attention information 210, which allows the playback device 200 to reduce power consumption by only combining the area of ​​interest including the fovea in the foveal area with the reduced image frame to generate a high-quality combined image frame 260 for playback. In addition, the playback system 200 is configured to apply edge smoothing to improve the quality across the combined image frame boundary in the foveal streaming.

[0036] exist Figure 3In the example of , the content delivery system 100 obtains an initial image frame at a first quality in box 310. The content delivery system 100 can obtain the initial image frame from an image stream captured directly using one or more camera devices and / or from a camera system communicatively coupled to the content delivery system 100 and / or the playback device 200. Additionally or alternatively, the content delivery system 100 can obtain the image from a storage device (such as a local storage device within the content delivery system 100) or a remote source (such as a network storage device) or from another storage device. Multiple media streams can be pre-labeled according to their corresponding types (e.g., 2D content, 3D content, focus in the fovea, surrounding points in the fovea setting, etc.). In some embodiments, information for selecting one or more frames from any one or more media streams is provided to the content delivery system 100.

[0037] At block 320, the content delivery system 100 generates a reduced image frame 140 from the initial image frame 130 at a second quality. Figure 1 As shown, the content delivery system 100 may divide the initial image frame 130 into a plurality of original sub-frames 131 to 139. The content delivery system 100 may determine that a portion of at least one sub-frame includes a fovea. As described above, the fovea may be an area in at least a portion of a sub-frame indicating an area of ​​interest in the initial image frame. The reduced image frame 140 may be a version of the initial image frame 130 reduced from the original size using the scaler 105. Thus, the reduced image frame 140 may include reduced versions of all the original sub-frames 131 to 139. For example, if the scaling factor is defined as N=2, the initial image frame 130 having a width 110A and a height 120A equal to 3600 pixels may be reduced to a reduced image frame 140 having a width 110B and a height 120B equal to 1800 pixels.

[0038] At box 330, the content delivery system 100 obtains one or more subframes of the initial image frame 130. The one or more subframes may be comparator image subframes generated by the content comparator 170. As described above, at the content comparator 170, the fovea in the subframes 131 to 139 of the initial image frame 130 is a target area, which is compared with the corresponding area in the subframes 161 to 169 of the enlarged image frame 160 to generate one or more comparator image subframes. In this case, the comparator image subframe is a subframe that represents the difference between the target area and the corresponding area. In other embodiments, at the content comparator 170, the subframes 131 to 139 of the initial image frame 130 are compared with the subframes 161 to 169 of the enlarged image frame 160 to generate the comparator image subframe. Therefore, the comparator image subframe is a subframe that represents the difference between the two groups of subframes.

[0039] In one or more embodiments, the initial image frame 130 may include a set of image parameters indicating a specific type of media content (such as 2D or 3D content) in the media stream. The image parameters may indicate that the media stream includes linear media content to be decoded and rendered using a 2D driver and / or a configuration for processing 2D content. The image parameters may indicate that the media stream includes interactive or immersive media content to be decoded and rendered using an interactive media driver and / or a configuration for processing interactive content. In some embodiments, the interactive content may be 2D content with a fovea to provide focus in an area of ​​interest. The interactive content may be 3D content with a fovea to immerse the viewer in a virtual environment.

[0040] In one or more embodiments, the set of image parameters indicates a quality allocation included in the media stream. In one example, the image parameters may indicate that the media stream includes media content including a specific resolution allocation. Additionally, the image parameters may indicate that the media stream includes one or more specific resolution allocations. In some embodiments, the resolution allocations may be different between frames of the same media stream. Additionally, the resolution allocations may be different while remaining within a common resolution range. In another example, three frames in a media stream may have different resolution allocations while remaining above a common resolution allocation threshold.

[0041] In one or more embodiments, the set of image parameters indicates a frame rate distribution included in the media stream. The image parameters may indicate that the media stream includes media content that includes a specific frame rate. In addition, the image parameters may indicate that the media stream includes one or more specific frame rates. In some embodiments, the frame rate between frames of the same media stream may be different. In addition, the frame rate may be different while remaining within a common frame rate range. For example, three groups of frames in a media stream may have different frame rates while remaining above a common frame rate threshold.

[0042] At block 340, the content delivery system 100 provides the reduced image frame and at least one of the subframes to the playback system 200 for playback. Figure 1 The described approach provides an encoded comparator image subframe and an encoded reduced image frame.

[0043] At block 350, the playback system 200 receives the downscaled image frame at the second quality. Figure 2 The method described above receives the encoded reduced image frame.

[0044] At block 360, the playback system 200 receives a plurality of sub-frames of the initial image frame. The sub-frames may have different qualities. Here, the playback system 200 may receive the sub-frames of the initial image frame in a manner similar to that of FIG. Figure 2The encoded comparator image subframes are received in the manner described. In some embodiments, these subframes represent the difference in the fovea between the subframes in the initial image frame 130 and the subframes in the enlarged image frame 160.

[0045] At block 370, the playback system 200 selects at least one of the subframes as a region of interest. The region of interest may correspond to the fovea in the foveal region. Figure 2 As described in the foregoing, the region of interest is selected based on head pose / gaze information obtained from at least one user. In some embodiments, the region of interest may correspond to an action region in the image frame. In this case, the "action" may be an area where attention-grabbing movement is expected to occur. The region of interest may be selected based on content identified in the initial image frame or from information obtained by a processor (such as a reference frame). Figure 6 The described processor 610 or 615) performs image analysis to indicate the action area to the playback system 200.

[0046] Finally, at block 380, the playback system 200 generates a combined image frame based on the reduced image frame and the selected at least one sub-frame. Figure 2 The combined image frame 260 (e.g., target frame) for playback is shown. Here, the playback system 200 generates a combined image frame to be rendered. Following the example mentioned in reference box 380, the playback system 200 can determine the rendering process of the combined image frame. For example, the playback system 200 can also receive the three-dimensional scene geometry and the mapping of the reduced image frame 140 and the higher quality sub-frame to the scene geometry from the content delivery system 100. In this case, the rendering process can include rendering the scene from the desired viewpoint by mapping the image frame and the higher quality sub-frame to the scene geometry. If necessary, the rendering process can also (e.g., using alpha technology) blend the image frame and the higher quality sub-frame. The rendering process can be predefined or dynamically configured according to one or more image parameters.

[0047] Figure 4 A flow chart of a technique in which multiple layers of a foveated stream are provided for playback according to one or more embodiments is shown. The technique may be referred to as Figure 1 The content delivery system 100 described herein is implemented as described herein. Figure 4 The various processes depicted in the present invention are shown in a particular order, but it should be understood that the processes can be performed in a different order. In addition, not all processes must be performed. For the purpose of explanation, various processes will be described in the context of specific components of specific devices; however, it should be understood that various processes can be performed by additional or alternative components or devices.

[0048] exist Figure 4In the example of , the content delivery system 100 is described as providing multiple encoded frame streams of different resolutions. As described above, the multi-layer foveated streaming technology can include providing multiple content streams at different qualities. In this regard, image quality can refer to resolution, frame rate, signal-to-noise ratio (SNR), dynamic range, contrast, color accuracy, distortion, etc.

[0049] The flowchart begins at block 410, where the content delivery system 100 obtains an initial image frame at a first resolution. The initial frame includes a plurality of pixels. The content delivery system 100 may capture or otherwise obtain the frame from one or more types of media streams. Figure 1 and Figure 3 As shown, the content delivery system 100 may identify a resolution associated with the initial image frame 130. In addition, the content delivery system 100 may identify a plurality of sub-frames 131-139 in the initial image frame 130.

[0050] The flowchart continues at block 420, where the content delivery system 100 generates a reduced image frame of a second resolution from the initial image frame. Figure 1 and Figure 3 The initial image frame is downscaled from a first resolution to a second resolution in the manner described in .

[0051] The flow chart continues at block 430 where the content delivery system 100 obtains one or more subframes of the initial image frame 130. Each of the subframes includes a portion of a pixel. The subframes may be comparator image subframes described as a result of finding differences between subframes 131 to 139 and subframes 161 to 169.

[0052] At box 440, the content delivery system 100 generates an enlarged image frame from the reduced image frame 140. As described above, the reduced image frame 140 is enlarged to obtain sub-frames 161 to 169. In box 450, the content delivery system 100 identifies a corresponding subset of pixels in the pixels for each of the sub-frames. The subset of pixels is a portion of the sub-frames 161 to 169 corresponding to the fovea found in the initial image frame 130. In box 460, the content delivery system 100 performs differential encoding on each of the sub-frames based on the corresponding subset of pixels. The comparator image sub-frame is obtained by using the content comparator 170, which compares the fovea in the initial image frame 130 with the corresponding position in the enlarged image frame 160.

[0053] The flowchart ends at block 470 , where the content delivery system 100 provides the downscaled image frame 140 and at least one of the subframes for playback. More specifically, the content delivery system 100 sends the encoded version of the comparator image subframe and the downscaled image frame 140 to the playback system 200 .

[0054] Figure 5 A flow chart of a technique according to one or more embodiments is shown in which a combined image frame is generated based on multiple layers of a foveated stream received for playback. The technique may be referred to as Figure 2 The playback system 200 described in the present invention is implemented. This technology can be referred to as Figure 2 The playback system 200 described herein is implemented. Figure 5 The various processes depicted in the present invention are shown in a particular order, but it should be understood that the processes can be performed in a different order. In addition, not all processes must be performed. For the purpose of explanation, various processes will be described in the context of specific components of specific devices; however, it should be understood that various processes can be performed by additional or alternative components or devices.

[0055] exist Figure 5 In the example of , playback system 200 is described as receiving multiple encoded frame streams of different resolutions. As described above, multi-layer foveated streaming techniques can include receiving multiple content streams of different qualities. In this regard, "quality" can refer to resolution, frame rate, noise, dynamic range, contrast, color accuracy, distortion, etc.

[0056] The flowchart begins at block 510, where the playback system 200 obtains head pose information and / or gaze information about the user. The head pose sensor and / or gaze tracking system ( Figure 5 ) may be configured to provide head pose / gaze information to a subframe selector 220 in the playback system 200.

[0057] In block 520, the playback system 200 receives a plurality of sub-frames of the initial image 130. The sub-frames may have different resolutions. According to one or more embodiments, the sub-frames may be reference frames. Figure 1 and Figure 2 Describes the encoded comparator image subframe.

[0058] At block 530, the playback system 200 receives a downscaled image frame 140 that is downscaled from the initial image frame. Depending on the implementation, the downscaled image frame 140 may be multiple orders of magnitude smaller in size than the initial image frame 130.

[0059] At block 540, the playback system 200 selects at least one of the subframes as a region of interest based on the head pose / gaze information. In some embodiments, the subframe selector 220 in the playback system 200 is configured to select the region of interest as a region in the encoded comparator image subframe that matches the user's head position and / or gaze.

[0060] The flowchart ends at block 550, where the playback system 200 generates a combined image frame based on the reduced image frame and the selected at least one sub-frame. At this stage, the playback system 200 combines the ROI 230 with the decoded reduced image frame 250 to obtain a reference image frame. Figure 2 The combined image frames of playback 280 are described.

[0061] refer to Figure 6 , which depicts a simplified block diagram of a multi-layer foveated streaming system 700 according to one or more embodiments of the present disclosure. The multi-layer foveated streaming system 700 may include and / or be a part of a multi-function device, such as a mobile phone, a tablet computer, a personal digital assistant, a portable music / video player, a wearable device such as a head-mounted device, a base station, a laptop computer, a desktop computer, a network device, or any other electronic device. In some embodiments, the multi-layer foveated streaming system 700 may include a content delivery system 100 and a playback system 200 that communicate with each other via a network 680 using network interfaces 630 and 645.

[0062] According to one or more embodiments, the content delivery system 100 can provide motion detection from a sensor 660, such as an inertial measurement unit ("IMU") sensor or other sensor that detects movement. The motion sensor 660 can detect changes in inertia that indicate a motion event. In this regard, the sensor data can be used to track motion parameters, and thresholds associated with these motion parameters can indicate that a motion event has occurred. The content delivery system 100 may include a processor 610 (e.g., at least one processor). In some embodiments, the processor 610 can be separate from the content delivery system 100 and can communicate with the content delivery system 100 across a network 680, such as a wired connection or a wireless short-range connection, etc. For example, in some embodiments, the processor 610 can be part of a smart accessory, such as a smart watch worn on a subject's wrist or arm, a smart headphone device worn on a subject's head, a smart hearing device worn on a subject's ear, or any other electronic device including a sensor 660 from which at least some motion can be determined. Processor 610 may be a central processing unit (CPU) or a system on a chip (such as those found in mobile devices), and may include one or more dedicated graphics processing units (GPUs). In addition, processor 610 may include multiple processors of the same or different types.

[0063] The content delivery system 100 may also include a memory 620 (e.g., a storage device). The memory 620 may include one or more different types of storage devices that may be used to perform device functions in conjunction with the processor 610. For example, the memory 620 may include a cache, a ROM, a RAM, or any type of transient or non-transitory computer-readable storage medium capable of storing computer-readable code. The memory 620 may store various programming modules for execution by the processor 610.

[0064] The content delivery system 100 may include a system configured to perform a reference Figures 1 to 5 The scaling module 622, encoding module 624, and decoding module 626 may perform one or more of the described scaling / encoding / decoding functions. The scaling module 622 may perform the Figure 1 The scaling module 622 may be configured to reduce the initial frame 130 to a reduced image frame 140. The plurality of frames may include image data representing different sets of image parameters. In addition, the processor 610 may be configured to determine whether the plurality of frames include different types of image parameters based on a scene analysis of the plurality of frames.

[0065] The encoding module 624 can execute reference Figure 1 The encoding module 624 may be configured to encode the comparator image subframe and the reduced image frame 140. The decoding module 626 may perform the reference Figure 1 The decoding module 626 may be configured to decode the reduced image frame 140.

[0066] The content delivery system 100 may include at least one camera 640 or other sensor from which the depth of the scene may be determined. In one or more embodiments, the camera 640 may be a conventional RGB camera, a depth camera, or other camera device capable of capturing image information. Additionally, the camera 640 may include a stereo or other multi-camera system, a time-of-flight camera system, or the like that captures images from which depth information of the scene may be determined.

[0067] According to one or more embodiments, playback system 200 is configured to combine sub-frames received from content delivery system 100. Playback system 200 may function as a system that receives combined media streams from content delivery system 100 and plays the sub-frames in a manner similar to that of FIG. Figure 2 , Figure 3 and Figure 5 A playback device that combines multiple subframes in the manner described in .

[0068] The playback system 200 may include a processor 615 (e.g., at least one processor). The processor 615 may perform one or more functions described with reference to the processor 615. The playback system 200 may also include a memory 625. The memory 625 may include one or more different types of storage devices that may be used to perform device functions in conjunction with the processor 615. As described with reference to the memory 620, the memory 620 may store various programming modules for execution by the processor 615. In some embodiments, the memory 625 may include a media playback module 635 that is configured to perform the media playback functions described with reference to the processor 615. Figure 2 One or more of the decoding functions described in decoder 205 in .

[0069] The media playback module 635 may be executed to perform the reference Figure 2 Specifically, the media playback module 635 can decode and render portions of the combined image frame 260 for subsequent playback. As described above, the media playback module 635 can be configured to perform one or more rendering operations based on a predefined rendering configuration for the foveal region. The playback system 200 may include a display 655 that is configured to show a visual representation of the rendered portion of the combined image frame 260 for playback.

[0070] Although content delivery system 100 is depicted as including the above-described multiple components, in one or more embodiments, various components may be distributed across multiple systems or devices. In particular, in one or more embodiments, one or more of scaling module 622, encoding module 624, and decoding module 626 may be distributed across multiple devices in different ways. Therefore, according to one or more embodiments, content delivery system 100 may not be required to perform one or more of the techniques described herein. Therefore, although certain operations are described herein with respect to the specific systems depicted, in one or more embodiments, various operations may be directed differently based on differently distributed functions. In addition, additional components may be used, and certain combinations of the functions of any component may also be combined.

[0071] Reference now Figure 7, which shows a simplified functional block diagram of an exemplary multifunctional electronic device 700 according to one or more embodiments. For example, the multi-layer foveated streaming system 600 may include one or more multifunctional electronic devices, or may have some or all of the described components of the multifunctional electronic devices described herein. The multifunctional electronic device 700 may include a processor 725, a display 730, a user interface 710, a device sensor 750, a graphics hardware 755, an image capture circuit 745, a device sensor (e.g., a proximity sensor / ambient light sensor, an accelerometer and / or a gyroscope), a microphone 715, an audio codec 720, a speaker 705, a communication circuit 765, a multi-layer foveated streaming system 600 (e.g., including a camera 640 or a camera system), a video codec 770 (e.g., supporting a digital image capture unit), a memory 775, a storage device 780, and a communication bus 760. The multifunctional electronic device 700 may be, for example, a digital camera or a personal electronic device such as a personal digital assistant (PDA), a personal music player, a mobile phone, or a tablet computer.

[0072] The processor 725 can execute instructions necessary for implementing or controlling the operation of many functions performed by the multifunctional electronic device 700 (such as the generation and / or processing of the media content types disclosed herein). The processor 725 can, for example, drive the display 730 and receive user input from the user interface 710. The user interface 710 can allow the user to interact with the multifunctional electronic device 700. For example, the user interface 710 can present various forms such as buttons, keypads, dials, click-type touch wheels, keyboards, display screens and / or touch screens. The processor 725 can also be, for example, a system on a chip, such as those present in a mobile device, and include a dedicated graphics processing unit (GPU). The processor 725 can be based on a reduced instruction set computer (RISC) or a complex instruction set computer (CISC) architecture or any other suitable architecture, and can include one or more processing cores. The graphics hardware 755 can be a dedicated computing hardware for processing graphics and / or assisting the processor 625 to process graphics information. In one embodiment, the graphics hardware 755 can include a programmable GPU.

[0073] In one or more embodiments, the image capture circuit 745 may include two (or more) lens assemblies (e.g., sensor elements 740A and 740B with corresponding lenses 735A and 735B), wherein each lens assembly may have a separate focal length. For example, one lens assembly may have a shorter focal length relative to the focal length of another lens assembly. Each lens assembly may have a separate associated sensor element. Alternatively, two or more lens assemblies may share a common sensor element. The image capture circuit 745 may cooperate with the multi-layer foveated streaming system 600 to capture still images and / or video images. The output from the image capture circuit 745 may be processed at least in part by the video codec 720 and / or the processor 725 and / or the graphics hardware 755. The images thus captured may be stored in the memory 775 and / or the storage device 780.

[0074] The memory 775 may include one or more different types of media used by the processor 725 and the graphics hardware 755 to perform device functions. For example, the memory 775 may include a memory cache, a read-only memory (ROM), and / or a random access memory (RAM). The storage device 780 may store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. The storage device 780 may include one or more non-transitory computer-readable storage media, which include, for example, disks (fixed hard disks, floppy disks, and removable disks) and tapes, optical media (such as CD-ROMs and digital video discs (DVDs)), and semiconductor storage devices (such as electrically programmable read-only memories (EPROMs) and electrically erasable programmable read-only memories (EEPROMs)). The memory 775 and the storage device 780 may be used to tangibly maintain computer program instructions or codes organized into one or more modules and written in any desired computer programming language. For example, when executed by the processor 725, such computer program code may implement one or more of the methods described herein.

[0075] Although Figures 1 to 7 Various configurations of components are shown, but other configurations may be used without departing from the scope of the present disclosure. For example, Figures 1 to 7 The various components in (for example, Figure 1 The content delivery system 100 and Figure 2 The playback system 200 of can be combined to create a single component. As another example, the function performed by a single component can be performed by two or more components.

[0076] The scope of the subject matter disclosed in this disclosure should be determined with reference to the appended claims, along with the full scope of equivalents to which the claims are entitled.In the appended claims, the terms "including" and "in which" are used as the plain-English equivalents of the respective terms "comprising" and "wherein."

Claims

1. A non-transitory computer readable medium comprising instructions that, when executed by at least one computer processor, cause the at least one computer processor to: obtaining an initial image frame of a first resolution, wherein the initial image frame comprises a first plurality of pixels; generating a reduced image frame of a second resolution from the initial image frame; Obtaining one or more subframes of the initial image frame, wherein each of the one or more subframes includes a portion of the plurality of pixels; as well as sending the reduced image frame and at least one of the one or more subframes to a playback device, The reduced image frame and the at least one sub-frame of the one or more sub-frames can be combined to form a target frame including sub-frames of different resolutions.

2. The non-transitory computer readable medium of claim 1 , wherein to obtain the one or more subframes, the at least one computer processor is further caused to: generating an upscaled image frame from the downscaled image frame, wherein the upscaled image frame comprises a second plurality of pixels; for each subframe of the one or more subframes, identifying a corresponding subset of pixels in the second plurality of pixels; as well as performing differential encoding on each of the one or more subframes based on the corresponding subset of pixels, Wherein the at least one subframe of the one or more subframes comprises one or more differentially encoded subframes.

3. The non-transitory computer readable medium of claim 2, wherein to generate the zoomed-in image frame, the at least one computer processor is further caused to: encoding the reduced image frame to obtain an encoded reduced image frame; applying the encoded reduced image frame to a decoder to obtain a decoded reduced image frame; and A sealer is applied to the decoded downscaled image to obtain the upscaled image frame. 4 . The non-transitory computer-readable medium of claim 1 , wherein the initial image frame comprises a foveated image frame generated from image data captured from a plurality of cameras. 5 . The non-transitory computer readable medium of claim 1 , wherein the at least one subframe of the one or more subframes corresponds to a region of interest in the initial image frame. 6 . The non-transitory computer-readable medium of claim 5 , wherein the region of interest is determined based on head pose information or gaze information obtained by the playback device.

7. A non-transitory computer readable medium comprising instructions that, when executed by at least one computer processor, cause the at least one computer processor to: receiving, at a playback device, a downscaled image frame having a first resolution; receiving a plurality of sub-frames of an initial image frame, the reduced image frame being reduced from the initial image frame, wherein the plurality of sub-frames comprise different resolutions; selecting at least one subframe among the plurality of subframes as a region of interest; as well as A combined image frame is generated based on the downsized image frame and the selected at least one sub-frame.

8. The non-transitory computer readable medium of claim 7, wherein the at least one computer processor is further caused to: Obtaining head posture information or gaze information of a user of the playback device; and The at least one subframe of the plurality of subframes is selected based on the head pose information or the gaze information.

9. The non-transitory computer readable medium of claim 7, wherein the at least one computer processor is further caused to: Identifying head posture information or gaze information of a user of the playback device; and A request for the plurality of subframes is sent based on the head pose information or the gaze information.

10. The non-transitory computer readable medium of claim 7, wherein to generate the combined image frame, the at least one computer processor is further caused to: decoding the downscaled image frame; and The at least one subframe among the plurality of subframes is decoded according to the downscaled image frame.

11. The non-transitory computer readable medium of claim 10, wherein the at least one of the plurality of subframes is differentially encoded based on a corresponding subset of pixels in an upscaled image frame having a second resolution generated from the downscaled image frame.

12. The non-transitory computer readable medium of claim 7, further causing the at least one computer processor to: determining a power target for the playback device; selecting a subframe size based on the power target; and A request for the at least one subframe of the plurality of subframes is sent according to the selected subframe size.

13. The non-transitory computer readable medium of claim 7, wherein the at least one computer processor is further caused to: The combined image frame is displayed on a display device of the playback device.

14. A non-transitory computer readable medium comprising instructions that, when executed by at least one computer processor, cause the at least one computer processor to: obtaining an initial image frame at a first quality, wherein the initial image frame comprises a first plurality of pixels; generating a reduced image frame from the initial image frame at a second quality; Obtaining one or more subframes of the initial image frame, wherein each of the one or more subframes includes a portion of the plurality of pixels; as well as sending the reduced image frame and at least one of the one or more subframes to a playback device, wherein the reduced image frame and the at least one subframe of the one or more subframes are capable of being combined to form a target frame including subframes of different qualities, and The target frame includes an area having a first quality surrounded by an area having a second quality.

15. The non-transitory computer readable medium of claim 14, wherein to obtain the one or more subframes, the at least one computer processor is further caused to: generating an upscaled image frame from the downscaled image frame, wherein the upscaled image frame comprises a second plurality of pixels; for each subframe of the one or more subframes, identifying a corresponding subset of pixels in the second plurality of pixels; as well as performing differential encoding on each of the one or more subframes based on the corresponding subset of pixels, Wherein the at least one subframe of the one or more subframes comprises one or more differentially encoded subframes.

16. The non-transitory computer readable medium of claim 15, wherein to generate the zoomed-in image frame, the at least one computer processor is further caused to: encoding the reduced image frame to obtain an encoded reduced image frame; applying the encoded reduced image frame to a decoder to obtain a decoded reduced image frame; and A sealer is applied to the decoded downscaled image to obtain the upscaled image frame. 17 . The non-transitory computer-readable medium of claim 14 , wherein the initial image frame comprises a foveated image frame generated from image data captured from a plurality of cameras.

18. The non-transitory computer readable medium of claim 14, wherein the at least one subframe of the one or more subframes corresponds to a region of interest in the initial image frame.

19. The non-transitory computer readable medium of claim 18, wherein the region of interest is the region having a first quality.

20. The non-transitory computer readable medium of claim 18, wherein the region of interest is determined based on head pose information or gaze information obtained by the playback device.

Images