Cloud-based extended reality system with depth-based region of interest encoding

Abstract

To encode objects in a frame that appear closest to a user with more detail without increasing the overall bits used to encode the frame, a system includes a depth-based region circuitry. The depth-based region circuitry is configured to identify regions of interest (ROIs) in a frame representing an extended reality (XR) scene based on depth values of one or more graphics object in the frame being equal to or less than one or more depth thresholds. Further, the system includes an encoder configured to encode the identified ROIs in the frame using respective quantization parameters (QPs) each associated with corresponding depth thresholds.

Description

CLOUD-BASED EXTENDED REALITY SYSTEM WITH DEPTH-BASED REGION OF INTEREST ENCODINGBACKGROUND

[0001] Cloud-based extended reality (XR) systems include servers configured to provide an encoded XR stream to a head-worn device (HWD). To this end, these servers execute XR applications that are each configured to generate a set of frames representative of an XR scene. The servers then encode the set of frames and provide, over a network, the encoded frames as an encoded XR stream to an HWD. The HWD then decodes the encoded XR stream to produce a set of decoded frames and presents the set of decoded frames to a user. To help improve user experience, some servers are configured to provide a set of frames representing a right view of an XR scene and another set of frames representing a left view of the XR scene in an encoded XR stream to the HWD. However, providing both sets of frames in an XR stream increases the bandwidth needed to send the encoded XR stream over the network. Such an increase in bandwidth increases the likelihood of increased latency or packet loss in the encoded XR stream which negatively impacts user experience.BRIEF DESCRIPTION OF THE DRAWINGS

[0002] The present disclosure may be better understood, and its numerous features and advantages are made apparent to those skilled in the art by referencing the accompanying drawings. The use of the same reference symbols in different drawings indicates similar or identical items.

[0003] FIG. 1 is a block diagram of a cloud-based extended reality (XR) system employing depth-based region of interest (ROI) encoding, in accordance with some embodiments.

[0004] FIG. 2 is a flow diagram for an example depth-based ROI encoding operation, in accordance with some embodiments.

[0005] FIG. 3 is a block diagram of an example frame including depth-based ROIs, in accordance with some embodiments.

[0006] FIG. 4 is a block diagram of an example list of ROIs in a frame, in accordance with some embodiments.

[0007] FIG. 5 is a flow diagram of an example method for depth-based ROI encoding, in accordance with some embodiments.SUMMARY OF EMBODIMENTS

[0008] Techniques and systems described herein are directed to encoding objects in a frame that appear closest to a user with more detail without increasing the overall bits used to encode the frame by using depth-based region of interest (ROI) encoding. According to an example embodiment, a computer-implemented method includes, based on a first depth value of a graphics object in a frame being equal to or less than a depth threshold, identifying an ROI in the frame. Additionally, the method can include encoding, at an encoder, the ROI using a first quantization parameter (QP) associated with the depth threshold.

[0009] In embodiments, the method can also include determining a priority associated with the ROI based on the depth threshold. The method may also include selecting the first QP based on the priority associated with the ROI. Additionally, the method can include encoding a remainder of the frame using a remainder QP, wherein the remainder of the frame omits the ROI and the remainder QP is different from the first QP. The first QP may be lower than the remainder QP. As well, the method can include comparing the depth value of the graphics object to a plurality of depth thresholds, wherein the plurality of depth thresholds includes the depth threshold. Each depth threshold of the plurality of depth thresholds can be associated with a respective depth value range.

[0010] The method may also, in embodiments, include providing a list of ROIs identifying the ROI and a priority associated with the ROI to the encoder. Further, the method can include based on a second depth value of a second graphics object in the frame, identifying a second ROI in the frame. Also, the method may include encoding, at an encoder, the second ROI using a second QP based on the second depth value of the second graphics object. The first QP may be lower than the second QP. The method can also include transmitting, by a network, an encodedframe with the encoded ROI to a head-worn device (HWD). Further, the frame can include a depth map.

[0011] According to another example embodiment, a computer-implemented method includes identifying a plurality of ROIs in a frame based on depth values of a plurality of graphics objects in the frame. Further, the method can include determining, for each ROI of the plurality of ROIs in the frame, a respective priority based on depth values of one or more graphics objects of the plurality of graphics objects within a corresponding ROI. Additionally, the method can include encoding each ROI of the plurality of ROIs using a respective QP based on the respective priority of the corresponding ROI.

[0012] In embodiments, the method may also include encoding a first ROI of the plurality of ROIs using a first QP based on a first priority associated with the first ROI and a second ROI of the plurality of ROIs using a second QP based on a second priority associated with the second ROI. The first priority can represent a higher priority than the second priority. Further, the method can include encoding a remainder of the frame using a remainder QP, wherein the remainder of the frame omits the plurality of ROIs and the remainder QP is higher than the respective QPs used to encode each ROI of the plurality of ROIs.

[0013] The method may, in embodiments, also include comparing the depth values of the plurality of graphics objects to one or more depth thresholds. A respective ROI of the plurality of ROIs may be identified in response to one or more depth values of a graphics object of the plurality of graphics objects within the respective ROI being equal to or less than the one or more depth thresholds. The method can additionally include transmitting, by a network, an encoded frame with the encoded plurality of ROIs to an HWD. The frame may include a depth map.

[0014] In another example embodiments, a system includes a depth-based region circuitry and an encoder. The encoder can be configured to perform any method disclosed herein.

[0015] According to another example embodiment, a system includes an HWD configured to receive an encoded frame encoded according to any method disclosedherein. Further, the system can include a decoder configured to decode the encoded frame.DETAILED DESCRIPTION

[0016] Within some cloud-based extended reality (XR) systems, a head-worn device (HWD) is configured to display an XR scene, such as a virtual reality (VR), augmented reality (AR), or mixed reality (MR) scene, to a user. To this end, the HWD is configured to present a first set of frames to a left eye of the user and a second set of frames to the right eye of the user. To help support the HWD in presenting the XR scene, a cloud-based XR system includes one or more servers each configured to execute an XR application. While executing the XR application, a server is configured to generate a frame representing an XR scene including one or more graphics objects. For example, the server is configured to generate a first set of frames each presenting a first view (e.g., left view) of an XR scene and a second set of frames each presenting a second view (e.g., right view) of the XR scene. Further, the server includes one or more encoders configured to encode each frame from the first set of frames and each frame from the second set of frames according to one or more codecs to produce a set of encoded frames. The encoders then provide the set of encoded frames to a streaming circuitry of the server configured to packetize the set of encoded frames and transmit each encoded packet over a network to the HWD as part of an XR stream. After receiving an encoded packet of the XR stream from the server, the HWD then decodes the encoded packet to produce a frame representing the first view (e.g., left view), a frame representing the second view (e.g., right view) or both and displays the decoded frames.

[0017] To help compensate for movement by the user of an HWD while the HWD displays one or more decoded frames, the server is further configured to obtain depth data for each generated frame of the first set of frames and each generated frame of the second set of frames. This depth data, for example, indicates depth values of one or more graphics objects within the XR scene represented by a frame. The servers then provide the determined depth data as part of the XR stream to the HWD. In response to receiving the depth data, the HWD is configured to perform one or more reprojection techniques. These reprojection techniques, for example, include the HWD modifying one or more frames indicated in a received XR stream so as tomatch a pose of the HWD of the user based on the movement of the HWD and the generated depth data for the frames. However, providing an XR stream that includes multiple sets of encoded frames and depth data for each set of encoded frames increases the bandwidth needed for the servers to transmit the XR stream over the network, increasing the likelihood that packets of the XR stream are lost or delayed and negatively impacting user experience. To this end, systems and techniques disclosed herein are directed to depth-based region of interest (ROI) encoding.

[0018] To perform such depth-based ROI encoding, for example, a server includes a depth-based region circuitry configured to identify ROIs in a frame based on the depth values of the graphics objects in the frame. To this end, for example, the depth-based region circuitry is configured to receive depth data for each frame of the first set of frames (e.g., frames representing a left view of the XR scene) and for each frame of the second set of frames (e.g, frame representing a right view of the XR scene). In response to receiving the depth data for a frame, the depth-based region circuitry then compares the depth values of each graphics object indicated in the depth data to one or more depth thresholds. In response to one or more depth values of a graphics object indicated in the depth data of a frame being equal to or less than one or more depth thresholds, the depth-based region circuitry determines an ROI within the frame that includes the graphics object. For example, the depth-based region circuitry generates a bounding box representing the ROI around the graphics object. After determining the ROI within the frame, the depth-based region circuitry then determines a priority of the ROI based on the depth values of the graphics object within the ROI. For example, the depth-based region circuitry determines a priority of the ROI based on which depth thresholds are equal to or greater than the depth values of the graphics objects within the ROI. The priority of an ROI, for example, includes data indicating how close the graphics objects within the ROI appear to a user with ROIs having the highest priority including graphics objects appearing closest to a user and ROIs having a lowest priority including graphics objects appearing farthest from the user. Once the depth-based region circuitry has determined the priority of the ROI, the depth-based region circuitry updates a list of ROIs for the frame. The list of ROIs includes data indicating the location of an ROI within a frame and the priority of the ROI.

[0019] After the depth-based region circuitry has updated the list of ROIs for a frame to include each ROI within the frame, the depth-based region circuitry provides the list of ROIs to an encoder of the server. That is to say, once the depth-based region circuitry has compared the depth values of each graphics objects within a frame to one or more depth thresholds, the depth-based region circuitry provides the list of ROIs for the frame to an encoder. Based on the list of ROIs for the frame, the encoder then adjusts the quantization parameters (QPs) used to encode the frame. For example, to help account for fluctuating network bandwidth, the encoder is configured to encode each frame of the first and second sets of frames and their corresponding depth data using a respective quantization parameter (QP). That is to say, the encoder is configured to encode each frame using a respective framelevel QP. Based on the list of ROIs for a frame, the encoder is configured to adjust the frame-level QP of a frame when encoding one or more ROIs in the frame. As an example, in response to the priority of a corresponding ROI indicating that the graphics objects within the corresponding ROI appear close to the user (e.g., in response to the corresponding ROI having a high priority), the encoder is configured to use a QP lower than the frame-level QP when encoding the corresponding ROI. Further, in response to the priority of a corresponding ROI indicating that the graphics objects within the corresponding ROI appear far from the user (e.g., in response to the corresponding ROI having a low priority) the encoder is configured to use a QP higher than the frame-level QP when encoding the corresponding ROI. As such, the encoder is configured to encode regions of a frame appearing closest to the user using more bits and encode regions of a frame appearing farthest to the user using less bits. By encoding in this way, objects in the frame appearing closest to the user will be encoded with more detail (e.g., more bits) without increasing the overall bits used to encode the frame due to the objects in the frame appearing farthest from the user being encoding with less detail (e.g., less bits). Further, encoding objects in the frame appearing closest to the user with more detail improves the detail of the frame presented to the user, improving user experience.

[0020] Referring now to FIG. 1 , a cloud-based XR system 100 employing depthbased ROI encoding is presented, in accordance with some embodiments. Cloudbased XR system 100 includes one or more servers 102 communicatively coupled toan HWD 114 by at least one network 112. Network 112 is configured to allow for the transmission and reception of data between the servers 102 and HWD 114 and includes, for example, wired and wireless networks, such as Ethernet, the Internet, Wi-Fi, or any combination thereof. In embodiments, HWD 114, for example, is configured to display an XR scene to a user. The XR scene displayed to the user includes, for example, a VR scene, AR scene, MR scene, or any combination thereof that includes one or more rendered graphics objects. To provide such an XR scene to a user, in some embodiments, HWD 114 is configured to display one or more frames representing a first view (e.g., left view) of the XR scene (e.g., left view frames) to a first (e.g., left) eye of the user and one or more frames representing a second (e.g., right) view of the XR scene (e.g., right view fames) to a second (e.g., right) eye of the user. As an example, in some embodiments, HWD 114 is configured to direct a first light representing a left view frame to a left eye of the user such that the left view frame is displayed to the user and direct a second light representing a right view frame to a right eye of the user such that the right view frame is displayed to the user.

[0021] To support HWD 114 in displaying an XR scene to a user, cloud-based XR system 100 includes one or more servers 102 configured to execute an XR application 104. XR application 104, for example, includes a VR application, AR application, MR application, or any combination thereof configured to render one or more frames representing an XR scene to be presented to the user. This XR scene, for example, includes one or more VR scenes, AR scenes, MR scenes, or any combination thereof that each include one or more rendered graphics objects to be presented to a user. As an example, while executing XR application 104, server 102 is configured to render a first set of frames (e.g., left view frames 118) each representing a first view (e.g., left view) of an XR scene including one or more rendered graphics objects. Further, while executing XR application 104, server 102 is configured to render a second set of frames (e.g., right view frames 120) each representing a second view (e.g., right view) of the XR scene that includes one or more graphics objects. In embodiments, after generating one or more left view frames 118, right view frames 120, or both, server 102 is configured to encode each generated frame. To this end, in embodiments, server 102 includes encoder 108. Encoder 108 includes, for example, one or more hardware encoders, one or moresoftware encoders, or both configured to encode a generated frame (e.g., left view frame 118, right view frame 120) through, for example, compression, reorganization, and manipulation based on one or more codecs implemented by encoder 108. Such codecs include, for example, H.264, H.265, VP9, AV1 , or any combination thereof, to name a few. In embodiments, to help account for a fluctuating amount of available bandwidth on network 112, encoder 108 is configured to encode each frame (e.g., each left view frame 118, each right view frame 120) using a respective quantization parameter (QP). For example, in some embodiments, each frame generated by server 102 is associated with a respective frame-level QP that includes data indicating the QP to be used when encoding the frame. One of skill in the art will appreciate that a higher frame-level QP indicates more aggressive quantization while encoding a frame, resulting in an encoded frame with lower quality and fewer bits. Likewise, a lower frame-level QP indicates a less aggressive quantization while encoding a frame, resulting in an encoded frame with more detail (e.g., higher quality) and more bits.

[0022] After encoding one or more left view frames 118, right view frames 120, or both to produce one or more encoded frames, encoder 108 provides these encoded frames to streaming circuitry 126 included in server 102. Streaming circuitry 126, for example, is configured to packetize the encoded frames provided from encoder 108 to produce a set of encoded packets, also referred to herein as XR stream 116. Streaming circuitry 126 then transmits the set of encoded packets over network 112 to HWD 114. In response to receiving one or more encoded packets, HWD 114 is configured to decode the encoded packets to produce one or more decoded frames (e.g., decoded left view frames 118, decoded right view frames 120). To this end, HWD 114 includes decoder 128 configured to decode one or more encoded packets based on one or more codecs (e.g., H.264, H.265, VP9, AV1 ). After decoding the encoded packets, HWD 114 then directs light representing one or more left view frames 118 to a first (e.g., left) eye of a user and light representing one or more right view frames 120 to a second (e.g., right) eye of the user. In this way, the left view frames 118 and right view frames 120 used to display an XR scene to a user of HWD 114 are rendered at server 102 rather than HWD 114, reducing the processing resources required for HWD 114 to display an XR scene to the user.

[0023] In some embodiments, movement of HWD 114, movement of a user, or both affect the display of the XR scene to the user. For example, movement of HWD 114, movement of a user, or both increases the likelihood of perceived latency in a displayed XR scene. As an example, HWD 114 moving from a first pose to a second pose (e.g., current pose) increases the likelihood of perceived latency in a displayed XR scene. To help compensate for such movement, in embodiments, XR application 104 is further configured to generate depth data, for example, a depth map, for each generated frame. As an example, XR application 104 is configured to generate left view depth data 122 for each left view frame in the set of left view frames 118 and generate right view depth data 124 for each right view frame in the set of right view frames 120. The depth data (e.g., left view depth data 122, right view depth data 124) of a frame, for example, indicates one or more depth values (e.g., z-values) for each graphics object within the frame (e.g., the graphics objects within the XR scene included in the frame). As an example, the depth data of a frame includes a depth map. Once the depth data (e.g., left view depth data 122, right view depth data 124) is generated for a corresponding frame, encoder 108 is configured to encode the depth data and the corresponding frame. Streaming circuitry 126 then packetizes the encoded frame and encoded depth data into encoded packets and transmits the encoded packets to HWD 114 via network 112. After decoding the encoded packets to produce a decoded frame and corresponding depth data, HWD 114 uses the depth data to perform one or more reprojection techniques on the decoded frame. These reprojection techniques include, for example, reducing the perceived latency of the decoded frame by warping the decoded frame to match a current pose of HWD 114. However, providing encoded packets including the encoded depth data (e.g., left view depth data 122, right view depth data 124) along with the encoded set of left view frames 118 and encoded set of right view frames 120, increases the bandwidth needed to transmit the encoded packets (e.g., XR stream 116) over network 112 to HWD 114. Because additional bandwidth is needed to transmit encoded packets that include depth data, further bits cannot be devoted to increasing the quality of the encoded frames included in the encoded packets without risking packet delays or packet loss.

[0024] To this end, in embodiments, encoder 108 is configured to improve the quality of objects in a frame appearing closest to a user without further increasing thebandwidth demands of the encoded frame. For example, according to embodiments, encoder 108 is configured to encode one or more frames and corresponding depth data (e.g., left view frames 118, right view frames 120, left view depth data 122, right view depth data 124) using depth-based ROI encoding. To perform depth-based ROI encoding, server 102 includes depth-based region circuitry 106 configured to identify one or more depth-based ROIs in a frame, corresponding depth data, or both. For example, for each left view frame of the set of left view frames 118 and corresponding left view depth data 122 and for each right view frame of the set of right view frames 120 and corresponding right view depth data 124, depth-based region circuitry 106 is configured to identify one or more ROIs in a frame based on the depth data associated with the frame. As an example, for each graphics object in a frame, depth-based region circuitry 106 compares one or more depth values of the graphic objects as indicated by the depth data (e.g., left view depth data 122, right view depth data 124) of the frame to one or more depth thresholds 130 each including one or more predetermined values. According to embodiments, each depth threshold 130 includes a value indicating a range of depths. For example, a first depth threshold 130 includes a predetermined value indicating a first depth range (e.g., lowest depth value range), a second depth threshold 130 includes a predetermined value indicating a second depth value range that is higher (e.g., deeper) than the lowest depth range, and a third depth threshold 130 includes a predetermined value indicating a third depth value range that is higher (e.g., deeper) than the second depth value range. In embodiments, for example, depth values in a higher depth range indicate objects appear farther from a user in a scene when compared to depth values of lower depth. In embodiments, for example, depth thresholds 130 can include any number of predetermined values each indicating a respective depth value range.

[0025] In response to a graphics object of a frame having a depth value equal to or less than one or more depth thresholds 130 (e.g., equal to or less than one or more predetermined values of the depth thresholds 130), depth-based region circuitry 106 is configured to determine an ROI in the frame, corresponding depth data (e.g., depth map), or both based on a location of the graphics object. That is to say, an ROI of the frame, corresponding depth data, or both are identified in response to a graphics object of a frame having a depth value equal to or less than one or more depththresholds 130. For example, depth-based region circuitry 106 generates a bounding box representing an ROI around the graphics object. In embodiments, depth-based region circuitry 106 is configured to determine an ROI in the frame, corresponding depth data, or both based on two or more graphics objects that have similar depth values (e.g., depth values within a predetermined range). For example, in response to a first graphics object of a frame having a depth value equal to or less than one or more depth thresholds 130, depth-based region circuitry 106 determines an ROI in the frame, corresponding depth data, or both based on the location of the first graphics object and one or more similar graphics objects (e.g., graphics objects having depth values within a predetermined range from the depth values of the first graphics object). As an example, depth-based region circuitry 106 generates a bounding box representing an ROI around the first graphics object and one or more similar graphics objects. After determining an ROI in a frame or corresponding depth data, depth-based region circuitry 106 is configured to update a list of ROIs for the frame or depth data to include an identifier of the ROI (e.g., data indicating the ROI) and a location of the ROI within the frame or depth data (e.g., data indicating the portion of the frame or depth data represented by the ROI).

[0026] Further, in embodiments, for each ROI determined in a frame or depth data, depth-based region circuitry 106 is configured to determine a priority of the ROI. The priority of an ROI, for example, includes a value indicating how close graphics objects within the ROI appear to a user. For example, a higher priority of an ROI indicates that the graphics objects within the ROI appear closer to a user (e.g., have shallower depth values) while a lower priority of an ROI indicates that the graphics objects within the ROI appear farther to the user (e.g., have deeper depth values). To determine a priority of an ROI, depth-based region circuitry 106 is configured to determine the lowest depth threshold 130 equal to or greater than one or more depth values of the graphics objects within the ROI. That is to say, depth-based region circuitry 106 is configured to determine the depth threshold 130 including a value representing the lowest range of depths that is equal to or greater than one or more depth values of the graphics objects within the ROI. Based on the lowest depth threshold 130 equal to or greater than one or more depth values of the graphics objects within the ROI, depth-based region circuitry 106 determines the priority of the ROI. For example, in response to one or more depth values of the graphics objectswithin the ROI being equal to or less than a depth threshold 130 representing a lowest depth range, depth-based region circuitry 106 assigns a highest priority to the ROI as the graphics objects within the ROI appear closest to the user. As another example, in response to one or more depth values of the graphics objects within the ROI being equal to or less than a depth threshold 130 representing a second depth range higher than (e.g., deeper) than the first lowest depth range, depth-based region circuitry 106 assigns a second priority to the ROI lower than the highest priority. As yet another example, in response to one or more depth values of the graphics objects within the ROI being equal to or less than a depth threshold 130 representing a depth range higher than the second depth range, depth-based region circuitry 106 assigns a third priority to the ROI that is lower than the second priority. In this way, depthbased region circuitry 106 assigns higher priorities to ROIs including graphics objects that appear closer to the user and lower priorities to ROIs including graphics objects that appear farther from the user. After determining the priority of an ROI, depthbased region circuitry 106 updates the list of ROIs for the frame or depth data to indicate the priority of the ROI.

[0027] After depth-based region circuitry 106 has determined each ROI within a frame or depth data, depth-based region circuitry 106 provides the list of ROIs for a frame to encoder 108. That is to say, after depth-based region circuitry 106 has compared the depth values of each graphics objects in the frame or depth data to one or more depth thresholds 130, depth-based region circuitry 106 provides the list of ROIs for a frame to encoder 108. Using the list of ROIs for a frame, depth data, or both (e.g., left view frame 118, corresponding left view depth data 122, right view frame 120, corresponding right view depth data 124), encoder 108 then encodes the frame, depth data, or both. For example, based on the respective priority of an ROI indicated in the list of ROIs for a frame, encoder 108 modifies the frame-level QP of the frame or depth data to encode the ROI within the frame or depth data to produce, for example, an encoded ROI. As an example, in response to the list of ROIs of a frame or depth data indicating an ROI has a highest priority, the encoder 108 decreases the frame-level QP to a first QP (e.g., lowest QP) so as to encode the ROI with more detail than the frame-level QP. As another example, in response to the list of ROIs of a frame or depth data indicating an ROI has second priority that is lower than the highest priority, the encoder 108 modifies the frame-level QP to a secondQP higher than the first QP so as to encode the ROI with less detail than graphics objects in an ROI with the highest priority. As yet another example, in response to the list of ROIs of a frame or depth data indicating an ROI has a third priority lower than the second priority the encoder 108 modifies the frame-level QP to a third QP that is higher than the second QP so as to encode the ROI with less detail than graphics objects in an ROI with the highest priority or second priority.

[0028] Additionally, in embodiments, encoder 108 is configured to encode the remainder of the frame or depth data by increasing the frame-level QP. That is to say, for each area of the frame or depth data not included in an ROI identified by the list of ROIs for the frame, encoder 108 uses a QP (e.g., remainder QP) higher than the frame-level QP. By using a QP higher than the frame level QP for the remainder of the frame or depth data (e.g., higher than the respective QPs to encode the ROIs), encoder 108 encodes graphics objects having depth values greater than each depth threshold 130 (e.g., graphics objects that appear farthest from the user) using less bits such that the graphics objects in the remainder of the frame or depth data have less detail than the graphics objects within the depth-based ROIs in the frame or depth data. In this way, encoder 108 is configured to encode graphics objects of the frame or depth data appearing closest to the user using more bits and encode graphics objects of the frame appearing farthest to the user using less bits. As such, graphics objects in an ROI of the frame appearing closest to the user will be encoded with more detail (e.g., more bits) without increasing the overall bits used to encode the frame and corresponding depth data (e.g., left view frame 1 18, left view depth data 122, right view frame 120, right view depth data 124) due to the graphics objects in the frame and depth data appearing farthest from the user being encoded with less detail (e.g., less bits). That is to say, encoder 108 encodes the frame and corresponding depth data such that graphics objects in an ROI of the frame appearing closest to the user will be encoded with more detail (e.g., more bits) without increasing the overall bits used to encode the frame and corresponding depth data when compared to encoding the entirety of the frame and corresponding depth data using a frame-level QP.

[0029] Referring now to FIG. 2, an example depth-based ROI encoding operation200, in accordance with some embodiments. In embodiments, depth-based ROIencoding operation 200 first includes server 102 executing XR application 104. While executing XR application 104, server 102 is configured to generate a left view frame 215 that includes one or more rendered graphics objects and that represents a first view (e.g., left view) of an XR scene. According to some embodiments, left view frame 215 is one frame of the set of left view frames 118. Additionally, when executing XR application 104, server 102 is configured to generate a right view frame 225 that includes one or more rendered graphics objects and that represents a second view (e.g., right view) of the XR scene. In some embodiments, right view frame 225 is one frame of the set of right view frames 120. After generating left view frame 215 and right view frame 225, server 102 provides left view frame 215 and right view frame 225 to encoder 108. According to embodiments, while executing XR application 104, server 102 is configured to generate depth data for left view frame 215 (e.g., left view depth data 235) and right view frame 225 (e.g., right view depth data 245). The depth data of a frame (e.g., left view depth data 235, right view depth data 245) indicates the depth values (e.g., z-values) for each graphics object within that frame. For example, left view depth data 235 includes a depth map that indicates the depth values for each graphics object in left view frame 215. As another example, right view depth data 245 includes a depth map that indicates the depth values of each graphics object in right view frame 225. After generating left view depth data 235, right view depth data 245, or both, server 102 provides left view depth data 235, right view depth data 245, or both to encoder 108 to be encoded.

[0030] Additionally, in embodiments, in response to generating left view depth data 235, right view depth data 245, or both, example operation 200 includes server 102 providing left view depth data 235, right view depth data 245, or both to depth-based region circuitry 106. After receiving left view depth data 235, depth-based region circuitry 106 is configured to identify one or more ROIs (e.g., left view ROIs 201) within left view frame 215, left view depth data 235, or both based on left view depth data 235. For example, depth-based region circuitry 106 is configured to compare the depth values of each graphics object in left view frame 215 as indicated in left view depth data 235 to one or more depth thresholds 130. In response to one or more depth values of a graphics object in left view frame 215 being equal to or less than one or more depth thresholds 130, depth-based region circuitry 106 is configured to determine a left view ROI 201 in left view frame 215 based on the location of thegraphics object. As an example, depth-based region circuitry 106 generates a bounding box representing a left view ROI 201 around the graphics object. In some embodiments, depth-based region circuitry 106 is configured to determine a left view ROI 201 in left view frame 215 based on two or more graphics objects in left view frame 215 having similar depth values (e.g., depth values within a predetermined range). As an example, in response to a first graphics object of left view frame 215 having one or more depth values equal to or less than a depth threshold 130, depthbased region circuitry 106 generates a bounding box representing a left view ROI 201 around the first graphics object and one or more similar graphics objects (e.g., graphics objects having depth values within a predetermined range from the depth values of the first graphics object). After determining a left view ROI 201 , depthbased region circuitry 106 is configured to update a list of ROIs in left view 285 associated with left view frame 215. The list of ROIs in left view 285 includes, for example, data identifying one or more left view ROIs 201 and data indicating the location of one or more left view ROIs 201 within left view frame 215, left view depth data 235, or both.

[0031] Similarly, depth-based region circuitry 106 is configured to identify one or more ROIs (e.g., right view ROIs 203) within right view frame 225, depth data 245, or both based on right view depth data 245. As an example, depth-based region circuitry 106 compares the depth values of each graphics object in right view frame 225 (e.g., as indicated in right view depth data 245) to one or more depth thresholds 130. In response to one or more depth values of a graphics object in right view frame 225 being equal to or less than one or more depth thresholds 130, depth-based region circuitry 106 is configured to determine a right view ROI 203 in right view frame 225 based on the location of the graphics object. In some embodiments, depth-based region circuitry 106 is configured to determine a right view ROI 203 in right view frame 225 based on two or more graphics objects in right view frame 225 having similar depth values (e.g., depth values within a predetermined range). After determining a right view ROI 203, depth-based region circuitry 106 is configured to update a list of ROIs in right view 275 associated with right view frame 225. The list of ROIs in right view 275 includes, for example, data identifying one or more right view ROIs 203 and data indicating the location of one or more right view ROIs 203 within right view frame 225, right view depth data 245, or both.

[0032] In embodiments, example operation 200 includes depth-based region circuitry 106 determining a respective ROI priority 207 for each identified left view ROI 201 and each identified right view ROI 203. An ROI priority 207, for example, includes data (e.g., a value) indicating how close one or more graphics objects within an ROI (e.g., left view ROI 201 , right view ROI 203) appear to a user scene with a higher ROI priority 207 (e.g., higher value) indicating the graphics objects within an ROI are closer to the user and a lower ROI priority 207 (e.g., lower value) indicating the graphics objects within an ROI are farther from the user. To determine an ROI priority 207, depth-based region circuitry 106 is configured to determine the lowest depth threshold 130 (e.g., depth threshold 130 representing the shallowest range of depth values) equal to or greater than one or more depth values of the graphics objects within the ROI. Based on the lowest depth threshold 130 equal to or greater than one or more depth values of the graphics objects within the ROI, depth-based region circuitry 106 determines the priority of the ROI. As an example, based on one or more depth values of the graphics objects within the ROI being equal to or less than a depth threshold 130 representing a lowest (e.g., shallowest) depth range, depth-based region circuitry 106 assigns an ROI priority indicating a highest priority to the ROI as the graphics objects within the ROI appear closest to the user. As another example, based on one or more depth values of the graphics objects within the ROI being equal to or less than a depth threshold 130 representing a second depth range higher (e.g., deeper) than the lowest depth range, depth-based region circuitry 106 assigns an ROI priority 207 indicating a second priority to the ROI that is lower than the highest priority. As yet another example, based on one or more depth values of the graphics objects within the ROI being equal to or less than a depth threshold 130 representing a third depth value range that is higher (e.g., deeper) than the second depth value range, depth-based region circuitry 106 assigns an ROI priority 207 indicating a third priority to the ROI that is lower than the second priority. In response to determining the ROI priority 207 of a left view ROI 201 , depth-based region circuitry 106 updates the list of ROIs in left view 285 to indicate the ROI priority 207 of the left view ROI 201. Similarly, in response to determining the ROI priority 207 of a right view ROI 203, depth-based region circuitry 106 updates the list of ROIs in right view 275 to indicate the ROI priority 207 of the right view ROI 203.

[0033] Once depth-based region circuitry 106 has determined an ROI priority 207 for each left view ROI 201 of left view frame 215, depth-based region circuitry 106 provides the list of ROIs in left view 285 to encoder 108. Further, once depth-based region circuitry 106 has determined an ROI priority 207 for each right view ROI 203 of right view frame 225, depth-based region circuitry 106 provides the list of ROIs in right view 275 to encoder 108. In embodiments, example operation 200 includes encoder 108 encoding left view frame 215 and corresponding left view depth data 235 (e.g., a depth map) based on list of ROIs in left view 285 and encoding right view frame 225 and corresponding right view depth data 245 (e.g., a depth map) based on list of ROIs in right view 275. As an example, in some embodiments, encoder 108 is configured to encode each frame (e.g., left view frame 215, right view frame 225) and corresponding depth data (e.g., left view depth data 235, right view depth data 245) based on a respective frame-level QP 209. According to embodiments, encoder 108 is configured to determine a respective frame-level QP 209 for a frame or depth data based on, for example, a complexity of the frame or depth data. Further, in embodiments, encoder 108 is configured to modify the frame-level QP 209 of a frame or depth data when encoding one or more ROIs (e.g., left view ROI 201 , right view ROI 203) of a frame or depth data. To this end, encoder 108 is configured to modify the frame-level QP 209 of a frame or depth to encode an ROI of a frame or depth data based on a list of ROIs in the frame or depth data (e.g., list of ROIs in left view 285, list of ROIs in right view 275). As an example, when encoding a frame or depth data, encoder 108 first identifies one or more ROIs within the frame or depth data based on a respective list of ROIs in the frame or depth data. For each identified ROI, encoder 108 then determines the ROI priority 207 of the ROI from the list of ROIs in the frame or depth data. When encoding an identified ROI within the frame or depth data, encoder 108 adjusts the frame-level QPs of the frame or depth data based on the respective ROI priority 207 associated with the ROI.

[0034] As an example, in response to the ROI priority 207 of an ROI within a frame or depth data indicating a highest priority (e.g., indicating that the depth values of the graphic objects within the ROI are within a lowest depth range), encoder 108 is configured to decrease the frame-level QP 209 of the frame or depth data to a first level (e.g., lowest level QP) such that the graphics objects within the ROI are encoded with more bits (e.g., greater detail) than if the graphics objects wereencoded using the frame-level QP 209. Further, in response to the ROI priority 207 of an ROI within a frame or depth data indicating a second priority lower that is lower than the highest priority (e.g., indicating that the depth values of the graphic objects within the ROI are within a depth range higher than the lowest depth range), encoder 108 is configured to modify the frame-level QP 209 of the frame or depth data to a second level higher than the first level such that the graphics objects within the ROI are encoded with less bits (e.g., less detail) than if the graphics objects were encoded using the first level (e.g., lowest level) QP. Additionally, in some embodiments, in response to the ROI priority 207 of an ROI within a frame or depth data indicating a priority lower than the second priority (e.g., indicating that the depth values of the graphic objects within the ROI are within a depth range higher than the second and first depth ranges), encoder 108 is configured to modify the frame-level QP 209 of the frame or depth data to a third level higher than the first and second levels such that the graphics objects within the ROI are encoded with less bits (e.g., less detail) than if the graphics objects were encoded using the first level (e.g., lowest level QP) or second level. According to some embodiments, encoder 108 is configured to encode the remainder of a frame or depth data (e.g., left view frame 215, right view frame 225, right view depth data 245, left view depth data 235) by increasing the frame-level QP. That is to say, for each area of the frame or depth data not included in an ROI (e.g., left view ROIs 201 , right view ROIs 203) identified by the list of ROIs (e.g., list of ROIs in left view 285, list of ROIs in right view 275) for the fame or depth data, encoder 108 uses a QP (e.g., remainder QP) higher than the frame-level QP. In this way, encoder 108 is configured to encode graphics objects in an ROI of the frame or corresponding depth data that appear closer to the user using more bits without increasing the overall bits used to encode the frame or corresponding depth data (e.g., left view frame 215, left view depth data 235, right view frame 225, right view depth data 245) due to encoder 108 being configured to encode graphics objects in the frame or depth data that appear farther from the user using fewer bits.

[0035] After each left view ROI 201 of left view frame 215, corresponding left view depth data 235, or both are encoded based on list of ROIs in left view 285, encoder 108 produces encoded left view video frame 265, encoded left view depth data 267, or both. An encoded left view video frame 265, for example, includes encoded datarepresenting a left view frame 215 of the set of left view frames 118. Encoded left view depth data 267, for example, includes encoded data (e.g., an encoded depth map) corresponding to a left view frame 215 of the set of left view frames 118. Further, after each right view frame 225, corresponding right view depth data 245, or both are encoded based on list of ROIs in right view 275, encoder 108 produces encoded right view video frame 255, encoded right view depth data 257, or both. An encoded right view video frame 255, for example, includes encoded data representing a right view frame 225 of the set of left view frames 120. Encoded right view depth data 257, for example, includes encoded data (e.g., an encoded depth map) corresponding to a right view frame 225 of the set of left view frames 120. After producing encoded right view frame 255, corresponding encoded right view depth data 257, encoded left view frame 265, corresponding encoded left view depth data 267, or any combination thereof, encoder 108 then provides encoded right view frame 255, corresponding encoded right view depth data 257, encoded left view frame 265, corresponding encoded left view depth data 267, or any combination thereof to streaming circuitry 126. Streaming circuitry 126 is configured to then packetize encoded right view frame 255, corresponding encoded right view depth data 257, encoded left view frame 265, corresponding encoded left view depth data 267, or any combination thereof to produce XR stream 116 and provides XR stream 116 to HWD 114 via network 112.

[0036] Referring now to FIG. 3, an example frame 300 including depth-based ROIs is presented. According to some embodiments, example frame 300 includes a video frame that includes data representing the video of an XR scene, for example, left view frame 215 or right view frame 225. In other embodiments, for example, frame 300 includes a depth map that includes data representing the depth values of an XR scene, for example, left view depth data 235 or right view depth data 245. In embodiments, example frame 300 is encoded by encoder 108 using example operation 200.

[0037] Example frame 300 includes a first graphics object 302 having one or more depth values (e.g., z-values) within a first (e.g., shallowest) depth value range associated with a depth threshold 130. Such depth values within the first depth value range are demonstrated in FIG. 3 by first graphics object 302 having a first level (e.g.,darkest level) of shading. Due to first graphics object 302 having depth values within the lowest depth value range, for example, first graphics object 302 appears closer to a user than graphics objects having depth values within higher (e.g., deeper) depth value ranges. Further, according to embodiments, due to first graphics object 302 having depth values within the lowest depth value range (e.g., due to the first graphics object 302 having depth values equal to or less than a depth threshold 130 associated with the lowest depth value range), depth-based region circuitry 106 identifies a first ROI 316 around the first graphic object 302 having a ROI priority 207 indicating a first (e.g., highest) priority. Additionally, example frame 300 includes a second graphics object 304 having one or more depth values within a second depth value range associated with a depth threshold 130 that is greater than (e.g., deeper) than the first depth value range. Such depth values within the second depth value range are demonstrated in FIG. 3 by first graphics object 302 having a second level of shading lighter than the first level of shading. Because second graphics object 304 has depth values within the second depth value range, for example, second graphics object 304 appears farther to a user than the first graphics object 302. Also, in embodiments, due to the second graphics object 304 having depth values within the second depth value range (e.g., due to the first graphics object 302 having depth values equal to or less than a depth threshold 130 associated with the second depth value range), depth-based region circuitry 106 identifies a second ROI 318 around the second graphic object 304 having an ROI priority 207 indicating a second priority lower than the first priority.

[0038] In embodiments, example frame 300 further includes a third graphics object 306 and fourth graphics object 308 each having one or more depth values within a third depth value range associated with a depth threshold 130 that is greater than (e.g., deeper) than the second depth value threshold. Such depth values within the third depth value range are demonstrated in FIG. 3 by third graphics object 306 and fourth graphics object 308 each having a third level of shading lighter than the second level of shading. Due to the third graphics object 306 and fourth graphics object 308 each having depth values within the third depth value range, for example, third graphics object 306 and fourth graphics object 308 both appear farther to a user than the first graphics object 302 and second graphics object 304. Additionally, in embodiments, due to the third graphics object 306 and fourth graphics object 308each having depth values within the third depth value range (e.g., due to the third graphics object 306 and fourth graphics object 308 each having depth values equal to or less than a depth threshold 130 associated with the third depth value range), depth-based region circuitry 106 identifies a third ROI 320 around the third graphics object 306 and the fourth graphics object 308 having an ROI priority 207 indicating a third priority that is lower than the first priority and the second priority. Further, in some embodiments, example frame 300 includes fifth graphics object 310, sixth graphics object 312, and seventh graphics object 314 each including depth values that are not equal to or greater than one or more depth thresholds 130. Because fifth graphics object 310, sixth graphics object 312, and seventh graphics object 314 each include depth values that are not equal to or less than one or more depth thresholds 130, fifth graphics object 310, sixth graphics object 312, and seventh graphics object 314 are in a remainder 322 of example frame 300 outside of any ROI 316, 318, 320.

[0039] Referring now to FIG. 4, an example list of ROIs 400 in a frame is presented. According to embodiments, list of ROIs 400 is a list of ROIs in right view 275 or list of ROIs in left view 285 and is used by encoder 108 to encode left view frame 215 and left view depth data 235 or right view frame 225 and right view depth data 245, respectively during example operation 200. In embodiments, list of ROIs 400 includes a data structure (e.g., table, array, vector) including one or more entries 420. Though the example embodiment in FIG. 4 shows the example list of ROIs 400 including three entries (420-1 , 420-2, 420-N) representing an N number of entries, in other embodiments, example list of ROIs 400 can include any number of entries 420. In embodiments, each entry 420 is associated with a respective ROI in a frame (e.g., left view ROI 201 , right view ROI 203). To this end, each entry 420 includes a respective identifier (405-1 , 405-2, 405-N) with each identifier 405 including data identifying, labeling, or both a respective ROI within a frame. Additionally, for example, each entry 420 includes location data (410-1 , 410-2, 410-N) that includes the location of a respective ROI within a frame. For example, location data 410 includes data indicating a position within a frame where a respective ROI is located. According to some embodiments, each entry 420 includes priority data (415-1 , 415-2, 415-N) with each priority data 415 indicating the ROI priority 207 of a respective ROI.

[0040] Referring now to FIG. 5, an example method 500 for depth-based ROI encoding is presented, in accordance with some embodiments. In embodiments, method 500 includes server 102 rendering a frame (e.g., left view frame 215, right view frame 225) at block 505. For example, in embodiments, server 102 is configured to execute XR application 104. While executing XR application 104, server 102 is configured to generate a frame that includes one or more graphics objects representing at least a portion of an XR scene. After generating the frame, at block 510, server 102 is configured to determine the depth data (e.g., left view depth data 235, right view depth data 245) for each graphics object within the frame. That is to say, for each graphics object within the frame, server 102 generates one or more depth values associated with the graphics object. At block 515, depth-based region circuitry 106 is configured to compare the depth values of a first graphics object of the frame to one or more depth thresholds 130. In response to one or more depth values of the first graphics object being equal to or less than one or more depth thresholds 130, at block 520, depth-based region circuitry 106 determines an ROI (e.g., left view ROI 201 , right view ROI 203) within the frame based on a location of the first graphics object. For example, depth-based region circuitry 106 generates a bounding box representing an ROI around the first graphics object. As another example, depthbased region circuitry 106 generates a bounding box representing the ROI around the first graphics object and one or more similar graphics objects in the frame (e.g., graphics objects having depth values within a predetermined threshold from the depth values of the first graphics object). Once depth-based region circuitry 106 has determined an ROI in the frame, depth-based region circuitry 106 updates a list of ROIs (e.g., list of ROIs in right view 275, list of ROIs in left view 285) with an identifier for the determined ROI, a location of the determined ROI within the frame, or both.

[0041] After determining the ROI in the frame, at block 525, depth-based region circuitry 106 is configured to determine a priority (e.g., ROI priority 207) of the ROI. To this end, for example, based on the lowest depth threshold 130 equal to or greater than one or more depth values of the graphics objects within the ROI, depth-based region circuitry 106 determines the priority of the ROI. As an example, based on one or more depth values of the graphics objects within the ROI being equal to or less than a depth threshold 130 representing a lowest (e.g., shallowest) depth range, depth-based region circuitry 106 assigns an ROI priority indicating a highest priorityto the ROI as the graphics objects within the ROI appear closest to the user. As another example, based on one or more depth values of the graphics objects within the ROI being equal to or less than a depth threshold 130 representing a second depth range higher (e.g., deeper) than the lowest depth range, depth-based region circuitry 106 assigns an ROI priority indicating a second priority to the ROI that is lower than the highest priority. At block 530, after determining the priority of the ROI, depth-based region circuitry 106 updates the list of ROIs with the priority of the ROI. At block 535, depth-based region circuitry 106 determines whether an ROI determination has been made for each graphics object within the frame. That is to say, depth-based region circuitry 106 determines whether one or more depth values of each graphics object within the frame have been compared to a depth threshold 130. If depth values of one or more graphics objects within the frame have not been compared to one or more depth thresholds 130, at block 540, depth-based region circuitry moves on to a next graphics object within the frame and, at block 515, compares the one or more depth values of the next graphics object to one or more depth thresholds 130. If depth values of all the graphics objects within the frame have been compared to one or more depth thresholds 130, at block 545, depth-based region circuitry 106 provides the list of ROIs to encoder 108.

[0042] Based on the list of ROIs, encoder 108 is configured to encode the frame, corresponding depth data (e.g., depth map), or both. For example, for each ROI identified in the list of ROIs, encoder 108 determines the priority of the ROI from the list of ROIs. When encoding an ROI identified in the list of ROIs within the frame, corresponding depth data, or both, encoder 108 adjusts the frame-level QP (e.g., frame-level QP 209) of the frame based on the priority associated with the ROI to encode the ROI. For example, in response to the priority of an ROI indicating a highest priority (e.g., indicating that the depth values of the graphic objects within the ROI are within a lowest depth range), encoder 108 is configured to decrease the frame-level QP to a first level (e.g., lowest level QP) such that the graphics objects within the ROI are encoded with more bits (e.g., greater detail) than if the graphics objects were encoded using the frame-level QP. Further, in response to the ROI priority 207 of an ROI within a frame indicating a second priority lower that is lower than the highest priority (e.g., indicating that the depth values of the graphic objects within the ROI are within a depth range higher than the lowest depth range), encoder108 is configured to modify the frame-level QP to a second level higher than the first level such that the graphics objects within the ROI are encoded with less bits (e.g., less detail) than if the graphics objects were encoded using the first level (e.g., lowest level) QP. Additionally, encoder 108 is configured to encode the remainder of the frame by increasing the frame-level QP. That is to say, for each area of the frame not included in an ROI identified in the list of ROIs, encoder 108 uses a QP higher than the frame-level QP to encode these areas.

[0043] In some embodiments, certain aspects of the techniques described above may be implemented by one or more processors of a processing system executing software. The software comprises one or more sets of executable instructions stored or otherwise tangibly embodied on a non-transitory computer-readable storage medium. The software can include the instructions and certain data that, when executed by the one or more processors, manipulate the one or more processors to perform one or more aspects of the techniques described above. The non-transitory computer-readable storage medium can include, for example, a magnetic or optical disk storage device, solid-state storage devices such as Flash memory, a cache, random access memory (RAM), or other non-volatile memory device or devices, and the like. The executable instructions stored on the non-transitory computer-readable storage medium may be in source code, assembly language code, object code, or other instruction format that is interpreted or otherwise executable by one or more processors.

[0044] A computer-readable storage medium may include any storage medium, or combination of storage media, accessible by a computer system during use to provide instructions and / or data to the computer system. Such storage media can include but is not limited to, optical media (e.g., compact disc (CD), digital versatile disc (DVD), Blu-Ray disc), magnetic media (e.g., floppy disc, magnetic tape, or magnetic hard drive), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or Flash memory), or microelectromechanical systems (MEMS)-based storage media. The computer- readable storage medium may be embedded in the computing system (e.g., system RAM or ROM), fixedly attached to the computing system (e.g., a magnetic hard drive), removably attached to the computing system (e.g., an optical disc or UniversalSerial Bus (USB)-based Flash memory) or coupled to the computer system via a wired or wireless network (e.g., network accessible storage (NAS)).

[0045] Note that not all of the activities or elements described above in the general description are required, that a portion of a specific activity or device may not be required, and that one or more further activities may be performed, or elements included, in addition to those described. Still further, the order in which activities are listed is not necessarily the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, one of ordinary skill in the art appreciates that various modifications and changes can be made without departing from the scope of the present disclosure as set forth in the claims below. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present disclosure.

[0046] Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and any feature(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature of any or all the claims. Moreover, the particular embodiments disclosed above are illustrative only, as the disclosed subject matter may be modified and practiced in different but equivalent manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design herein shown, other than as described in the claims below. It is therefore evident that the particular embodiments disclosed above may be altered or modified and all such variations are considered within the scope of the disclosed subject matter. Accordingly, the protection sought herein is as set forth in the claims below.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method comprising: based on a first depth value of a graphics object in a frame being equal to or less than a depth threshold, identifying a region of interest (ROI) in the frame; and encoding, at an encoder, the ROI using a first quantization parameter (QP) associated with the depth threshold.

2. The method of claim 1 , further comprising: determining a priority associated with the ROI based on the depth threshold.

3. The method of claim 2, further comprising: selecting the first QP based on the priority associated with the ROI.

4. The method of any of claims 1 to 3, further comprising: encoding a remainder of the frame using a remainder QP, wherein the remainder of the frame omits the ROI and the remainder QP is different from the first QP.

5. The method of claim 4, wherein the first QP is lower than the remainder QP.

6. The method of any of claims 1 to 5, further comprising: comparing the depth value of the graphics object to a plurality of depth thresholds, wherein the plurality of depth thresholds includes the depth threshold.

7. The method of claim 6, wherein each depth threshold of the plurality of depth thresholds is associated with a respective depth value range.

8. The method of any of claims 1 to 7, further comprising: providing a list of ROIs identifying the ROI and a priority associated with the ROI to the encoder.

9. The method of any of claims 1 to 8, further comprising: based on a second depth value of a second graphics object in the frame, identifying a second ROI in the frame; and encoding, at an encoder, the second ROI using a second QP based on the second depth value of the second graphics object.

10. The method of claim 9, wherein the first QP is lower than the second QP.11 . The method of any of claims 1 to 10, further comprising: transmitting, by a network, an encoded frame with the encoded ROI to a head- worn device (HWD).

12. The method of any of claims 1 to 11 , wherein the frame includes a depth map.

13. A computer-implemented method, comprising: identifying a plurality of regions of interest (ROIs) in a frame based on depth values of a plurality of graphics objects in the frame; determining, for each ROI of the plurality of ROIs in the frame, a respective priority based on depth values of one or more graphics objects of the plurality of graphics objects within a corresponding ROI; and encoding each ROI of the plurality of ROIs using a respective quantization parameter (QP) based on the respective priority of the corresponding ROI.

14. The method of claim 13, further comprising: encoding a first ROI of the plurality of ROIs using a first QP based on a first priority associated with the first ROI and a second ROI of the plurality of ROIs using a second QP based on a second priority associated with the second ROI.

15. The method of claim 14, wherein the first priority represents a higher priority than the second priority.

16. The method of claim 15, wherein the first QP is lower than the second QP.

17. The method of any of claims 13 to 16, further comprising: encoding a remainder of the frame using a remainder QP, wherein the remainder of the frame omits the plurality of ROIs and the remainder QP is higher than the respective QPs used to encode each ROI of the plurality of ROIs.

18. The method of any of claims 13 to 17, further comprising: comparing the depth values of the plurality of graphics object to one or more depth thresholds, wherein a respective ROI of the plurality of ROIs is identified in response to one or more depth values of a graphics object of the plurality of graphics objects within the respective ROI being equal to or less than the one or more depth thresholds.

19. The method of any of claims 13 to 18, further comprising: transmitting, by a network, an encoded frame with the encoded plurality of ROIs to a head-worn device (HWD).

20. The method of any of claims 13 to 19, wherein the frame includes a depth map.21 . A system comprising: a depth-based region circuitry; and an encoder, wherein the system is configured to perform the method of any of claims 1 to 20.

22. A system comprising: a head-worn device (HWD) configured to receive an encoded frame encoded according to the method of any of claims 1 to 20; and a decoder configured to decode the encoded frame.

Images