Sort segments of an image for encoding and sending to a display device

Abstract

The present disclosure relates to sorting segments of an image for encoding and transmitting to a display device. Methods, apparatus, systems, and articles are disclosed for sorting multiple segments of an image for encoding and wirelessly transmitting to a display device. The apparatus includes a segment sorter for arranging the segments in an encoding order for encoding. The encoding order is different from a scan line order, and the encoding order causes a center segment of the multiple segments to occupy a first position in the encoding order. The first position is before a second position that the center segment occupies in the scan line order, and the center segment corresponds to the center of the image. The apparatus further includes: an encoder for encoding the segments in the encoding order; and a wireless transmitter for transmitting the encoded segments to the display device in the encoding order in which the encoded segments are encoded.

Description

Technical Field

[0001] The present disclosure generally relates to virtual reality systems, and more particularly, to sorting segments of an image for encoding and transmission to a display device. Background Art

[0002] Recent advancements in virtual reality technology have enabled virtual reality systems to provide a greatly improved user experience. As a result, virtual reality systems are becoming increasingly popular and will soon become a major product for home entertainment systems, theme park attractions, work-related training tools, educational methods, and the like. Summary of the Invention

[0003] Embodiments of the present disclosure provide an apparatus for sorting multiple segments of an image for encoding and transmission to a display device. The apparatus includes: a segment sorter for arranging the segments in an encoding order for encoding, the encoding order being different from the scan line order, and the encoding order being such that a central segment of the multiple segments occupies a first position in the encoding order, the first position being before a second position that the central segment occupies in the scan line order, the central segment corresponding to the center of the image; an encoder for encoding the segments in the encoding order; and a wireless transmitter for transmitting the encoded segments to the display device in the encoding order in which the encoded segments are encoded.

[0004] Embodiments of the present disclosure also provide one or more non-transitory machine-readable storage media including machine-readable instructions that, when executed, cause one or more processors to perform at least the following operations: arranging a set of image regions in a non-scan line order; encoding the image regions in a non-scan line order, using the non-scan line order such that a central image region of the set of image regions is encoded earlier than the central image region would be encoded in the scan line order, and such that peripheral image regions of the set of image regions are encoded later than the peripheral image regions would be encoded in the scan line order; and providing the encoded image regions to a transmitter in a non-scan line order for transmission to a display unit.

[0005] Embodiments of the present disclosure also provide a device for sorting multiple image tiles. The device includes: means for positioning multiple image tiles in a non-scan line order, the multiple image tiles including a central image tile corresponding to the center of the image, the central image tile occupying an encoding position earlier than the position that the central image tile occupies in the scan line order in the non-scan line order; means for encoding the multiple image tiles in a non-scan line order; and means for transmitting the encoded image tiles to a display in a non-scan line order.

[0006] Embodiments of the present disclosure also provide a method for sorting multiple segments of an image for encoding and transmitting to a display device, the method comprising: arranging the segments in an encoding order for encoding, the encoding order being different from the scan line order, and the encoding order being such that a central segment of the multiple segments occupies a first position in the encoding order, the first position being before a second position that the central segment occupies in the scan line order, and the central segment corresponding to the center of the image; encoding the segments in the encoding order; and transmitting the encoded segments to the display device in the encoding order in which the encoded segments are encoded. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 An example virtual reality system is shown in accordance with the teachings of the present disclosure, the example virtual reality system including an example virtual reality console for sorting segments of an image for encoding and transmitting to an example head-mounted display of the virtual reality system.

[0008] Figure 2 is Figure 1 an example virtual reality console of Figure 1 and a block diagram of an example head-mounted display unit of

[0009] Figure 3 is an example image divided into nine image segments.

[0010] Figure 4A is an example image segment queue including image segments arranged in scan line order.

[0011] Figure 4B is an example image segment queue including image segments arranged in a first enhanced encoding order.

[0012] Figure 5 is a table showing the scan line decoding order, an example first enhanced encoding order, and the time period difference between the encoding of each image segment and the decoding of each image segment.

[0013] Figure 6A is a graph showing an example decoding success rate achieved for each image segment when using the scan line encoding order.

[0014] Figure 6B is a graph showing an example decoding success rate achieved for each image segment when using the first enhanced encoding order to encode / transmit the image segments.

[0015] Figure 7 is an example image segment queue including image segments arranged in a second enhanced encoding order.

[0016] Figure 8A table showing the scan line decoding order, an example of the second enhanced encoding order, and the time period difference between the encoding of each image segment and the decoding of each image segment.

[0017] Figure 9 A graph showing an example of the decoding success rate achieved for each image segment when encoding / transmitting image segments using the second enhanced encoding order.

[0018] Figure 10 Represents what can be executed to achieve Figure 1 and / or Figure 2 A flowchart of example machine-readable instructions for an example virtual reality console.

[0019] Figure 11 Represents what can be executed to achieve Figure 1 and / or Figure 2 A flowchart of example machine-readable instructions for an example head-mounted display unit.

[0020] Figure 12 Is constructed to execute Figure 10 Instructions to achieve Figure 1 and / or Figure 2 A block diagram of an example processing platform for an example virtual reality console.

[0021] Figure 13 Is constructed to execute Figure 11 Instructions to achieve Figure 1 and / or Figure 2 A block diagram of an example processing platform for an example head-mounted display unit.

[0022] The figures are not drawn to scale. In general, the same reference numerals will be used throughout the figures and the accompanying written description to refer to the same or like parts. Detailed Description

[0023] Due to recent advancements in underlying technologies that have greatly improved the user experience, it is predicted that virtual reality systems will become a major product in home entertainment. In current wireless virtual reality systems, a virtual reality console communicates with a head-mounted display worn by a user via a wireless link. The information transmitted via the wireless link includes images that give the user / wearer the illusion of occupying virtual reality when displayed on the screen of the head-mounted display.

[0024] In traditional virtual reality systems, as the orientation of a head-mounted display changes, the bandwidth / actual throughput of the wireless link over which a virtual reality console communicates with the head-mounted display changes dynamically. Thus, as the user / wearer's head moves, the orientation of the head-mounted display changes and the quality of the wireless link changes. Unfortunately, in some cases, the change in orientation causes the quality of the wireless link to degrade sufficiently that the transmission of image data is adversely affected. In some cases, one or more segments / regions / portions of an image frame displayed on the head-mounted display are dropped during transmission, and / or the transmission of one or more image frames is delayed. A decoder of the head-mounted display unit decodes the received image segments to render an image frame (also referred to as an image) on the head-mounted display. When an image segment corresponding to the current image is dropped or received too late to be decoded in time, the decoder causes image data of the corresponding image segment from a previously displayed image to be rendered on the display. Additionally or alternatively, one or more other error mitigation, error handling algorithms may be used to compensate for the missing image segments. While such error correction / mitigation techniques help to mitigate the adverse effects caused by dropped or delayed image segments, the dropped / delayed segments may still have an adverse impact on the user experience. In some such instances, the image rendered using the current and previously transmitted image segments may have artifacts discernible to the human eye and sufficient to impair the illusion of the virtual reality experience. Thus, there has remained a need to improve the manner in which image segments are encoded / transmitted to the head-mounted display unit.

[0025] The virtual reality techniques disclosed herein greatly improve the throughput of the wireless link between a virtual reality console and a head-mounted display by arranging image segments in a non-scanline order for encoding and transmission. In traditional systems, an image to be sent from a console to a head-mounted display is segmented / partitioned into smaller segments (also referred to herein as portions, regions, partitions, etc.). The segmentation / partitioning causes the image to be divided into a set of rows, which are further divided into columns, thereby forming a grid of image segments. The image segments are numbered starting from the top row, and each segment in the top row is assigned a number from left to right until the end of the top (first) row is reached. Next, numbering starts again at the leftmost segment of the second row. Numbering proceeds in this manner (row by row, top to bottom, and left to right within a row) until all image segments are assigned a number. This is referred to herein as the scanline order / sorting of image segments because the image segments are sorted in the manner in which an image is scanned (e.g., top to bottom, left to right). Thus, in an image divided into nine segments included in three rows and three columns, the three segments of the top row are numbered one, two, and three from left to right. The three segments of the second row are numbered four, five, and six from left to right, and the three segments of the third row are numbered seven, eight, and nine from left to right.

[0026] In a traditional virtual reality system, numbered segments are encoded (and then sent) in scan line order. For example, segment 1 is encoded first, then segments 2, 3, 4, 5, 6, 7, 8, and finally segment 9. The encoded segments are also sent by the virtual reality console to the head-mounted display in scan line order (e.g., from 1 to 9).

[0027] However, the center of the image is typically more visually important to the user as it holds the user's gaze. Additionally, due to the design of the lens included in the head-mounted display, the content fidelity is greatest at the center of the image. Thus, in the above example of a nine-segment image, image segment 5, which includes content located at the center of the image, is more important (compared to other image segments) for the integrity of the image displayed at the head-mounted display unit. However, as described above, in a traditional virtual reality system, segment 5 is encoded and transmitted to the head-mounted display after segments 1, 2, 3, and 4.

[0028] As described above, the movement of the head of the user / wearer of the head-mounted display unit changes the orientation of the head-mounted display unit, thereby degrading the condition of the wireless link. In some cases, the reorientation of the head-mounted display unit will trigger a beamforming operation by which the wireless transmitter of the virtual reality console attempts to realign the wireless beam (through which the wireless link communicates with the head-mounted display unit). As a result, the transmitter of the virtual reality console stops sending image segments until the beam is realigned. This transmission pause operation can last for several milliseconds, thus delaying the transmission of image segments. Unfortunately, the delay caused by beam realignment may cause one or more image segments to arrive at the decoder of the head-mounted display unit too late, which may in turn cause the transmission of the image segment to fail. Additionally, referring to the above example of a nine-segment image, since image segment 5 includes content located at the center of the image, the failure of the transmission of image segment 5 may have a greater impact on the integrity of the resulting image rendered at the head-mounted display (compared to the transmission failure of other segments).

[0029] The virtual reality system disclosed herein improves the user experience by enabling the display of richer and higher-quality image / video content. The display of richer and higher-quality image / video content is achieved by arranging image segments in a non-scanline order for encoding and transmission to a head-mounted display. In some examples, the image segments are sorted in a manner that provides a greater amount of time for an image segment having central image content to reach the head-mounted display unit. In this way, higher-priority image segments that include more important image content are more likely to be received (e.g., not discarded) at the head-mounted display and received within an amount of time sufficient to allow decoding (e.g., without unacceptable latency). As a result, the images rendered by the head-mounted display of the disclosed virtual reality system have fewer anomalies and provide a richer and more immersive virtual reality experience to the user.

[0030] The techniques disclosed herein are applicable not only to virtual reality systems but also to mixed reality and augmented reality systems. In fact, the techniques disclosed herein can be used to improve any system that wirelessly transmits images / image segments to a remote display unit.

[0031] Figure 1 is an example implementation of an example virtual reality system 100 that includes an example virtual reality console 110 that wirelessly communicates with an example virtual reality head-mounted display 120 worn by a user 130. In some examples, the virtual reality console 110 wirelessly transmits image data to the head-mounted display 120, which uses the image data to render a corresponding image for the user 130 to view.

[0032] Figure 2 is Figure 1 a block diagram of an example implementation of an example virtual reality console 110 and an example head-mounted display unit 120. In some examples, the virtual reality console 110 includes an example image processor 202, an example splitter 206, an example segment queue 208, and an example segment sorter 210, an example encoder 212, and an example wireless transmitter 214 that communicate with an example storage device (e.g., memory) 204 that stores software instructions and / or image / video data. In some examples, the example head-mounted display unit 120 includes an example wireless receiver 216, an example decoder storage device 217, an example decoder 218, and an example display / screen / lens 220 on which an image is rendered.

[0033] In some examples, example image processor 202 processes, generates, and / or selects image frames to be transmitted to the head-mounted display 120 for rendering there. In some examples, image processor 202 identifies to example image segmenter 206 the locations in example storage device 204 where the image frames are stored. Image segmenter 206 responds to the location information by segmenting / partitioning / dividing the image frames into a set of image segments (also referred to herein as image parts, image regions, image partitions, image tiles, etc.).

[0034] Figure 3 An example image frame 300 divided into a set of image segments is shown. In some examples, example image segmenter 206 segments / divides the image frame 300 into a set of rows (e.g., top row 304, middle row 306, and bottom row 308) and a set of columns (e.g., first column 310, second column 312, and third column 314), thereby forming a grid that defines the image segments, to segment / divide the image frame into image segments. In some examples, the image can be divided into any number of rows / columns or divided in any desired manner. For reference purposes, the image segments are referenced using numbers that ascend row-by-row starting from the top row and increment in a left-to-right manner within each row. As a result, the image segments in the top row are referred to as the first image segment 318, the second image segment 316, and the third image segment 320. Similarly, the image segments in the middle row are referred to as the fourth image segment 322, the fifth image segment 324, and the sixth image segment 326. And, the image segments in the bottom row are referred to as the seventh image segment 328, the eighth image segment 330, and the ninth image segment 332.

[0035] For reference purposes, image segments 316 - 332 are also described as being associated with one of three layers. The first layer includes the most central image segment (e.g., fifth image segment 324), the second layer includes the image segments that share a boundary with the most central, fifth image segment (e.g., second image segment 318, fourth image segment 322, sixth image segment 326, and eighth image segment 330), and the third layer includes the image segments that are peripheral to the most central fifth image segment 324 (e.g., first image segment 316, third image segment 320, seventh image segment 328, and ninth image segment 332). As can be seen from Figure 3 the peripheral image segments include the image content located at the corners of the image frame 300.

[0036] In some examples, the example splitter 206 places the image segments in a traditional (also known as scanline) order into the example segment queue 208. In some examples, the scanline is the order in which traditional virtual reality systems encode / transmit image segments. Thus, the first segment 316 occupies the first position in the scanline order and in the segment queue 208, the second segment 318 occupies the second position in the scanline order and in the segment queue 208, the third segment 320 occupies the third position in the scanline order and in the segment queue 208, and so on.

[0037] In some examples, when the image segments are placed in the segment queue 208 in scanline order, the example image splitter 206 notifies the example segment sorter 210. In response to this notification, the segment sorter 210 rearranges the image segments in the segment queue 208 according to a first example enhanced encoding order.

[0038] Figure 4A Illustrates image segments placed in the example segment queue 208 in scanline order 410, and Figure 4B Illustrates the image segments in a first example enhanced encoding order 420. In some examples, the example segment sorter 210 rearranges the image segments in the segment queue 208 into a first enhanced encoding order such that the fifth image segment 324 occupies the first position, the second image segment 318 occupies the second position, the first image segment 316 occupies the third position, the fourth image segment 322 occupies the fourth position, the third image segment 320 occupies the fifth position, the sixth image segment 326 occupies the sixth position, the eighth image segment 330 occupies the seventh position, the seventh image segment 328 occupies the eighth position, and the ninth image segment 332 occupies the ninth position. After the segment sorter 210 has rearranged the image segments into a non-scanline order, the segment sorter 210 notifies the example encoder 212, which extracts and encodes the image segments from the segment queue 208 in the first enhanced encoding order. The encoder 212 provides the encoded image segments to the example wireless transmitter 214 in the first enhanced encoding order. The wireless transmitter 214 transmits the encoded segments to the example head-mounted display unit 120 in the first enhanced encoding order.

[0039] The example wireless receiver 216 of the example head-mounted display unit 120 receives the image segments in a first example enhanced encoding order. In some examples, the example decoder 218 decodes the image segments in the above-mentioned scanline order (e.g., the first decodes the first image segment 316, the second decodes the second image segment 320, the third decodes the third image segment 322, and so on). The decoder 218 provides the decoded image segments to the screen / display / lens 220 of the head-mounted display unit 120 in scanline order for rendering thereon.

[0040] Figure 5 shows the scan line order in which an example decoder 218 decodes image segments (see Figure 5 the first row 510 of ) and the first example enhanced encoding order in which the image segments are encoded / sent to the head-mounted display unit 120 (see Figure 5 the second row 520 of ). Since the scan line order and the first enhanced encoding order are different, Figure 5 also identifies the time difference between the encoding / sending of each image segment and the decoding of each image segment (see Figure 5 the third row 530 of ). Figure 5 The time difference of the third row 530 of is expressed as a multiple of the time period "t0", where the image segment is expected to reach the example decoder 218 (see Figure 2 ) within this time period "t0" after being sent by the example wireless transmitter 214 (see Figure 2 ). A negative number is used to indicate the instance where the sending time of the image segment occurs after the set decoding time.

[0041] As Figure 5 shown in the first row 510 of, the scan line decoding order indicates that the first decodes the first image segment 316, the second decodes the second image segment 318, the third decodes the third image segment 320, and so on until all nine image segments are decoded and displayed. As Figure 5 shown in the second row 520 of, the first enhanced encoding order indicates that the first encodes the fifth image segment 324, the second encodes the second image segment 318, the third encodes the first image segment 316, the fourth encodes the fourth image segment 322, the fifth encodes the third image segment 320, the sixth encodes the sixth image segment 326, the seventh encodes the eighth image segment 328, the eighth encodes the seventh image segment 330, and the ninth encodes the ninth image segment 332. Figure 5 The third row 530 of shows the effect of decoding image segments using the scan line order when using the first enhanced encoding order to encode / send each image segment.

[0042] As Figure 5 shown in the third row 530 of, when using the first enhanced encoding order to encode / send segments of an image, compared to the example decoder 218 (see Figure 2) In comparison with preparing to decode the fifth image segment 324, the fifth image segment 324 was encoded / transmitted four time periods (4t0) earlier. In contrast, when encoding / transmitting using the scan line order, the fifth image segment 324 was encoded / transmitted and decoded in the same time period. Therefore, compared with encoding and transmitting image segments using the scan line order, the first enhanced encoding order provides additional time for the fifth image segment 324 to reach the decoder 218, thus increasing the likelihood of successfully decoding the fifth image segment 324. In contrast, when encoding / transmitting using the first enhanced encoding order, the second image segment 318 was encoded / transmitted in the same time period as the decoder 218 would be preparing to decode the second image segment 318. The same is true when encoding / transmitting using the scan line order. Therefore, for the second image segment 318, using the first enhanced encoding order has the same success likelihood as using the scan line order.

[0043] Still referring to Figure 5 the third row 530, when using the first enhanced encoding order, the first image segment 316 was encoded / transmitted two time periods later compared with the time period when the decoder 218 was preparing to start decoding the first image segment 316. In contrast, when encoding / transmitting using the scan line order, the first image segment 316 was encoded / transmitted and decoded in the same time period. Therefore, compared with encoding using the scan line order, using the first enhanced encoding order has a negative impact on the likelihood of successfully decoding the first image segment 316. When encoding / transmitting using the first enhanced encoding order, the fourth image segment 322 was encoded / transmitted and decoded in the same time period. Similarly, when encoding / transmitting using the scan line order, the fourth image segment 322 was encoded / transmitted and decoded in the same time period. Therefore, compared with using the scan line order, using the first enhanced encoding order does not change the likelihood of successfully decoding the fourth image segment 322. When encoding / transmitting using the first enhanced encoding order, the third image segment 320 was encoded / transmitted two time periods later compared with the time when the decoder 218 was preparing to start decoding the third image segment 320. When encoding / transmitting using the scan line order, the third image segment 316 was encoded / transmitted and decoded in the same time period. Therefore, compared with encoding using the scan line order, using the first enhanced encoding order has a negative impact on the likelihood of successfully decoding the third image segment 320.

[0044] Still referring to Figure 5, when encoding / transmitting using the first enhanced encoding order, the sixth image segment 326 is encoded / transmitted and decoded in the same time period. The same is true for the sixth image segment 326 when using the scan line order. Therefore, compared with encoding using the scan line order, using the first enhanced encoding order does not change the possibility of successfully decoding the sixth image segment 326. In contrast, when encoding / transmitting using the first enhanced encoding order, the eighth image segment 330 is encoded / transmitted one time period earlier compared to the time when the example decoder 218 (see Figure 2 ) is ready to start decoding the eighth image segment 330. Conversely, when encoding / transmitting using the scan line order, the eighth image segment 316 is encoded / transmitted and decoded in the same time period. Therefore, compared with encoding using the scan line order, using the first enhanced encoding order allows the eighth image segment 330 more time to reach the decoder 218, thus having a positive impact on the possibility of successfully decoding the eighth image segment 330.

[0045] Still referring to Figure 5 , when encoding / transmitting using the first enhanced encoding order, the seventh image segment 328 is encoded / transmitted one time period later compared to the time when the example decoder 218 (see Figure 2 ) is ready to start decoding the seventh image segment 328. Conversely, when encoding / transmitting using the scan line order, the seventh image segment 316 is encoded / transmitted and decoded in the same time period. Therefore, compared with the amount of time provided when using the scan line order, using the first enhanced encoding order allows the seventh image segment 328 less time to reach the decoder 218. As a result, compared with encoding using the scan line order, using the first enhanced encoding order has a negative impact on the possibility of successfully decoding the seventh image segment 328.

[0046] When encoding / transmitting using the first enhanced encoding order, the ninth image segment 332 is encoded / transmitted and decoded in the same time period. The same is true when using scan line encoding. Therefore, compared with encoding using the scan line order, using the first enhanced encoding order does not change the possibility of successfully decoding the ninth image segment 332.

[0047] To further quantify the effects of using the first enhanced encoding order versus the scan line order for encoding / transmitting image segments, Figure 6A a graph is presented showing an example success rate of receiving and decoding each of nine image segments when encoding / transmitting (and decoding) the nine image segments in scan line order. In the example shown, when encoding / transmitting and decoding the image segments using the scan line order, the success rate of receiving and decoding for each of the nine image segments is 90%. Fig. Figure 6Bpresents a graph showing an example success rate of receiving and decoding each of nine image segments when encoding / transmitting image segments in a first example enhanced encoding order and decoding in a scan line order. In the example shown, as Figure 6B shown, when using the first enhanced encoding order, the success rate of receiving and decoding the fifth image segment is 100%. When using the first enhanced encoding order, the success rate of receiving and decoding the second, fourth, sixth, and ninth image segments is 90%. This is expected because the second, fourth, sixth, and ninth image segments occupy the same positions in both the conventional order and the modified order. When using the first enhanced encoding order, the success rate of receiving and decoding the first, third, and seventh image segments is 80% or lower. As Figure 5 shown in the table of, this decrease in success rate from 90% to 80% or lower is caused by the delay in the transmission of the first, third, and seventh image segments in the first enhanced encoding order relative to the decoding time of the first, third, and seventh image segments. Thus, for the fifth image segment that includes the center of the image, the success rate achieved when using the first enhanced encoding order is improved, and for the eighth image segment (which includes non-peripheral image content), the success rate achieved when using the first enhanced encoding order is also improved. Because the fifth and eighth image segments include non-peripheral image content that is more important for the rendering of the image, the increased likelihood of successfully decoding these segments provides an important benefit to the rendering of the image composed of the image segments. Although the success rate of the first, third, and seventh image segments drops from 90% to 80% or lower, each of the first, third, and seventh image segments includes peripheral image content that is less important for rendering, and thus, this is tolerable.

[0048] Figure 7Shows an example content of the example segment queue 208 when arranging example image segments in a second enhanced encoding order. In some examples, for any image segment, a decoding success rate below 80% is not tolerated. In some such examples, the example segment sorter 210 can alternatively be configured to arrange the image segments in the example segment queue 208 using a second example enhanced encoding order. In some such examples, in response to a notification from the example image splitter 206 indicating that a segment has been placed in the queue 208, the segment sorter 210 rearranges the image segments into a second enhanced encoding order. In some examples, when the segment sorter 210 arranges the image segments in a second enhanced modification order, the second image segment 318 occupies the first position in the segment queue 208, the first image segment 316 occupies the second position, the fifth image segment 324 occupies the third position, the third image segment 320 occupies the fourth position, the fourth image segment 322 occupies the fifth position, the sixth image segment 326 occupies the sixth position, the eighth image segment 330 occupies the seventh position, the seventh image segment 328 occupies the eighth position, and the ninth image segment 332 occupies the ninth position. After the segment sorter 210 has rearranged the image segments in the segment queue 208 into an enhanced modification order, the example segment sorter 210 notifies the example encoder 212 (see Figure 2 ), and the example encoder 212 extracts and encodes the image segments from the segment queue 208 in a second enhanced encoding order. The encoder 212 provides the encoded image segments to the example wireless transmitter 214, and the example wireless transmitter 214 wirelessly transmits the encoded image segments to the example head-mounted display unit 120.

[0049] As described above, the example wireless receiver 216 of the example head-mounted display unit 120 receives the image segments in a second enhanced encoding order. In some examples, the example decoder 218 of the head-mounted display unit 120 decodes the image segments in the above-described scan line decoding order (e.g., the first decodes the first image segment, the second decodes the second image segment, the third decodes the third image segment, and so on). Upon decoding, the decoder 218 provides the image segments to the screen / display / lens 220 of the head-mounted display unit 120 in the scan line decoding order for rendering thereon.

[0050] Figure 8 Table 800 is for comparing the scan line order (first row 810) of the decoder decoding the image segments with the example second enhanced encoding order (second row 820) in which the image segments are encoded / transmitted to the head-mounted display unit 120. Since the scan line order and the second enhanced encoding order are different, Figure 8The time difference (in multiples of the time period "t0") between encoding / transmitting each image segment and the example decoder 218 being ready to decode each image segment is also identified (third row 830). Negative numbers are used to indicate instances where the transmission of an image segment occurs after the time when the decoder 218 is ready to decode that image segment.

[0051] As Figure 8 shown in the first row 810 of Figure 2 (see Figure 8 ), the example decoder 218 decodes the image segments in scan line order such that the first decodes the first image segment 316, the second decodes the second image segment 318, the third decodes the third image segment 320, and so on until all nine image segments are decoded and displayed. As

[0052] Figure 8 shown in the second row 820 of Figure 2 (see

[0053] ), the second enhanced encoding order indicates that the first encodes the second image segment 318, the second encodes the first image segment 316, the third encodes the fifth image segment 324, the fourth encodes the third image segment 320, the fifth encodes the fourth image segment 322, the sixth encodes the sixth image segment 326, the seventh encodes the eighth image segment 330, the eighth encodes the seventh image segment 328, and the ninth encodes the ninth image segment 332.

[0052] Figure 8 The third row 830 of Figure 2 (see

[0053] ) shows the effect of decoding the image segments in scan line order when using the second enhanced encoding order to encode / transmit each image segment. Using the second enhanced encoding order, the second image segment 318 is encoded / transmitted one time period (1t0) earlier compared to when the decoder 218 (see Figure 2 ) is ready to decode the second image segment 318. In contrast, when using scan line order for encoding / transmission and decoding, the second image segment is encoded / transmitted and decoded in the same time period. Thus, when using the enhanced encoding order for encoding / transmission, the second image segment has a greater likelihood of being successfully decoded compared to scan line order. When using the second enhanced encoding order for encoding / transmission, the first image segment 316 is encoded / transmitted one time period later compared to when the decoder 218 is ready to decode the first image segment 316. In contrast, when using scan line order, the first image segment is encoded / transmitted and decoded in the same time period. Therefore, using the second enhanced encoding order has a negative impact on the likelihood of the first image segment being successfully decoded compared to using scan line order for encoding / transmission.

[0053] Still referring to Figure 8For the third row 830, when using the second enhanced encoding order, the fifth image segment 324 is encoded / transmitted two time periods earlier compared to the time period when the decoder 218 is ready to start decoding the fifth image segment 324. In contrast, when using the scan line order for encoding / transmission, the fifth image segment 324 is encoded / transmitted and decoded in the same time period. Therefore, compared with encoding using the scan line order, using the second enhanced encoding order has a positive impact on the possibility of successfully decoding the fifth image segment 324. On the contrary, when using the second enhanced encoding order for encoding / transmission, the third image segment 320 is encoded / transmitted one time period later compared to when the decoder 218 is ready to decode the third image segment 320. When using the scan line order for encoding / transmission, the third image segment is encoded / transmitted and decoded in the same time period. Therefore, compared with using the scan line order, using the second enhanced encoding order has a negative impact on the possibility of successfully decoding the third image segment 320.

[0054] When using the second enhanced encoding order for encoding / transmission, compared with the time when the exemplary decoder 218 (see Figure 2 ) is ready to start decoding the fourth image segment 322, the fourth image segment 322 is encoded / transmitted one time period later. In contrast, when using the scan line order for encoding / transmission, the fourth image segment 322 is encoded / transmitted and decoded in the same time period. Therefore, compared with encoding using the scan line order, using the second enhanced encoding order has a negative impact on the possibility of successfully decoding the fourth image segment 322.

[0055] Still referring to Figure 8 , when using the second enhanced encoding order for encoding / transmission, the sixth image segment 326 is encoded / transmitted and decoded in the same time period. The same is true for the sixth image segment 326 when using the scan line order. Therefore, compared with encoding using the scan line order, using the second enhanced encoding order does not change the possibility of successfully decoding the sixth image segment 326. When using the second enhanced encoding order for encoding / transmission, compared with the time when the exemplary decoder 218 (see Figure 2 ) is ready to start decoding the eighth image segment, the eighth image segment 330 is encoded / transmitted one time period earlier. When using the scan line order for encoding / transmission, the eighth image segment 330 is encoded / transmitted and decoded in the same time period. Therefore, compared with encoding using the scan line order, using the second enhanced encoding order allows the eighth image segment 330 more time to reach the decoder 218, thus having a positive impact on the possibility of successfully decoding the eighth image segment 330.

[0056] When using the second enhanced encoding order for encoding / transmission, compared with the exemplary decoder 218 (see Figure 2)The seventh image segment 328 is encoded / transmitted at a time that is one time period later than the time when decoding of the seventh image segment 328 is about to start. In contrast, when encoding / transmission is performed in scan line order, the seventh image segment 328 is encoded / transmitted and decoded in the same time period. Thus, compared to the amount of time provided when using scan line order, using the second enhanced encoding order results in less time for the seventh image segment 328 to reach the decoder 218. As a result, using the second enhanced encoding order has a negative impact on the likelihood of successfully decoding the seventh image segment 328 compared to encoding using scan line order.

[0057] When using the second enhanced encoding order for encoding / transmission, the ninth image segment 332 is encoded / transmitted and decoded in the same time period. The same is true when using scan line encoding. Thus, using the second enhanced encoding order does not change the likelihood of successfully decoding the ninth image segment 332 compared to encoding using scan line order.

[0058] Thus, using the second enhanced encoding order to encode and transmit image segments (compared to using scan line order) increases the likelihood of successfully decoding the second, fifth, and eighth image segments, has no effect on the likelihood of successfully decoding the sixth and ninth image segments, and has an adverse effect on the likelihood of successfully decoding the first, third, fourth, and seventh image segments. As described above, when using the second enhanced encoding order to encode / transmit image segments, the likelihood of successfully decoding some image segments (e.g., the first, third, fourth, and seventh image segments) is adversely affected. In some examples, this adverse effect is reduced when the example encoder 212 (see Figure 2 )(compared to the example decoder 218 decoding the received image segments) encodes faster. For example, in some examples, the encoder 212 encodes at a rate that takes about 5 milliseconds (ms) of encoding time per image frame, and the encoder is producing data to be displayed at a rate of about 11.1 ms (i.e., the encoder takes 5 ms to encode an image that takes 11 ms to be displayed). This time difference allows more time for the image segments to arrive, thus having a positive impact on the likelihood of successfully transmitting and decoding the image segments.

[0059] Figure 9 is a chart 900 showing the example decoding success rate for each of nine image segments when encoding / transmitting image segments in the second example enhanced encoding order and decoding in scan line order. As Figure 9As shown in the example of Figure 8 it is shown, the decrease in the success rate from 90% to 80% can be attributed to the delay in the transmission of the first, third, fourth, and seventh image segments in the second enhanced coding order relative to the decoding time of the first, third, fourth, and seventh image segments.

[0060] Therefore, for the second, fifth, and eighth image segments (e.g., layer 1 and layer 2) that include image content located at or adjacent to the center of the image, the success rate achieved when using the second enhanced coding order is improved. Since the second, fifth, and eighth image segments include content located at or adjacent to the center of the image, these image segments include important content and are more important for the rendering of the image. As a result, the increased likelihood of successfully decoding the second, fifth, and eighth image segments brings important benefits to the virtual reality achieved by the virtual reality system. In addition, although the success rate of the first, third, fourth, and seventh image segments decreases from 90% to 80%, each of the first, third, fourth, and seventh image segments includes peripheral image content that is less important for rendering, and thus, this is tolerable. In addition, the second enhanced coding order never causes the success rate to drop below 80%.

[0061] Although an example manner of implementing [[ID= 74]]Figure 2 the virtual reality console 110 and the head-mounted display unit 120 is shown in Figure 1 it is shown, Figure 2One or more of the elements, processes, and / or devices shown may be combined, divided, rearranged, omitted, eliminated, and / or implemented in any other way. Additionally, example image processor 202, example storage device 204, example image splitter 206, example segment queue 208, example segment sorter 210, example encoder 212, example wireless transmitter 214, example wireless receiver 216, example decoder storage device 217, example image decoder 218, example display 220, and / or more generally Figure 1 example virtual reality console 110 and head-mounted display unit 120 of Figure 1 may be implemented by hardware, software, firmware, and / or any combination of hardware, software, and / or firmware. Thus, for example, example image processor 202, example storage device 204, example image splitter 206, example segment queue 208, example segment sorter 210, example encoder 212, example wireless transmitter 214, example wireless receiver 216, example decoder storage device 217, example image decoder 218, example display 220, and / or more generally example virtual reality console 110 and head-mounted display unit 120 may be implemented by one or more analog or digital circuits, logic circuits, (one or more) programmable processors, (one or more) programmable controllers, (one or more) graphics processing units (GPUs), (one or more) digital signal processors (DSPs), (one or more) application specific integrated circuits (ASICs), (one or more) programmable logic devices (PLDs), and / or (one or more) field programmable logic devices (FPLDs). When reading any apparatus or system claims of this patent to cover pure software and / or firmware implementations, at least one of example image processor 202, example storage device 204, example image splitter 206, example segment queue 208, example segment sorter 210, example encoder 212, example wireless transmitter 214, example wireless receiver 216, example decoder storage device 217, example image decoder 218, and / or example display 220 is explicitly defined to include a non-transitory computer-readable storage device or storage disk having software and / or firmware, e.g., a memory, digital versatile disk (DVD), optical disk (CD), Blu-ray disk, etc. Additionally, Figure 2 example image processor 202, example storage device 204, example image splitter 206, example segment queue 208, example segment sorter 210, example encoder 212, example wireless transmitter 214, example wireless receiver 216, example decoder storage device 217, example image decoder 218, example display 220 of Figure 2 may include one or more elements, processes, and / or devices (in addition to Figure 2 those shown in Figure 2 , or instead of Figure 2those shown in), and / or may include more than one or any or all of the illustrated elements, processes, and devices. As used herein, the phrase "communicate" (including its variants) includes direct communication and / or indirect communication through one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or continuous communication, but additionally also includes selective communication in a periodic interval, a scheduled interval, an aperiodic interval, and / or a one-time event manner.

[0062] Figure 10 illustrates an example hardware logic and / or machine-readable instructions for implementing Figure 2 of the virtual reality console 110, and Figure 11 illustrates an example hardware logic or machine-readable instructions for implementing Figure 2 of the head-mounted display unit 120. The machine-readable instructions may be a program or a part of a program for execution by a processor (e.g., the processor 1212 shown in the example processor platform 1200 discussed below). The program may be embodied in software stored on a non-transitory computer-readable storage medium (e.g., a CD-ROM, a floppy disk, a hard disk drive, a DVD, a Blu-ray disc, or a memory associated with the processor 1212), and the entire program and / or portions thereof may alternatively be executed by a device other than the processor 1212, and / or be embodied in firmware or dedicated hardware. Additionally, although the example program is described with reference to Figure 12 and Figure 10 and 11 the flowcharts shown in, many other methods for implementing the example virtual reality console 110 and / or the head-mounted display unit 120 may alternatively be used. For example, the order of execution of the boxes may be changed, and / or some of the described boxes may be changed, eliminated, or combined. Additionally or alternatively, any or all of the boxes may be implemented by one or more hardware circuits (e.g., discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.) configured to perform the corresponding operations without executing software or firmware.

[0063] As described above, Figure 10 and Figure 11The example process can be implemented using executable instructions (e.g., computer and / or machine-readable instructions) stored on a non-transitory computer and / or machine-readable medium (e.g., a hard disk drive, flash memory, read-only memory, optical disk, digital versatile disk, cache, random access memory, and / or any other storage device or storage disk where information is stored for any duration (e.g., for an extended period of time for information storage, for permanent information storage, for short instances of information storage, for temporarily buffering information, and / or for caching information)). As used herein, the term non-transitory computer-readable medium is expressly defined to include any type of computer-readable storage device and / or storage disk, and to exclude propagated signals and transmission media.

[0064] As used herein, "comprising" and "including" (and all forms and tenses thereof) are open-ended terms. Thus, whenever a claim uses any form of "comprising" or "including" (e.g., comprises, includes, has, contains, etc.) as a preamble or in any type of claim, it should be understood that additional elements, terms, etc. may exist without falling outside the scope of the corresponding claim or the reference. As used herein, when the phrase "at least" is used as a transitional term in, for example, the preamble of a claim, it is open-ended in the same way that the terms "comprising" and "including" are open-ended. When used in the form of, for example, A, B, and / or C, the term "and / or" refers to any combination or subset of A, B, C, e.g., (1) A alone, (2) B alone, (3) C alone, (4) A and B, (5) A and C, (6) B and C, (7) A and B and C.

[0065] Figure 10 Program 1000 begins at block 1002, where example processor 202 (see Figure 2 ) identifies an image to be encoded and sent from example virtual reality console 110 (see Figure 1 and Figure 2 and sent to example head-mounted display unit 120 (see ) and Figure 1 and 2 ). In some examples, processor 202 places the image to be encoded in example storage device 204. Example image splitter 206 (see Figure 2)The image frames stored in the example storage device 204 are divided into segments (block 1004). In some examples, the image splitter 206 divides the image into nine image segments by dividing the image into a grid with three rows and three columns. In some examples, the nine image segments include non-overlapping image content. In some examples, the example first image segment, second image segment, and third image segment are located in the top row of the image grid, the first image segment is the leftmost segment of the top row, the second image segment is the middle segment of the top row, and the third image segment is the rightmost segment of the top row. In some examples, the example fourth image segment, fifth image segment, and sixth image segment are located in the middle row of the image grid, the fourth image segment is the leftmost segment of the middle row, the fifth image segment is the middle segment of the middle row, and the sixth image segment is the rightmost segment of the middle row. In some examples, the example seventh image segment, eighth image segment, and ninth image segment are located in the bottom row of the image grid, and the seventh image segment is the leftmost segment of the bottom row, the eighth image segment is the middle segment of the bottom row, and the ninth image segment is the rightmost segment of the bottom row.

[0066] The example image splitter 206 places the image segments into the example segment queue 208 in a scan line order (see Figure 2 ) (block 1006). As a result, the first segment is placed in the first position of the segment queue 208, the second segment is placed in the second position of the segment queue 208, and the third segment is placed in the third position of the segment queue 208. Additionally, the fourth segment is placed in the fourth position of the segment queue 208, the fifth segment is placed in the fifth position of the segment queue 208, the sixth image segment is placed in the sixth position of the segment queue 208, the seventh image segment is placed in the seventh position in the segment queue 208, the eighth image segment is placed in the eighth position in the segment queue 208, and the ninth image segment is placed in the ninth position in the segment queue 208. Additionally, in some examples, the image splitter 206 notifies the example segment sorter 210 that the image segments have been placed into the image queue 208 (also at block 1006).

[0067] The example segment sorter 210 rearranges the order in which image segments are stored in the segment queue 208 (block 1008). In some examples, the segment sorter 210 rearranges the image segments such that the image segments are in a first enhanced encoding order (e.g., the fifth image segment occupies the first position, the second image segment occupies the second position, the first image segment occupies the third position, the fourth image segment occupies the fourth position, the third image segment occupies the fifth position, the sixth image segment occupies the sixth position, the eighth image segment occupies the seventh position, the seventh image segment occupies the eighth position, and the ninth image segment occupies the ninth position). In some examples, the example segment sorter 210 arranges the image segments in a second enhanced encoding order (e.g., the second image segment occupies the first position, the first image segment occupies the second position, the fifth image segment occupies the third position, the third image segment occupies the fourth position, the fourth image segment occupies the fifth position, the sixth image segment occupies the sixth position, the eighth image segment occupies the seventh position, the seventh image segment occupies the eighth position, and the ninth image segment occupies the ninth position). In some examples, the segment sorter 210 arranges the image segments in the segment queue 208 such that one or more image segments associated with the first layer and / or the second layer are moved to a more forward position in the segment queue 208 (compared to the position occupied by the same image segments in scanline order). In some such examples, the first layer and the second layer include image segments located at the center of the image (e.g., the fifth image segment), or image segments that share a boundary with segments located at the center of the image. In some examples, the segment sorter 210 arranges the order of the image segments in the segment queue 208 such that one or more corresponding image segments that include image content in the third layer (e.g., located at the corners of the image) are moved to a more rearward position in the segment queue 208 (compared to the position occupied by the corresponding image segments when arranged in scanline order).

[0068] After the example segment sorter 210 has reordered the image segments, the example encoder 212 encodes the image segments in the same order in which the image segments are stored in the segment queue 208 (e.g., the first encoded image segment occupies the first position, the second encoded image segment occupies the second position, the third encoded image segment occupies the third position, and so on) (block 1010). The encoder 212 provides each encoded image segment to the example transmitter 214 in the same order in which the image segments are encoded (see Figure 2 )(block 1012), and the transmitter 214 wirelessly transmits the image segments in the same order in which the image segments are encoded (block 1014). After the image segments have been encoded, the program returns to block 1002 and subsequent blocks to encode the next image, or if no other images are to be encoded / transmitted (as determined at block 1016), the program 1000 ends.

[0069] Figure 11 The process 1100 begins at block 1102, where the example head mounted display unit 120 (see Figure 1 and Figure 2 ) of an example receiver 216 (see Figure 2 ) from the virtual reality console 110 (see Figure 1 and Figure 2 ) of an example transmitter 214 (see Figure 2 ) wirelessly receives one or more image segments. In some examples, receiver 216 places the received image segments into decoder storage 217 and sets a decoder counter variable "M" equal to 1 (block 1102). In some such examples, example receiver 216 notifies decoder 218 that the encoded image segments have been received and stored (also at block 1104).

[0070] In response to the notification, the image decoder 218 identifies which of the first, second, third, fourth, fifth, sixth, seventh, eighth, and ninth image segments to decode first based on the decoding order to be used and based on the value of the counter variable "M" (block 1106). In some such examples, during a first execution of the routine 1100, the image decoder 218 identifies the number of the first image segment to be decoded based on the decoder counter variable M being equal to 1. During subsequent executions of the routine 1100, the image decoder 218 identifies the number of the image segment to be decoded based on the current value of the decoder counter variable M. Thus, if the decoder counter variable "M" is equal to 2, the decoder identifies the number of the second image segment to be decoded based on the order to be used. If the decoder counter variable "M" is equal to 3, the decoder identifies the number of the third image segment to be decoded based on the order to be used, and so on. The image decoder 218 begins a decoding process by which the stored image segments received from the virtual reality console are decoded (block 1104). In some examples, the image decoder 218 is configured to decode the image segments in scanline order. In some examples, the image decoder 218 is configured to decode the image segments in a first enhancement coding order, a second enhancement coding order, or any other desired order. For example, emerging image / video display technologies (e.g., displays compliant with the VESA DisplayID 2.0 specification) have the ability to output pixels from the decoder in an arbitrary order (e.g., non-scanline order). Figure 2The image decoder 218 can similarly be configured to decode / output image segments for display in any order. In some examples, the image decoder is configured to output image segments in an order that decodes and displays the image segments with the most important content as early as possible. In some examples, the order of encoding / sending and then decoding / displaying the image segments (e.g., first enhanced encoding order, second enhanced encoding order, etc.) is selected to ensure that the image segments with the most important content pass through the wireless link in a timely manner and have the highest possible decoding success rate, thereby enriching the user experience by providing better quality images on the head-mounted display.

[0071] In some examples, to assist the image decoder 208 in selecting the appropriate image segments to decode based on the decoding order employed (e.g., scan line order, first example enhanced order, second example enhanced order, etc.), example image segments are sent together with information identifying the numbers of the image segments. This information can be included in headers, tags, etc. In some such examples, the image decoder 208 uses the segment number information to identify the image segment numbers and places the image segments in the appropriate order for decoding. The image segment numbers are also used to identify how the content included in the image segments will be displayed (relative to the content included in other image segments). Any other technique can be used to identify the numbers of the image segments to the decoder for decoding the image segments in the desired order and displaying the image segments appropriately.

[0072] When identifying the image segments to decode, the example image decoder 218 determines whether the identified image segments have been received (e.g., stored in the decoder storage device) (block 1108). If the identified image segments have been received, the image decoder 218 decodes the identified image segments and provides the decoded content of the identified image segments to the example display 220 (see Figure 2 ) for rendering at the example display 220 (block 1110). If the identified image segments have not been received, the image decoder 218 provides the decoded content of image segments having the same numbers as the identified image segments but corresponding to earlier transmitted frames / images to the display 220 for rendering at the display 220 (block 1112).

[0073] In some examples, after having decoded / displayed an identified image segment or the corresponding image segment of a previous image frame that has been displayed, image decoder 218 determines whether a counter variable M is equal to the total number of image segments associated with the current frame (block 1114). If not, there are more image segments to be decoded for the current frame, and image decoder increments the counter variable M (block 1116), then returns to block 1106 and subsequent blocks to decode and display the next image segment in the order of use. If it is determined (at block 1114) that the counter variable M is equal to the total number of image segments associated with the current frame, decoding of the current frame is completed. As a result, image decoder 218 determines whether to decode / display another set of image segments associated with another image (block 1118). If there is another set of image segments to be decoded / displayed, program 1100 returns to block 1104 and subsequent blocks. If there is no other set of image segments to be decoded / displayed, program 1100 ends.

[0074] Figure 12 is constructed to execute Figure 10 instructions to implement Figure 1 and Figure 2 block diagram of an example processor platform 1200 of a virtual reality console 110. Processor platform 1200 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a mobile phone, a smartphone, a tablet (e.g., iPad TM )), a personal digital assistant (PDA), an Internet device, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a gaming console, a personal video recorder, a set-top box, or any other type of computing device.

[0075] The illustrated example of processor platform 1200 includes a processor 1212. The illustrated example of processor 1212 is hardware. For example, processor 1212 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor can be a semiconductor-based (e.g., silicon-based) device. In this example, the processor implements example image processor 202, example image segmenter 206, example segment queue 208, example segment sorter 210, and example encoder 212.

[0076] The illustrated example of processor 1212 includes local memory 1213 (e.g., a cache). The illustrated example of processor 1212 communicates via a bus 1218 with a main memory that includes volatile memory 1214 and non-volatile memory 1216. Volatile memory 1214 can be implemented by synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), Dynamic random access memory and / or any other type of random access memory device. The non-volatile memory 1216 can be implemented by flash memory and / or any other desired type of memory device. Access to the main memories 1214 and 1216 is controlled by a memory controller.

[0077] The processor platform 1200 of the illustrated example also includes interface circuitry 1220. The interface circuitry 1220 can be implemented by any type of interface standard, e.g., an Ethernet interface, a Universal Serial Bus (USB), a Bluetooth interface, a Near Field Communication (NFC) interface, and / or a PCI express interface.

[0078] In the illustrated example, one or more input devices 1222 are connected to the interface circuitry 1220. The (one or more) input devices 1222 allow a user to input data and / or commands into the processor 1212. The (one or more) input devices can be implemented by, for example, an audio sensor, a microphone, a camera (a still camera or a video camera), a keyboard, a button, a mouse, a touch screen, a trackpad, a trackball, an isopoint, and / or a voice recognition system.

[0079] One or more output devices 1224 are also connected to the interface circuitry 1220 of the illustrated example. The output devices 1224 can be implemented by, for example, a display device (e.g., a light-emitting diode (LED), an organic light-emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-plane switching (IPS) display, a touch screen, etc.), a haptic output device, a printer, and / or a speaker. Thus, the interface circuitry 1220 of the illustrated example generally includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0080] The interface circuitry 1220 of the illustrated example also includes communication devices, e.g., a transmitter, a receiver, a transceiver, a modem, a residential gateway, a wireless access point, and / or a network interface, for facilitating data exchange by communicating with external machines (e.g., any type of computing device) via a network 1226. The communication can be carried out by, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a field wireless system, a cellular telephone system, etc. The interface circuitry 1220 is used to implement Figure 2 the example wireless transmitter 214. In some examples, the interface 1220 circuitry uses WiFi defined in IEEE standard 802.11 for communication. In other examples, the interface circuitry 1220 uses WiGig (as defined in 802.11ad and / or 802.11ay) to transmit image segments.

[0081] The processor platform 1200 of the illustrated example also includes one or more mass storage devices 1228 for storing software and / or data. Examples of such mass storage devices 1228 include floppy disk drives, hard disk drives, compact disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives. Figure 12 Any of the storage devices can be used to implement the example storage device 204 and the example segment queue 208.

[0082] Figure 12 The machine-executable instructions 1232 (e.g., program 1000) can be stored on the mass storage device 1228, volatile memory 1214, non-volatile memory 1216, and / or a removable non-transitory computer-readable storage medium (e.g., a CD or DVD).

[0083] Figure 13 is constructed to execute Figure 11 the instructions (e.g., program 1100) to implement Figure 1 and Figure 2 a block diagram of an example processor platform 1300 of an example head-mounted display unit 120. The processor platform 1300 can be, for example, a head-mounted device or any other type of wearable device.

[0084] The processor platform 1300 of the illustrated example includes a processor 1312. The processor 1312 of the illustrated example is hardware. For example, the processor 1312 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor can be a semiconductor-based (e.g., silicon-based) device. In this example, the processor implements Figure 2 an example image decoder 218.

[0085] The processor 1312 of the illustrated example includes local memory 1313 (e.g., a cache). The processor 1312 of the illustrated example communicates with a main memory including volatile memory 1314 and non-volatile memory 1316 via a bus 1318. The volatile memory 1314 can be implemented by synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), dynamic random access memory and / or any other type of random access memory device. The non-volatile memory 1316 can be implemented by flash memory and / or any other desired type of memory device. Access to the main memories 1314 and 1316 is controlled by a memory controller.

[0086] The processor platform 1300 of the illustrated example also includes interface circuitry 1320. The interface circuitry 1320 can be implemented by any type of interface standard, e.g., an Ethernet interface, a Universal Serial Bus (USB), a Bluetooth interface, a Near Field Communication (NFC) interface, and / or a PCI express interface.

[0087] In the illustrated example, one or more input devices 1322 are connected to the interface circuitry 1320. The (one or more) input devices 1322 allow a user to input data and / or commands into the processor 1312. The (one or more) input devices can be implemented by, for example, an audio sensor, a microphone, a camera (a still camera or a video camera), a button, and / or a voice recognition system.

[0088] One or more output devices 1324 are also connected to the interface circuitry 1320 of the illustrated example. The output device 1324 can be implemented by, for example, a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-plane switching (IPS) display, a touch screen, etc.), a haptic output device, a printer, and / or a speaker. Thus, the interface circuitry 1320 of the illustrated example generally includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor. The interface circuitry 1320 is used to implement Figure 2 the example display 220.

[0089] The interface circuitry 1320 of the illustrated example also includes communication devices, e.g., a transmitter, a receiver, a transceiver, a modem, a residential gateway, a wireless access point, and / or a network interface, for facilitating the exchange of data by communicating with external machines (e.g., any type of computing device) via a network 1326. The communication can be carried out by, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a field wireless system, a cellular telephone system, etc. The interface circuitry 1320 is used to implement Figure 2 the example wireless receiver 216. In some examples, the interface 1320 circuitry uses WiFi defined in IEEE standard 802.11 for communication. In other examples, the interface circuitry 1320 uses WiGig (as defined in, e.g., 802.11ad and / or 802.11ay) to transmit image segments.

[0090] The processor platform 1300 of the illustrated example also includes one or more mass storage devices 1328 for storing software and / or data. Examples of such mass storage devices 1328 include a floppy disk drive, a hard disk drive, a compact disk drive, a Blu-ray disk drive, a redundant array of independent disks (RAID) system, and a digital versatile disk (DVD) drive. Figure 13Any storage device can be used to implement Figure 2 the example decoder storage device 217 of

[0091] Figure 13 The machine-executable instructions 1332 (e.g., program 1100) can be stored on the mass storage device 1328, volatile memory 1314, non-volatile memory 1316, and / or a removable non-transitory computer-readable storage medium (e.g., CD or DVD).

[0092] From the foregoing, it can be understood that example methods, apparatuses, and articles for increasing throughput between a virtual reality console and a head-mounted display unit of a virtual reality system have been disclosed. The disclosed virtual reality console prioritizes encoding and transmitting image segments that include visually important portions of an image (including, e.g., a center image segment having content including the center of the image). Some examples additionally prioritize image segments that share a boundary with the center image segment. Some example systems, methods, and apparatuses disclosed herein balance the prioritization of the center image segment and the timely transmission of other image segments. Such example systems, methods, and apparatuses are configured to encode and transmit image segments such that the center image segment is encoded / transmitted earlier than encoding the center segment using a scan line order, while maintaining a desired (e.g., 80%) success rate of transmitting / decoding non-center image segments. Encoding / transmitting the center image segment and image segments contiguous to the center image segment earlier (compared to encoding / transmitting these image segments using a scan line order) results in an improved “virtual reality” experience for the user because the most important image content is more likely to be successfully decoded and thus included in the image displayed on the head-mounted display unit.

[0093] It should be understood that although the image to be encoded is shown for exemplary purposes as being divided into nine segments, the example image segmenter 206, example segment sorter 210, example encoder 212, example transmitter 214, exemplary receiver 216, exemplary decoder 218 can be configured to operate on image frames divided into any number of image segments. In examples using more or fewer than nine segments, the numbering of the image segments that include content at the center of the image, at the periphery of the image, and non-peripheral of the image will of course be numbered differently (compared to the same segments being numbered in the first enhanced encoding order and the second enhanced encoding order). Regardless of the number of segments into which the image is divided, the same concept applies, that is, the first enhanced encoding order gives the highest priority to transmitting the center image segment (thus placing the center segment as the first encoded / transmitted image segment), while the second enhanced encoding order gives a higher priority to the center image segment and non-peripheral image segments (compared to the priority given to the same segments in scan line order) and does not come at the cost of excessive delay for the peripheral image segments. Thus, the first enhanced encoding order encodes / transmits the center image segment first, thereby giving the center image segment the greatest likelihood of successful arrival and decoding, and the second enhanced encoding attempts to balance the increased priority of the center image segment and the timely arrival of all image segments.

[0094] The following further examples are disclosed herein.

[0095] Example 1 is an apparatus for sorting multiple segments of an image for encoding and transmission to a display device. The apparatus of Example 1 includes a segment sorter for arranging the segments in an encoding order for encoding. The encoding order is different from the scan line order, and in the encoding order, the center segment of the multiple segments occupies the first position, which is before the second position that the center segment occupies in the scan line order, and the center segment corresponds to the center of the image. The apparatus further includes: an encoder for encoding the segments in the encoding order; and a wireless transmitter for transmitting the encoded segments to the display device in the encoding order in which the encoded segments are encoded.

[0096] Example 2 includes the apparatus of Example 1. In Example 2, the encoding order further causes the peripheral segments of the multiple segments to occupy the third position in the order, which is at least one of the following positions: 1) the position that is the same as the position that the peripheral segment occupies in the scan line order, or 2) a position after the fourth position that the peripheral segment occupies in the scan line order, and the peripheral segment does not share a boundary with the center segment.

[0097] Example 3 includes the apparatus of Example 2. In Example 3, the peripheral segment is the first peripheral segment of the multiple peripheral segments, and the multiple peripheral segments correspond to the corners of the image.

[0098] Example 4 includes the apparatus of Example 2. In Example 4, the encoding order also causes at least one non-peripheral segment among the plurality of segments to occupy the fifth position in the order, the fifth position being before the sixth position that the non-peripheral segment occupies in the scan line order, and at least one non-peripheral segment shares a boundary with the central segment.

[0099] Example 5 includes the apparatus of any one of Examples 1-4, and further includes a splitter for forming a plurality of segments by dividing an image. In Example 5, the splitter stores the plurality of segments in a queue in scan line order.

[0100] Example 6 includes the apparatus of Example 5. In Example 6, the splitter divides the image into a grid including a top row, a bottom row, and a middle row, a left column, a middle column, and a right column, and the plurality of segments includes nine segments. In Example 6, the first segment is the leftmost segment in the top row, the second segment is the middle segment in the top row, the third segment is the rightmost segment in the top row, the fourth segment is the leftmost segment in the middle row, the fifth segment is the middle segment in the middle row, the sixth segment is the rightmost segment in the middle row, the seventh segment is the leftmost segment in the bottom row, the eighth segment is the middle segment in the bottom row, and the ninth segment is the rightmost segment in the bottom row. In Example 6, the scan line order is the first segment, the second segment, the third segment, the fourth segment, the fifth segment, the sixth segment, the seventh segment, the eighth segment, and the ninth segment, and the encoding order is the second segment, the first segment, the fifth segment, the third segment, the fourth segment, the sixth segment, the eighth segment, the seventh segment, and the ninth segment.

[0101] Example 7 includes the apparatus of any one of Examples 1-6. In Example 7, the portion of the image does not include overlapping image content.

[0102] Example 8 includes one or more non-transitory machine-readable storage media storing machine-readable instructions that, when executed, cause one or more processors to arrange a set of image regions at least in a non-scan line order and encode the image regions in a non-scan line order. In Example 8, using the non-scan line order causes the central image region of the set of image regions to be encoded earlier compared to when the central image region would be encoded in scan line order, and causes the peripheral image regions of the set of image regions to be encoded later compared to when the peripheral image regions would be encoded in scan line order. In Example 8, the instructions further cause one or more processors to provide the encoded image regions to a transmitter for transmission to a display unit in a non-scan line order.

[0103] Example 9 includes the one or more non-transitory machine-readable storage media of Example 8. In Example 9, the peripheral image region is the first peripheral region among a plurality of peripheral regions, and the plurality of peripheral regions correspond to the corners of the image.

[0104] Example 10 includes one or more non-transitory machine-readable storage media of Example 8. In Example 10, using a non-scanline order also causes non-peripheral image regions in the set of image regions to be encoded earlier than the non-peripheral image regions would be encoded in scanline order.

[0105] Example 11 includes one or more non-transitory machine-readable storage media of any one of claims 8-10. In Example 11, the instructions also cause one or more processors to divide an image into non-overlapping image regions to form a set of image regions and store the set of image regions in a queue.

[0106] Example 12 includes one or more non-transitory machine-readable storage media of Example 11. In Example 12, to divide the image, the instructions cause one or more processors to divide the image into a grid including a top row, a bottom row, and middle rows, a left column, middle columns, and a right column. Additionally, the set of image regions includes nine image regions. In Example 12, the first image region is the leftmost image region in the top row, the second image region is the middle image region in the top row, the third image region is the rightmost image region in the top row, the fourth image region is the leftmost image region in the middle row, the fifth image region is the middle image region in the middle row, the sixth image region is the rightmost image region in the middle row, the seventh image region is the leftmost image region in the bottom row, the eighth image region is the middle image region in the bottom row, and the ninth image region is the rightmost image region in the bottom row. Additionally, in Example 12, the non-scanline order includes the second segment in the first position, the first segment in the second position, the fifth segment in the third position, the third segment in the fourth position, the fourth segment in the fifth position, the sixth segment in the sixth position, the eighth segment in the seventh position, the seventh segment in the eighth position, and the ninth segment in the ninth position.

[0107] Example 13 is a device for sorting a plurality of image tiles. The device of Example 13 includes means for positioning a plurality of image tiles in a non-scanline order. In Example 13, the plurality of image tiles includes a center image tile corresponding to the center of an image, and the center image tile occupies an earlier encoding position in the non-scanline order than the position the center image tile occupies in scanline order. The device also includes means for encoding the plurality of image tiles in a non-scanline order and means for sending the encoded image tiles to a display in a non-scanline order.

[0108] Example 14 includes the device of Example 13. In Example 14, the non-scanline order is a first encoding order in which a center image tile having content corresponding to the center of an image is encoded and sent before other image tiles in the plurality of image tiles are encoded and sent.

[0109] Example 15 includes the apparatus of Example 13. In Example 15, the non-scanline order is the second encoding order, in which a central image tile having content corresponding to the center of the image appears earlier than it appears in the scanline order, and a peripheral image tile having content corresponding to a corner of the image appears later than it appears in the scanline order.

[0110] Example 16 includes the apparatus of any one of Examples 13-15. In Example 16, the plurality of image tiles further includes a peripheral image tile corresponding to the periphery of the image. The peripheral image tile occupies an encoding position later than the position occupied by the peripheral image tile in the scanline order in the non-scanline order.

[0111] Example 17 includes the apparatus of any one of Examples 13-15. Additionally, the apparatus of Example 17 further includes means for dividing an image to form a plurality of image tiles.

[0112] Example 18 includes the apparatus of Example 17. In Example 18, the means for dividing divides the image into a grid including a top row, a bottom row, and a middle row, a left column, a middle column, and a right column. In Example 18, the plurality of image tiles includes nine image tiles. The first image tile is the leftmost image tile in the top row, the second image tile is the middle image tile in the top row, the third image tile is the rightmost image tile in the top row, the fourth image tile is the leftmost image tile in the middle row, the fifth image tile is the middle image tile in the middle row, the sixth image tile is the rightmost image tile in the middle row, the seventh image tile is the leftmost image tile in the bottom row, the eighth image tile is the middle image tile in the bottom row, and the ninth image tile is the rightmost image tile in the bottom row. In Example 18, the scanline order is the first image tile, the second image tile, the third image tile, the fourth image tile, the fifth image tile, the sixth image tile, the seventh image tile, the eighth image tile, and the ninth image tile, and the first encoding order is the second image tile, the first image tile, the fifth image tile, the third image tile, the fourth image tile, the sixth image tile, the eighth image tile, the seventh image tile, and the ninth image tile.

[0113] Example 19 includes the apparatus of any one of Examples 13-18. In Example 19, the image tiles do not include overlapping image content.

[0114] Example 20 includes the apparatus of any one of Examples 13-18. In Example 20, the apparatus is included in a virtual reality console, and the encoded image tiles are wirelessly transmitted.

[0115] Example 21 is a method for sorting multiple segments of an image for encoding and sending to a display device. The method of Example 21 includes arranging the segments in an encoding order for encoding. The encoding order is different from the scan line order, and the encoding order causes the central segment of the multiple segments to occupy a first position in the encoding order, the first position being before a second position that the central segment occupies in the scan line order, and the central segment corresponding to the center of the image. Example 21 also includes encoding the segments in the encoding order and sending the encoded segments to the display device in the encoding order in which the encoded segments are encoded.

[0116] Example 22 includes the method of Example 21. In Example 22, the encoding order also causes the peripheral segments of the multiple segments to occupy a third position in the order, the third position being at least one of the following positions: 1) a position the same as the position that the peripheral segment occupies in the scan line order, or 2) a position after a fourth position that the peripheral segment occupies in the scan line order. In Example 22, the peripheral segments do not share a boundary with the central segment.

[0117] Example 23 includes the method of Example 22. In Example 23, the peripheral segment is the first peripheral segment of the multiple peripheral segments, and the multiple peripheral segments correspond to the corners of the image.

[0118] Example 24 includes the method of Example 22. In Example 24, the encoding order causes at least one non-peripheral segment of the multiple segments to occupy a fifth position in the order, the fifth position being before a sixth position that the non-peripheral segment occupies in the scan line order, and the at least one non-peripheral segment shares a boundary with the central segment.

[0119] Example 25 includes the method of any one of Examples 21 - 24. The method of Example 25 also includes dividing the image to form multiple segments and storing the segments in a queue in scan line order.

[0120] Although certain example methods, apparatuses, and articles have been disclosed herein, the scope of coverage of this patent is not limited thereto. Instead, the patent covers all methods, apparatuses, and articles that fall entirely within the scope of the claims of this patent.

Claims

1. An apparatus for ordering a plurality of fragments of an image for encoding and transmitting to a display device, the apparatus comprising: a fragment sorter for rearranging the plurality of fragments having a scan line order for encoding in a coding order that is different from the scan line order, the coding order being such that a center fragment of the plurality of fragments occupies a first position in the coding order, the first position preceding a second position in the scan line order occupied by the center fragment, the center fragment corresponding to a center of the image, and the coding order being such that at least one non-peripheral fragment of the plurality of fragments that shares a border with the center fragment occupies a fifth position in the order, the fifth position preceding a sixth position in the scan line order occupied by the non-peripheral fragment; an encoder for encoding the segments in the coding order; as well as A wireless transmitter is configured to transmit the encoded segments to the display device in the encoding order in which they were encoded.

2. The device according to claim 1, wherein The encoding order further causes a peripheral fragment of the plurality of fragments to occupy a third position in the order, the third position being at least one of: 1) the same position as the position occupied by the peripheral fragment in the scan line order, or 2) a position after the fourth position occupied by the peripheral fragment in the scan line order, the peripheral fragment not sharing a border with the central fragment.

3. The device according to claim 2, wherein The peripheral segment is a first peripheral segment of a plurality of peripheral segments corresponding to corners of the image.

4. The apparatus according to claim 1, further comprising: A segmenter is configured to form the plurality of segments by dividing the image, the segmenter being configured to store the plurality of segments in a queue in a scan line order.

5. The device according to claim 4, wherein The segmenter divides the image into a grid including a top row, a bottom row, and a middle row, a left column, a middle column, and a right column, and the plurality of segments include nine segments, a first segment being the leftmost segment in the top row, a second segment being the middle segment in the top row, a third segment being the rightmost segment in the top row, a fourth segment being the leftmost segment in the middle row, a fifth segment being the middle segment in the middle row, a sixth segment being the rightmost segment in the middle row, a seventh segment being the leftmost segment in the bottom row, an eighth segment being the middle segment in the bottom row, and a ninth segment being the rightmost segment in the bottom row, the scan line order being the first segment, the second segment, the third segment, the fourth segment, the fifth segment, the sixth segment, the seventh segment, the eighth segment, and the ninth segment, and the encoding order being the second segment, the first segment, the fifth segment, the third segment, the fourth segment, the sixth segment, the eighth segment, the seventh segment, and the ninth segment.

6. The device according to claim 1, wherein The portion of the image does not include overlapping image content.

7. One or more non-transitory machine-readable storage media comprising machine-readable instructions that, when executed, cause one or more processors to perform at least the following operations: Rearranging a set of image regions having a scanline order in a non-scanline order; encoding the image regions in the non-scanline order, using the non-scanline order such that a central image region of the group of image regions is encoded earlier than the central image region would be encoded in the scanline order, such that peripheral image regions of the group of image regions are encoded later than the peripheral image regions would be encoded in the scanline order, and further such that non-peripheral image regions of the group of image regions that share a border with the central image region are encoded earlier than the non-peripheral image regions would be encoded in the scanline order; as well as The encoded image region is provided to a transmitter in said non-scanline sequence for transmission to a display unit.

8. The one or more non-transitory machine-readable storage media of claim 7, wherein: The peripheral image area is a first peripheral area among a plurality of peripheral areas corresponding to corners of the image.

9. The one or more non-transitory machine-readable storage media of claim 7, wherein: The instructions further cause the one or more processors to: dividing the image into non-overlapping image regions to form the set of image regions; and The set of image regions is stored in a queue.

10. The one or more non-transitory machine-readable storage media of claim 9, wherein: In order to divide the image, the instructions cause the one or more processors to divide the image into a grid including a top row, a bottom row, and a middle row, a left column, a middle column, and a right column, and the group of image areas includes nine image areas, the first image area is the leftmost image area in the top row, the second image area is the middle image area in the top row, the third image area is the rightmost image area in the top row, the fourth image area is the leftmost image area in the middle row, the fifth image area is the middle image area in the middle row, the sixth image area is the rightmost image area in the middle row, the seventh image area is the leftmost image area in the bottom row, the eighth image area is the middle image area in the bottom row, and the ninth image area is the rightmost image area in the bottom row, and the non-scan line order includes the second segment being at the first position, the first segment being at the second position, the fifth segment being at the third position, the third segment being at the fourth position, the fourth segment being at the fifth position, the sixth segment being at the sixth position, the eighth segment being at the seventh position, the seventh segment being at the eighth position, and the ninth segment being at the ninth position.

11. A device for sorting a plurality of image tiles, the device comprising: means for positioning the plurality of image tiles in a non-scanline order, the plurality of image tiles comprising a central image tile corresponding to a center of an image, the central image tile occupying an earlier coding position in the non-scanline order than a position occupied by the central image tile in the scanline order, the plurality of image tiles further comprising non-peripheral image tiles sharing a border with the central image tile, the non-peripheral image tiles occupying an earlier coding position in the non-scanline order than a position occupied by the non-peripheral image tiles in the scanline order; means for encoding said plurality of image tiles in said non-scanline order; as well as Means for sending the encoded image tiles to a display in said non-scanline order.

12. The apparatus according to claim 11, wherein The non-scanline order is a first coding order in which a center image tile having content corresponding to a center of the image is encoded and transmitted before other image tiles of the plurality of image tiles are encoded and transmitted.

13. The apparatus according to claim 11, wherein The non-scanline order is a second encoding order in which a central image tile having content corresponding to a center of the image appears earlier than the central image tile appears in the scanline order, and peripheral image tiles having content corresponding to corners of the image appear later than the peripheral image tiles appear in the scanline order.

14. The apparatus according to claim 11, wherein The plurality of image tiles further comprise peripheral image tiles corresponding to a periphery of the image, the peripheral image tiles occupying later coding positions in the non-scanline order than positions occupied by the peripheral image tiles in the scanline order.

15. The apparatus of claim 11, further comprising means for segmenting the image to form the plurality of image tiles.

16. The apparatus according to claim 15, wherein The apparatus for segmenting divides the image into a grid comprising a top row, a bottom row, and a middle row, a left column, a middle column, and a right column, and the plurality of image tiles comprises nine image tiles, a first image tile being the leftmost image tile in the top row, a second image tile being the middle image tile in the top row, a third image tile being the rightmost image tile in the top row, a fourth image tile being the leftmost image tile in the middle row, a fifth image tile being the middle image tile in the middle row, a sixth image tile being the rightmost image tile in the middle row, and a seventh image tile being the bottom row. The leftmost image tile in the row, the eighth image tile is the middle image tile in the bottom row, the ninth image tile is the rightmost image tile in the bottom row, the scan line order is the first image tile, the second image tile, the third image tile, the fourth image tile, the fifth image tile, the sixth image tile, the seventh image tile, the eighth image tile, and the ninth image tile, and the first enhanced coding order is the second image tile, the first image tile, the fifth image tile, the third image tile, the fourth image tile, the sixth image tile, the eighth image tile, the seventh image tile, and the ninth image tile.

17. The apparatus according to claim 11, wherein The image tiles do not include overlapping image content.

18. The apparatus according to claim 11, wherein The device is included in a virtual reality console, and the encoded image tiles are transmitted wirelessly.

19. A method for ordering a plurality of fragments of an image for encoding and transmission to a display device, the method comprising: rearranging the plurality of segments having a scan line order for encoding in a coding order that is different from the scan line order, the coding order being such that a center segment of the plurality of segments occupies a first position in the coding order, the first position preceding a second position occupied by the center segment in the scan line order, the center segment corresponding to a center of the image, and the coding order being such that at least one non-peripheral segment of the plurality of segments that shares a border with the center segment occupies a fifth position in the order, the fifth position preceding a sixth position occupied by the non-peripheral segment in the scan line order; encoding the segments in the coding order; and The encoded segments are sent to the display device in the encoding order in which they were encoded.

20. The method according to claim 19, wherein The encoding order further causes a peripheral fragment of the plurality of fragments to occupy a third position in the order, the third position being at least one of: 1) the same position as the position occupied by the peripheral fragment in the scan line order, or 2) a position after the fourth position occupied by the peripheral fragment in the scan line order, the peripheral fragment not sharing a border with the central fragment.

21. The method according to claim 20, wherein The peripheral segment is a first peripheral segment of a plurality of peripheral segments corresponding to corners of the image.

22. The method of claim 19, further comprising: dividing the image to form the plurality of segments; as well as The fragments are stored in a queue in scanline order.

Images