Image slice transformation methods and electronic devices

Abstract

This disclosure provides an image slice transformation method and electronic device. The method includes: receiving an i-th image slice output by an image encoder, wherein the i-th image slice belongs to N image slices divided from an image frame, where i is an index and N is an integer; obtaining a slice header of the i-th image slice; and transforming the i-th image slice into an i-th subframe by modifying the slice header of the i-th image slice. Therefore, receiver latency can be reduced, which can correspondingly improve the user experience.

Description

Technical Field

[0001] This disclosure generally relates to an image transformation mechanism, and more specifically, to an image slicing transformation method and an electronic device. Background Technology

[0002] See Figure 1 This illustrates a common video streaming mechanism. As illustrated, video streaming is generally frame-based, where image frames can only be decoded after the encoder has output encoded image frames. In this case, subsequent processing (e.g., rendering) is postponed until the encoder has output image frames, resulting in longer latency.

[0003] In some image processing specifications (e.g., H.264), although the encoder can output image slices of an image frame, subsequent programs still need to wait until all image slices of the image frame have been output by the encoder, which may result in longer latency. Summary of the Invention

[0004] Therefore, this disclosure relates to an image slicing transformation method and an electronic device that can be used to solve the above-mentioned technical problems.

[0005] Embodiments of this disclosure provide an image slice transformation method, which is suitable for electronic devices. The method includes: receiving an i-th image slice output by an image encoder, wherein the i-th image slice belongs to N image slices divided from an image frame, i is an index, and N is an integer; obtaining a slice header of the i-th image slice; and transforming the i-th image slice into an i-th subframe by modifying the slice header of the i-th image slice.

[0006] Embodiments of this disclosure provide an electronic device comprising a storage circuit and a processor. The storage circuit stores program code. The processor is coupled to the non-transitory storage circuit and accesses the program code to execute: receiving an i-th image slice output by an image encoder, wherein the i-th image slice belongs to N image slices divided from an image frame, i being an index and N being an integer; obtaining a slice header of the i-th image slice; and transforming the i-th image slice into an i-th subframe by modifying the slice header of the i-th image slice. Attached Figure Description

[0007] The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0008] Figure 1 This illustrates a common video streaming mechanism.

[0009] Figure 2A schematic diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure is shown.

[0010] Figure 3 A flowchart illustrating an image slicing transformation method according to an embodiment of the present disclosure is shown.

[0011] Figure 4 A schematic diagram illustrating the initial macroblock index of a modified slice header according to an embodiment of the present disclosure is shown.

[0012] Explanation of icon numbers

[0013] 200: Electronic devices;

[0014] 202: Storage circuit;

[0015] 204: Processor;

[0016] 0, 1, 2, 3…509, 510…1019, 1020: macroblocks;

[0017] S310, S320, S330: Steps;

[0018] IM1: Image frame. Detailed Implementation

[0019] Reference will now be made in detail to the preferred embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals are used in the drawings and description to refer to the same or similar parts.

[0020] See Figure 2 This diagram illustrates an electronic device according to exemplary embodiments of the present disclosure. In some embodiments, the electronic device 200 may be any computer device and / or smart device, and the electronic device 200 may belong to a video streaming system including a transmitter and a receiver. In one embodiment, the electronic device 200 may be a transmitter of a video streaming system. In this case, the electronic device 200 may generate image frames and use an image encoder to encode the image frames. The electronic device 200 may then transmit the image frames to a receiver for decoding and displaying the image frames as visual content.

[0021] In one embodiment, the electronic device 200 may be a receiver of a video streaming system. In this case, the electronic device 200 may receive encoded image frames from a transmitter of the video streaming system, and the electronic device 200 may decode the image frames and display the image frames as visual content.

[0022] In various embodiments, the mechanism by which the transmitter and receiver encode / decode image frames can be based on any known image encoding / decoding algorithm. For a better understanding of the concepts of this disclosure, H.264 will be used as an example of an image encoding / decoding algorithm used by the transmitter and receiver, but this disclosure is not limited thereto.

[0023] exist Figure 2 In the electronic device 200, there are storage circuitry 202 and processor 204. Storage circuitry 202 is one or a combination of the following: static or mobile random access memory (RAM), read-only memory (ROM), flash memory, hard disk, or any other similar device, and storage circuitry 202 records program code and / or multiple modules executable by processor 204.

[0024] Processor 204 may be coupled to storage circuit 202, and processor 204 may be, for example, a graphics processing unit (GPU), a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc.

[0025] In embodiments of this disclosure, processor 204 may access modules and / or program code stored in storage circuit 202 to implement the image slicing transformation method provided in this disclosure, which will be further discussed below.

[0026] See Figure 3 The diagram illustrates a flowchart of an image slicing transformation method according to an embodiment of the present disclosure. The method of this embodiment can be derived from... Figure 2 The electronic device 200 in the text executes, and will be combined with the following text. Figure 2 The components shown in the figure Figure 3 The details of each step in the process are described.

[0027] In one embodiment, an image frame can be divided into N image slices (N is an integer), and an image encoder can encode the image slices and output the image slices.

[0028] Therefore, in step S310, the processor 204 receives the i-th image slice output by the image encoder, where i is the index.

[0029] In step S320, the processor 204 obtains the slice header of the i-th image slice. In one embodiment, since it is assumed that the image encoder operates based on H.264, the i-th image slice may contain multiple macroblocks and a slice header. In one embodiment, the slice header of the i-th image slice may contain a sequence parameter set and an initial macroblock index. Based on H.264, the initial macroblock index may be a variable named first_mb_in_slice, and the sequence parameter set may contain the height parameter of the i-th image slice, which may be a variable named pic_height_in_map_units_minus1. The definitions of the sequence parameter set, the height parameter, and the initial macroblock index can be found in the H.264 specification and will not be repeated here.

[0030] In H.264, a receiver cannot decode and / or render any of the N image slices associated with an image frame until all N image slices have been received. That is, a received image slice cannot be decoded and / or rendered by the receiver alone, which increases the latency of the video stream. However, if an image slice can be modified to treat it as an independent frame (which may be called a subframe), then the receiver can decode / render the subframe individually.

[0031] Therefore, in step S330, the processor 204 transforms the i-th image slice into the i-th subframe by modifying the slice header of the i-th image slice. In one embodiment, the processor 204 can modify the slice header of the i-th image slice by modifying the height parameter (i.e., pic_height_in_map_units_minus1) and the initial macroblock index (i.e., first_mb_in_slice) of the slice header of the i-th image slice.

[0032] In one embodiment, processor 204 may modify the height parameter of the slice header based on the height of the i-th image slice and the size of the macroblock. In one embodiment, processor 204 may divide the height of the i-th image slice by the height of one of the macroblocks to obtain a quotient (called Q). Then, processor 204 may modify the height parameter of the slice header to Q-1.

[0033] For example, suppose the height of the i-th image slice is 1600 pixels and the macroblock size is 16×16. In this case, the processor 204 can obtain a quotient of 100 by calculating 1600 / 16. Then, the processor 204 can modify the height parameter of the slice header to Q-1 (i.e., 99), but this disclosure is not limited thereto.

[0034] In one embodiment, processor 204 may modify the initial macroblock index of the slice header to a predetermined index, which may be, for example, 0. Specifically, based on H.264, the initial macroblock index is initially used to characterize the order of the first macroblocks of the i-th image slice in an image frame. After modifying the initial macroblock index of the slice header of the i-th image slice to 0, the i-th image slice can be considered an independent subframe. To better understand the above concepts, [further explanation will be provided]. Figure 4 As an example.

[0035] See Figure 4 This diagram illustrates the initial macroblock index of a modified slice header according to an embodiment of the present disclosure. Figure 4 In the image frame IM1, it can be divided into four image slices: image slice 0, image slice 1, image slice 2, and image slice 3. Each image slice can contain 510 macroblocks. For example, image slice 0 can contain macroblocks 0 to 509 of image frame IM1, and image slice 1 can contain macroblocks 510 to 1019 of image frame IM1. The order of macroblocks in image slice 2 and image slice 3 can be understood based on the above teachings and will not be elaborated further here.

[0036] Since the first macroblock in image slice 0 (i.e., macroblock 0) is ordered 0, the original value indicated by the initial macroblock index in the slice header of image slice 0 can be 0. Since the first macroblock in image slice 1 (i.e., macroblock 510) is ordered 510, the original value indicated by the initial macroblock index in the slice header of image slice 1 can be 510. Since the first macroblock in image slice 2 (i.e., macroblock 1020) is ordered 1020, the original value indicated by the initial macroblock index in the slice header of image slice 2 can be 1020.

[0037] exist Figure 4In the given scenario, when processor 204 receives image slice 0, it can modify the initial macroblock index in the slice header of image slice 0 to 0 and modify the height parameter to the corresponding Q-1, making image slice 0 with the modified header a subframe 0. When processor 204 receives image slice 1, it can modify the initial macroblock index in the slice header of image slice 1 to 0 and modify the height parameter to the corresponding Q-1, making image slice 1 with the modified header a subframe 1. When processor 204 receives image slice 2, it can modify the initial macroblock index in the slice header of image slice 2 to 0 and modify the height parameter to the corresponding Q-1, making image slice 2 with the modified header a subframe 2. When processor 204 receives image slice 3, it can modify the initial macroblock index in the slice header of image slice 3 to 0 and modify the height parameter to the corresponding Q-1, making image slice 3 with the modified header a subframe 3.

[0038] Based on the above teachings, the processor 204 can transform the i-th image slice into the i-th subframe by modifying the slice header of the i-th image slice. In one embodiment, since the length of the subframe header is specified as a predetermined length in the H.264 specification, the processor 204 can determine whether the length of the modified slice header of the i-th image slice meets the predetermined length. In one embodiment, in response to determining that the length of the modified slice header of the i-th image slice is shorter than the predetermined length, the processor 204 can adjust the length of the modified slice header of the i-th image slice by filling the modified slice header with specific data bits (e.g., 0) to meet the predetermined length. In this case, the i-th image slice with the modified and filled slice header can be regarded as the i-th subframe, but this disclosure is not limited thereto.

[0039] In one embodiment, when the electronic device 200 is implemented as a transmitter in a video streaming system, the electronic device 200 can transmit the i-th subframe to the receiver of the video streaming system. Since the i-th image slice has already been transformed into the i-th subframe, the receiver of the video streaming system can decode the i-th subframe and render the decoded i-th subframe immediately after receiving it. That is, the receiver does not need to wait for other subframes associated with the image frame before it can begin decoding and rendering the i-th subframe. Because the receiver can begin decoding the i-th subframe earlier, the receiver latency can be reduced.

[0040] In another embodiment, when the electronic device 200 is implemented as a receiver of a video streaming system, the electronic device 200 can decode the i-th subframe, and render and decode the i-th subframe immediately after transforming the i-th image slice into the i-th subframe. Therefore, the latency of the electronic device 200 can be reduced.

[0041] In other embodiments, the processor 204 can perform similar processing on other image slices of the N image slices. Figure 3 Other operations provided herein are used to transform these image slices into corresponding subframes. For example, after receiving the (i+1)th image slice from N image slices from the image encoder, the processor 204 can obtain the slice header of the (i+1)th image slice and transform the (i+1)th image slice into the (i+1)th subframe by modifying the slice header of the (i+1)th image slice, and the details of which can be referred to the teachings above, and will not be repeated here.

[0042] In one embodiment, when the electronic device 200 is implemented as a receiver of a video streaming system, the electronic device 200 can determine whether the aforementioned N image slices have been transformed into corresponding N subframes, and whether each subframe has been decoded and rendered. If so, the processor 204 can merge the rendered N subframes into visual content and provide the visual content.

[0043] For example, suppose electronic device 200 is a head-mounted display used to provide virtual reality content to its wearer. In this case, electronic device 200 can display the merged subframes as visual content for the wearer to view. Because electronic device 200 can operate with low latency, the user experience for the wearer can be improved accordingly.

[0044] In summary, embodiments of this disclosure can transform each image slice into a corresponding subframe by modifying its slice header. Since each subframe can be decoded and / or rendered independently, the receiver of the video streaming system does not need to wait for other subframes associated with the image frame. Therefore, the receiver can begin decoding each subframe earlier, and thus reduce receiver latency, which can correspondingly improve the user experience.

[0045] It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the invention without departing from the scope or spirit of the invention. In view of the foregoing, it is intended that this invention cover modifications and variations thereof, provided that such modifications and variations fall within the scope of the appended claims and their equivalents.

Claims

1. An image slicing transformation method, suitable for electronic devices, characterized in that, include: Receive the i-th image slice output by the image encoder, wherein the i-th image slice belongs to N image slices divided from the image frame, i is an index and N is an integer, wherein the i-th image slice includes multiple macroblocks; Obtain the slice header of the i-th image slice, wherein the slice header of the i-th image slice includes a height parameter and an initial macroblock index; as well as Transforming the i-th image slice into the i-th subframe by modifying the slice header of the i-th image slice includes: The height parameter of the slice header is modified based on the height of the i-th image slice and the size of the plurality of macroblocks; as well as Modify the initial macroblock index of the slice header to a predetermined index.

2. The image slice transformation method according to claim 1, wherein the step of modifying the height parameter of the slice header based on the height of the i-th image slice and the size of the plurality of macroblocks includes: The height of the i-th image slice is divided by the height of one of the plurality of macroblocks to obtain a quotient, wherein the quotient is represented by Q; as well as Modify the height parameter of the slice header to Q-1.

3. The image slicing transformation method according to claim 1, wherein the predetermined index is 0.

4. The image slicing transformation method according to claim 1, wherein after the step of modifying the initial macroblock index of the slice header to the predetermined index, the method further includes: In response to determining that the length of the modified slice header of the i-th image slice is shorter than a predetermined length, the length of the modified slice header of the i-th image slice is adjusted by filling the modified slice header with specific data bits to satisfy the predetermined length.

5. The image slicing transformation method according to claim 1 further includes: Receive the (i+1)th image slice from the N image slices from the image encoder; Obtain the slice header of the (i+1)th image slice; as well as The (i+1)th image slice is transformed into the (i+1)th subframe by modifying the slice header of the (i+1)th image slice.

6. The image slicing transformation method according to claim 5 further includes: In response to determining that the N image slices have been transformed into N subframes and that each subframe has been decoded and rendered, the rendered N subframes are merged into visual content and the visual content is provided.

7. The image slicing transformation method according to claim 1, wherein the electronic device belongs to a video streaming system.

8. An electronic device, characterized in that, include: Storage circuitry stores program code; as well as The processor, coupled to the storage circuitry, accesses the program code to: Receive the i-th image slice output by the image encoder, wherein the i-th image slice belongs to N image slices divided from the image frame, i is an index and N is an integer, wherein the i-th image slice includes multiple macroblocks; Obtain the slice header of the i-th image slice, wherein the slice header of the i-th image slice includes a height parameter and an initial macroblock index; The i-th image slice is transformed into the i-th subframe by modifying the slice header of the i-th image slice; The height parameter of the slice header is modified based on the height of the i-th image slice and the size of the plurality of macroblocks; as well as Modify the initial macroblock index of the slice header to a predetermined index.

9. The electronic device of claim 8, wherein when the height parameter of the slice header is modified based on the height of the i-th image slice and the size of the plurality of macroblocks, the processor performs: Divide the height of the i-th image slice by the height of one of the plurality of macroblocks to obtain a quotient, wherein the quotient is represented by Q; and Modify the height parameter of the slice header to Q-1.

10. The electronic device according to claim 8, wherein the predetermined index is 0.

11. The electronic device of claim 8, wherein after modifying the initial macroblock index of the slice header to the predetermined index, the processor further performs: In response to determining that the length of the modified slice header of the i-th image slice is shorter than a predetermined length, the length of the modified slice header of the i-th image slice is adjusted by filling the modified slice header with specific data bits to satisfy the predetermined length.

12. The electronic device of claim 8, wherein the processor further performs: Receive the (i+1)th image slice from the N image slices from the image encoder; Obtain the slice header of the (i+1)th image slice; and The (i+1)th image slice is transformed into the (i+1)th subframe by modifying the slice header of the (i+1)th image slice.

13. The electronic device of claim 12, wherein the processor further performs: In response to determining that the N image slices have been transformed into N subframes and that each subframe has been decoded and rendered, the rendered N subframes are merged into visual content and the visual content is provided.

14. The electronic device of claim 8, wherein the electronic device is a transmitter of a video streaming system.

15. The electronic device of claim 8, wherein the electronic device is a receiver of a video streaming system, and the electronic device decodes the i-th subframe and renders the decoded i-th subframe.

Images