Methods and apparatus for image translation

Abstract

Embodiments of this application relate to methods and apparatus for image translation. One method includes converting (299) a pixel data frame into a first frame segmentation unit. The method further includes cropping (310) and shifting (312) the frame segmentation unit in response to X,Y coordinates to generate a second frame segmentation unit. The method also includes outputting (314) the second frame segmentation unit.

Description

[0001] Cross-referencing related applications

[0002] This application claims priority to U.S. Provisional Application No. 62 / 928,772, filed on October 31, 2019, entitled “Microsecond XY Image Translation,” which is incorporated herein by reference in its entirety. Technical Field

[0003] This invention relates to a method and apparatus for image translation. Background Technology

[0004] In augmented reality (AR) / virtual reality (VR) applications, a user wearing head-mounted goggles moves his or her head to view different parts of an image displayed in the goggles. One or more sensors in the head-mounted goggles detect the orientation of the user's head position and thus changes in the user's field of vision. A graphics processing unit (GPU) receives signals from the sensors and calculates pixel data frames relative to the user's current field of vision, which will be displayed on a display built into the AR / VR goggles. The rate at which video content can be updated is limited by the GPU's frame rate (also known as the playback speed). Summary of the Invention

[0005] In one example, a method includes converting a pixel data frame into frame partitioning units. The method further includes cropping and shifting a first frame partitioning unit in response to pixel coordinates to generate a second frame partitioning unit. The method also includes outputting the second frame partitioning unit.

[0006] In another example, a device includes a frame converter configured to convert video frames into corresponding frame partitioning units. A translation circuit is coupled to the frame converter. The translation circuit has inputs and an output. The translation circuit is configured to: crop and shift a first frame partitioning unit in response to pixel coordinates to generate a second frame partitioning unit; and output the second frame partitioning unit. Attached Figure Description

[0007] To describe the various examples in detail, reference will now be made to the accompanying drawings, in which:

[0008] Figure 1A Illustrated examples of display systems that include translation circuitry.

[0009] Figure 1B This illustration shows another example of a display system with a translation circuit.

[0010] Figure 2The diagram illustrates the relationship between the view windows within the reconstructed frame and the view windows within the reconstructed frame.

[0011] Figure 3 This diagram illustrates an example flowchart of a method for cropping and shifting individual bit planes of a reproduced frame.

[0012] Figure 4 Further illustrations are provided to explain the cutting and shifting operations.

[0013] Figure 5 The diagram illustrates the shifting of the position plane from one observation window to another.

[0014] Figure 6 The diagram illustrates a translation circuit that modifies the displayed image within the reproducible frame interval.

[0015] Figure 7 Further illustration of the translation circuit that performs changes to the displayed image within the reproduced frame interval.

[0016] Figure 8 The diagram illustrates the XY shift process performed on individual segments of a multi-segmented image.

[0017] Figure 9 This demonstrates an example of a translation circuit. Detailed Implementation

[0018] The described example pertains to allowing updates to image content to be displayed during the interval between replayed video frames, based on frame division units (explained below). An example of a frame division unit includes bit planes and subframes, further explained below. A viewport contains a subset of all pixel values ​​of the replayed frame. Viewport pixels are displayed on a display device within a head-mounted display (for example). Detected movement of the head-mounted display during the replayed frame interval causes a corresponding "shift" of the viewport within the replayed frame. Due to the shift of the viewport, the image content being displayed is also shifted.

[0019] Figure 1AAn exemplary display system 100 is shown. The display system 100 can be implemented as a head-mounted device. As shown, the display system 100 includes a sensor 112, a display device 114, a graphics processing unit (GPU) 120, and a display controller 130. The display system 100 can be an augmented reality / virtual reality (AR / VR) headset that may include a separate display device 114 for each of the user's eyes. The display device 114 includes any suitable type of display mechanism, such as a digital micromirror display (DMD), a liquid crystal on silicon (LCOS) display, a miniature light-emitting diode (micro-LED) display, etc. The sensor 112 may include one or more accelerometers and can detect the orientation / direction of the user's field of vision. The sensor 112 sends sensor signals 115 to the GPU. The GPU 120 receives data 111 indicating the content to be displayed on the display device 114 and reproduces individual frames 121 of pixel data. The content 111 is generated by a microprocessor 113 that executes software (e.g., a video game). The rendering process, performed by the GPU, generates color values ​​(e.g., red, green, and blue) for each pixel in a frame. The rendering process may involve applying textures or other types of visual effects to the content to be displayed. The GPU provides pixel data frames to the display controller 130, and pixel data frames 121 are stored in memory 132 located within or coupled to the display controller. The GPU 120 updates pixel data frames 121 based on the content and sensor signals 115 indicating the user's field of view. The rate at which the GPU 120 updates pixel data frames 121 in response to changes in content and sensor signals 115 is limited by its rendering speed (frame rate).

[0020] Display controller 130 reads pixel data frames (labeled pixel data frame 151) from memory 132 and converts pixel data frames 151 into bit planes 153 to be displayed on display device 114. Display controller 130 includes frame converter 133 and translation circuitry 134. Frame converter 133 retrieves pixel data frames 151 from memory 132 and converts pixel data frames 151 into frame partitioning units. In one exemplary embodiment, frame converter 133 includes digital circuitry including, for example, logic gates, registers, flip-flops, etc. As described above, an example of a frame partitioning unit is a bit plane. A frame contains an array of pixel color values. For example, each pixel contains red, green, and blue values. Each color value contains multiple bits (e.g., 8 bits). A bit plane is a sequence of bit values ​​across a pixel array for the same position of a given color. For example, all bits 2 of the red pixel value for a given frame constitute a bit plane. In an instance where each of the three colors is represented by 8 bits in the frame, there are 24 bit planes for that frame (8 bit planes for red, 8 bit planes for green, and 8 bit planes for blue). Another example of a frame partitioning unit is a subframe. A subframe is any segment of the original reproduced frame supplied by the GPU, which may be displayed one or more times along with other subframes during the GPU's frame interval to form the final image on the display device. Reference bit planes are discussed below, but the examples apply to other types of frame partitioning units.

[0021] Translation circuit 134 processes bit plane 153 from frame converter 133 and provides the resulting processed bit plane data 139 to display device 114. Furthermore, as explained below, translation circuit 134 shifts the view window within an individual bit plane based on new pixel coordinates (e.g., X, Y coordinates) 155 provided to translation circuit 134 by GPU 120 (which itself is determined based on sensor signal 115 from sensor 112). In this example, GPU 120 sends the new XY coordinates without sending a new data frame. Display device 114 receives bit plane data 139 and generates an image based on this data.

[0022] The translation circuit 134 may include digital circuitry, which is synthesized using circuit synthesis tools based on the disclosed functionality attributed to said digital circuitry. In another embodiment, the translation circuit 134 includes a hardware processor that executes firmware to perform the described functionality. Figure 9 An exemplary implementation of the translation circuit is shown in the figure and described below.

[0023] Figure 1B This demonstrates another instance of the display system 150. The display system 150 is architecturally similar to... Figure 1AThe display system 100 differs from 150 in that, for display system 150, the input of frame converter 133 is coupled to the output of GPU 120, and the output of frame converter 133 is coupled to memory 132. Thus, pixel data frames 121 from GPU 120 are provided to frame converter 133, which converts the pixel data frames 121 into bit plane 173 for storage in memory 132. Translation circuitry 134 then processes bit plane 175 from memory 132 as described herein.

[0024] Each pixel within a reconstructed frame contains (for example) one or more components of red, green, and blue—each color component containing multiple bits (e.g., 8 bits). Each color within a reconstructed frame contains multiple bit planes. A bit plane for a given color contains bit values ​​for the corresponding bits across all pixels within a given reconstructed frame. For example, for an 8-bit RGB representation (i.e., 8 bits for red, 8 bits for green, and 8 bits for blue), red comprises 8 bit planes within a single reconstructed frame. One bit plane contains bit values ​​[0] across the reconstructed frame for each red pixel value. Another plane contains bit values ​​[1] across the reconstructed frame for each red pixel value, and so on. Similarly, there are also 8 blue bit planes and 8 green bit planes. Translation circuitry 134 operates on individual bit planes (or other types of frame division units, such as subframes).

[0025] Multiple bit planes are displayed for each frame, and thus for a given frame rate, and these multiple bit planes are integrated over time by the user's sensory system (eyes / brain) to form the final image seen by the observer. The XY translation circuit 134 performs shifting on individual bit planes rather than the entire frame. When the sensor 112 detects a change in the user's head position or orientation, the bit planes of the currently displayed frame can be shifted before the next frame is available for processing by the display controller. Thus, the display system 100 advantageously allows shift updates to occur at a bit plane rate that is faster than the slower GPU frame rate. In this way, the display system 100 achieves a faster shift update rate on the displayed image without the power penalty of running the GPU at a faster playback speed / frame rate.

[0026] Figure 2The diagram illustrates the viewing window 220 within the reproduced frame 210. The viewing window 220 defines the portion of the reproduced frame 210 displayed on the display device 114. The reproduced frame 210 is generated by the GPU 120 and stored in the memory 132 of the display controller. In one example, the reproduced frame comprises a pixel array. Each pixel is represented by three color values ​​(e.g., red (R), green (G), and blue (B)). Each color value comprises a set of bits. In one example, each pixel includes 8 bits of red value, 8 bits of green value, and 8 bits of blue value, which can be displayed sequentially at a sufficiently high rate, allowing the human eye and brain to integrate the flashing pixel colors into any of a multitude of desired colors. Any of the multitude of colors can be formed by combinations of red, green, or blue pairs.

[0027] The viewing window 220 is smaller than the reproduced frame, which means that the viewing window 220 contains a subset of all pixel data of the reproduced frame. Figure 2 The illustration shows the observation window 220 located at an instance position, centered within the reproduced frame 210. The border 215 within the reproduced frame 210 contains pixel data outside the default position of the observation window 220 and therefore not displayed on the display device 114 (unless the user moves their head). When the sensor 112 detects a movement of the user's field of view, new XY coordinates are sent from the GPU 120 to a translation circuit 134, which responds by shifting the observation window in the direction of the change in the user's field of view. As the observation window 220 shifts, some pixels within the border 215 are displayed on the display device 114, while some pixels in the default observation window are discarded.

[0028] Figure 3 The flowchart 300 illustrates a method implemented by display system 100 or 150. At block 298, the method includes generating video frames. This operation may be performed by, for example, microprocessor 113, as explained above. At block 299, the method includes converting the video frames (e.g., via frame converter 133) into bit planes, or other types of frame partitioning units. Sensor signals (as indicated by block 301) may be received, indicating that pixel bit planes should be clipped and shifted.

[0029] At box 302, the translation circuit 134 awaits the start of the bit plane. The start of the bit plane is indicated by the bit plane start signal 135. Figure 1A and 1BThe bit plane start signal is generated by the display controller 130 and acknowledged before the first bit plane data segment is provided to the translation circuit 134 for processing. The bit plane start signal 135 notifies the translation circuit 134 to reinitialize its logic to prepare for each new input bit plane. At block 304, the bit plane start signal is received by the translation circuit 134.

[0030] At block 306, the translation circuit 134 determines whether to enable or disable its shift function. In one embodiment, the display controller 130 is programmable to perform the disclosed bit-plane shift functionality by, for example, writing a programmable value to a corresponding configuration register. If the shift function is not already enabled, flow control proceeds to block 314, where the unshifted bit plane is provided to the display device 114 for display thereon.

[0031] However, if the shift function is enabled, then at box 308, the method includes calculating the shift value based on the XY coordinates provided by GPU 120. An example procedure for calculating the shift value is described below. At box 310, a specific row and / or column within the bit plane is clipped based on the calculated shift value, and at box 312, the remaining (unclipped) bits of the bit plane are shifted based on the shift value. The resulting clipped and shifted bit plane is then sent to display 114 for display thereon.

[0032] Figure 4 Diagrammatic explanation based on Figure 3 Method 300 performs operations 310 and 312 to clip and shift the bit plane. The original rendered frame 210 from GPU 120 is displayed in view 410. The default position of view window 220 is also displayed as centered within rendered frame 210. In this example, the user's head and therefore field of view have moved upwards and to the left, causing view window 220 to shift towards the new view window 412. The new view window 412 now contains most, but not all, of the pixels from the default view window 220, as well as pixels 413 and 414 of the border 215, which now fall within the new view window 412.

[0033] according to Figure 3 The bit plane columns 415 and 416 and rows 417 and 418 outside the new viewing window 412 are clipped (e.g., deleted, marked as ignored, etc.). The resulting bit plane after clipping rows 417 and 418 is shown in view 420, and the bit plane after clipping columns 415 and 416 is shown in view 430. The resulting clipped bit plane data 430 is now mapped to the provided XY coordinates and sent to the display device 114.

[0034] Figure 5This shift operation is illustrated. For illustrative purposes, the pixels of display 114 are shown as representing a single pixel P1. Before the shift, P1 has display coordinates (X1, Y1). As the user's field of view shifts upwards, P1 shifts downwards and to the right to reach new coordinates (X2, Y2). (Return to Reference) Figure 4 The bit plane following the cropping and shifting operation (430) will be provided to the display device 114. Therefore, the XY shift of the pixels is calculated based on the viewpoint shift, which is determined by the GPU 120 based on sensor signals 115 from the display system 100. Upon the start of a new replay frame from the GPU, the view window is immediately initialized to its default (e.g., centered position) for the new replay frame.

[0035] Figure 3 And such as Figure 5 The method 300, further illustrated in the diagram, implements changes to the displayed content during frame intervals, such as... Figure 6 The diagram in the image is explained. Figure 6 The time series of reconstructed frames N, N+1, N+2, and N+3, calculated by GPU 120, are displayed. Each reconstructed frame begins after a VSYNC pulse, as shown at positions 601, 602, and 603. The frame interval is the time between VSYNC pulses, as shown at position 605.

[0036] During frame interval 605, GPU 120 determines changes in the field of view (based on sensor signal 115) at discrete time intervals (e.g., 10 times / frame interval 605) as indicated at 610. If the GPU detects a change in the user's field of view during the frame interval, then translation circuitry 134 is executed. Figure 3 Method 300 involves cropping and shifting one or more of the constituent bit planes containing the corresponding reproduced frames. This allows for updating of the displayed image during any given frame interval (e.g., as illustrated at 615).

[0037] Figure 7The time series of reconstructed frames N, N+1, N+2, and N+3, calculated by GPU 120, are also illustrated. Reference numeral 710 identifies the boundaries of each reconstructed frame, and reference numerals 720 and 740 identify the viewing window within each frame. As reflected in reference numerals 721 and 722, due to a detected change in the user's gaze (via sensor 112), viewing window 720 has shifted upward and to the right within frame 710. Frame 740 is the subsequent frame N+2, and in this example, viewing window 720 is reset to the center of the new frame 740. As reflected in reference numerals 741 and 742, due to a detected change in the user's gaze (via sensor 112), viewing window 720 has again shifted upward and to the right within frame 710. Viewing window 720 may or may not be reset to the center of each new reconstructed frame. Figure 7 The skipped update time period is also illustrated. When the skipped update time period occurs, the previously shifted pixel bit plane is reused.

[0038] Figure 8 The illustration shows that controller 130 generates an image comprising multiple segmented regions (e.g., segmented regions A, B, C, and D). Furthermore, XY translation functionality performed by translation circuit 134 can be executed for each individual segmented region, and the resulting cropped and shifted bit-planes can be joined to form the final translated image.

[0039] Figure 9An exemplary embodiment of the translation circuit 134 is shown. In this example, the translation circuit 134 includes an XY coordinate receiver 902, a subframe start detector 904, an image data cropping / shifting circuit 906, and a cropping / shifting calculation circuit 908. The XY coordinate receiver 902, subframe start detector 904, image data cropping / shifting circuit 906, and cropping / shifting calculation circuit 908 may include digital circuitry, including logic gates, flip-flops, registers, etc. The subframe start detector 904 has an input 912 via which it receives frame partitioning units 153 / 175 from the frame converter 133 or the memory 132, as described above. Upon receiving the frame partitioning unit, the subframe start detector 904 immediately acknowledges the bit plane start signal 135 destined for the coordinate receiver 902. The XY coordinate receiver 902 has an input 911 via which it receives the XY coordinates 155 from the GPU 120. XY coordinate receiver 911 is coupled to clip / shift calculation circuit 908 and provides the received XY coordinates (shown as signal 903) to clip / shift calculation circuit 908. Subframe start detector 904 also generates and provides data 905 indicating a default viewing window (e.g., viewing window 220) and shift margin 215. Data 905 may contain XY coordinates defining the boundaries of the default viewing window and the shift margin. Clip / shift calculation circuit 908 generates its hardware clip / shift value 907 derived from XY coordinate 903, and image data clip / shift circuit 906 clips and shifts the frame division unit based on hardware clip / shift value 907 and data 905 (as described above), and outputs the resulting clipped and shifted frame division unit 139 to be sent to a display device.

[0040] In one application, and as explained above, sensor 112 is included in a head-mounted headset (e.g., AR / VR) and operates to track the movement of the user's head. In another instance, the sensor is attached to a display device (e.g., a projector) or otherwise is part of said display device. In this latter case, the sensor may be an accelerometer and operates to track unintentional and undesirable vibrations or impacts to the display device in order to stabilize the image being displayed. The technique for stabilizing the image produced by the display device is the process described above. The display device may be a display device within a mobile communication device (e.g., a cellular phone). Whether the image is shifted due to the AR / VR user moving his or her head to attempt to view a previously undisplayed portion of the image, or whether the image is shifted to account for unintentional movement of the display device, the disclosed processing of the frame division unit is the same.

[0041] The term "couple" is used throughout this specification. This term may encompass connection, communication, or signaling paths that achieve a functional relationship consistent with the description of the invention. For example, if device A generates a signal to control device B to perform an action, then in a first instance, device A is coupled to device B, or in a second instance, device A is coupled to device B via an intervening component C (if the intervening component C substantially does not alter the functional relationship between device A and device B, such that a control signal generated by device A controls device B).

[0042] Modifications are possible in the described embodiments, and other embodiments are possible within the scope of the claims.

Claims

1. A method comprising: Video frames are generated at frame intervals using a graphics processing unit (GPU). The video frames are received via a frame converter; The video frame is converted into a first-order plane using the frame converter. Pixel coordinates are received via a translation circuit; Based on the pixel coordinates, the first bit plane is cropped and shifted by the translation circuit to generate the second bit plane. The cropping and shifting of the first bit plane occurs within a time period less than the frame interval used to generate the video frame. as well as The second bit plane is output through the translation circuit.

2. The method according to claim 1, wherein outputting the second bit plane includes outputting the second bit plane to a display device, the display device being: a digital micromirror device (DMD) display, a liquid crystal on silicon (LCOS) display, or a micro light-emitting diode (LED) display.

3. The method of claim 1, wherein receiving the pixel coordinates includes receiving the pixel coordinates from a sensor, wherein the sensor is a sensor of a display device.

4. An apparatus comprising: A graphics processing unit (GPU) is configured to generate video frames at frame intervals. Frame converter, the frame converter being configured to: Receive the video frame; The video frames are converted into corresponding first-order planes; and Output the first bit plane; and Translation circuitry, coupled to the frame converter, is configured to: Receive the first bit plane; Receive pixel coordinates; Based on the pixel coordinates, the first bit plane is cropped and shifted to generate a second bit plane, and the cropping and shifting of the first bit plane occurs within a time period shorter than the frame interval used to generate the video frame; and Output the second bit plane.

5. The device of claim 4, further comprising a display device coupled to the translation circuit, wherein the display device comprises a digital micromirror device (DMD) display, a liquid crystal on silicon (LCOS) display, or a micro light-emitting diode (LED) display.

6. The device of claim 4, further comprising a sensor, wherein the device is a head-mounted device, and the sensor is a sensor within the head-mounted device.

7. The device of claim 4, further comprising a sensor, wherein the device is a projector, and the sensor is a sensor of the projector, and the sensor is configured to determine the orientation of the head-mounted goggles.

8. The device of claim 4, further comprising an accelerometer, wherein the pixel coordinates are based on signals from the accelerometer.

9. The device of claim 4, further comprising a memory coupled to the frame converter, the memory being configured to store the video frames, and the frame converter being configured to retrieve the video frames from the memory.

10. The device of claim 4, further comprising a memory coupled to the frame converter and the translation circuitry, the memory being configured to store the first bit plane from the frame converter, and the translation circuitry being configured to retrieve the first bit plane from the memory.

11. An apparatus comprising: A graphics processing unit (GPU) is configured to generate video frames at frame intervals. A sensor coupled to the GPU, the sensor being configured to generate sensor signals, and the GPU being configured to generate pixel coordinates based on the sensor signals; A frame converter coupled to the GPU, the frame converter being configured to convert the video frames into corresponding first-order planes; Display device; Translation circuitry, coupled to the frame converter, coupled to the GPU, and coupled to the display device, is configured to: Receive the pixel coordinates; Based on the pixel coordinates, the first bit plane is cropped and shifted to generate the second bit plane, and the cropping and shifting of the first bit plane occurs within a time period less than the frame interval used to generate the video frame; as well as The second bit plane is output to the display device.

12. The device of claim 11, wherein the device is a virtual reality or augmented reality head-mounted device.

13. The device of claim 11, wherein the sensor is a sensor within a head-mounted display.

14. The device of claim 11, wherein the sensor is an accelerometer.

15. The device of claim 11, wherein the sensor is configured to generate the sensor signal in response to a change in the orientation of the head-mounted display.

16. The device of claim 11, further comprising a memory coupled to the GPU and the frame converter, the memory being configured to store the video frames, and the frame converter being configured to retrieve the video frames from the memory.

17. The device of claim 11, further comprising a memory coupled to the frame converter and the translation circuitry, the memory being configured to store the first bit plane from the frame converter, and the translation circuitry being configured to retrieve the first bit plane from the memory.

Images