Extended display apparatus, display method, electronic device, medium, and program product

Abstract

The present disclosure provides an extended display apparatus for a terminal device, a display method for an extended display apparatus for a terminal device, an electronic device, a non-transitory computer-readable storage medium, and a computer program product. The extended display apparatus for a terminal device comprises a central processing unit, a graphics processing unit, a three-dimensional optical device, a display screen driving unit, and a display screen. The central processing unit is configured to obtain a left-eye image and a right-eye image. The graphics processing unit is configured to fuse the left-eye image and the right-eye image to generate fused three-dimensional display data. The display screen driving unit is configured to write a plurality of pixel values of the fused three-dimensional display data into a plurality of respective sub-pixels of the display screen. The display screen is configured to enable the plurality of sub-pixels to emit light, so as to emit a directional light beam toward the left eye and a directional light beam toward the right eye via the three-dimensional optical device.

Description

Extended display devices, display methods, electronic devices, media, and program products Technical Field

[0001] The embodiments of this disclosure relate to the field of display devices, and more specifically to extended display devices for terminal devices, display methods for extended display devices for terminal devices, electronic devices, non-transitory computer-readable storage media, and computer program products. Background Technology

[0002] In today's digital entertainment era, two-dimensional images or videos can no longer satisfy users' pursuit of immersive, three-dimensional visual experiences. Three-dimensional image or video technology, as a representative of the next generation of display technology, simulates the principle of human eyes viewing the real world, and can present images with depth and three-dimensionality on the screen, greatly enhancing the viewing experience.

[0003] With the emergence of terminal devices such as mobile phones, viewing two-dimensional images or videos on these devices has become commonplace. However, the screens of these devices are relatively small and cannot meet the growing demand for viewing experiences. Furthermore, these devices typically cannot provide three-dimensional images or videos, preventing users from having a truly immersive viewing experience in such scenarios.

[0004] Summary of the Invention

[0005] According to one aspect of this disclosure, at least one embodiment provides an extended display device for a terminal device, including a central processing unit, a graphics processing unit, a three-dimensional optical device, a display driving unit, and a display screen; wherein the central processing unit is configured to acquire a left-eye image and a right-eye image; the graphics processing unit is configured to fuse the left-eye image and the right-eye image to generate fused three-dimensional display data; the display driving unit is configured to write multiple pixel values ​​of the fused three-dimensional display data into multiple sub-pixels of the display screen, and the display screen is configured to cause the multiple sub-pixels to emit light so as to emit directional light beams directed toward the left eye and directional light beams directed toward the right eye through the three-dimensional optical device.

[0006] According to one aspect of this disclosure, at least one embodiment provides a display method for an extended display device of a terminal device, comprising: obtaining a left-eye image and a right-eye image; fusing the left-eye image and the right-eye image to generate fused three-dimensional display data; fusing the left-eye image and the right-eye image to generate fused three-dimensional display data; and writing multiple pixel values ​​of the fused three-dimensional display data into multiple sub-pixels of the display screen of the extended display device, such that the multiple sub-pixels emit light to emit directional light beams directed toward the left eye and directional light beams directed toward the right eye through the three-dimensional optical devices of the extended display device.

[0007] According to another aspect of this disclosure, at least one embodiment provides an electronic device, including: a memory for storing computer instructions; and a processor for reading the computer instructions from the memory and performing a method according to at least one embodiment of this disclosure.

[0008] According to another aspect of this disclosure, at least one embodiment provides a non-transitory computer-readable storage medium having computer instructions stored thereon, wherein, when executed by a processor, the computer instructions cause the processor to perform a method according to at least one embodiment of this disclosure.

[0009] According to another aspect of this disclosure, at least one embodiment provides a computer program product including computer instructions, wherein, when executed by a processor, the computer instructions cause the processor to perform a method according to at least one embodiment of this disclosure. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 illustrates a schematic diagram of an application scenario according to at least one embodiment of the present disclosure.

[0012] Figure 2 shows a block diagram of an extended display device of a terminal device according to at least one embodiment of the present disclosure.

[0013] Figure 3 shows a block diagram of another embodiment of the extended display device of a terminal device according to at least one embodiment of the present disclosure.

[0014] Figure 4 illustrates a schematic diagram of a central processing unit calibrating fused 3D display data generated by a graphics processing unit based on calibration data and the position and / or orientation of the user's eyes or head, according to at least one embodiment of the present disclosure.

[0015] Figure 5 shows a flowchart of a display method of an extended display device of a terminal device according to at least one embodiment of the present disclosure.

[0016] Figure 6 shows a block diagram of an exemplary electronic device according to at least one embodiment of the present disclosure.

[0017] Figure 7 illustrates a schematic diagram of a non-transitory computer-readable storage medium according to at least one embodiment of the present disclosure.

[0018] Figures 8A-8C illustrate a connection method between an extended display device and a terminal device according to at least one embodiment of the present disclosure.

[0019] Figures 9A-9B illustrate two ways of connecting an extended display device and a terminal device according to at least one embodiment of the present disclosure. Detailed Implementation

[0020] Referring now to specific embodiments of this disclosure, examples of which are illustrated in the accompanying drawings. Although this application will be described in conjunction with specific embodiments, it will be understood that it is not intended to limit this application to the described embodiments. Rather, it is intended to cover variations, modifications, and equivalents included within the spirit and scope of this disclosure. It should be noted that the method steps described herein can be implemented by any functional block or functional arrangement, and any functional block or functional arrangement can be implemented as a physical entity or a logical entity, or a combination of both.

[0021] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0022] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0023] Today, users typically view two-dimensional images or videos on mobile devices such as smartphones. These images or videos may originate from social media apps, online streaming platforms, video platforms, gaming platforms, and so on. However, users cannot obtain a truly immersive three-dimensional viewing experience in this usage scenario.

[0024] In existing technologies, fused 3D display data is typically generated on the terminal device and then sent to a 3D display screen to display a 3D stereoscopic effect. However, if the terminal device only has 2D images or videos, or if it lacks the capability to fuse 2D images (or images from the left and right eyes) into 3D display data, a 3D stereoscopic viewing experience cannot be achieved. Furthermore, since most terminal devices are portable, their processing units cannot easily perform the real-time fusion of 2D images (or images from the left and right eyes) into 3D display data.

[0025] The defects and problems existing in the above-mentioned prior art solutions are also the result of the inventor's careful research after practical and creative labor. The discovery process of the above problems and the solutions proposed by at least one embodiment disclosed below for the above problems are all creative contributions of the inventor during the invention process.

[0026] According to at least one embodiment of this disclosure, the extended display device can convert two-dimensional images or videos displayed on a terminal device such as a mobile phone into three-dimensional images or videos, or the extended display device can directly receive the left-eye and right-eye images of the terminal device, fuse them into fused three-dimensional display data, and display it on the extended display device to obtain a better three-dimensional viewing experience. According to at least one embodiment of this disclosure, regardless of whether the image or video playback application on the mobile phone provides a control method for operating the displayed three-dimensional images or videos, the displayed three-dimensional images or videos can be operated, for example, by operating the mobile phone's application or touching the screen of the extended display device, such as playing, pausing, stopping, fast forwarding, rewinding, speed adjustment, screen zooming, etc. That is, according to at least one embodiment of this disclosure, it is not merely a display screen for playing images or videos sent by a mobile phone, but a user-friendly intelligent extended display device for converting two-dimensional to three-dimensional images or videos. According to at least one embodiment of this disclosure, it can also provide two-dimensional and three-dimensional switching display, supporting the display of both two-dimensional and three-dimensional images or videos. According to at least one embodiment of this disclosure, head tracking / eye tracking of the user can also be provided via a camera or sensor to offer the user fused 3D display data that better matches their visual perception. According to at least one embodiment of this disclosure, the influence of optical errors of the 3D optics in different extended display device hardware and / or hardware errors of the camera or sensor on the 3D display effect can also be eliminated. According to at least one embodiment of this disclosure, a good connection can also be established with terminal devices such as mobile phones to obtain display data and / or operation commands from the terminal device.

[0027] Figure 1 illustrates a schematic diagram of an application scenario according to at least one embodiment of the present disclosure.

[0028] When users view 2D images / videos on devices such as mobile phones, they often encounter insufficient immersion due to screen size limitations. As users demand greater depth and a more three-dimensional feel from content and crave the immersive experience of 3D videos, a technological challenge arises: how to perfectly present 2D or 3D video content from mobile phones on a 3D-enabled screen, while ensuring convenience, compatibility, and cost-effectiveness in this process.

[0029] Figure 1 illustrates a solution to the aforementioned problem. A user may operate an application on terminal device 110 to browse or play two-dimensional display data, such as images / videos. This application can be a general-purpose application for displaying two-dimensional data, such as a long video application, a short video application, a social media platform, a photo album application, etc., or it can be a specialized application designed specifically for the extended display device 120. In this case, the user may want to view three-dimensional videos, three-dimensional photos, etc.

[0030] Users can send display data and / or operation commands from the terminal device 110 to the extended display device 120, or perform other interactions with the extended display device 120, through the interface between the terminal device 110 and the extended display device 120. The interface may include, for example, a wired connection interface 140 (e.g., a Universal Serial Bus (USB) Type-A interface, a USB Type-B interface, a Micro USB interface, a USB Type-C interface, a Lightning interface, etc.), or a short-range wireless communication interface 130 (e.g., a Wireless Fidelity (WiFi) interface, a Bluetooth interface, a ZigBee interface, etc.). Display data may be two-dimensional images, two-dimensional videos, or other display data. Operation commands may be various commands that manipulate the display data, such as play, pause, stop, fast forward, rewind, playback speed adjustment, screen zoom, etc.

[0031] The extended display device 120 can have a three-dimensional display panel for three-dimensional display. The three-dimensional display panel can include a display screen, a display screen driving unit, and three-dimensional optics. The display screen driving unit can drive the sub-pixels of the display screen to emit light, which, through the three-dimensional optics, emits directional light beams directed towards the left eye and the right eye, respectively, and is displayed to the viewing user. This allows the left and right eyes to see two images from different angles, achieving a three-dimensional display and a naked-eye 3D effect. Here, the sub-pixels of the display screen can include light-emitting diodes (LEDs). For a full-color LED display screen, a pixel is typically composed of multiple color LEDs to form a single pixel capable of displaying multiple colors. The three-dimensional optics can include lenticular lenses, parallax barriers, and diffractive light backlights.

[0032] The extended display device 120 can have eye-tracking or head-tracking capabilities. Eye-tracking technology monitors the user's gaze movement using specialized sensors, employing infrared light reflection and video ophthalmography, and utilizing multiple cameras to track reflected light. This process identifies the user's gaze focus and optimizes image rendering and interactive experience accordingly. Head-tracking technology tracks the user's head movements using stereo cameras or other sensors and feeds the information back to the computing unit. The computing unit adjusts the image display based on the head position and movement information to ensure the user always sees the correct view.

[0033] The extended display device 120 may have a touch screen, and users can directly use their fingers or other media (such as a stylus) to click, drag, zoom and other operations on the touch screen to perform various commands for displaying data, such as play, pause, stop, fast forward, rewind, speed up playback, zoom the screen, etc.

[0034] The extended display device 120 can support switching between two-dimensional and three-dimensional displays, and can support displaying two-dimensional images or videos as well as three-dimensional images or videos. This can be achieved by driving three-dimensional optical devices through the display screen driving unit of the extended display device 120.

[0035] This solves the problem of users wanting to view three-dimensional visual effects when viewing two-dimensional images or videos on terminal devices such as mobile phones.

[0036] Figure 2 shows a block diagram of an extended display device 220 of a terminal device 210 according to at least one embodiment of the present disclosure.

[0037] As shown in Figure 2, the extended display device 220 of the terminal device 210 may include a central processing unit (CPU) 221, a graphics processing unit (GPU) 222, a display driving unit 223, a three-dimensional optical device 224, a display screen 225, and a three-dimensional optical device driver 226 (optional).

[0038] The communication interface between the terminal device 210 and the extended display device 220 may include a short-range wireless communication interface (such as a WiFi interface, Bluetooth interface, ZigBee interface, etc.) and / or a wired interface (such as a USB Type A interface, USB Type B interface, USB Type C interface, Lightning interface, etc.). In this way, the extended display device can establish a reliable connection with a terminal device, such as a mobile phone, to obtain display data and / or operation commands from the terminal device.

[0039] The central processing unit (CPU) 221 can be configured to acquire left-eye and right-eye images. Here, the left-eye and right-eye images can come from the terminal device 210. For example, the terminal device 210 can use a local conversion algorithm to convert the 2D display data into left-eye and right-eye images and send them to the CPU 221 of the extended display device 220. Alternatively, it can directly capture the left-eye and right-eye images using a binocular camera and send them to the CPU 221 of the extended display device 220. The method and process of acquiring the left-eye and right-eye images are not limited here. Alternatively, the left-eye and right-eye images can be obtained by the CPU 221 from the 2D display data from the terminal device 210. For example, the terminal device 210 can send the 2D display data to the CPU 221 of the extended display device 220 without converting the 2D display data into left-eye and right-eye images, and the CPU 221 can use a local conversion algorithm to convert the 2D display data into left-eye and right-eye images. Here, conversion algorithms can, for example, analyze depth information in a 2D image to generate a depth map. Depth information can be obtained through various methods, including but not limited to: professionals manually labeling different regions in an image based on its content; or automatically analyzing image content using computer vision and machine learning algorithms to estimate the depth information of each pixel or region. This method relies on factors such as feature points, texture, and lighting in the image. The depth map describes the distance from each pixel in the image to the observer. Then, based on the depth map, the 2D image can be converted into stereoscopic left-eye and right-eye images. For example, a 2D image can be converted into a 3D model using 3D reconstruction techniques based on the depth map. This involves mapping pixels in the 2D image to corresponding points in 3D space and constructing the 3D contours and surfaces of the object. In the 3D model, different views are generated for the left and right eyes based on the principle of binocular parallax, which can be achieved through image transformation and interpolation algorithms. The positions and viewing angles of the two eyes are simulated in 3D space, and the two images seen by the left and right eyes are rendered separately. Accurate determination of the user's binocular position and / or viewing angle allows for better determination of the left and right eye images to suit the user's naked-eye 3D viewing experience. Further details on how to improve the accuracy of determining the user's binocular position and / or viewing angle will follow.

[0040] The graphics processing unit (GPU) 222 can be configured to fuse left-eye and right-eye images to generate fused 3D display data, and send this fused 3D display data to the display driver unit. Fusion refers to combining the contents of the left-eye and right-eye images from both perspectives into a single (3D image) according to an optical model. This image is then written to the corresponding sub-pixels on the display screen by the display driver unit to realize the pixel information for the left and right eye perspectives. The optical model can be based on the binocular parallax principle of the human visual system, ensuring that the left eye only sees the left-eye image and the right eye only sees the right-eye image, while also ensuring that these two images are correctly fused in the brain.

[0041] The display driving unit 223 is configured to write multiple pixel values ​​of the fused three-dimensional display data into multiple sub-pixels of the display screen 225 respectively.

[0042] In color display, a pixel of three-dimensional display data can correspond to a red pixel value, a blue pixel value, and a green pixel value (specifically, a voltage value). This data is written into the red LED, blue LED, and green LED of a sub-pixel of the display screen, so that all the LEDs of all these sub-pixels of the display screen emit light due to the voltage.

[0043] The display screen 225 is configured to emit a plurality of sub-pixels to emit directional beams toward the left eye and directional beams toward the right eye through the three-dimensional optical device 224.

[0044] The three-dimensional optical device 224 can be a lenticular lens. A lenticular lens allows light from the sub-pixels of the display screen to be emitted separately towards the left and right eyes, forming a stereoscopic image. Other three-dimensional optical devices, such as parallax barriers and diffraction backlights, can also achieve similar effects, but will not be described in detail here. The three-dimensional optical device 224 can be placed above the display screen 225, that is, between the display screen 225 and the human eye.

[0045] The left and right eye sub-pixel information of the GPU 222 is displayed through the display screen 225. The display screen 225 may include a liquid crystal display, an organic light-emitting diode display, a micro light-emitting diode display, etc., and is not limited thereto. The display screen 225 can communicate with the graphics processing unit GPU 222 through interfaces such as the Mobile Industry Processor Interface (MIPI), Embedded DisplayPort (EDP), and Low Voltage Differential Signaling (LVDS). This allows the left and right eyes to see two images from different angles, achieving a naked-eye 3D display effect.

[0046] Thus, according to at least one embodiment of this disclosure, a two-dimensional image or video displayed on a terminal device such as a mobile phone can be converted into a three-dimensional image or video on an extended display device, or the extended display device can directly receive the left-eye and right-eye images of the terminal device, fuse them into fused three-dimensional display data, and display them on the extended display device to obtain a better three-dimensional viewing experience, without the terminal device such as a mobile phone having three-dimensional images or videos or having the function of converting two-dimensional images or videos into three-dimensional images or videos.

[0047] In some embodiments, the central processing unit (CPU) 221 may be configured to receive operation commands that manipulate the fused 3D display data and control the graphics processing unit (GPU) 222 to perform corresponding operations. The operation commands may include those from the terminal device.

[0048] Operation commands can include, for example, playing, pausing, stopping, fast-forwarding, rewinding, speeding up playback, and zooming the screen on the fused 3D display data. For instance, if the operation command is to play the fused 3D display data, the CPU 221 can continuously send left-eye and right-eye images to the GPU 222, causing the GPU 222 to continuously generate fused 3D display data, thus enabling playback of the fused 3D display data. Conversely, if the operation command is to pause the fused 3D display data, the CPU 221 can pause sending left-eye and right-eye images to the GPU 222, causing the GPU 222 to pause generating fused 3D display data, thus pausing the fused 3D display data. Other operation commands cause the CPU 221 to perform corresponding operations to achieve the desired effect.

[0049] In other words, the extended display device 220 is not merely a pure display terminal for playing display data on the terminal device 210 (in a pure display terminal, the operation of display data is performed by the terminal device, and the terminal device executes the operation to achieve the result, such as pausing the screen, while the pure display terminal is only responsible for displaying the paused screen). Instead, the extended display device 220 has the function of independently operating the display data, enabling it to interpret the meaning of operation commands and, based on the interpreted meaning of the operation commands, independently operate the fused 3D display data to be displayed by the extended display device 220 in conjunction with the terminal device 210.

[0050] This feature also allows for an expansion of the types of operation commands from the terminal device, enabling corresponding operations to be performed in response to operation commands not present in the application of the terminal device 210 itself. For example, if the application of the terminal device 210 itself does not support 3x fast forward, a new gesture or touch method (such as continuous two-finger touch on the touch screen of the terminal device 210) can be designed as an operation command, allowing the CPU 221 of the extended display device 220 to interpret the operation command as 3x fast forward and perform 3x fast forward operation on the fused 3D display data to be displayed.

[0051] This function also enables the reception and interpretation of operation commands from devices other than the terminal device 210 (such as the touch screen or buttons of the extended display device 220 itself or other remotely linked devices), so as to operate the fused 3D display data to be displayed by the extended display device 220 independently of the terminal device 210.

[0052] Operation commands can include commands from the display screen.

[0053] In some embodiments, the display screen 225 may be a touch display screen, and the central processing unit (CPU) 221 may be configured to receive operation commands from the touch display screen 225. Here, the operation commands from the terminal device 210 may be the same as or different from the operation commands from the touch display screen 225 (for example, the operation command may not include 3x speed playback, but may include 3x speed playback). The touch display screen can, for example, use I... 2 The C interface is used to communicate with the central processing unit CPU 221.

[0054] According to at least one embodiment of this disclosure, regardless of whether the image or video playback application on the mobile phone provides a control method for operating the displayed three-dimensional image or video, the displayed three-dimensional image or video can be operated, for example, by operating the mobile phone application or touching the screen of the extended display device. This operation includes functions such as playing, pausing, stopping, fast forwarding, rewinding, speeding up playback, zooming the screen, etc. In other words, according to at least one embodiment of this disclosure, it is not just a display screen for playing images or videos sent by the mobile phone, but a smart extended display device that can be easily operated by the user to convert two-dimensional to three-dimensional.

[0055] The device 220 may also include a three-dimensional optics driver 226 to combine with a switchable three-dimensional optics device, such as a switchable lenticular lens 224, to achieve two-dimensional and three-dimensional switching functionality. The principle of the switchable lenticular lens three-dimensional display is the same as that of the lenticular lens, but a liquid crystal layer is added to achieve the two-dimensional / three-dimensional switching function. The liquid crystal layer can dynamically control the light by controlling the arrangement and orientation of liquid crystal molecules under the action of an electric field. In some embodiments, the operation command or operation command includes a two-dimensional and three-dimensional switching command to indicate whether to display two-dimensional display data or fuse three-dimensional display data. The central processing unit CPU 221 can be configured to, in response to the two-dimensional and three-dimensional switching command indicating the display of two-dimensional display data, instruct the display drive unit 223 to drive the three-dimensional optics device 224 to provide diffused light; and in response to the two-dimensional and three-dimensional switching command indicating the display of fused three-dimensional display data, instruct the display drive unit 223 to drive the three-dimensional optics device 224 to provide a directional beam directed to the left eye and a directional beam directed to the right eye.

[0056] On the one hand, when displaying two-dimensional data, the three-dimensional optical device driver 226 can control the liquid crystal layer in a state where the lenticular lens no longer produces a significant refraction or focusing effect on the light. At this time, the light emitted by the display screen is randomly scattered into the viewer's eyes in the form of diffuse light, forming a standard two-dimensional image. Specifically, the liquid crystal molecules in the liquid crystal layer may be in a uniformly aligned state, without changing the direction of light propagation, thereby ensuring that the entire screen presents a uniform two-dimensional image.

[0057] On the other hand, when displaying fused 3D display data, the 3D optical device driver 226 can adjust the arrangement of liquid crystal molecules within the liquid crystal layer according to a preset algorithm. These liquid crystal molecules are deflected under the influence of an electric field, forming specific optical structures, such as Fresnel liquid crystal sub-lenses. The combination of lenticular lenses and these dynamically formed liquid crystal sub-lenses precisely refracts and focuses light. By controlling the direction of light projection in different areas, slightly different images are received by the left and right eyes, thereby synthesizing a stereoscopic visual effect in the brain.

[0058] In this way, it is easy to switch between two-dimensional and three-dimensional effects, thereby being compatible with displaying two-dimensional display data and integrating three-dimensional display data, which makes this extended display device 220 have a wider range of application scenarios.

[0059] The display driver unit 223 can be connected via, for example, UART, USB, I... 2 It communicates with the central processing unit CPU 221 via an interface such as C. Note that the three-dimensional optical device driver 226 may also be a software module that drives the three-dimensional optical device 224, rather than hardware.

[0060] Thus, according to at least one embodiment of this disclosure, it is also possible to provide a two-dimensional and three-dimensional switching display, which can support the display of two-dimensional images or videos, as well as the display of three-dimensional images or videos.

[0061] The device 220 may also have a wired power supply or a rechargeable power source (such as a lithium battery) to provide power.

[0062] Figure 3 shows a block diagram of another embodiment of the extended display device 220 of the terminal device 210 according to at least one embodiment of the present disclosure.

[0063] The alignment and coordination between the 3D optical device 224 and the sub-pixels of the display screen 225 are crucial to the 3D display effect. However, during the actual manufacturing of the extended display device 220, hardware errors may occur between the 3D optical device and the sub-pixels of the display screen 225, causing a deviation between the image directed to the left eye and the image directed to the right eye, resulting in a poor 3D display effect. For example, the light emitted by a sub-pixel of the display screen should be directed to the left eye, but if the corresponding prism of the 3D optical device is misaligned or deviated from the position of that sub-pixel, the corresponding prism of the 3D optical device will be unable to direct the light emitted by that sub-pixel to the left eye, or the direction of the light will deviate from the direction of the left eye.

[0064] In some embodiments of this disclosure, the central processing unit (CPU) 221 may be configured to calibrate fused 3D display data based on calibration data. The calibration data may be pre-stored in a memory (not shown), so that the CPU 221 can retrieve the calibration data from the memory.

[0065] The calibration data may include data used to calibrate hardware errors of the sub-pixels of the 3D optics 224 and the display screen 225. For example, the central processing unit (CPU) 221 may calibrate the left-eye and right-eye images according to a predetermined algorithm based on the calibration data to correct hardware errors of the 3D optics 224. Note that hardware errors here are typically optical errors, i.e., errors in optical effects caused by errors in the size and position of internal hardware components. Alternatively, the CPU 221 may provide calibration data to the graphics processing unit (GPU) 222, and the GPU 222 may perform a fusion algorithm to fuse the left-eye and right-eye images into fused 3D display data for writing to the sub-pixels of the display screen, taking into account the calibration data from the CPU 221.

[0066] Specifically, in the fusion algorithm, an optical model of a three-dimensional optical device 224 (e.g., a prismatic prism) can first be established, including its geometry, refractive index, surface curvature, and possible manufacturing errors (such as angular deviations, shape distortions, etc.). Manufacturing errors can be measured or obtained using specialized optical design software or simulation tools. Then, based on this optical model, the manufacturing errors of, for example, the prismatic prism can be evaluated through ray tracing and optical simulation to predict how these errors will affect the generation of left-eye and right-eye images. The predetermined fusion algorithm can also, based on how these predicted errors affect the generation of left-eye and right-eye images, use image processing and computer vision algorithms to perform geometric correction on the generated left-eye and right-eye images to compensate for the effects of these errors, thereby correcting image distortion and deformation caused by, for example, angular deviations or shape distortions in the hardware of the prismatic prism.

[0067] As described above, it is important to improve the accuracy of determining the position and viewing angle of the user's eyes. The higher the accuracy of the position and viewing angle of the eyes, the more accurately the left and right eye images can be determined to adapt to the user's naked-eye 3D perception.

[0068] In some embodiments, the extended display device 220 may also include a camera or sensor 226 configured to capture or sense images of at least the user's eyes or head.

[0069] The central processing unit (CPU) 221 is configured to determine the position and / or orientation of the user's eyes or head based on images of at least the user's eyes or head. The CPU 221 can provide the position and / or orientation of the user's eyes or head to the graphics processing unit (GPU) 222 for calibration and fusion of 3D display data. Changes in the position and / or orientation of the user's eyes or head can be detected in real time. The GPU 222 can then use a fusion algorithm that takes into account these changes to fuse the 3D display data, providing more accurate information on the sub-pixels of the left and right eye images. This ensures that the image seen by the user always aligns with the user's eye position, thus providing a realistic 3D effect. The GPU 222 can be replaced by an image processing chip such as a neural processing unit (NPU) or a field-programmable gate array (FPGA). In some embodiments, a camera and sensors can also be used in combination to improve the accuracy of eye tracking and / or head tracking.

[0070] Thus, according to at least one embodiment of this disclosure, head tracking / eye tracking of the user can be provided via a camera or sensor to provide the user with fused 3D display data that is more in line with their eye's viewing experience.

[0071] Hardware errors may exist during the manufacturing of camera or sensor hardware and the assembly of display extension devices. These errors, such as camera deflection, offset, and position relative to the display screen, can lead to errors in the optical or sensing effects of the camera or sensor, resulting in errors in the aforementioned eye tracking and / or head tracking. In some embodiments, calibration data may further include data used to calibrate the hardware errors of the camera or sensor. The impact of these hardware errors on eye tracking and / or head tracking can be simulated by measuring them, and the effects of these hardware errors on eye tracking and / or head tracking can be eliminated through software algorithms.

[0072] Based on the aforementioned hardware errors, the graphics processing unit (GPU) 222 can run a fusion algorithm that takes these errors into account to adjust the sub-pixel data for the left and right eyes, and fuse them into fused 3D display data, which is then written to the sub-pixels of the display screen. In this way, by combining the display screen with 3D optics, the left and right eyes can see accurate images from different angles, thus achieving glasses-free 3D display.

[0073] Thus, according to at least one embodiment of this disclosure, it is possible to correct the hardware position deviation of the sub-pixels of the three-dimensional optical device and the display screen, as well as the hardware deviation of the camera itself used for eye / head tracking and the hardware position deviation between it and the display screen, and to eliminate the influence of optical errors of the three-dimensional optical device itself and / or hardware errors of the camera or sensor on the three-dimensional display effect.

[0074] Figure 4 illustrates a schematic diagram of a central processing unit calibrating fused 3D display data generated by a graphics processing unit based on calibration data and the position and / or orientation of the user's eyes or head, according to at least one embodiment of the present disclosure.

[0075] The graphics processing unit (GPU) 222 can generate a fused 3D display image based on the left-eye and right-eye images. This fused 3D display image can be a fusion of the left-eye and right-eye images. However, due to optical errors in the 3D optics and / or hardware errors in the camera or sensor, as well as the position and orientation of the user's head and / or eyes, the fused 3D display image may not provide a natural and realistic 3D effect for the user.

[0076] Therefore, the central processing unit (CPU) 221 can obtain calibration data. This calibration data may include data used to calibrate optical errors of the three-dimensional optics and / or data used to calibrate hardware errors of the camera or sensor. The CPU 221 can also calculate the position and / or orientation of the eyes and / or head by capturing and / or sensing images of the eyes or head through the camera / sensor 226.

[0077] The central processing unit (CPU) 221 can provide calibration data and / or the position and / or orientation of the eye and / or head calculated by the CPU 221 to the graphics processing unit (GPU) 222. The GPU 222 runs a fusion algorithm that takes into account the calibration data and / or the position and / or orientation data of the eye and / or head to provide a fused 3D display image, which is written to the sub-pixels of the display screen 225. Thus, the 3D optics 224, in conjunction with the sub-pixels of the display screen 225, can emit light suitable for the left eye and light suitable for the right eye, to provide the user with fused 3D display data that is more in line with their eye's viewing perspective.

[0078] Thus, according to at least one embodiment of this disclosure, the extended display device can convert two-dimensional images or videos displayed on a terminal device such as a mobile phone into three-dimensional images or videos, or the extended display device can directly receive the left-eye and right-eye images of the terminal device, fuse them into fused three-dimensional display data, and display it on the extended display device to obtain a better three-dimensional viewing experience. According to at least one embodiment of this disclosure, regardless of whether the image or video playback application on the mobile phone provides a control method for operating the displayed three-dimensional images or videos, the displayed three-dimensional images or videos can be operated, for example, by operating the mobile phone's application or touching the screen of the extended display device, such as playing, pausing, stopping, fast forwarding, rewinding, speed adjustment, zooming, etc. That is, according to at least one embodiment of this disclosure, it is not merely a display screen for playing images or videos sent by a mobile phone, but a user-friendly intelligent extended display device for converting two-dimensional to three-dimensional images or videos. According to at least one embodiment of this disclosure, it can also provide two-dimensional and three-dimensional switching display, supporting the display of both two-dimensional and three-dimensional images or videos. According to at least one embodiment of this disclosure, head tracking / eye tracking of the user can also be provided via a camera or sensor to offer the user fused 3D display data that better matches their visual perception. According to at least one embodiment of this disclosure, the influence of optical errors of the 3D optics in different extended display device hardware and / or hardware errors of the camera or sensor on the 3D display effect can also be eliminated. According to at least one embodiment of this disclosure, a good connection can also be established with terminal devices such as mobile phones to obtain display data and / or operation commands from the terminal device.

[0079] Figures 8A-8C illustrate a connection method between an extended display device and a terminal device according to at least one embodiment of the present disclosure.

[0080] Figure 8A shows a schematic diagram of the front view of a terminal device such as a mobile phone. As can be seen, the front of the mobile phone has a display screen.

[0081] Figure 8B shows a schematic diagram of the rear view of the extended display device. The extended display device has a display screen at the rear and also a housing for accommodating a mobile phone.

[0082] Figure 8C shows a schematic diagram of inserting a mobile phone into an extended display device. In this way, the extended display device can be easily attached to the mobile phone as a phone case, allowing the user to flip the phone over and view the fused 3D display data on the extended display device's screen from the back.

[0083] Figures 9A-9B illustrate two ways of connecting an extended display device and a terminal device according to at least one embodiment of the present disclosure.

[0084] Figure 9A shows the connection between the extended display device and the terminal device along the long side, allowing the extended display device to be opened like flipping through a book. The display screen of the extended display device can be located on the outside or inside. Figure 9B shows the connection between the extended display device and the terminal device along the short side, with the display screen of the extended display device also located on the outside or inside.

[0085] All of the above methods extend the size and shape of the display device to match the terminal device, allowing for a tight fit with the terminal device, thus achieving plug-and-play functionality, while being portable and not taking up much space.

[0086] In this way, users can easily view the fused 3D display data of the extended display device that can be closely integrated with the terminal device.

[0087] Next, FIG5 shows a flowchart of a display method 500 of an extended display device of a terminal device according to at least one embodiment of the present disclosure.

[0088] The execution subject of the method provided in this disclosure is generally a computer device with certain computing power, which may include a processor and a memory, the memory storing computer instructions; the processor is configured to execute the computer instructions in the memory to perform the method according to at least one embodiment of this disclosure.

[0089] As shown in Figure 5, the display method 500 of the extended display device of the terminal device may include steps 510, 520, 530 and 540.

[0090] In step 510, the left eye image and the right eye image are obtained.

[0091] In step 520, the left-eye image and the right-eye image are fused to generate fused 3D display data.

[0092] In step 530, the multiple pixel values ​​of the fused 3D display data are written into multiple sub-pixels of the display screen of the extended display device.

[0093] In step 540, the multiple sub-pixels emit light to emit directional beams toward the left eye and directional beams toward the right eye through the three-dimensional optics of the extended display device.

[0094] In some embodiments, the method 500 may further include: receiving an operation command to manipulate the fused three-dimensional display data, and controlling the graphics processing unit to perform the corresponding operation, wherein the operation command includes an operation command from the display screen and / or an operation command from the terminal device.

[0095] In some embodiments, the operation commands include a two-dimensional and three-dimensional switching command to indicate whether to display two-dimensional display data or fused three-dimensional display data. The method further includes: in response to the two-dimensional and three-dimensional switching command indicating the display of two-dimensional display data, driving a three-dimensional optics device of the extended display device to provide diffuse light; and in response to the two-dimensional and three-dimensional switching command indicating the display of fused three-dimensional display data, driving a three-dimensional optics device to provide a directional light beam directed towards the left eye and a directional light beam directed towards the right eye.

[0096] In some embodiments, the left-eye and right-eye images are obtained from the terminal device, or the left-eye and right-eye images are obtained by the central processing unit of the extended display device from two-dimensional display data from the terminal device.

[0097] Method 500 may also include: generating fused 3D display data based on calibration data.

[0098] In some embodiments, calibration data may include data used to calibrate the optical errors of a three-dimensional optical device.

[0099] In some embodiments, method 500 may further include: determining the position and / or orientation of the user's eyes or head based on images of at least the user's eyes or head captured or sensed by a camera or sensor of the extended display device, wherein the calibration data further includes the determined position and / or orientation of the user's eyes or head.

[0100] In some embodiments, the calibration data also includes data used to calibrate hardware errors of the camera or sensor.

[0101] Thus, according to at least one embodiment of this disclosure, the extended display device can convert two-dimensional images or videos displayed on a terminal device such as a mobile phone into three-dimensional images or videos, or the extended display device can directly receive the left-eye and right-eye images of the terminal device, fuse them into fused three-dimensional display data, and display it on the extended display device to obtain a better three-dimensional viewing experience. According to at least one embodiment of this disclosure, regardless of whether the image or video playback application on the mobile phone provides a control method for operating the displayed three-dimensional images or videos, the displayed three-dimensional images or videos can be operated, for example, by operating the mobile phone's application or touching the screen of the extended display device, such as playing, pausing, stopping, fast forwarding, rewinding, speed adjustment, zooming, etc. That is, according to at least one embodiment of this disclosure, it is not merely a display screen for playing images or videos sent by a mobile phone, but a user-friendly intelligent extended display device for converting two-dimensional to three-dimensional images or videos. According to at least one embodiment of this disclosure, it can also provide two-dimensional and three-dimensional switching display, supporting the display of both two-dimensional and three-dimensional images or videos. According to at least one embodiment of this disclosure, it is also possible to provide head tracking / eye tracking of the user via a camera or sensor, thereby providing the user with fused 3D display data that is more consistent with their eye's viewing experience. According to at least one embodiment of this disclosure, it is also possible to eliminate the influence of optical errors of the 3D optics in different extended display device hardware and / or hardware errors of the camera or sensor on the 3D display effect.

[0102] Figure 6 shows a block diagram of an exemplary electronic device according to at least one embodiment of the present disclosure.

[0103] The electronic device may include a processor 610 and a memory 620, the memory 620 being coupled to the processor 610 and storing computer instructions therein for performing steps of various methods of at least one embodiment of the present disclosure when executed by the processor 610.

[0104] The processor 610 may include, but is not limited to, one or more processors or microprocessors.

[0105] The memory 620 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (e.g., hard disk, floppy disk, solid-state drive, removable disk, CD-ROM, DVD-ROM, Blu-ray disc, etc.).

[0106] In addition, the electronic device may also include (but is not limited to) a data bus 630, an input / output (I / O) bus 640, an external device 650, and an input / output device 660 (e.g., a keyboard, mouse, speaker, monitor, etc.).

[0107] The processor 610 can communicate with external displays 650 and input / output devices 660 via the I / O bus 640.

[0108] In one embodiment, the at least one computer instruction may also be compiled into or comprise a computer program product or software product, wherein one or more computer instructions, when executed by a processor, perform the steps of the various functions and / or methods in the embodiments described herein.

[0109] Figure 7 illustrates a schematic diagram of a non-transitory computer-readable storage medium according to at least one embodiment of the present disclosure.

[0110] As shown in Figure 7, the non-transitory computer-readable storage medium 720 stores instructions, such as computer instructions 710. When the computer instructions 710 are executed by a processor, the various methods described above can be performed. Non-transitory computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (e.g., hard disks, floppy disks, solid-state drives, removable disks, CD-ROMs, DVD-ROMs, Blu-ray discs, etc.). For example, the non-transitory computer-readable storage medium 720 can be connected to a computing device such as a computer, and then, when the computing device executes the computer instructions 710 stored on the computer-readable storage medium 720, the various methods described above can be performed.

[0111] This disclosure may also include a computer program product that can perform the methods, steps, and operations given herein. For example, such a computer program product may be a computer software package, computer code instructions, or a computer-readable tangible medium having computer instructions tangibly stored (and / or encoded) thereon, which can be executed by a processor to perform the operations described herein. The computer program product may include packaging materials.

[0112] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The term “such as / for example” as used herein refers to the phrase “such as / for example but not limited to,” and is used interchangeably with it.

[0113] The flowcharts and method descriptions in this disclosure are merely illustrative examples and are not intended to require or imply that the steps of the various embodiments must be performed in the given order. As those skilled in the art will recognize, the steps in the above embodiments can be performed in any order. Words such as "then," "next," etc., are not intended to limit the order of the steps; these words are only used to guide the reader through the description of these methods. Furthermore, any reference to a singular element, such as the use of the articles "a," "one," or "the," is not to be construed as limiting that element to the singular.

[0114] Furthermore, the steps and apparatus in the various embodiments herein are not limited to any one embodiment. In fact, new embodiments can be conceived by combining relevant steps and apparatus in the various embodiments herein based on the concepts of this disclosure, and these new embodiments are also included within the scope of this disclosure.

[0115] The above methods can be implemented in hardware, software, firmware, or any combination thereof.

[0116] Furthermore, modules and / or other suitable means for carrying out the methods and techniques described herein can be downloaded from a server wirelessly when appropriate. Alternatively, the various methods described herein can be provided via a storage component so that the various methods are available when coupled to the storage component. Additionally, any other suitable techniques for providing the methods and techniques described herein to the device can be utilized.

[0117] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit at least one embodiment of the present disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. An extended display device for a terminal equipment, comprising a central processing unit, a graphics processing unit, a three-dimensional optical device, a display screen driving unit, and a display screen; wherein, The central processing unit is configured to acquire left-eye and right-eye images; The graphics processing unit is configured to fuse the left-eye image and the right-eye image to generate fused 3D display data; The display screen driving unit is configured to write multiple pixel values ​​of the fused 3D display data into multiple sub-pixels of the display screen, respectively. The display screen is configured such that the plurality of sub-pixels emit light to emit directional beams toward the left eye and directional beams toward the right eye through the three-dimensional optics.

2. The apparatus according to claim 1, wherein, The central processing unit is configured to receive operation commands that manipulate the fused 3D display data and control the graphics processing unit to perform corresponding operations. The operation commands include operation commands from the display screen and / or operation commands from the terminal device.

3. The apparatus of claim 2, further comprising a three-dimensional optical device driver, the operation command including a two-dimensional and three-dimensional switching command to indicate whether to display two-dimensional display data or the fused three-dimensional display data, the central processing unit being configured to: in response to the two-dimensional and three-dimensional switching command indicating the display of the two-dimensional display data, drive the three-dimensional optical device to provide diffuse light via the three-dimensional optical device driver; and in response to the two-dimensional and three-dimensional switching command indicating the display of the fused three-dimensional display data, drive the three-dimensional optical device to provide the directional beam directed to the left eye and the directional beam directed to the right eye via the three-dimensional optical device driver.

4. The apparatus of claim 1, wherein, The left-eye image and the right-eye image are from the terminal device, or the left-eye image and the right-eye image are obtained by the central processing unit from two-dimensional display data from the terminal device.

5. The apparatus of claim 1, wherein, The graphics processing unit is also configured to generate the fused 3D display data based on the calibration data.

6. The apparatus of claim 5, wherein, The calibration data includes data used to calibrate the optical errors of the three-dimensional optical device. 7.The apparatus of claim 5, further comprising a camera or a sensor configured to take or sense an image of at least an eye or a head of a user; the central processing unit is configured to determine a position and / or an orientation of the eye or the head of the user according to the image of at least the eye or the head of the user taken or sensed by the camera or the sensor, wherein, The calibration data also includes the determined position and / or orientation of the user's eyes or head.

8. The apparatus of claim 7, wherein, The calibration data also includes data used to calibrate the hardware errors of the camera or the sensor.

9. The apparatus of claim 1, wherein, The extended display device is the same size and shape as the terminal device and can be tightly integrated with the terminal device.

10. A display method for an extended display device of a terminal device, comprising: Obtain the left eye image and the right eye image; The left-eye image and the right-eye image are fused to generate fused 3D display data; The pixel values ​​of the fused 3D display data are written into multiple sub-pixels of the display screen of the extended display device. The plurality of sub-pixels emit light to emit directional beams toward the left eye and directional beams toward the right eye through the three-dimensional optics of the extended display device.

11. The method of claim 10, further comprising: Receive operation commands to manipulate the fused 3D display data, and control the extended display The graphics processing unit of the display device performs corresponding operations. The operation commands include operation commands from the display screen and / or operation commands from the terminal device.

12. The method of claim 11, wherein, The operation commands include two-dimensional and three-dimensional switching commands to indicate whether to display the two-dimensional display data or the fused three-dimensional display data. The method further includes: in response to the two-dimensional and three-dimensional switching command indicating the display of two-dimensional display data, driving the three-dimensional optical device to provide diffuse light; and in response to the two-dimensional and three-dimensional switching command indicating the display of the fused three-dimensional display data, driving the three-dimensional optical device to provide the directional beam directed to the left eye and the directional beam directed to the right eye.

13. The method of claim 10, wherein, The left-eye image and the right-eye image are from the terminal device, or the left-eye image and the right-eye image are obtained by the central processing unit of the extended display device from two-dimensional display data from the terminal device.

14. The method of claim 10, further comprising: The fused 3D display data is generated based on the calibration data.

15. The method of claim 14, wherein, The calibration data includes data used to calibrate the optical errors of the three-dimensional optical device.

16. The method of claim 14, further comprising: Based on images of at least the user's eyes or head captured or sensed by the camera or sensor of the extended display device, the position and / or orientation of the user's eyes or head are determined, wherein the calibration data further includes the determined position and / or orientation of the user's eyes or head.

17. The method of claim 16, wherein, The calibration data also includes data used to calibrate hardware errors of the camera or sensor.

18. An electronic device comprising: Memory, which stores computer instructions; At least one processor is configured to execute the computer instructions in the memory to perform the method according to any one of claims 10-17.

19. A non-transitory computer-readable storage medium having computer instructions stored thereon, wherein When executed by a processor, the computer instructions cause the processor to perform the method according to any one of claims 10-17.

20. A computer program product comprising computer instructions, wherein When executed by a processor, the computer instructions cause the processor to perform the method according to any one of claims 10-17.

Images