Human eye tracking device, method, 3D display device, and method

Abstract

This application relates to naked-eye 3D display technology, and discloses an eye tracking device, including: an eye tracker, including a first black-and-white camera configured to capture a first black-and-white image and a second black-and-white camera configured to capture a second black-and-white image; an eye tracking image processor configured to identify the presence of eyes based on at least one of the first black-and-white image and the second black-and-white image and determine the spatial position of the eyes based on the eyes identified in the first black-and-white image and the second black-and-white image. The above device can achieve high-precision determination of the spatial positions of the user's binocular eyes respectively and high-speed eye identification, or in other words, real-time eye tracking. This application also discloses an eye tracking method and a 3D display device and method.

Description

Technical Field

[0001] This application relates to autostereoscopic 3D display technology, for example, to human eye tracking devices and methods, and 3D display devices and methods. Background Art

[0002] 3D (stereoscopic) imaging is one of the hot technologies in the video industry, driving the technological transformation from flat panel display to 3D display. 3D display technology is a key link in the 3D imaging industry, mainly divided into two categories, namely glasses-type 3D display and autostereoscopic 3D display technology. Autostereoscopic 3D display technology is a technology that enables users to directly view 3D display images without wearing glasses. Compared with glasses-type 3D display, autostereoscopic 3D display belongs to free 3D display technology, reducing the constraints on users.

[0003] Autostereoscopic 3D display is viewpoint-based. Recently, multi-view autostereoscopic 3D display has also been proposed, so as to form a sequence of parallax images (frames) at different positions in space, enabling 3D image pairs with parallax relationships to enter the left and right eyes of a person respectively, thus bringing a 3D feeling to the user. For a traditional multi-view autostereoscopic 3D display with, for example, N viewpoints, multiple independent pixels on the display panel are used to project multiple viewpoints in space.

[0004] In some research progress, such multi-view display also provides a better viewing experience based on face or human eye tracking data. In some conventional face or human eye tracking devices, only the distance between the face and the screen is detected, and the viewpoint position where the human eyes are located is determined relying on a preset or default interpupillary distance. The accuracy of such identification is not high and cannot meet the requirements of high-quality autostereoscopic 3D display. For example, such face or human eye tracking devices cannot determine the actual spatial positions of the user's both eyes, nor can they determine whether the human eyes are tilted relative to the screen, which may cause viewpoint calculation errors.

[0005] Moreover, conventional face or human eye tracking devices also have a computational bottleneck when tracking at a high real-time rate, which affects the recognition speed or tracking speed.

[0006] This background art is only for facilitating the understanding of the relevant technologies in this field and is not regarded as an admission of the prior art. Summary of the Invention

[0007] To have a basic understanding of some aspects of the disclosed embodiments, a summary of the embodiments is given below. It is not intended to identify key / important constituent elements or depict the protection scope of the invention, but rather serves as a preamble to the subsequent detailed description.

[0008] Embodiments of this application are intended to provide human eye tracking devices and methods, and 3D display devices and methods.

[0009] In one solution, a human eye tracking device is provided, including: a human eye tracker, including a first black-and-white camera configured to capture a first black-and-white image and a second black-and-white camera configured to capture a second black-and-white image; a human eye tracking image processor configured to identify the presence of a human eye based on at least one of the first black-and-white image and the second black-and-white image and determine the spatial position of the human eye based on the human eye identified in the first black-and-white image and the second black-and-white image.

[0010] With such a human eye tracking device, it is possible to accurately determine the spatial positions of the user's two eyes respectively and achieve high-speed human eye recognition, or in other words, real-time human eye tracking.

[0011] In some embodiments, the human eye tracking device further includes a human eye tracking data interface configured to transmit human eye spatial position information indicating the spatial position of the human eye.

[0012] In some embodiments, the human eye tracker further includes an infrared emission device.

[0013] In some embodiments, the infrared emission device is configured to emit infrared light with a wavelength greater than or equal to 1.5 microns.

[0014] In some embodiments, the first black-and-white camera and the second black-and-white camera are configured to respectively capture a first black-and-white image sequence including the first black-and-white image and a second black-and-white image sequence including the second black-and-white image.

[0015] In some embodiments, the human eye tracking image processor includes a synchronizer configured to determine the first black-and-white image and the second black-and-white image that are time-synchronized for human eye recognition and determination of the human eye spatial position.

[0016] In some embodiments, the human eye tracking image processor includes: a buffer configured to buffer multiple first black-and-white images and second black-and-white images in the first black-and-white image sequence and the second black-and-white image sequence; a comparator configured to compare multiple first black-and-white images and second black-and-white images before and after in the first black-and-white image sequence and the second black-and-white image sequence; a decision maker configured to, when the comparator fails to identify the presence of a human eye in the current first black-and-white image and the second black-and-white image in the first black-and-white image sequence and the second black-and-white image sequence but identifies the presence of a human eye in the first black-and-white image and the second black-and-white image before or after, use the human eye spatial position determined based on the first black-and-white image and the second black-and-white image before or after as the current human eye spatial position.

[0017] Based on this, for example, when there are situations such as stuttering or frame skipping in the first or second black-and-white camera, a more coherent display screen can be provided for the user to ensure the viewing experience.

[0018] In another solution, a 3D display device is provided, including: a multi-viewpoint autostereoscopic 3D display screen, including a plurality of sub-pixels corresponding to a plurality of viewpoints; an eye tracking device as described above to obtain the spatial position of the human eye; a 3D processing device configured to determine the corresponding viewpoint according to the spatial position of the human eye obtained by the eye tracking device, and render the sub-pixels corresponding to the viewpoint of the multi-viewpoint autostereoscopic 3D display screen based on the 3D signal.

[0019] In some embodiments, the multi-viewpoint autostereoscopic 3D display screen includes a plurality of composite pixels, and each composite pixel among the plurality of composite pixels includes a plurality of composite sub-pixels, and each composite sub-pixel among the plurality of composite sub-pixels is composed of a plurality of sub-pixels corresponding to a plurality of viewpoints.

[0020] In some embodiments, the 3D processing device is communicatively connected to the eye tracking device.

[0021] In some embodiments, it further includes: a 3D shooting device configured to collect 3D images; the 3D shooting device includes a depth camera and at least two color cameras.

[0022] In some embodiments, the eye tracking device and the 3D shooting device are integrally arranged.

[0023] In some embodiments, the 3D shooting device is placed in front of the 3D display device.

[0024] In another solution, an eye tracking method is provided, including: shooting a first black-and-white image and a second black-and-white image; identifying the presence of the human eye based on at least one of the first black-and-white image and the second black-and-white image; determining the spatial position of the human eye based on the human eye identified in the first black-and-white image and the second black-and-white image.

[0025] In some embodiments, the eye tracking method further includes: transmitting eye spatial position information indicating the spatial position of the human eye.

[0026] In some embodiments, the eye tracking method further includes: when the first black-and-white camera or the second black-and-white camera is working, using an infrared emission device to emit infrared light.

[0027] In some embodiments, the eye tracking method further includes: respectively shooting a first black-and-white image sequence including the first black-and-white image and a second black-and-white image sequence including the second black-and-white image.

[0028] In some embodiments, the eye tracking method further includes: determining the first black-and-white image and the second black-and-white image with time synchronization.

[0029] In some embodiments, the human eye tracking method further includes: caching a plurality of first black-and-white images and second black-and-white images in the first black-and-white image sequence and the second black-and-white image sequence; comparing the plurality of first black-and-white images and second black-and-white images before and after in the first black-and-white image sequence and the second black-and-white image sequence; when the presence of a human eye is not recognized in the current first black-and-white image and second black-and-white image in the first black-and-white image sequence and the second black-and-white image sequence, and the presence of a human eye is recognized in the first black-and-white image and second black-and-white image before or after, using the human eye spatial position determined based on the first black-and-white image and second black-and-white image before or after as the current human eye spatial position.

[0030] In another solution, a 3D display method is provided, including: obtaining the human eye spatial position of a user; determining the corresponding viewing point according to the human eye spatial position; rendering sub-pixels corresponding to the viewing point of a multi-view autostereoscopic 3D display screen based on a 3D signal.

[0031] In some embodiments, the 3D display method further includes: providing a multi-view autostereoscopic 3D display screen, including a plurality of composite pixels, each composite pixel of the plurality of composite pixels including a plurality of composite sub-pixels, and each composite sub-pixel of the plurality of composite sub-pixels being composed of a plurality of sub-pixels corresponding to a plurality of viewing points.

[0032] The above general description and the following description are only exemplary and explanatory, and are not used to limit this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] One or more embodiments are exemplarily illustrated by corresponding drawings. These exemplary illustrations and the drawings do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements. The drawings do not constitute a scale limitation, and in which:

[0034] Figure 1A and Figure 1B is a schematic structural diagram of a 3D display device according to an embodiment of the present disclosure;

[0035] Figure 1C is a schematic structural diagram of a human eye tracking device according to an embodiment of the present disclosure;

[0036] Figure 2 is a schematic hardware structural diagram of a 3D display device according to an embodiment of the present disclosure;

[0037] Figure 3 is Figure 2 a schematic software structural diagram of the 3D display device shown;

[0038] Figure 4 is a schematic diagram of determining a human eye spatial position using a human eye tracking device according to an embodiment of the present disclosure;

[0039] Figures 5A to 5C It is a front schematic view of a 3D display device according to an embodiment of the present disclosure;

[0040] Figure 6A and Figure 6B It is a schematic diagram of the positional relationship between a user's face and a 3D display device according to an embodiment of the present disclosure;

[0041] Figure 7 It is a step schematic diagram of an eye tracking method according to an embodiment of the present disclosure;

[0042] Figure 8 It is a step schematic diagram of a 3D display method according to an embodiment of the present disclosure;

[0043] Figure 9 It is a schematic diagram of realizing the display of a multi-viewpoint autostereoscopic 3D display screen of a 3D display device by using the 3D display method according to an embodiment of the present disclosure, where each of the user's two eyes corresponds to one viewpoint.

[0044] Reference numerals:

[0045] 100: 3D display device; 101: processor; 122: register; 110: multi-view autostereoscopic 3D display screen; 120: 3D shooting device; 121: camera module; 121a: first color camera; 121b: second color camera; 121c: depth camera; 125: 3D image output interface; 126: 3D image processor; 130: 3D processing device; 131: buffer; 140: signal interface; 150: eye tracking device; 151: eye tracker; 151a: first black and white camera; 151b: second black and white camera; 154: infrared emission device; 152: eye tracking image processor; 155: synchronizer; 156: buffer; 157: comparator; 153: eye tracking data interface; CP: composite pixel; CSP: composite sub-pixel; 200: 3D display device; 201: processor; 202: external memory interface; 203: memory; 204: USB interface; 205: charging management module; 206: power management module; 207: battery; 210: multi-view autostereoscopic 3D display screen; 212: audio module; 213: speaker; 214: receiver; 215: microphone; 216: headphone interface; 217: key; 218: motor; 219: indicator; 220: 3D shooting device; 221: camera module; 222: register; 223: GPU; 224: codec; 225: 3D image output interface; 226: 3D image processor; 230: 3D processing device; 240: signal interface; 250: eye tracking device; 260: SIM card interface; 270: sensor module; 2701: proximity light sensor; 2702: ambient light sensor; 2703: pressure sensor; 2704: barometric pressure sensor; 2705: magnetic sensor; 2706: gravity sensor; 2707: gyroscope sensor; 2708: acceleration sensor; 2709: distance sensor; 2710: temperature sensor; 2711: fingerprint sensor; 2712: touch sensor; 2713: bone conduction sensor; 281: mobile communication module; 282: antenna; 283: wireless communication module; 284: antenna; 310: application layer; 320: framework layer; 330: core class library and Runtime; 340: kernel layer; T: distance between two black and white cameras; 401a: focal plane of the first black and white camera 151a; 401b: focal plane of the second black and white camera 151b; f: focal length; Oa: lens center of the first black and white camera 151a; Ob: lens center of the second black and white camera 151b; Za: optical axis of the first black and white camera 151a; Zb: optical axis of the second black and white camera 151b; R: user's right eye; L: user's left eye; P: user's interpupillary distance; α: tilt angle of the observer's face with respect to the multi-view autostereoscopic 3D display screen;XRa: The X-axis coordinate of the image of the user's right eye R in the focal plane 401a of the first black-and-white camera 151a; XRb: The X-axis coordinate of the image of the user's right eye R in the focal plane 401b of the second black-and-white camera 151b; XLa: The X-axis coordinate of the image of the user's left eye L in the focal plane 401a of the first black-and-white camera 151a; XLb: The X-axis coordinate of the image of the user's left eye L in the focal plane 401b of the second black-and-white camera 151b; DR: The distance between the user's right eye R and the multi-viewpoint autostereoscopic display; DL: The distance between the user's left eye L and the multi-viewpoint autostereoscopic display; 500: 3D display device; 600: 3D display device.; Detailed implementation manners

[0046] In order to understand the features and technical content of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration purposes only and are not used to limit the embodiments of the present disclosure.

[0047] In this document, "autostereoscopic display" relates to a technology in which a user (viewer) can observe a 3D display image on a flat panel display without wearing glasses for 3D display, including but not limited to "parallax barrier", "lenticular lens", and "directed backlight" technologies.

[0048] In this document, "multi-viewpoint" has its conventional meaning in the art, meaning that different images displayed by different pixels or sub-pixels of the display can be viewed at different positions (viewpoints) in space. In this document, multi-viewpoint will mean at least 3 viewpoints.

[0049] In this document, "grating" has a broad interpretation in the art, including but not limited to "parallax barrier" gratings and "lens" gratings, such as "lenticular lens" gratings.

[0050] In this document, "lens" or "lens grating" has its conventional meaning in the art, for example, including lenticular lenses and spherical lenses.

[0051] A conventional "pixel" means the smallest display unit in terms of its resolution when a 2D display or as a 2D display.

[0052] However, in some embodiments of the present disclosure, the "composite pixel" referred to when applied to the multi-viewpoint technology in the field of autostereoscopic 3D display refers to the smallest display unit when the autostereoscopic 3D display provides multi-viewpoint display, but it does not exclude that a single composite pixel for the multi-viewpoint technology may include or be presented as multiple pixels for 2D display. In the present disclosure, unless specifically stated as a composite pixel or a 3D pixel for "3D display" or "multi-viewpoint" applications, a pixel will refer to the smallest display unit for 2D display. Similarly, when describing the "composite sub-pixel" of an autostereoscopic 3D display for multi-viewpoints, it will refer to the composite sub-pixel of a single color presented in the composite pixel when the autostereoscopic 3D display provides multi-viewpoint display. In the present disclosure, the sub-pixel in the "composite sub-pixel" will refer to the smallest display unit of a single color, which often corresponds to the viewpoints.

[0053] In one aspect, there is provided an eye tracking device, comprising: an eye tracker, including a first black-and-white camera configured to capture a first black-and-white image and a second black-and-white camera configured to capture a second black-and-white image; an eye tracking image processor configured to identify the presence of an eye based on at least one of the first black-and-white image and the second black-and-white image and determine the spatial position of the eye based on the location of the eye present in the first black-and-white image and the second black-and-white image; and an eye tracking data interface configured to transmit the eye spatial position information of the spatial position of the eye.

[0054] With such an eye tracking device, it is possible to accurately determine the spatial positions of the user's binocular eyes respectively and identify the eyes at high speed, or in other words, to track the eyes in real time.

[0055] In some embodiments, the eye tracker further includes an infrared emission device.

[0056] In some embodiments, the infrared emission device is configured to emit infrared light with a wavelength greater than or equal to 1.5 micrometers.

[0057] In some embodiments, the first black-and-white camera and the second black-and-white camera are configured to capture a first sequence of black-and-white images and a second sequence of black-and-white images respectively.

[0058] In some embodiments, the eye tracking image processor includes a synchronizer configured to determine the first black-and-white image and the second black-and-white image that are time-synchronized.

[0059] In some embodiments, the eye tracking image processor includes: a buffer configured to buffer a plurality of first black-and-white images and second black-and-white images in the first sequence of black-and-white images and the second sequence of black-and-white images; and a comparator configured to compare a plurality of previous and subsequent first black-and-white images and second black-and-white images in the first sequence of black-and-white images and the second sequence of black-and-white images.

[0060] In some embodiments, the human eye tracking image processor is configured to, when the presence of a human eye is not recognized in the current first black-and-white image and the second black-and-white image in the first black-and-white image sequence and the second black-and-white image sequence, and the presence of a human eye is recognized in the first black-and-white image and the second black-and-white image before or after, use the human eye spatial position information determined based on the first black-and-white image and the second black-and-white image before or after as the current human eye spatial position information.

[0061] Based on this, for example, when the first or second black-and-white camera experiences freezing or frame skipping, etc., a more coherent display screen can be provided for the user to ensure the viewing experience.

[0062] In some embodiments, the first black-and-white camera and the second black-and-white camera are configured to capture the first black-and-white image sequence and the second black-and-white image sequence at a frequency of 24 frames per second or higher.

[0063] In another solution, a 3D display device is provided, including a multi-viewpoint autostereoscopic 3D display screen, a video signal interface (signal interface) configured to receive video frames of a 3D video signal (3D signal), a 3D processing device communicatively connected to the video signal interface, and the human eye tracking device as described above. The multi-viewpoint autostereoscopic 3D display screen includes a plurality of sub-pixels corresponding to a plurality of viewpoints. The 3D processing device is configured to render sub-pixels related to a predetermined viewpoint based on the video frames of the 3D video signal, and the predetermined viewpoint is determined by the real-time human eye spatial position information of the user.

[0064] In some embodiments, the multi-viewpoint autostereoscopic 3D display screen includes a plurality of composite pixels, and each of the plurality of composite pixels includes a plurality of composite sub-pixels. Each composite sub-pixel is composed of a plurality of same-color sub-pixels corresponding to a plurality of viewpoints.

[0065] In some embodiments, the 3D processing device is communicatively connected to the human eye tracking data interface of the human eye tracking device.

[0066] In some embodiments, the 3D display device further includes a 3D shooting device configured to capture 3D images. The 3D shooting device includes a camera assembly and a 3D image processor. The camera assembly includes a first color camera, a second color camera, and a depth camera.

[0067] In some embodiments, the human eye tracking device is integrated with the 3D shooting device.

[0068] In some embodiments, the 3D shooting device is a front camera device.

[0069] In another solution, a human eye tracking method is provided, including: capturing a first black-and-white image at a first position; capturing a second black-and-white image at a second position, where the second position is different from the first position; identifying the presence of a human eye based on at least one of the first black-and-white image and the second black-and-white image; determining the spatial position of the human eye based on the location of the human eye present in the first black-and-white image and the second black-and-white image; and transmitting the human eye spatial position information of the human eye spatial position.

[0070] In some embodiments, the human eye tracking method further includes: when the first or second black-and-white camera is working, emitting infrared light using an infrared emitting device.

[0071] In some embodiments, the human eye tracking method further includes: respectively capturing a first sequence of black-and-white images and a second sequence of black-and-white images.

[0072] In some embodiments, the human eye tracking method further includes: determining the first black-and-white image and the second black-and-white image with time synchronization.

[0073] In some embodiments, the human eye tracking method further includes: caching multiple first black-and-white images and second black-and-white images in the first sequence of black-and-white images and the second sequence of black-and-white images; comparing multiple previous and subsequent first black-and-white images and second black-and-white images in the first sequence of black-and-white images and the second sequence of black-and-white images.

[0074] In some embodiments, the human eye tracking method further includes: when the presence of a human eye is not recognized in the current first black-and-white image and the second black-and-white image in the first sequence of black-and-white images and the second sequence of black-and-white images, and the presence of a human eye is recognized in the previous or subsequent first black-and-white image and the second black-and-white image, using the human eye spatial position information determined based on the previous or subsequent first black-and-white image and the second black-and-white image as the current human eye spatial position information.

[0075] In some embodiments, the human eye tracking method further includes: capturing the first sequence of black-and-white images and the second sequence of black-and-white images at a frequency of 24 frames per second or higher.

[0076] In another solution, a 3D display method is provided, applicable to a 3D display device. The 3D display device includes a multi-view autostereoscopic 3D display screen, including multiple sub-pixels corresponding to multiple viewpoints. The 3D display method includes: transmitting a video frame of a 3D video signal; receiving or reading the real-time human eye spatial position information of a user, and the real-time human eye spatial position information is determined using the human eye tracking method described above; determining the viewpoint where the human eye is located based on the human eye spatial position information; and rendering related sub-pixels based on the viewpoint according to the received video frame of the 3D video signal.

[0077] In some embodiments, the 3D display method further includes: providing a multi-view autostereoscopic 3D display screen including a plurality of composite pixels, each of the plurality of composite pixels including a plurality of composite sub-pixels, and each composite sub-pixel being composed of a plurality of same-color sub-pixels corresponding to a plurality of viewpoints.

[0078] In another solution, there is provided a 3D display device including a processor and a memory storing program instructions, and further including a multi-view autostereoscopic 3D display screen, the processor being configured to execute the 3D display method as described above when executing the program instructions.

[0079] Figure 1A The schematic structural diagram of a 3D display device 100 according to an embodiment of the present disclosure is shown. Refer to Figure 1A , in an embodiment of the present disclosure, there is provided a 3D display device 100 including a multi-view autostereoscopic 3D display screen 110, a signal interface 140 configured to receive video frames of 3D video signals, a 3D processing device 130 communicatively connected to the signal interface 140, and an eye tracking device 150. The eye tracking device 150 is communicatively connected to the 3D processing device 130, whereby the 3D processing device 130 can directly receive eye tracking data.

[0080] The multi-view autostereoscopic 3D display screen 110 may include a display panel and a grating (not labeled) covering the display panel. In Figure 1A the illustrated embodiment, the multi-view autostereoscopic 3D display screen 110 may include m×n composite pixels CP and thus define an m×n display resolution. As Figure 1A shown, the multi-view autostereoscopic 3D display screen 110 includes m columns and n rows of composite pixels and thus defines an m×n display resolution.

[0081] In some embodiments, the resolution of m×n may be a resolution above full high definition (FHD), including but not limited to, 1920×1080, 1920×1200, 2048×1280, 2560×1440, 3840×2160, etc.

[0082] In some embodiments, the 3D processing device is communicatively connected to the multi-view autostereoscopic 3D display screen.

[0083] In some embodiments, the 3D processing device is communicatively connected to the driving device of the multi-view autostereoscopic 3D display screen.

[0084] By way of explanation and not limitation, each composite pixel CP includes a plurality of composite sub-pixels CSP, and each composite sub-pixel is composed of i same-color sub-pixels corresponding to i viewpoints, where i ≥ 3. In the embodiment shown in FIG. 1, i = 6, but it is conceivable that i can be other values. In the shown embodiment, the multi-view autostereoscopic 3D display screen can correspondingly have i (i = 6) viewpoints (V1-V6), but it is conceivable that it can correspondingly have more or fewer viewpoints.

[0085] By way of explanation and not limitation, in the embodiment shown in FIG. 1, each composite pixel includes three composite sub-pixels, and each composite sub-pixel is composed of 6 same-color sub-pixels corresponding to 6 viewpoints (i = 6). The three composite sub-pixels respectively correspond to three colors, namely red (R), green (G), and blue (B). That is to say, the three composite sub-pixels of each composite pixel respectively have 6 red, 6 green, or 6 blue sub-pixels. In the embodiment shown in FIG. 1, the composite sub-pixels in the composite pixel are arranged in parallel. Each composite sub-pixel includes sub-pixels in a single-row form. However, it is conceivable that there are different arrangements of the composite sub-pixels in the composite pixel or different arrangements of the sub-pixels in the composite sub-pixel. For example, each composite sub-pixel includes sub-pixels in a single-column or array form.

[0086] By way of explanation and not by way of limitation, for example Figure 1A As shown, the 3D display device 100 can be provided with a single 3D processing device 130. The single 3D processing device 130 simultaneously processes the rendering of the sub-pixels of each composite sub-pixel of each composite pixel of the autostereoscopic 3D display screen 110. In other embodiments, the 3D display device 100 can also be provided with more than one 3D processing device 130, which process the rendering of the sub-pixels of each composite sub-pixel of each composite pixel of the autostereoscopic 3D display screen 110 in parallel, serially, or in a combination of serial and parallel. Those skilled in the art will understand that more than one 3D processing device can be distributed and process the multi-row and multi-column composite pixels or composite sub-pixels of the autostereoscopic 3D display screen 110 in parallel in other ways, which fall within the scope of the embodiments of the present disclosure.

[0087] In some embodiments, the 3D processing device 130 can also optionally include a buffer 131 to buffer the received video frames.

[0088] In some embodiments, the 3D processing device is an FPGA or an ASIC chip or an FPGA or ASIC chipset.

[0089] Continue to refer to Figure 1A, the 3D display device 100 may further include a processor 101 communicatively connected to the 3D processing device 130 via a signal interface 140. In some embodiments shown herein, the processor 101 is included in or is a processor unit of a computer or a smart terminal, such as a mobile terminal. However, it is conceivable that in some embodiments, the processor 101 may be disposed outside the 3D display device. For example, the 3D display device may be a multi-view autostereoscopic 3D display with a 3D processing device, such as a non-smart autostereoscopic 3D TV.

[0090] For simplicity, in the exemplary embodiments of the 3D display device hereinafter, a processor is included inside. Based on this, the signal interface 140 is configured as an internal interface connecting the processor 101 and the 3D processing device 130. Referring to Figure 2 and Figure 3 The 3D display device 200 implemented in the form of a mobile terminal as shown can more clearly illustrate this structure. In some embodiments shown herein, the signal interface as an internal interface of the 3D display device may be an MIPI, mini-MIPI interface, LVDS interface, min-LVDS interface, or Display Port interface. In some embodiments, as Figure 1A shown, the processor 101 of the 3D display device 100 may further include a register 122. The register 122 may be configured to temporarily store instructions, data, and addresses. In some embodiments, the register 122 may be configured to receive information about the display requirements of the multi-view autostereoscopic 3D display screen 110

[0091] In some embodiments, the 3D display device 100 may further include a codec configured to decompress, encode and decode a compressed 3D video signal and send the decompressed 3D video signal to the 3D processing device 130 via the signal interface 140.

[0092] Referring to Figure 1B , Figure 1B the embodiment shown and Figure 1A the difference between the embodiment shown is that the 3D display device 100 further includes a 3D shooting device 120 configured to capture 3D images, and the human eye tracking device 150 is integrated in the 3D shooting device 120. It is also conceivable to integrate it into a conventional camera device of a processing terminal or a display device. As Figure 1B shown, the 3D shooting device 120 is configured as a front shooting device. The 3D shooting device 120 includes a camera assembly 121, a 3D image processor 126, and a 3D image output interface 125. The 3D shooting device 120 is integrated with the human eye tracking device 150.

[0093] As Figure 1BAs shown, the camera assembly 121 includes a first color camera 121a, a second color camera 121b, and a depth camera 121c. In some other embodiments not shown, the 3D image processor 126 may be integrated within the camera assembly 121. In some embodiments, the first color camera 121a is configured to obtain a first color image of a subject, the second color camera 121b is configured to obtain a second color image of the subject, and a composite color image of an intermediate point is obtained by synthesizing these two color images; the depth camera 121c is configured to obtain a depth image of the subject. The composite color image and the depth image form a video frame of a 3D video signal. In the embodiments of the present disclosure, the first color camera and the second color camera are the same color camera. In some other embodiments, the first color camera and the second color camera may also be different color cameras. In this case, in order to obtain a color composite image, the first and second color images may be calibrated or corrected. The depth camera 121c may be a time-of-flight (TOF) camera or a structured light camera. The depth camera 121c may be disposed between the first color camera and the second color camera.

[0094] In some embodiments, the 3D image processor 126 is configured to synthesize the first and second color images into a composite color image, and form a 3D image by combining the synthesized composite color image with the depth image. The formed 3D image is transmitted via the 3D image output interface 125 to the processor 101 of the 3D display device 100.

[0095] Optionally, the first and second color images and the depth image are directly transmitted via the 3D image output interface 125 to the processor 101 of the 3D display device 100, and the above processing such as synthesizing the composite color image and forming the 3D image is performed by the processor 101.

[0096] Optionally, the 3D image output interface 125 may also be communicatively connected to the 3D processing device 130 of the 3D display device 100, so that the above processing such as synthesizing the composite color image and forming the 3D image can be performed by the 3D processing device 130.

[0097] In some embodiments, at least one of the first color camera and the second color camera is a wide-angle color camera.

[0098] Continuing to refer Figure 1B , the eye tracking device 150 is integrated within the 3D shooting device 120 and includes an eye tracker 151, an eye tracking image processor 152, and an eye tracking data interface 153.

[0099] The eye tracker 151 includes a first black-and-white camera 151a and a second black-and-white camera 151b. The first black-and-white camera 151a is configured to capture a first black-and-white image, and the second black-and-white camera 151b is configured to capture a second black-and-white image. When the 3D shooting device 120 is front-mounted and the eye tracking device 150 is integrated into the 3D shooting device 120, the eye tracking device 150 is also front-mounted, and the shooting objects of the first black-and-white camera and the second black-and-white camera are the user's face.

[0100] In some embodiments, the eye tracking data interface 153 of the eye tracking device 150 is communicatively connected to the 3D processing device 130 of the 3D display device 100, whereby the 3D processing device 130 can directly receive the eye tracking data. In other embodiments, the eye tracking image processor 152 of the eye tracking device 150 can be communicatively connected to the processor 101 of the 3D display device 100, whereby the eye tracking data can be transmitted from the processor 101 to the 3D processing device 130 through the eye tracking data interface 153.

[0101] In some embodiments, the eye tracking device 150 is communicatively connected to the camera assembly 221, whereby the eye tracking data can be used when shooting 3D images.

[0102] Optionally, the eye tracker 151 is further provided with an infrared emission device 154. When the first or second black-and-white camera is operating, the infrared emission device 154 is configured to selectively emit infrared light to provide supplementary lighting when the ambient light is insufficient, such as when shooting at night, so as to capture the first or second black-and-white image that can identify the user's face and eyes even under weak ambient light conditions.

[0103] In some embodiments, the eye tracking device 150 or the processing terminal or display device integrated with the eye tracking device can be configured to, when the first or second black-and-white camera is operating, based on the received light sensing signal, such as when it is detected that the light sensing signal is lower than a given threshold, control the turning on of the infrared emission device or adjust its intensity. In some embodiments, the light sensing signal is received from an ambient light sensor integrated in the processing terminal or display device, such as the ambient light sensor 2702.

[0104] Optionally, the infrared emission device 154 is configured to emit infrared light with a wavelength greater than or equal to 1.5 micrometers, that is, long-wave infrared light. Compared with short-wave infrared light, long-wave infrared light has weaker skin penetration ability and thus less harm to the human eye.

[0105] The first black-and-white image and the second black-and-white image captured are transmitted to the eye-tracking image processor 152. Exemplarily, the eye-tracking image processor is configured to have a visual recognition function, such as a face recognition function, and is configured to recognize a face and identify the eyes based on at least one of the two black-and-white images, and determine the eye spatial position based on the positions of the eyes present in the two black-and-white images. In the embodiments of the present disclosure, the first black-and-white camera and the second black-and-white camera are the same black-and-white camera. In some other embodiments, the first black-and-white camera and the second black-and-white camera may also be different black-and-white cameras. In this case, in order to determine the eye spatial position, the first black-and-white image and the second black-and-white image can be calibrated or corrected.

[0106] In some embodiments, at least one of the first black-and-white camera and the second black-and-white camera is a wide-angle black-and-white camera.

[0107] Figure 4 A top view of a geometric relationship model for determining the eye spatial position using two black-and-white cameras is schematically shown. In Figure 4 the illustrated embodiment, the first black-and-white camera and the second black-and-white camera are the same black-and-white camera, and thus have the same focal length f; the optical axis Za of the first black-and-white camera 151a is parallel to the optical axis Zb of the second black-and-white camera 151b, and the focal plane 401a of the first black-and-white camera 151a and the focal plane 401b of the second black-and-white camera 151b are in the same plane and perpendicular to the optical axes of the two black-and-white cameras. Based on the above settings, the line connecting the lens centers Oa and Ob of the two black-and-white cameras is parallel to the focal planes of the two black-and-white cameras. In Figure 4 the illustrated embodiment, a top view of the geometric relationship model of the XZ plane is shown with the direction of the line connecting the lens centers Oa to Ob of the two black-and-white cameras as the X-axis direction and the optical axis direction of the two black-and-white cameras as the Z-axis direction.

[0108] In Figure 4 the illustrated embodiment, the lens center Oa of the first black-and-white camera 151a is taken as its origin, and the lens center Ob of the second black-and-white camera 151b is taken as its origin. R and L respectively represent the user's right eye and left eye, XRa and XRb are the X-axis coordinates of the user's right eye R imaged in the focal planes 401a and 401b of the two black-and-white cameras respectively, and XLa and XLb are the X-axis coordinates of the user's left eye L imaged in the focal planes 401a and 401b of the two black-and-white cameras respectively. In addition, the distance T between the two black-and-white cameras and their focal length f are also known. According to the geometric relationship of similar triangles, the distances DR and DL of the right eye R and the left eye L from the plane where the two black-and-white cameras are located as set above can be obtained as follows:

[0109]

[0110]

[0111] And it can be obtained that the inclination angle α of the line connecting the observer's two eyes and the plane where the two black-and-white cameras are arranged as above, and the observer's interpupillary distance or pupil distance P are respectively:

[0112]

[0113]

[0114] In Figure 4 the illustrated embodiment, the line connecting the observer's two eyes, that is, the plane of the observer's face and the plane where the two black-and-white cameras are arranged as above are inclined to each other, and the inclination angle is α; when the plane of the observer's face and the plane where the two black-and-white cameras are arranged as above are parallel to each other, that is, when the observer looks straight at the two black-and-white cameras, the inclination angle α is zero.

[0115] As described above, in some embodiments of this article, the 3D display device 100 may be a computer or a smart terminal, such as a mobile terminal. However, it can be envisioned that in some embodiments, the 3D display device 100 may also be a non-smart display terminal, such as a non-smart autostereoscopic TV. In Figure 5A , Figure 5B and Figure 5C schematic diagrams of 3D display devices 500 respectively configured as a smart phone, a tablet computer, and a non-smart display are shown, which have a multi-view autostereoscopic display screen 510, a front 3D shooting device, and the 3D shooting device integrates an eye tracking device. In Figures 5A to 5C the illustrated embodiment, the 3D shooting device 120 including two color cameras 121a, 121b and a depth camera 121c and the integrated eye tracking device 150 including two black-and-white cameras 151a, 151b are arranged in the same plane as the multi-view autostereoscopic display screen 510 of the 3D display device 500. Therefore, in Figure 4 the distances DR and DL between the user's right eye R and left eye L and the plane where the two black-and-white cameras are arranged as above, which are exemplarily obtained in the illustrated embodiment, are the distances between the user's right eye R and left eye L and the multi-view autostereoscopic display screen, and the inclination angle α of the observer's face and the plane where the two black-and-white cameras are arranged as above is the inclination angle of the observer's face and the multi-view autostereoscopic display screen.

[0116] Referring to Figure 6A , a schematic diagram of a user looking straight at or looking level at the multi-view autostereoscopic display screen of the 3D display device 600 is shown, that is, the plane where the user's face is located and the plane where the display screen is located are parallel to each other, the distances DR and DL between the user's two eyes and the display screen are the same, and the inclination angle α is zero.

[0117] Referring toFigure 6B , which shows a schematic diagram of the user's face tilted relative to the multi-view autostereoscopic 3D display screen of the 3D display device 600, that is, the plane where the user's face is located is not parallel to the plane where the display screen is located, the distances DR and DL between the user's two eyes and the display screen are different, and the tilt angle α is not zero.

[0118] In some embodiments, the eye tracking data interface 153 is configured to transmit the tilt angle or parallelism of the user's two eyes relative to the eye tracking device 150 or the multi-view autostereoscopic 3D display screen 110. This can be beneficial for more precisely presenting 3D images, which will be described below.

[0119] For example, the eye spatial position information DR, DL, α, and P obtained as above by way of example is transmitted to the 3D processing device 130 through the eye tracking data interface 153. The 3D processing device 130 determines the viewpoints where the user's two eyes are located and provided by the multi-view autostereoscopic 3D display screen 110, that is, the predetermined viewpoints, based on the received eye spatial position information.

[0120] For example, the eye spatial position information DR, DL, α, and P obtained as above by way of example can also be directly transmitted to the processor 101 of the 3D display device 100, and the 3D processing device 130 receives / reads the eye spatial position information from the processor 101 through the eye tracking data interface 153.

[0121] In some embodiments, the first black-and-white camera 151a is configured to capture a first sequence of black-and-white images, which includes a plurality of first black-and-white images arranged in chronological order, and the second black-and-white camera 151b is configured to capture a second sequence of black-and-white images, which includes a plurality of second black-and-white images arranged in chronological order.

[0122] In some embodiments, the eye tracking image processor 152 includes a synchronizer 155, which is configured to determine the first black-and-white image and the second black-and-white image that are time-synchronized in the first sequence of black-and-white images and the second sequence of black-and-white images. The first black-and-white image and the second black-and-white image determined to be time-synchronized are used for the recognition of the eyes and the determination of the eye spatial position.

[0123] In some embodiments, the eye tracking image processor 152 includes a buffer 156 and a comparator 157. The buffer 156 is configured to buffer a plurality of first black-and-white images and second black-and-white images arranged in chronological order in the first sequence of black-and-white images and the second sequence of black-and-white images respectively. The comparator 157 is configured to compare the plurality of first black-and-white images and second black-and-white images taken in chronological order in the first sequence of black-and-white images and the second sequence of black-and-white images. Through the comparison, for example, it can be judged whether the eye spatial position changes or whether the eyes are still within the viewing range, etc.

[0124] In some embodiments, the human eye tracking image processor 152 further includes a decision maker (not shown), configured to, when the comparator does not recognize the presence of a human eye in the current first black-and-white image and the second black-and-white image in the first black-and-white image sequence and the second black-and-white image sequence but recognizes the presence of a human eye in the first black-and-white image and the second black-and-white image before or after, use the human eye spatial position determined based on the first black-and-white image and the second black-and-white image before or after as the current human eye spatial position. This situation is, for example, when the user briefly turns their head. In this case, it is possible that the user's face and eyes cannot be recognized briefly.

[0125] Exemplarily, several first black-and-white images and second black-and-white images in the first black-and-white image sequence and the second black-and-white image sequence are stored in the cache segments of the buffer 156. In some cases, a human face and human eyes cannot be recognized from the currently cached first black-and-white image and second black-and-white image, yet a human face and human eyes can be recognized from the first black-and-white image and the second black-and-white image before or after that are cached. In this case, the human eye spatial position information determined based on the first black-and-white image and the second black-and-white image that are after, that is, taken later than, the current first black-and-white image and second black-and-white image can be used as the current human eye spatial position information; alternatively, the human eye spatial position information determined based on the first black-and-white image and the second black-and-white image that are before, that is, taken earlier than, the current first black-and-white image and second black-and-white image can be used as the current human eye spatial position information. In addition, the human eye spatial position information determined based on the above-mentioned first black-and-white images and second black-and-white images before and after that can recognize a human face and human eyes can be averaged, data-fitted, interpolated, or processed by other methods, and the resulting result can be used as the current human eye spatial position information.

[0126] In some embodiments, the first black-and-white camera and the second black-and-white camera are configured to capture the first black-and-white image sequence and the second black-and-white image sequence at a frequency of 24 frames per second or higher. Exemplarily, it is captured at a frequency of 30 frames per second. Exemplarily, it is captured at a frequency of 60 frames per second.

[0127] In some embodiments, the first black-and-white camera and the second black-and-white camera are configured to capture at the same frequency as the refresh frequency of the display screen of the 3D display device.

[0128] As mentioned above, the 3D display device provided by the embodiments of the present disclosure can be a 3D display device including a processor. In some embodiments, the 3D display device can be configured as a smart cellular phone, a tablet computer, a smart TV, a wearable device, a vehicle-mounted device, a laptop computer, a ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc.

[0129] Exemplarily, Figure 2The schematic diagram of the hardware structure of the 3D display device 200 implemented as a mobile terminal, such as a tablet computer or a smart cellular phone, is shown. The 3D display device 200 may include a processor 201, an external storage interface 202, an (internal) memory 203, a universal serial bus (USB) interface 204, a charging management module 205, a power management module 206, a battery 207, a mobile communication module 281, a wireless communication module 283, antennas 282 and 284, an audio module 212, a speaker 213, a receiver 214, a microphone 215, a headphone interface 216, keys 217, a motor 218, an indicator 219, a subscriber identity module (SIM) card interface 260, a multi-view autostereoscopic 3D display screen 210, a 3D processing device 230, a signal interface 240, a 3D shooting device 220, a sensor module 230, etc. The 3D shooting device 220 may include a camera assembly 221, a 3D image output interface 225, and a human eye tracking device 250. The sensor module 270 may include a proximity light sensor 2701, an ambient light sensor 2702, a pressure sensor 2703, a barometric pressure sensor 2704, a magnetic sensor 2705, a gravity sensor 2706, a gyroscope sensor 2707, an acceleration sensor 2708, a distance sensor 2709, a temperature sensor 2710, a fingerprint sensor 2711, a touch sensor 2712, a bone conduction sensor 2713, etc.

[0130] It can be understood that the structure schematically shown in the embodiments of the present disclosure does not limit the 3D display device 200. In other embodiments of the present disclosure, the 3D display device 200 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0131] The processor 201 may include one or more processing units. For example, the processor 201 may include an application processor (AP), a modulation and demodulation processor, a baseband processor, registers 222, a graphics processing unit (GPU) 223, an image signal processor (ISP), a controller, a memory, a video codec 224, a digital signal processor (DSP), a baseband processor, a neural network processor (NPU), etc., or a combination thereof. Among them, different processing units may be independent devices or integrated in one or more processors.

[0132] A cache may also be provided in the processor 201, configured to store instructions or data that the processor 201 has just used or recycled. When the processor 201 needs to use the instructions or data again, it can be directly called from the memory.

[0133] In some embodiments, the processor 201 may include one or more interfaces. The interfaces may include an Inter-Integrated Circuit (I2C) interface, an Integrated Circuit Built-in Audio (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General-Purpose Input / Output (GPIO) interface, a Subscriber Identity Module (SIM) interface, a Universal Serial Bus (USB) interface, etc.

[0134] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 201 may include multiple groups of I2C buses. The processor 201 may be communicatively connected to the touch sensor 2712, the charger, the flash, the 3D shooting device 220 or its camera assembly 221, the eye tracking device 250, etc. respectively through different I2C bus interfaces.

[0135] Both the I2S interface and the PCM interface can be used for audio communication.

[0136] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is configured to connect the processor 201 to the wireless communication module 283.

[0137] In Figure 2 In the illustrated embodiment, the MIPI interface can be configured to connect the processor 201 to the multi-view autostereoscopic 3D display 210. Additionally, the MIPI interface can also be configured to connect to peripheral devices such as the camera assembly 221, the eye tracking device 250, etc.

[0138] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be configured to connect the processor 201 to the 3D shooting device 220 or its camera assembly 221, the multi-view autostereoscopic 3D display 110, the wireless communication module 283, the audio module 212, the sensor module 270, etc.

[0139] The USB interface 204 is an interface that complies with the USB standard specification and can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 204 can be configured to connect to a charger to charge the 3D display device 200, and can also be used to transfer data between the 3D display device 200 and peripheral devices. It can also be configured to connect to a headset to play audio through the headset.

[0140] It should be understood that the interface connection relationships among the modules illustrated in the embodiments of the present disclosure are merely illustrative and do not constitute a structural limitation on the 3D display device 200.

[0141] The wireless communication function of the 3D display device 200 can be implemented by antennas 282 and 284, a mobile communication module 281, a wireless communication module 283, a modulation and demodulation processor, a baseband processor, etc.

[0142] Antennas 282 and 284 are configured to transmit and receive electromagnetic wave signals. Each antenna in the 3D display device 200 can be configured to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas.

[0143] The mobile communication module 281 can provide solutions for wireless communications such as 2G / 3G / 4G / 5G applied to the 3D display device 200. The mobile communication module 281 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 281 can receive electromagnetic waves through the antenna 282, filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 281 can also amplify the signal modulated by the modulation and demodulation processor and convert it into electromagnetic waves through the antenna 282 and radiate it out. In some embodiments, at least some functional modules of the mobile communication module 281 can be provided in the processor 201. In some embodiments, at least some functional modules of the mobile communication module 282 and at least some modules of the processor 201 can be provided in the same device.

[0144] The wireless communication module 283 can provide solutions for wireless communications such as wireless local area network (WLAN), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc. applied to the 3D display device 200. The wireless communication module 283 can be one or more devices integrating at least one communication processing module. The wireless communication module 283 receives electromagnetic waves through the antenna 284, frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 201. The wireless communication module 283 can also receive the signals to be sent from the processor 201, frequency-modulate and amplify them, and convert them into electromagnetic waves through the antenna 284 and radiate them out.

[0145] In some embodiments, the antenna 282 of the 3D display device 200 is coupled to the mobile communication module 281, and the antenna 284 is coupled to the wireless communication module 283, such that the 3D display device 200 can communicate with the network and other devices through wireless communication technologies. The wireless communication technologies may include at least one of Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division-Synchronous Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), Bluetooth (BT), Global Navigation Satellite System (GNSS), Wireless Local Area Network (WLAN), Near Field Communication (NFC), Frequency Modulation (FM), or Infrared (IR) technology, etc. The GNSS may include at least one of Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), or Satellite Based Augmentation System (SBAS).

[0146] In some embodiments, the external interface configured to receive 3D video signals may include the USB interface 204, the mobile communication module 281, the wireless communication module 283, or a combination thereof. In addition, other feasible interfaces configured to receive 3D video signals may also be contemplated, such as the above-mentioned interfaces.

[0147] The memory 203 may be configured to store computer-executable program code, and the executable program code includes instructions. The processor 201 executes various functional applications and data processing of the 3D display device 200 by running the instructions stored in the memory 203. The memory 203 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, applications required for at least one function (such as a sound playback function, an image playback function, etc.). The data storage area may store data created during the use of the 3D display device 200 (such as audio data, phone book, etc.). In addition, the memory 203 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a Universal Flash Storage (UFS), etc.

[0148] The external memory interface 202 may be configured to connect to an external memory card, such as a Micro SD card, to implement the storage capacity expansion of the 3D display device 200. The external memory card communicates with the processor 201 through the external memory interface 202 to achieve the data storage function.

[0149] In some embodiments, the memory of the 3D display device may include the (internal) memory 203, an external memory card connected to the external memory interface 202, or a combination thereof. In other embodiments of the present disclosure, the signal interface may also adopt different internal interface connection methods or combinations thereof in the above embodiments.

[0150] In the embodiments of the present disclosure, the camera assembly 221 can acquire images or videos in 2D or 3D and output the acquired videos via the 3D image output interface 225. The eye tracking device 250 can determine the spatial positions of the user's binoculars. The camera assembly 221, the 3D image output interface 225, and the eye tracking device 250 together form a 3D shooting device 220.

[0151] In some embodiments, the 3D display device 200 realizes the display function through the signal interface 240, the 3D processing device 230, the eye tracking device 250, the multi-view autostereoscopic 3D display screen 210, and the application processor, etc.

[0152] In some embodiments, the 3D display device 200 may include a GPU, for example, configured to process 3D video images within the processor 201 and also capable of processing 2D video images.

[0153] In some embodiments, the 3D display device 200 further includes a video codec 224, configured to compress or decompress digital videos.

[0154] In some embodiments, the signal interface 240 is configured to output video frames of 3D video signals processed by the GPU or the codec 224 or both, such as decompressed 3D video signals, to the 3D processing device 230.

[0155] In some embodiments, the GPU or the codec 224 is integrated with a format adjuster.

[0156] The multi-view autostereoscopic 3D display screen 210 is configured to display 3D (stereoscopic) images or videos, etc. The multi-view autostereoscopic 3D display screen 210 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini-LED, a Micro-LED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc.

[0157] In some embodiments, the eye tracking device 250 is communicatively connected to the 3D processing device 230, so that the 3D processing device 230 can render corresponding sub-pixels in the composite pixels (composite sub-pixels) based on the eye tracking data. In some embodiments, the eye tracking device 250 can also be connected to the processor 201, for example, bypass-connected to the processor 201.

[0158] In some embodiments, the 3D image output interface 225 of the 3D shooting device 220 can be communicatively connected to the processor 201 or the 3D processing device 230.

[0159] The 3D display device 200 can implement audio functions through the audio module 212, speaker 213, receiver 214, microphone 215, headphone jack 216, and application processor, etc. For example, music playback, recording, etc. The audio module 212 is configured to convert digital audio information into an analog audio signal for output, and is also configured to convert an analog audio input into a digital audio signal. The audio module 212 can also be configured to encode and decode audio signals. In some embodiments, the audio module 212 can be disposed in the processor 201, or some functional modules of the audio module 212 can be disposed in the processor 201. The speaker 213 is configured to convert an audio electrical signal into a sound signal. The 3D display device 200 can listen to music or hands-free calls through the speaker 213. The receiver 214, also known as the "earpiece", is configured to convert an audio electrical signal into a sound signal. When the 3D display device 200 answers a call or voice message, the voice can be listened to by bringing the receiver 214 close to the human ear. The microphone 215 is configured to convert a sound signal into an electrical signal. The headphone jack 216 is configured to connect a wired headphone. The headphone jack 216 can be a USB interface 204, or a 3.5 mm Open Mobile 3D Display Device Platform (OMTP) standard interface, or a Cellular Telecommunications Industry Association (CTIA) standard interface.

[0160] The keys 217 include a power-on key, volume keys, etc. The keys 217 can be mechanical keys or touch keys. The 3D display device 200 can receive key inputs and generate key signal inputs related to the user settings and function controls of the 3D display device 200.

[0161] The motor 218 can generate vibration prompts. The motor 218 can be configured for incoming call vibration prompts or touch vibration feedback.

[0162] The SIM card interface 260 is configured to connect a SIM card. In some embodiments, the 3D display device 200 uses an eSIM, that is, an embedded SIM card.

[0163] The ambient light sensor 2702 is configured to sense the ambient light brightness. The 3D display device 200 can adjust the brightness of the multi-view autostereoscopic 3D display screen 210 or assist in human eye tracking according to the sensed ambient light brightness. For example, when the ambient light brightness is relatively dim, the human eye tracking device 250 activates the infrared emission device. The ambient light sensor 2702 can also be configured to adjust the white balance during black and white camera shooting.

[0164] The pressure sensor 2703 is configured to sense a pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 2703 can be disposed on the multi-view autostereoscopic 3D display screen 210, which falls within the scope of the embodiments of the present disclosure.

[0165] The barometric pressure sensor 2704 is configured to measure barometric pressure. In some embodiments, the 3D display device 200 calculates the altitude based on the barometric pressure value measured by the barometric pressure sensor 2704 to assist in positioning and navigation.

[0166] The magnetic sensor 2705 includes a Hall sensor.

[0167] The gravity sensor 2706 is a sensor that converts motion or gravity into an electrical signal, and is mainly configured to measure parameters such as tilt angle, inertial force, impact, and vibration.

[0168] The gyroscope sensor 2707 can be configured to determine the motion posture of the 3D display device 200.

[0169] The acceleration sensor 2708 can detect the magnitude of the acceleration of the 3D display device 200 in various directions (generally three axes).

[0170] The distance sensor 2709 can be configured to measure distance

[0171] The temperature sensor 2710 can be configured to detect temperature.

[0172] The fingerprint sensor 2711 is configured to collect fingerprints. The 3D display device 200 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access to application locks, fingerprint photography, fingerprint answering of incoming calls, etc.

[0173] The touch sensor 2712 can be disposed in the multi-view autostereoscopic 3D display screen 210. The touch sensor 2712 and the multi-view autostereoscopic 3D display screen 210 form a touch screen, also known as a "touch panel".

[0174] The bone conduction sensor 2713 can acquire vibration signals.

[0175] The charging management module 205 is configured to receive a charging input from a charger. Among them, the charger can be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 205 can receive the charging input from a wired charger through the USB interface 204. In some embodiments of wireless charging, the charging management module 205 can receive the wireless charging input through the wireless charging coil of the 3D display device 200.

[0176] The power management module 206 is configured to connect to the battery 207, the charging management module 205, and the processor 201. The power management module 206 receives inputs from at least one of the battery 207 or the charging management module 205, and powers the processor 201, the memory 203, the external memory, the multi-view autostereoscopic 3D display 210, the camera module 221, and the wireless communication module 283, etc. In some other embodiments, the power management module 206 and the charging management module 205 may also be provided in the same device.

[0177] The software system of the 3D display device 200 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. The embodiments shown in the present disclosure take the Android system with a layered architecture as an example to exemplarily illustrate the software structure of the 3D display device 200. However, it can be envisioned that the embodiments of the present disclosure can be implemented in different software systems, such as operating systems.

[0178] Figure 3 is Figure 2 The schematic diagram of the software structure of the 3D display device 200 shown. The layered architecture divides the software into several layers. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely the application layer 310, the framework layer 320, the core class libraries and runtime 330, and the kernel layer 340.

[0179] The application layer 310 may include a series of application packages. Such as Figure 3 shown, the application packages may include applications such as Bluetooth, WLAN, navigation, music, camera, calendar, call, video, gallery, map, short message, etc. According to the 3D video display method of the embodiments of the present disclosure, for example, it can be implemented in a video application.

[0180] The framework layer 320 provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The framework layer includes some predefined functions. For example, in some embodiments of the present disclosure, functions or algorithms for recognizing the collected 3D video images and algorithms for processing images, etc. may be included in the framework layer.

[0181] Such as Figure 3 shown, the framework layer 320 may include a resource manager, a telephone manager, a content manager, a notification manager, a window manager, a view system, an installation package manager, etc.

[0182] Android Runtime includes core libraries and a virtual machine. Android Runtime is responsible for the scheduling and management of the Android system.

[0183] The core library consists of two parts: one is the functional functions to be called by the Java language, and the other is the core library of Android.

[0184] The application layer and the framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the framework layer as binary files. The virtual machine is configured to perform functions such as management of object life cycles, stack management, thread management, security and exception management, and garbage collection.

[0185] The core class library can include multiple functional modules. For example: 3D graphics processing library (e.g., OpenGL ES), surface manager, image processing library, media library, graphics engine (e.g., SGL), etc.

[0186] The kernel layer 340 is the layer between the hardware and the software. The kernel layer at least includes a camera driver, an audio-video interface, a call interface, a Wifi interface, a sensor driver, power management, and a GPS interface.

[0187] Here, taking a 3D display device as a mobile terminal having Figure 2 and Figure 3 the structure shown as an example, embodiments of 3D video transmission and display in the 3D display device are described; however, it can be envisioned that in other embodiments, more or fewer features may be included or the features may be changed.

[0188] In some embodiments, for example, a 3D display device 200 such as a mobile terminal, such as a tablet computer or a smart cellular phone, receives, for example, a compressed 3D video signal from a network, such as a cellular network, a WLAN network, or Bluetooth, by means of a mobile communication module 281 and an antenna 282 or a wireless communication module 283 and an antenna 284 as an external interface. The compressed 3D video signal is, for example, subjected to image processing, codec encoding and decoding, and decompression by a GPU 223, and then the decompressed 3D video signal is sent to a 3D processing device 230 by means of a signal interface 240, such as a MIPI interface or a mini-MIPI interface, as an internal interface. And, real-time human eye spatial position information of the user is obtained by a human eye tracking device 250. A predetermined viewing point is determined based on the human eye spatial position information. The 3D processing device 230 renders the sub-pixels of the display screen correspondingly for the predetermined viewing point, thereby realizing 3D video playback.

[0189] In other embodiments, the 3D display device 200 reads a compressed 3D image signal stored in an (internal) memory 203 or reads an external memory card through an external memory interface 202, and realizes 3D image playback through corresponding processing, transmission, and rendering.

[0190] In some other embodiments, the 3D display device 200 receives the 3D images captured by the camera assembly 221 and transmitted via the 3D image output interface 225, and implements 3D image playback through corresponding processing, transmission, and rendering.

[0191] In some embodiments, the playback of the above 3D images is implemented in a video application in the Android system application layer 310.

[0192] Embodiments of the present disclosure may also provide a human eye tracking method, which is implemented by using the human eye tracking device in the above embodiments.

[0193] Reference Figure 7 , in some embodiments, the human eye tracking method includes:

[0194] S701: Capture a first black-and-white image and a second black-and-white image;

[0195] S702: Identify the presence of a human eye based on at least one of the first black-and-white image and the second black-and-white image;

[0196] S703: Determine the spatial position of the human eye based on the human eyes identified in the first black-and-white image and the second black-and-white image.

[0197] Exemplarily, the first black-and-white image is captured at a first position, and the second black-and-white image is captured at a second position, where the first position is different from the second position.

[0198] In some embodiments, the human eye tracking method further includes: transmitting human eye spatial position information indicating the spatial position of the human eye.

[0199] In some embodiments, the human eye tracking method further includes: when the first black-and-white camera or the second black-and-white camera is working, an infrared emitting device emits infrared light.

[0200] In some embodiments, the human eye tracking method further includes: respectively capturing a first black-and-white image sequence including the first black-and-white image and a second black-and-white image sequence including the second black-and-white image.

[0201] In some embodiments, the human eye tracking method further includes: determining the first black-and-white image and the second black-and-white image with time synchronization.

[0202] In some embodiments, the human eye tracking method further includes: caching multiple first black-and-white images and second black-and-white images in the first black-and-white image sequence and the second black-and-white image sequence; comparing the multiple first black-and-white images and second black-and-white images before and after in the first black-and-white image sequence and the second black-and-white image sequence; when the presence of a human eye is not recognized in the current first black-and-white image and second black-and-white image in the first black-and-white image sequence and the second black-and-white image sequence but the presence of a human eye is recognized in the first black-and-white image and second black-and-white image before or after, using the human eye spatial position determined based on the first black-and-white image and second black-and-white image before or after as the current human eye spatial position.

[0203] In some embodiments, the human eye tracking method includes: capturing the first black-and-white image sequence and the second black-and-white image sequence at a frequency of 24 frames per second or higher.

[0204] Embodiments of the present disclosure may further provide a 3D display method.

[0205] Reference Figure 8 , in some embodiments, the 3D display method includes:

[0206] S801: Obtain the human eye spatial position of the user;

[0207] S802: Determine the corresponding viewpoints according to the human eye spatial position;

[0208] S803: Render the sub-pixels corresponding to the viewpoints of the multi-view autostereoscopic 3D display screen based on the 3D signal.

[0209] In some embodiments, the 3D display method further includes: providing a multi-view autostereoscopic 3D display screen, including a plurality of composite pixels, each of the plurality of composite pixels including a plurality of composite sub-pixels, and each of the plurality of composite sub-pixels being composed of a plurality of sub-pixels corresponding to a plurality of viewpoints.

[0210] Exemplarily, when it is determined based on the human eye spatial position that each of the user's two eyes corresponds to one viewpoint, generate images of the two viewpoints where the user's two eyes are located based on the video frames of the 3D video signal, and render the sub-pixels corresponding to these two viewpoints in the composite sub-pixels.

[0211] Reference Figure 9 , in the illustrated embodiment, the user's right eye is at the second viewpoint V2, and the left eye is at the fifth viewpoint V5. Generate images of these two viewpoints V2 and V5 based on the video frames of the 3D video signal, and render the sub-pixels corresponding to these two viewpoints in the composite sub-pixels.

[0212] In some embodiments, when determining the tilt angle or parallelism of the user's binoculars relative to the multi-viewpoint autostereoscopic display based on the spatial position of the human eyes, a targeted or customized display image can be provided for the user to enhance the user's viewing experience.

[0213] The systems, devices, modules, or units illustrated in the above embodiments can be implemented by various possible entities. A typical implementation entity is a computer or its processor or other components. The computer can be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, a smart TV, an Internet of Things system, a smart home, an industrial computer, a single-chip microcomputer system, or a combination of these devices. In a typical configuration, a computer may include one or more processors (CPUs), an input / output interface, a network interface, and a memory. The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM).

[0214] The methods, programs, systems, devices, etc. in the embodiments of the present application can be executed or implemented in a single or multiple networked computers, and can also be practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks are executed by remote processing devices connected through a communication network.

[0215] Those skilled in the art should understand that the embodiments of this specification can be provided as a method, a system, or a computer program product. Therefore, the embodiments of this specification can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects.

[0216] Those skilled in the art can conceive that the implementation of the functional modules / units or controllers and the related method steps illustrated in the above embodiments can be achieved in a software, hardware, or software / hardware combination manner. For example, it can be implemented in a pure computer-readable program code manner, or partially or fully by logically programming the method steps to enable the controller to implement the same function in hardware, including but not limited to logic gates, switches, application-specific integrated circuits, programmable logic controllers (such as FPGAs), and embedded microcontrollers.

[0217] In some embodiments of the present application, components of a device are described in the form of functional modules / units. It is conceivable that multiple functional modules / units are implemented in one or more "combined" functional modules / units and / or one or more software and / or hardware. It is also conceivable that a single functional module / unit is implemented by a combination of multiple sub-functional modules or sub-units and / or multiple software and / or hardware. The division of functional modules / units may be only a logical function division. In the implementation, multiple modules / units may be combined or integrated into another system. In addition, the connections of the modules, units, devices, systems and their components described herein include direct or indirect connections, covering feasible electrical, mechanical, and communication connections, especially including wired or wireless connections between various interfaces, including but not limited to HDMI, radar, USB, WiFi, and cellular networks.

[0218] In embodiments of the present application, the technical features, flowcharts, and / or block diagrams of methods and programs can be applied to corresponding devices, equipment, systems, and their modules, units, and components. Conversely, the embodiments and features of devices, equipment, systems, and their modules, units, and components can be applied to the methods and programs according to the embodiments of the present application. For example, computer program instructions can be loaded into a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine that has the corresponding functions or features implemented in one or more processes of a flowchart and / or one or more blocks of a block diagram.

[0219] The methods and programs according to the embodiments of the present application can be stored in a computer-readable memory or medium in the form of computer program instructions or programs, which can guide a computer or other programmable data processing devices to work in a specific manner. Embodiments of the present application also relate to a readable memory or medium storing methods, programs, and instructions that can implement the embodiments of the present application.

[0220] Unless explicitly stated, the actions or steps of the methods and programs according to the embodiments of the present application do not necessarily have to be executed in a specific order and can still achieve the desired results. In some embodiments, multi-tasking and parallel processing are also possible or may be advantageous.

[0221] The exemplary systems and methods of the present application have been specifically shown and described with reference to the above embodiments, which are only examples of the best mode for implementing the systems and methods. Those skilled in the art can understand that various changes can be made to the embodiments of the systems and methods described herein when implementing the systems and / or methods without departing from the spirit and scope of the present application defined in the appended claims.

Claims

1. An eye tracking device, comprising: An eye tracker, including a first black-and-white camera configured to capture a first black-and-white image and a second black-and-white camera configured to capture a second black-and-white image. Among them, the first black-and-white camera and the second black-and-white camera are the same. The eye tracker further includes an infrared emission device configured to provide supplementary lighting; An ambient light sensor for receiving a light induction signal, configured to control the activation or adjustment of the infrared emission device when it detects that the light induction signal is lower than a given threshold; An eye tracking image processor, configured to identify the presence of a user's eyes based on at least one of the first black-and-white image and the second black-and-white image and determine the eye spatial position based on the eyes identified in the first black-and-white image and the second black-and-white image. The eye spatial position is determined according to the following formula: Wherein, DR and DL are the distances between the user's right eye R and left eye L and the plane where the first and second black-and-white cameras are located respectively; XRa and XRb are the X-axis coordinates of the user's right eye R imaged in the focal planes of the first and second black-and-white cameras respectively; XLa and XLb are the X-axis coordinates of the user's left eye L imaged in the focal planes of the first and second black-and-white cameras respectively; T is the distance between the first and second black-and-white cameras, f is the focal length f of the first and second black-and-white cameras; α is the inclination angle between the line connecting the user's two eyes and the plane where the first and second black-and-white cameras are located; P is the distance between the user's two eyes or the interpupillary distance.

2. The human eye tracking device according to claim 1, characterized in that, It further includes an eye tracking data interface configured to transmit eye spatial position information indicating the eye spatial position.

3. The human eye tracking device according to claim 1, wherein The infrared emission device is configured to emit infrared light with a wavelength greater than or equal to 1.5 micrometers.

4. The human eye tracking device according to any one of claims 1 to 3, characterized in that, The first black-and-white camera and the second black-and-white camera are configured to capture a first black-and-white image sequence including the first black-and-white image and a second black-and-white image sequence including the second black-and-white image respectively.

5. The human eye tracking device according to claim 4, characterized in that, The eye tracking image processor includes a synchronizer configured to determine the first black-and-white image and the second black-and-white image with time synchronization for eye recognition and determination of the eye spatial position.

6. The human eye tracking device according to claim 5, characterized in that, The eye tracking image processor includes: A buffer configured to buffer multiple first black-and-white images and second black-and-white images in the first black-and-white image sequence and the second black-and-white image sequence; A comparator configured to compare multiple front and rear first black-and-white images and second black-and-white images in the first black-and-white image sequence and the second black-and-white image sequence; A decision maker configured to, when the comparator does not recognize the presence of eyes in the current first black-and-white image and the second black-and-white image in the first black-and-white image sequence and the second black-and-white image sequence but recognizes the presence of eyes in the first black-and-white image and the second black-and-white image before or after, use the eye spatial position determined based on the first black-and-white image and the second black-and-white image before or after as the current eye spatial position.

7. A 3D display device, comprising: A multi-viewpoint autostereoscopic 3D display screen, including multiple sub-pixels corresponding to multiple viewpoints; The eye tracking device according to any one of claims 1 to 6 to obtain the eye spatial position; A 3D processing device is configured to determine a corresponding viewing point according to the human eye spatial position obtained by the human eye tracking device, and render sub-pixels corresponding to the viewing point on the multi-view autostereoscopic 3D display based on a 3D signal.

8. The 3D display device according to claim 7, wherein The multi-view autostereoscopic 3D display includes a plurality of composite pixels, each of the plurality of composite pixels includes a plurality of composite sub-pixels, and each of the plurality of composite sub-pixels is composed of a plurality of sub-pixels corresponding to a plurality of viewing points.

9. The 3D display device according to claim 7, wherein The 3D processing device is communicatively connected to the human eye tracking device.

10. The 3D display device according to any one of claims 7 to 9, characterized in that, Further included: A 3D shooting device configured to collect 3D images; The 3D shooting device includes a depth camera and at least two color cameras.

11. The 3D display device according to claim 10, wherein The human eye tracking device is integrally provided with the 3D shooting device.

12. The 3D display device according to claim 11, characterized in that, The 3D shooting device is placed in front of the 3D display device.

13. A human eye tracking method includes: Taking a first black-and-white image and a second black-and-white image respectively through the same first and second black-and-white cameras, wherein when taking the first black-and-white image and the second black-and-white image, in response to detecting that the light sensing signal is lower than a given threshold, controlling the turning on or adjustment of the infrared emission device for fill light; Identifying the presence of a human eye based on at least one of the first black-and-white image and the second black-and-white image; Determining a human eye spatial position based on the human eye identified in the first black-and-white image and the second black-and-white image, and the human eye spatial position is determined according to the following formula: wherein, DR and DL are the distances between the user's right eye R and left eye L and the plane where the first and second black-and-white cameras are located respectively; XRa and XRb are the X-axis coordinates of the user's right eye R imaged in the focal planes of the first and second black-and-white cameras respectively; XLa and XLb are the X-axis coordinates of the user's left eye L imaged in the focal planes of the first and second black-and-white cameras respectively; T is the distance between the first and second black-and-white cameras, f is the focal length of the two black-and-white cameras; α is the inclination angle between the line connecting the user's two eyes and the plane where the first and second black-and-white cameras are located; P is the distance between the user's two eyes or the interpupillary distance.

14. The human eye tracking method according to claim 13, wherein Further included: Transmitting human eye spatial position information indicating the human eye spatial position.

15. The human eye tracking method according to any one of claims 13 to 14, characterized in that, Further included: Taking a first black-and-white image sequence including the first black-and-white image and a second black-and-white image sequence including the second black-and-white image respectively.

16. The human eye tracking method according to claim 15, wherein Further included: Determining the first black-and-white image and the second black-and-white image with time synchronization.

17. The human eye tracking method according to claim 16, wherein Further included: Caching a plurality of first black-and-white images and second black-and-white images in the first black-and-white image sequence and the second black-and-white image sequence; Comparing a plurality of previous and subsequent first black-and-white images and second black-and-white images in the first black-and-white image sequence and the second black-and-white image sequence; When the presence of a human eye is not identified in the current first black-and-white image and the second black-and-white image in the first black-and-white image sequence and the second black-and-white image sequence through comparison, and the presence of a human eye is identified in the previous or subsequent first black-and-white images and second black-and-white images, taking the human eye spatial position determined based on the previous or subsequent first black-and-white images and second black-and-white images as the current human eye spatial position.

18. A 3D display method includes: Obtain the human eye spatial position of the user by using the human eye tracking method according to any one of claims 13 to 17; Determine the corresponding viewing point according to the human eye spatial position; Render the sub-pixels corresponding to the viewing point on the multi-view autostereoscopic display based on the 3D signal.

19. The 3D display method according to claim 18, wherein Further comprising: Provide the multi-view autostereoscopic display, including a plurality of composite pixels, each of the plurality of composite pixels includes a plurality of composite sub-pixels, and each of the plurality of composite sub-pixels is composed of a plurality of sub-pixels corresponding to a plurality of viewing points.

Images