Electronic device and wearable display device
By setting multiple light sources and cameras at the edge of the folded optical circuit lens and adjusting the camera working mode with the controller, the problem of optical device arrangement in electronic devices under the folded optical circuit lens is solved, improving the eye image quality and tracking accuracy, while reducing the Power consumption.
Patent Information
- Application Number
- CN202310814922.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-04
- Publication Date
- 2025-01-07
AI Technical Summary
In electronic devices that employ folding optical circuit lenses, it is difficult to arrange the camera between the lens and the display screen, resulting in optical device arrangement problems and other potential problems.
An electronic device is designed, including the arrangement of multiple light sources and cameras at the edge of the lens, which are arranged along the horizontal or vertical axis of the lens to capture high-quality eye images and perform eye tracking based on the image by a controller , control the camera's working mode to save power consumption.
It achieves that the feasibility and quality of eye images are ensured without affecting the lens characteristics, improves the accuracy of eye position determination and tracking, and reduces the power consumption of electronic devices.
Smart Images
Figure CN119270496A_ABST
Abstract
Description
Technical Field
[0001] Example embodiments of the present disclosure generally relate to the field of optical imaging, and in particular, to an electronic device and a wearable display device including such an electronic device. Background Art
[0002] In recent years, research has begun on the use of eye tracking in the field of extended reality (XR), including virtual reality (VR), augmented reality (AR), mixed reality (MR), etc. For example, for AR, VR or MR display devices, eye tracking can be used for imaging display, interactive control, target recognition, identity authentication, health monitoring, social interaction and multi-person collaboration, and provide users with a clearer and smoother viewing experience. Summary of the invention
[0003] In a first aspect of the present disclosure, an electronic device is provided, comprising: a lens; a plurality of light sources, disposed on the lens or adjacent to an edge of the lens and configured to illuminate the eyes of a user; at least one camera, disposed adjacent to an edge of the lens and configured to capture an image of the eye; and a controller, configured to perform eye tracking for the user based on the image captured by the at least one camera.
[0004] In some embodiments, the lens comprises a folded light path lens.
[0005] In some embodiments, at least a portion of the plurality of light sources are embedded in the lens.
[0006] In some embodiments, the at least one camera includes a first camera disposed adjacent an end of the lens along a lateral axis of the lens.
[0007] In some embodiments, the at least one camera includes a second camera disposed adjacent an end of the lens along a longitudinal axis of the lens.
[0008] In some embodiments, the controller is further configured to control an operating mode of the at least one camera based on a position of a predetermined reference portion of the user's eye in the image, the operating mode being associated with at least power consumption of the at least one camera.
[0009] In some embodiments, the lens, the plurality of light sources, the at least one camera, and at least a portion of the controller are integrated with each other.
[0010] In some embodiments, the plurality of light sources include light emitting diodes and / or vertical cavity surface emitting lasers.
[0011] In a second aspect of the present disclosure, a wearable display device is provided, which includes the electronic device described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:
[0013] Figure 1 A schematic diagram showing an embodiment of an electronic device according to an embodiment of the present disclosure;
[0014] Figure 2 A schematic diagram showing another embodiment of an electronic device according to an embodiment of the present disclosure;
[0015] Figure 3 A schematic diagram showing another embodiment of an electronic device according to an embodiment of the present disclosure;
[0016] Figure 4 A schematic diagram showing an embodiment of the relative position between an electronic device and an eye according to an embodiment of the present disclosure; and
[0017] Figure 5 A schematic diagram showing an embodiment of a user wearing an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0019] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.
[0020] In addition, it should be understood that the spatial descriptions used in this document (e.g., "upper", "lower", "left", "right", "lateral", "longitudinal", etc.) are for illustrative purposes only, and the actual implementation of the structures described in this document can be spatially arranged in any direction or manner.
[0021] In recent years, Extended Reality (XR) has been widely studied and applied. XR integrates virtual content and real scenes by combining hardware devices and a variety of technical means to provide users with a unique sensory experience. XR includes virtual reality VR, augmented reality AR, mixed reality MR, etc. VR is a virtual world that uses computers to simulate three-dimensional space, providing users with an immersive experience in vision, hearing, touch, etc. AR allows the real environment and virtual objects to be superimposed in the same space in real time and exist simultaneously. MR is a new visualization environment that integrates the real world and the virtual world, where objects in the physical real world scene coexist with objects in the virtual world in real time.
[0022] In some scenarios, electronic devices can be implemented in AR, VR or MR display devices, such as AR, VR or MR glasses. When a user wears a display device with an electronic device, the electronic device can be used for imaging display, interactive control, target recognition, identity authentication, health monitoring, social interaction and multi-person collaboration. By using electronic devices, the imaging system can present clearer and smoother images to users.
[0023] At present, folded light path lenses, or pancake lenses, are beginning to be used in the display devices described above. This type of lens usually refers to a small, lightweight, thin lens or lens assembly, which mainly uses the principle of polarized light and adopts a folded light path design. After the image source enters the semi-reflective and semi-transparent beam splitter, the light is folded back and forth between the lens, phase delay plate, and reflective polarizing film for many times, and finally emitted from the reflective polarizing film into the human eye.
[0024] Compared with traditional Fresnel lenses, folded optical path lenses can improve edge imaging distortion and enhance imaging contrast, clarity and fineness. In addition, folded optical path lenses can adjust the diopter, so that myopic users can see images clearly without wearing myopic glasses.
[0025] Due to the thin and light characteristics of the folded optical path lens, it is difficult to arrange the camera of the electronic device between the lens and the display screen. Therefore, an embodiment of the present disclosure provides an electronic device to solve or at least partially solve the optical device arrangement problem and / or other potential problems caused by the use of the folded optical path lens.
[0026] The following combination Figures 1 to 5 The principles of the present disclosure are explained in detail.
[0027] like Figure 1 As shown, the electronic device 100 capable of implementing the eye tracking function according to an embodiment of the present disclosure may include a lens 110. The electronic device 100 may be, for example, a helmet, glasses, or other appropriate wearable or portable device for VR, AR, or XR. In some scenarios, such as when the electronic device 100 is applied to a wearable display device, the electronic device 100 may include lenses 110 arranged in pairs. It should be understood that in some other scenarios, such as when analyzing user behavior based on eye tracking, the lens 110 may also be provided in a single piece.
[0028] In the embodiment of the present disclosure, the lens 110 can be implemented as a folded light path lens or lens group, which can also be referred to as a pancake lens in the context of the present disclosure. It should be understood that any element with the same or similar structure and optical properties that can refract the light path can also be used in the electronic device 100 of the present disclosure.
[0029] A plurality of light sources 120 are arranged at the lens 110. In some embodiments, Figure 1 As shown, a plurality of light sources 120 are arranged around the edge of the lens 110. In other embodiments, as shown in FIG. Figure 2 As shown, a plurality of light sources 120 are arranged in the middle portion of the lens 110. Alternatively or additionally, Figure 2 The plurality of light sources 120 shown in FIG. 1 may be embedded in the lens 110 .
[0030] In an embodiment of the present disclosure, the plurality of light sources 120 may be implemented as light emitting diodes (LEDs). Alternatively or additionally, the plurality of light sources 120 may also be implemented as vertical cavity surface emitting lasers (VCSELs). It should be understood that the scope of the present disclosure is not limited thereto. Other light emitting elements with suitable sizes, structures, and features may be configured as the plurality of light sources in the electronic device of the present disclosure.
[0031] When the lens 110 of the electronic device 100 is placed in front of the user's eyes, the multiple light sources in the electronic device 100 can be used to illuminate the user's eyes. Figure 4 As shown, the eye tracking device 100 includes a lens 110 and a plurality of light sources 120 arranged at the edge of the lens 110. The light emitted by the light source 120 illuminates the eye 200, especially illuminates the area adjacent to the pupil 210 and / or the iris 220 to generate glint 230.
[0032] The electronic device 100 may further include a camera 130. The camera 130 is used to capture images of the user's eyes. In some embodiments, the camera 130 may be disposed adjacent to the end of the lens 110 along the horizontal axis H of the lens 110. When the user wears the electronic device 100, the direction of the horizontal axis H is substantially consistent with the direction of the line connecting the two eyes of the user. In the case where the lens 110 is a common ellipse-like shape, the horizontal axis H may generally be the long axis of the ellipse.
[0033] Figure 1-Figure 3 The examples shown in follow this configuration. Figure 1 and Figure 2 In the example shown, the camera 130 is arranged at the distal end of the lens 110 along the horizontal axis H of the lens 110. At this time, when the user wears the electronic device 100, the camera 130 will be located adjacent to the bridge of the user's nose. Figure 3 In the example shown, the camera 130 is arranged at both the proximal end and the distal end of each lens 110 along the horizontal axis H of the lens 110. At this time, when the user wears the electronic device 100, the camera 130 at the proximal end will be positioned adjacent to the user's nose bridge, and the camera at the distal end will be positioned adjacent to the user's temporal portion (i.e., temple).
[0034] In the case where the camera 130 is arranged at the distal end of the lens 110 along the horizontal axis H of the lens 110, as shown in FIG. Figure 5 As shown, when the user 310 wears the wearable display device 300, in some embodiments, the camera 130 can be located at the nose of the user 300, such as between the bridge of the nose and the inner corner of the user's eye.
[0035] Since the acquisition angles for eye images at the nose and temples are relatively wide, in such an embodiment, by placing the camera at such a position, higher quality eye images can be better captured, thereby improving the accuracy of eye position determination and eye tracking.
[0036] It should be understood that the camera 130 of the electronic device 100 of the present disclosure is not limited to Figures 1 to 3 Without violating the principle of the present disclosure, the position and / or number of the camera 130 may be arbitrarily modified or adjusted.
[0037] Through the embodiments described above in conjunction with the accompanying drawings, the arrangement of the camera 130 and the light source 120 of the electronic device 100 is realized in the scenario of using a specific lens. Thus, on the one hand, it is ensured that the structure of the electronic device 100 does not affect the characteristics of the lens, and on the other hand, it is also ensured that the feasibility of collecting eye images and the quality of the images are ensured.
[0038] In an embodiment of the present disclosure, the electronic device 100 may further include a controller 140. The controller 140 may be integrated into the camera 130 or at least may be coupled to the camera 130 to obtain an eye image captured by the camera 130. Based on the image captured by the camera 130. The controller 140 may perform eye tracking for the user. For example, the controller 140 may be implemented as a custom processing chip integrated with a camera sensor. In this way, the data transmission delay from the camera 130 to the controller 140 may be saved. Alternatively, the controller 140 may be implemented as a separate computing unit or on the same computing unit as the eye tracking algorithm.
[0039] Eye tracking may, for example, involve tracking the position of the center of the pupil, the gaze angle, and / or the eye movement trajectory of the user. For example, by determining the position and / or size of a predetermined reference part of the user's eye (e.g., the pupil) in the image, the controller 140 may determine the center of the pupil and capture the gaze direction of the current user. Alternatively or additionally, by determining the position and / or size of a predetermined reference part of the user's eye in the image, the controller 140 may control the operating mode of the camera 130. The operating mode may, for example, be related to the power consumption of the camera 130.
[0040] For example, the working mode of the camera 130 may include an image output mode of the camera. For example, if the pixel and / or resolution of the image output by the camera 130 is higher, the power consumed by the camera 130 is greater. Once the position and / or size of a predetermined reference part of the user's eye, such as the pupil, in the image captured by the camera 130 does not meet a predetermined condition, such as the pupil is not captured or only a partial pupil image is captured, it may not have much reference value for the result of eye tracking. In this case, it is desirable to reduce the power consumption of the camera.
[0041] Therefore, in some embodiments, the controller 140 may determine the eye position through the image captured by the camera 130. If the controller 140 determines that the position and / or size of the predetermined reference part of the user's eye in the image meets the predetermined condition, the controller 140 causes the camera 130 to output the image in a manner of changing the image size, such as by scaling or cropping. This mode of outputting the image may, for example, involve a cropping mode. In this mode, the camera 130 may output a cropped image based on a region of interest.
[0042] In other embodiments, if the controller 140 determines that the position and / or size of the predetermined reference part of the user's eye in the image meets the predetermined condition, the controller 140 causes the camera 130 to output the image in a manner of reducing pixels and / or resolution. Such an output image mode may, for example, incorporate a binning mode.
[0043] Thus, the controller 140 can adjust the working mode of the camera according to the acquired image, thereby saving power consumption during the use of the electronic device. In the application scenario where the electronic device is arranged on a wearable device, it is very beneficial to reduce the power consumption. Because wearable devices often use a rechargeable power supply, it is expected to reduce the charging frequency of the power supply and extend the continuous use time of the user.
[0044] It should be understood that although the Figures 1 to 5 The electronic device according to the embodiment of the present disclosure is described. Without violating the principles of the present disclosure, the structure of the electronic device may be modified in any suitable manner, such as by adding, removing, or replacing components. Figures 1 to 5 At least a portion of the lens, camera, light source, and controller of the described electronic device may be integrally arranged.
[0045] In addition, an embodiment of the present disclosure further provides a wearable display device, which may include the electronic device described above. Figure 5 As shown, the wearable display device 300 may include the electronic device 100. The wearable display device 300 may be, for example, the AR, VR or MR display device mentioned above, including but not limited to a head-mounted display, smart glasses, a user behavior analysis device, and the like.
[0046] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. An electronic device (100), comprising: A lens (110); a plurality of light sources (120) disposed on the lens (110) or adjacent to an edge of the lens (110) and configured to illuminate the eyes (200) of a user; at least one camera (130) disposed adjacent to an edge of the lens (110) and configured to capture an image of the eye (200); as well as A controller (140) is configured to perform eye tracking for the user based on the image captured by the at least one camera (130).
2. The electronic device according to claim 1, wherein the lens (110) comprises a folded light path lens.
3. The electronic device (100) according to claim 1, wherein at least a portion of the plurality of light sources (120) are embedded in the lens (110).
4. The electronic device (100) of claim 1, wherein the at least one camera (130) comprises a first camera disposed adjacent to an end of the lens (110) along a transverse axis (H) of the lens (110).
5. The electronic device (100) of claim 1, wherein the at least one camera (130) comprises a second camera disposed adjacent to an end of the lens (110) along a longitudinal axis (L) of the lens (110).
6. The electronic device (100) according to claim 1, wherein the controller (140) is further configured to: Based on the position of a predetermined reference part of the user's eye (200) in the image, an operating mode of the at least one camera (130) is controlled, the operating mode being associated at least with power consumption of the at least one camera (130).
7. The electronic device (100) according to claim 1, wherein at least a portion of the lens (110), the plurality of light sources (120), the at least one camera (130), and the controller (140) are integrated with each other.
8. The electronic device (100) of claim 1, wherein the plurality of light sources (120) include light emitting diodes and / or vertical cavity surface emitting lasers.
9. A wearable display device (300), comprising the electronic device (100) according to any one of claims 1 to 7.