Near-to-eye display apparatus, control method, electronic device, and storage medium

Abstract

The invention provides a near-to-eye display device, a control method, electronic equipment and a storage medium, and the near-to-eye display device comprises a lens assembly which comprises a left lens and a right lens; the light guide structure is arranged on the periphery of the lens assembly, a light inlet area, a first light outlet area and a second light outlet area are arranged on the light guide structure, the first light outlet area is correspondingly arranged on the peripheral side of the left lens, and the second light outlet area is correspondingly arranged on the peripheral side of the right lens; the infrared light emitter is correspondingly arranged in the light inlet area, the infrared light emitter is configured to emit infrared light, and the infrared light emitted from the first light outlet area and the second light outlet area can be reflected into infrared reflected light by human eyes; and the infrared light receiver is arranged corresponding to the infrared light emitter, and the infrared light receiver is used for receiving the infrared reflected light. The near-to-eye display device achieves the purpose of eyeball tracking, and meets the requirements of attractive appearance and light weight at the same time.

Description

Technical Field

[0001] The present invention relates to the field of extended reality, and in particular to a near-eye display device, a control method, an electronic device and a storage medium. Background Art

[0002] Extended Reality (XR) display technology generally includes Augmented Reality (AR) display technology, Virtual Reality (VR) display technology, and Mixed Reality (MR) display technology. As one of the product forms of electronic terminals for the application of extended reality display technology, near-eye display devices can create a virtual world for users or combine the real and virtual worlds to create a new visual environment. They have broad application prospects in important fields such as military, medical, education, games, and life.

[0003] Currently, in order to improve the interactive efficiency and immediacy of AR glasses, the industry has begun to integrate eye tracking technology. This technology usually relies on the deployment of multiple infrared light emitters, and uses the directional reflection principle of these light sources to enable the camera to capture the direction of the pupil's line of sight. By calculating the convergence angle between the two pupils, the processor can accurately determine the user's gaze point, thereby realizing the eye tracking function. However, in the pursuit of a thin and light design of near-eye display devices, the above solution faces significant challenges: the addition of infrared light emitters greatly occupies the internal space of the frame, and the power supply and signal transmission line layout of the infrared light emitter is complex, which seriously restricts the appearance and lightweight design of the near-eye display device. Summary of the invention

[0004] Embodiments of the present application provide a near-eye display device, a control method, an electronic device, and a storage medium. The near-eye display device achieves the purpose of eye tracking while meeting the requirements of aesthetic appearance and light weight.

[0005] The embodiment of the present application provides a near-eye display device, comprising:

[0006] A lens assembly, the lens assembly comprising a left lens and a right lens;

[0007] A light guide structure, the light guide structure is arranged at the periphery of the lens assembly, and is provided with a light inlet area, a first light outlet area and a second light outlet area, the first light outlet area is correspondingly arranged at the periphery of the left lens, and the second light outlet area is correspondingly arranged at the periphery of the right lens;

[0008] an infrared light emitter, the infrared light emitter being arranged correspondingly in the light inlet area, the infrared light emitter being configured to emit infrared light, the infrared light being able to enter the light guide structure from the light inlet area, and then being emitted from the first light exit area and the second light exit area, and the infrared light emitted from the first light exit area and the second light exit area being able to be reflected by the human eye into infrared reflected light;

[0009] An infrared light receiver, the infrared light receiver is arranged corresponding to the infrared light transmitter, and the infrared light receiver is used to receive the infrared reflected light.

[0010] In some embodiments, the light-guiding structure has a first surface and a second surface, wherein the first surface and the second surface extend along the length of the light-guiding structure and are arranged opposite to each other, wherein the first surface is configured to face the human eye, and a plurality of dots are arranged on the second surface, wherein the dots are configured to reflect infrared light to form the first light-emitting area and the second light-emitting area on the first surface.

[0011] In some embodiments, the dots are convex dots or concave dots; or, the dots are in the shape of a pyramid, a frustum, a cone or a truncated cone.

[0012] In some embodiments, the light guide structure further has a third surface, the third surface is arranged at the end of the light guide structure, the third surface is respectively connected to the first surface and the second surface, and the third surface forms the light inlet area. In some embodiments, the light guide structure includes a first light guide segment and a second light guide segment, the first light guide segment is arranged on the peripheral side of the left lens, and a plurality of the first light exit areas are arranged on the first light guide segment; the second light guide segment is arranged on the peripheral side of the right lens, and a plurality of the second light exit areas are arranged on the second light guide segment; the infrared light emitter is arranged between the left lens and the right lens, and the third surface of the first light guide segment and the third surface of the second light guide segment face the infrared light emitter.

[0013] In some embodiments, the near-eye display device further includes a frame and a nose pad, wherein the nose pad is connected to the frame, the frame is connected to the lens assembly, and the infrared light receiver is disposed on the nose pad.

[0014] The present application also provides a control method, which is applied to the above-mentioned near-eye display device. The control method includes:

[0015] Acquire a virtual display screen, wherein the virtual display screen has a plurality of icons, each of which corresponds to an application program;

[0016] Calculating the gaze point of the human eye according to the infrared reflected light;

[0017] Determine a target icon among the multiple icons according to the gaze point;

[0018] Get the trigger instruction;

[0019] The application corresponding to the target icon is triggered according to the trigger instruction.

[0020] In some embodiments, after determining the target icon among the multiple icons according to the gaze point, the method further includes: controlling the target icon to be in a highlighted state, a bold state, or a protruding state.

[0021] The present application also provides an electronic device, including:

[0022] A first acquisition module, the first acquisition module is used to acquire a virtual display screen, the virtual display screen has a plurality of icons, each of the icons corresponds to an application;

[0023] A calculation module, the calculation module is used to calculate the gaze point of the human eye according to the infrared reflected light;

[0024] a determination module, the determination module being used to determine a target icon among the multiple icons according to the gaze point;

[0025] A second acquisition module, the second acquisition module is used to acquire a trigger instruction;

[0026] A trigger module is used to trigger the application corresponding to the target icon according to the trigger instruction.

[0027] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above control method are implemented.

[0028] The near-eye display device, control method, electronic device and storage medium provided by the embodiment of the present application have achieved significant improvements and innovations in the field of eye tracking technology. Specifically, the near-eye display device integrates a lens assembly, a light guide structure, an infrared light emitter and an infrared light receiver to achieve an efficient and compact eye tracking function. The lens assembly in the device is composed of a left lens and a right lens, and the light guide structure is arranged on the periphery of the lens assembly. The light guide structure has a light inlet area, a first light outlet area and a second light outlet area, which provides a precise path for the transmission and distribution of infrared light. Among them, the light inlet area is located between the left lens and the right lens, and the first light outlet area and the second light outlet area are respectively arranged on the peripheral side of the left lens and the right lens. The embodiment of the present application only uses a small number of infrared light emitters, and they are arranged in the light inlet area of ​​the light guide structure. The infrared light emitter can emit infrared light. After the infrared light enters the light guide structure through the light inlet area, it is efficiently guided to the first light outlet area and the second light outlet area, and is emitted from these two areas. The human eye reflects these emitted infrared lights to form infrared reflected light, which is then received by the infrared light receiver. This design not only achieves efficient transmission and distribution of infrared light, but also greatly meets the requirements of near-eye display devices for aesthetics and lightweight while achieving the purpose of eye tracking. Compared with traditional solutions, the embodiment of the present application significantly reduces the number of infrared light emitters and the complexity of layout, providing strong support for further miniaturization and lightweighting of near-eye display devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0030] Figure 1 A first structural schematic diagram of a near-eye display device provided in an embodiment of the present application.

[0031] Figure 2 A schematic diagram of the working principle of the infrared light receiver provided in an embodiment of the present application.

[0032] Figure 3 A schematic diagram of the first structure of the light guide structure provided in an embodiment of the present application.

[0033] Figure 4 A second structural schematic diagram of the light guiding structure provided in an embodiment of the present application.

[0034] Figure 5 A second structural schematic diagram of the near-eye display device provided in an embodiment of the present application.

[0035] Figure 6 A flow chart of a control method provided in an embodiment of the present application.

[0036] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0038] The embodiments of the present application provide a near-eye display device, a control method, an electronic device, and a storage medium, wherein the near-eye display device achieves the purpose of eye tracking while meeting the requirements of aesthetic appearance and light weight.

[0039] See also Figure 1 , Figure 1 A first structural schematic diagram of a near-eye display device provided in an embodiment of the present application.

[0040] The embodiment of the present application provides a near-eye display device 100, which is generally used in augmented, mixed or virtual reality applications and can provide users with close-range visual display. The near-eye display device 100 in the embodiment of the present application is a device that is configured at a certain distance in front of the user's eyes (generally 2-5 cm) to directly present the display content in front of the user's eyes, such as optical perspective glasses in augmented reality (AR) devices, etc. The near-eye display device 100 includes but is not limited to head-mounted devices such as AR glasses and AR helmets, and its application range is wide and flexible.

[0041] The near-eye display device 100 includes a lens assembly 10 , a light guide structure 20 , an infrared light emitter 30 , and an infrared receiver.

[0042] The lens assembly 10 includes a left lens 11 and a right lens 12 , and is used to present images or information to a user.

[0043] The light guide structure 20 is disposed on the periphery of the lens assembly 10 , which not only avoids occupying the effective display area of ​​the lens, but also ensures that the infrared light can be accurately transmitted to the periphery of the lens.

[0044] The light guide structure 20 is provided with a light inlet area 21, a first light outlet area 22 and a second light outlet area 23. The light inlet area 21 is correspondingly arranged between the left lens 11 and the right lens 12, the first light outlet area 22 is correspondingly arranged on the peripheral side of the left lens 11, and the second light outlet area 23 is correspondingly arranged on the peripheral side of the right lens 12. The first light outlet area 22 and the second light outlet area 23 are used to uniformly emit infrared light to the pupil position of the human eye 200.

[0045] The infrared light emitter 30 is a device capable of emitting infrared light. The infrared light emitter 30 is correspondingly arranged in the light inlet area 21 of the light guide structure 20, and the infrared light emitter 30 is configured to emit infrared light. Infrared light is an electromagnetic wave with a wavelength outside the visible spectrum, which belongs to invisible light and is often used in applications such as eye tracking and remote control.

[0046] The infrared light can enter the light guide structure 20 from the light inlet area 21, and then be emitted from the first light exit area 22 and the second light exit area 23. The infrared light emitted from the first light exit area 22 and the second light exit area 23 can be reflected by the human eye 200 to become infrared reflected light. More specifically, after the infrared light enters the light guide structure 20 from the light inlet area 21, it can be totally reflected and propagated, and then be emitted from the first light exit area 22 and the second light exit area 23.

[0047] The number of the first light emitting areas 22 or the second light emitting areas 23 is greater than or equal to 1. Furthermore, in order to improve the accuracy and robustness of eye tracking, in actual applications, the number of the first light emitting areas 22 is often greater than or equal to 3, and the number of the second light emitting areas 23 is also greater than or equal to 3. Such a design not only expands the coverage of infrared light, but also improves the accuracy of the three-dimensional position estimation of the eye through multi-point measurement. For example, under complex lighting conditions or when the user performs rapid eye movements, multiple light emitting areas can provide more data points, which helps the algorithm to better identify and filter noise, thereby outputting more stable and reliable eye tracking results.

[0048] See also Figure 2 , Figure 2 Schematic diagram of the working principle of the infrared light receiver provided in the embodiment of the present application. The infrared light receiver 40 is used to receive infrared light. The infrared light receiver 40 is arranged corresponding to the infrared light transmitter 30, and the infrared light receiver 40 is used to receive infrared reflected light, thereby being able to capture the line of sight direction of the user's human eye 200. By calculating the convergence angle between the two pupils, the processor can accurately determine the user's gaze point, thereby realizing the eye tracking function.

[0049] In some scenarios, when the near-eye display device 100 is a device equipped with augmented reality (AR) technology, its functions and application scenarios are greatly expanded. AR technology enables users to overlay virtual information on the view of the real world, bringing an immersive interactive experience to users. In this context, the near-eye display device 100 not only needs to clearly present the virtual display screen in front of the user's eyes, but also needs to be able to accurately identify the user's gaze point and trigger the corresponding application according to the user's intention.

[0050] In a virtual display screen, there are usually multiple icons, each of which corresponds to a specific application. These icons may represent different functions or services, such as map navigation, information prompts, game entertainment, etc. By looking at an icon, the user can express his or her desire to use the application corresponding to the icon.

[0051] In order to achieve this function, the near-eye display device 100 can identify the user's gaze point using a small amount of infrared light emitters 30. The infrared light emitter 30 can emit infrared light. After the infrared light enters the light guide structure 20 through the light inlet area 21, it is efficiently guided to the first light outlet area 22 and the second light outlet area 23, and emitted from these two areas. The human eye 200 reflects these emitted infrared lights to form infrared reflected light, which is then received by the infrared light receiver 40, and the processor can calculate the user's gaze point. By comparing the gaze point and the icon position in the virtual screen, the processor can accurately determine the icon that the user is looking at, that is, the target icon. Once the target icon is determined, the processor can wait for the user's trigger instruction. The trigger instruction can be issued by the user through a specific gesture, voice command or other means. When the user issues a trigger instruction, the processor triggers the application corresponding to the target icon according to the content of the instruction. For example, if the user is looking at the map navigation icon and issues a "start" instruction, the processor will start the map navigation application to provide the user with real-time route guidance.

[0052] Therefore, the near-eye display device 100 in the embodiment of the present application not only realizes efficient transmission and distribution of infrared light by reducing the number of infrared light emitters 30, but also greatly satisfies the requirements of the near-eye display device 100 for aesthetic appearance and lightweight while achieving the purpose of eye tracking. Compared with the traditional solution, the embodiment of the present application significantly reduces the number and layout complexity of the infrared light emitters 30, providing strong support for the further miniaturization and lightweighting of the near-eye display device 100.

[0053] See also Figure 3 as well as Figure 4 , Figure 3 This is a schematic diagram of the first structure of the light guide structure provided in the embodiment of the present application. Figure 4A second structural schematic diagram of the light guide structure provided in an embodiment of the present application. The light guide structure 20 may be a translucent flexible film that is easy to bend and wrap around the left lens 11 and / or the right lens 12. The light guide structure 20 may be a light guide film (LGF), which is a low-cost and low-energy light-transmitting material. The thickness of the light guide film may be 0.1 mm to 0.2 mm, for example, 0.125 mm, 0.15 mm, or 0.175 mm.

[0054] The light guide structure 20 has a first surface 24 and a second surface 25. The first surface 24 is the outer surface of the light guide structure 20 facing the observer, and in this case specifically refers to the portion facing the human eye 200. The second surface 25 is the other side surface opposite to the first surface 24, and is generally used to set structures or elements that affect the propagation of light. The first surface 24 and the second surface 25 extend along the length of the light guide structure 20 and are arranged opposite to each other. The first surface 24 is configured to face the human eye 200. A plurality of mesh dots 29 are arranged on the second surface 25. The mesh dots 29 can destroy the total reflection transmission of light, so that infrared light is transmitted from the light guide structure 20.

[0055] In some cases, mesh dots 29 are configured to reflect infrared light to form first light exit area 22 and second light exit area 23 on first surface 24 .

[0056] In other cases, when the first surface 24 is the outer surface of the light-guiding structure 20 facing the observer, the dots 29 can also be set on the first surface 24 so that the infrared light transmitting dots 29 form a first light-emitting area 22 and a second light-emitting area 23 from the first surface 24 of the light-guiding structure 20.

[0057] When infrared light is irradiated onto these mesh dots 29, the mesh dots 29 guide the light in a specific direction, so that the infrared light can form two or more clear light exit areas on the first surface 24, namely, the first light exit area 22 and the second light exit area 23. It can be seen that by adjusting the shape, layout and number of the mesh dots 29, the distribution of the light exit areas can be flexibly changed to adapt to different device sizes and shapes. Finally, this design is also reliable and stable, and can maintain stable performance during long-term use.

[0058] The mesh dots 29 may be convex dots or concave dots. When the mesh dots 29 are convex dots, they help to enhance the light intensity in a specific direction, and are suitable for applications that require improved light focusing effects.

[0059] The shape of the mesh dots 29 can be a pyramid, a prism, a cone or a truncated cone. When the mesh dots 29 are cones or truncated cones, the mesh dots 29 are conducive to maintaining light clarity and directionality, helping to reduce unnecessary scattering of infrared light, thereby improving light transmission efficiency and image quality. The arrangement density of the mesh dots 29 can gradually increase as the distance between the mesh dots 29 and the infrared light emitter 30 increases.

[0060] By properly designing the shape of the dots 29 (convex or concave, and specific geometric shapes such as pyramids, frustums, cones or truncated cones), the propagation path of light can be precisely controlled to meet the requirements of different applications for light quality and efficiency.

[0061] The light guide structure 20 also has a third surface 26, which is disposed at the end of the light guide structure 20, and is connected to the first surface 24 and the second surface 25, respectively, and the third surface 26 forms a light inlet area 21. The third surface 26 forms a specific light inlet area 21, so that infrared light can efficiently enter the interior of the light guide structure 20, which facilitates total reflection of the infrared light in the light guide structure 20, and finally exits from the first light exit area 22 and the second light exit area 23, thereby realizing low-loss and high-speed infrared transmission inside the light guide structure 20.

[0062] The light guide structure 20 includes a first light guide segment 27 and a second light guide segment 28. The first light guide segment 27 is arranged around the left lens 11 and is mainly responsible for the transmission of light signals corresponding to the left eye area. A plurality of first light exit areas 22 are arranged on the first light guide segment 27. The second light guide segment 28 is arranged around the right lens 12 and is mainly responsible for the transmission of light signals corresponding to the right eye area. A plurality of second light exit areas 23 are arranged on the second light guide segment 28. This bilaterally symmetrical layout is not only beautiful, but also ensures that users can obtain uniform and efficient light signal coverage regardless of the viewing angle.

[0063] The infrared light emitter 30 is disposed between the left lens 11 and the right lens 12. The third surface 26 of the first light guide segment 27 and the third surface 26 of the second light guide segment 28 face the infrared light emitter 30, presenting an efficient integrated design, that is, one infrared light emitter 30 corresponds to two light guide segments (the first light guide segment 27 and the second light guide segment 28). This design not only optimizes the spatial layout, but also significantly improves the transmission efficiency of the infrared light signal and the overall performance of the device. Since the infrared light emitter 30 is located at the center of the lens group, it can transmit signals to the first light guide segment 27 and the second light guide segment 28 at a relatively balanced angle. This balanced transmission method helps to reduce the attenuation of the signal inside the light guide structure 20, improve the strength and stability of the signal, and thus ensure the reliability and efficiency of the infrared light transmission process.

[0064] See also Figure 5 , Figure 5A second structural schematic diagram of a near-eye display device provided in an embodiment of the present application. The near-eye display device 100 also includes a frame 50 and a nose pad 60, the frame 50 is connected to the lens assembly 10, and the nose pad 60 is connected to the frame 50. The nose pad 60 is a key component connecting the frame 50 and the user's nose bridge, and its design is directly related to the stability and comfort of wearing. The nose pad 60 is generally made of a soft material with memory function, such as silicone or highly elastic plastic. The infrared light receiver 40 is arranged on the nose pad 60, which is responsible for capturing the infrared light reflected by the user's eyeball, and then analyzing and calculating the user's gaze point position. Placing the infrared light receiver 40 on the nose pad 60 can maintain a relatively fixed spatial relationship with the user's human eye 200 more directly and stably than other positions. This design greatly reduces the relative position change between the receiver and the eye caused by slight movement of the head or difference in facial morphology, thereby significantly improving the accuracy and reliability of calculating the gaze point. This not only optimizes the device's interactive experience, making eye tracking control more responsive and accurate, but also provides a solid foundation for implementing more complex interactive logic, such as automatic page turning and cursor movement under gaze tracking.

[0065] The number of the infrared light receivers 40 can be two, which are denoted as a first infrared light receiver 40 and a second infrared light receiver 40. The first infrared light receiver 40 and the second infrared light receiver 40 are mirror-symmetrical in layout. Such a design not only ensures the balance of reception, but also facilitates the separate tracking of both eyes.

[0066] The first infrared light receiver 40 is specifically used to receive the infrared reflected light reflected from the left eye, which is called the first infrared reflected light. Correspondingly, the second infrared receiver is responsible for receiving the infrared reflected light reflected from the right eye, which is the second infrared reflected light. These two beams of reflected light carry key information about the movement of the eyeball. By capturing and analyzing them, the movement trajectory and position information of the user's eyeball can be obtained.

[0067] With the help of advanced image processing algorithms, the gaze point of the eyeball can be calculated based on these reflected lights, and the movement of the eyeball can be tracked in real time. It is worth mentioning that there is usually an angle between the first infrared reflected light and the second infrared reflected light. Due to the reversibility principle of light - that is, the path of light is reversible during propagation - the reflected light will return along the same path (or approximately the same path) as the incident light. Therefore, when the two beams of reflected light are extended, they will theoretically intersect at one point, and the position of this point is the user's current gaze point.

[0068] In actual applications, this technology can accurately determine the icon the user is looking at, i.e. the target icon, by comparing the gaze point with the icon position in the virtual screen. Once the target icon is determined, the system can wait for the user's trigger command, thus achieving a more natural and intuitive human-computer interaction.

[0069] In order to improve the efficiency and accuracy of capturing infrared reflected light, the infrared receiver usually uses a highly sensitive infrared camera or infrared camera. These devices can accurately capture the reflected infrared reflected light and form a clear infrared image of the eyeball. By further processing and analyzing these images, key data such as the eyeball's movement trajectory, position information, and gaze point can be obtained.

[0070] In some embodiments, the near-eye display device 100 further includes a processor, which is connected to the infrared light transmitter 30 and the infrared light receiver 40 respectively, and is used to perform the following steps: obtaining a virtual display screen, the virtual display screen has multiple icons, each icon corresponds to an application; calculating the gaze point of the human eye 200 according to the infrared reflected light; and determining a target icon from the multiple icons according to the gaze point. S4, obtaining a trigger instruction; triggering the application corresponding to the target icon according to the trigger instruction.

[0071] The near-eye display device 100 may also include a light engine, which is responsible for emitting light and forming an image. The light engine in this embodiment may include various types such as LED miniaturization and matrix technology display screen (Micro LED display screen), OLED miniaturization and matrix technology display screen (Micro OLED display screen), laser beam scanning display screen (LBS display screen) and digital mirror display screen (DMD display screen). These advanced display screen technologies ensure the high resolution and color accuracy of virtual images, providing users with a more realistic and vivid visual experience. The light engine is also called a micro-projection optical machine, optical machine, etc. If these terms are not specifically specified, they can be used interchangeably in this application. The display screen size of the light engine can be 0.4 to 0.7".

[0072] The near-eye display device 100 may also include components such as sensors, circuits, output devices, and batteries or solar cells. The sensors can monitor the user's head movements or eye movements in real time, thereby adjusting the position and angle of the virtual image to further enhance the user experience. The circuit is responsible for connecting the various components to achieve information transmission and processing. The output device is used to display virtual images or issue sound prompts to the user. Batteries or solar cells provide continuous power support for the device to ensure long-term operation of the device.

[0073] In addition, the near-eye display device 100 can also adopt designs such as wraparound lenses, adaptable glasses or auxiliary frames to meet the wearing needs of different users. At the same time, the device can also be equipped with a strap for fixing to ensure that the user can maintain stability and comfort during use.

[0074] The present application also provides a control method, which is applied to the near-eye display device 100 in the above embodiment. The near-eye display device 100 is a wearable device, which is usually equipped with transparent or semi-transparent lenses, and is used to project virtual display images directly onto the user's retina, thereby providing an immersive visual experience. It is often used in augmented reality (AR), virtual reality (VR) or mixed reality (MR) applications. The wearable device has an eye tracking function. Please refer to Figure 6 , Figure 6 The control method provided in the embodiment of the present application is a flow chart of the control method. The control method includes the following steps.

[0075] S1. Obtain a virtual display screen, where the virtual display screen has multiple icons, and each icon corresponds to an application.

[0076] Icons are small pictures or symbols in a virtual display screen, and applications can be used to perform specific tasks, files, functions or operations.

[0077] It is understandable that in the control method provided in the embodiment of the present application, a virtual display screen is first obtained, which includes multiple icons, each of which is associated with a specific application. This design allows the user to browse and select applications directly through the visual interface without leaving the virtual environment.

[0078] The virtual display screen can be displayed through the high resolution and perspective display technology of the near-eye display device 100, ensuring the clarity and readability of the icon. At the same time, the size, shape and color of the icon can also be personalized according to the user's preferences, improving the user experience.

[0079] S2. Calculate the gaze point of the human eye 200 according to the infrared reflected light.

[0080] The gaze point refers to the position where the user's eyeball is looking at a specific moment, which is usually obtained by the infrared reflection principle in the above embodiment. In the above embodiment, only a small number of infrared light emitters 30 are used and they are arranged correspondingly in the light inlet area 21 of the light guide structure 20. The infrared light emitter 30 is capable of emitting infrared light. After the infrared light enters the light guide structure 20 through the light inlet area 21, it is efficiently guided to the first light exit area 22 and the second light exit area 23, and is emitted from these two areas. The human eye 200 reflects these emitted infrared lights to form infrared reflected lights, which are then received by the infrared light receiver 40 arranged between the left lens 11 and the right lens 12.

[0081] In order to improve the efficiency and accuracy of capturing infrared reflected light, the infrared receiver usually uses a highly sensitive infrared camera or infrared camera. The infrared camera or infrared camera can accurately capture the reflected infrared reflected light and form a clear infrared image of the eyeball. By further processing and analyzing these images, key data such as the eyeball's movement trajectory, position information, and gaze point can be obtained.

[0082] In some cases, the near-eye display device 100 further includes a storage module, and after the infrared receiver forms a clear infrared image of the eyeball, the infrared image of the eyeball is stored in the storage module. The gaze point of the human eye 200 is subsequently calculated based on the infrared image of the eyeball.

[0083] There is usually an angle between the first infrared reflected light and the second infrared reflected light. Due to the reversibility of light - that is, the path of light is reversible during propagation - the reflected light will return along the same path (or approximately the same path) as the incident light. Therefore, when the two reflected lights are extended, they will theoretically intersect at one point, and the position of this point is the user's current gaze point.

[0084] S3. Determine a target icon among multiple icons according to the gaze point.

[0085] By comparing the gaze point with the icon position in the virtual screen, the icon the user is looking at, i.e. the target icon, can be accurately determined, and the user can be waited for to issue a trigger command. This process realizes the intuitive expression of user intention, i.e. the user only needs to look at the target icon and issue a simple trigger command (such as blinking, head movement, etc.) to start the corresponding application.

[0086] S4. Get trigger instruction.

[0087] Once the target icon is determined, the user's trigger instruction can be waited for. The trigger instruction can be issued by the user through a specific gesture, voice command or other means.

[0088] For example, the trigger instruction may be blinking, prolonged gazing, the user's head movement, or capturing the user's gestures through the second camera to obtain confirmation or denial information and complete the interaction with the near-eye display device 100.

[0089] S5. Trigger the application corresponding to the target icon according to the trigger instruction.

[0090] The trigger instruction can be a specific gesture made by the user, a voice command issued, or other innovative methods. Specifically, the trigger instruction can be a blink of the user, a long gaze on the icon, or a subtle movement of the user's head. Furthermore, the second camera can also be used to capture and analyze the user's gestures to obtain the user's confirmation or denial information. Through this simple and convenient trigger instruction, the user can interact with the near-eye display device 100 more naturally and smoothly, thereby greatly improving the convenience and comfort of the user experience.

[0091] In some embodiments, after determining the target icon among multiple icons according to the gaze point in step S3, the control method further includes: controlling the target icon to be in a highlighted state, a bold state, or a protruding state.

[0092] The highlight state refers to a visual expression that increases the brightness of an icon to make it more conspicuous in the virtual display screen.

[0093] The bold state refers to the visual expression of changing the thickness of the icon lines, which is usually used to enhance the icon's recognition and visual effect.

[0094] The protruding state refers to changing the depth or height of the icon in the virtual display screen so that it looks like it protrudes from the background, thereby attracting the user's attention.

[0095] In the embodiment of the present application, after determining the target icon among multiple icons according to the gaze point, the control method is not limited to simply identifying the user's intention, but further enhances the user experience through visual feedback. Specifically, the control method controls the target icon to be highlighted, bolded or protruded as an immediate response to the user's gaze behavior.

[0096] By highlighting, bolding or protruding the target icon, users can more quickly identify the focus of the current operation and reduce the trouble of searching back and forth between multiple icons. This intuitive visual cue reduces the user's cognitive burden and makes the operation process smoother. In a complex virtual display screen, there may be multiple similar or adjacent icons. By changing the visual state of the target icon, users can more accurately judge their operation object and avoid the possibility of accidental touch or misoperation.

[0097] In some scenarios, for example, users wear AR glasses at home and control smart home devices through the virtual display screen in the glasses. AR glasses present a virtual control interface containing multiple icons to users, which correspond to various smart home devices in the home, such as lights, air conditioners, curtains, etc. The glasses capture the infrared reflected light of the user's eyeballs through the built-in infrared receiver and accurately calculate the user's gaze point. According to the position of the gaze point, the glasses determine the target icon that the user wants to operate in the virtual display screen, such as the air conditioner icon. The user gives a trigger instruction through voice commands or eye movements (such as blinking). After receiving the trigger instruction, the glasses automatically open the corresponding smart home application and perform the operation specified by the user, such as adjusting the air conditioner temperature.

[0098] In other scenarios, for example, when users wear AR glasses at tourist attractions, they can obtain navigation information and introductions to scenic spots through the virtual display screen in the glasses. AR glasses present users with a virtual navigation interface containing multiple icons, which correspond to important locations in the scenic spot, such as entrances, exits, restrooms, shops, etc. The glasses capture the infrared reflected light of the user's eyes through infrared sensors and determine the user's gaze point in real time. According to the location of the gaze point, the glasses determine the target icon that the user wants to know in the virtual display screen, such as the restroom icon. At this time, the target icon is highlighted, indicating to the user that it is selected. The user gives a trigger command through simple eye movements (such as staring for a while) or head movements. After receiving the trigger command, the glasses automatically open the corresponding navigation application to provide users with detailed route guidance and introductions to scenic spots.

[0099] An embodiment of the present application provides an electronic device, which includes the following modules.

[0100] The first acquisition module is used to acquire a virtual display screen. The virtual display screen has a plurality of icons, and each icon corresponds to an application.

[0101] A calculation module is used to calculate the gaze point of the human eye 200 according to the infrared reflected light.

[0102] The determination module is used to determine a target icon among multiple icons according to a gaze point.

[0103] The second acquisition module is used to acquire the trigger instruction.

[0104] The trigger module is used to trigger the application corresponding to the target icon according to the trigger instruction.

[0105] In some embodiments, the electronic device further includes a control module, and the control module is used to control the target icon to be in a highlighted state, a bold state, or a protruding state.

[0106] See also Figure 7 , Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0107] The electronic device 300 may include a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, a power supply 303, an input unit 304, and other components. Those skilled in the art will appreciate that Figure 7 The structure of the electronic device 300 shown in the figure does not constitute a limitation on the electronic device 300, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0108] The processor 301 is the control center of the electronic device 300. It uses various interfaces and lines to connect various parts of the entire electronic device 300. By running or executing software programs and / or modules stored in the memory 302 and calling data stored in the memory 302, it executes various functions of the electronic device 300 and processes data, thereby monitoring the electronic device 300 as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor 301 and a modem processor 301, wherein the application processor 301 mainly processes the operation processor 301, the user interface and the application program, and the modem processor 301 mainly processes wireless communication. It can be understood that the above-mentioned modem processor 301 may not be integrated into the processor 301.

[0109] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and data processing by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device 300, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage 302 device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0110] The electronic device 300 also includes a power supply 303 for supplying power to various components. Preferably, the power supply 303 can be connected to the system logic through the power supply 303 management system, so as to manage charging, discharging, and power consumption management through the power supply 303 management system. The power supply 303 may also include one or more DC or AC power supplies 303, recharging systems, power supply 303 device debugging circuits, power supply 303 converters or inverters, power supply 303 status indicators, and other arbitrary components.

[0111] The electronic device 300 may further include an input unit 304, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0112] Although not shown, the electronic device 300 may also include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 301 in the electronic device 300 will load the executable files corresponding to the processes of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application programs stored in the memory 302, thereby implementing the steps in any one of the control methods provided in the embodiments of the present application.

[0113] Acquire a virtual display screen, wherein the virtual display screen has multiple icons, each of which corresponds to an application; calculate the gaze point of the human eye 200 according to the infrared reflected light; determine a target icon from the multiple icons according to the gaze point; acquire a trigger instruction; and trigger the application corresponding to the target icon according to the trigger instruction.

[0114] In the above, by combining the near-eye display device 100 and the control method, efficient and intuitive control of the application in the virtual display screen is achieved. This method not only improves the user's operating efficiency and convenience, but also brings a more natural and immersive interactive experience to the user. In addition, the near-eye display device 100 in the embodiment of the present application can significantly reduce the number and layout complexity of the infrared light emitters 30, providing strong support for the further miniaturization and lightweighting of the near-eye display device 100.

[0115] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0116] A person skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by the processor 301.

[0117] To this end, the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above control method are implemented. The specific implementation of each of the above operations can be referred to the previous embodiments, and will not be repeated here.

[0118] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0119] Since the instructions stored in the computer-readable storage medium can execute the steps in any control method provided in the present application, the beneficial effects that can be achieved by any control method provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0120] The near-eye display device 100, control method, electronic device and storage medium provided in the embodiment of the present application have achieved significant improvements and innovations in the field of eye tracking technology. Specifically, the near-eye display device 100 integrates a lens assembly 10, a light guide structure 20, an infrared light emitter 30 and an infrared light receiver 40, and realizes an efficient and compact eye tracking function. The lens assembly 10 in the device is composed of a left lens 11 and a right lens 12, and the light guide structure 20 is arranged on the periphery of the lens assembly 10. The light guide structure 20 has a light inlet area 21, a first light outlet area 22 and a second light outlet area 23, which provides a precise path for the transmission and distribution of infrared light. Among them, the light inlet area 21 is located between the left lens 11 and the right lens 12, and the first light outlet area 22 and the second light outlet area 23 are respectively arranged on the peripheral sides of the left lens 11 and the right lens 12. The embodiment of the present application only uses a small number of infrared light emitters 30, and they are arranged correspondingly in the light inlet area 21 of the light guide structure 20. The infrared light emitter 30 is capable of emitting infrared light. After the infrared light enters the light guide structure 20 through the light inlet area 21, it is efficiently guided to the first light outlet area 22 and the second light outlet area 23, and is emitted from these two areas. The human eye 200 reflects these emitted infrared lights to form infrared reflected light, which is then received by the infrared light receiver 40 disposed between the left lens 11 and the right lens 12. This design not only realizes the efficient transmission and distribution of infrared light, but also greatly satisfies the requirements of the near-eye display device 100 for aesthetic appearance and lightweight while achieving the purpose of eye tracking. Compared with the traditional solution, the embodiment of the present application significantly reduces the number and layout complexity of the infrared light emitters 30, providing strong support for the further miniaturization and lightweighting of the near-eye display device 100.

[0121] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0122] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features.

[0123] The near-eye display device, control method, electronic device and storage medium provided in the embodiments of the present application are introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the present application. At the same time, for those skilled in the art, according to the ideas of the present application, there will be changes in the specific implementation methods and scope of application. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A near-eye display device, characterized in that: include: A lens assembly, the lens assembly comprising a left lens and a right lens; A light guide structure, the light guide structure is arranged at the periphery of the lens assembly, and is provided with a light inlet area, a first light outlet area and a second light outlet area, the first light outlet area is correspondingly arranged at the periphery of the left lens, and the second light outlet area is correspondingly arranged at the periphery of the right lens; an infrared light emitter, the infrared light emitter being arranged correspondingly in the light inlet area, the infrared light emitter being configured to emit infrared light, the infrared light being able to enter the light guide structure from the light inlet area, and then being emitted from the first light exit area and the second light exit area, and the infrared light emitted from the first light exit area and the second light exit area being able to be reflected by the human eye into infrared reflected light; An infrared light receiver, the infrared light receiver is arranged corresponding to the infrared light transmitter, and the infrared light receiver is used to receive the infrared reflected light.

2. The near-eye display device according to claim 1, characterized in that: The light-guiding structure has a first surface and a second surface, wherein the first surface and the second surface extend along the length of the light-guiding structure and are arranged opposite to each other, wherein the first surface is configured to face the human eye, and a plurality of grid points are arranged on the second surface, wherein the grid points are configured to reflect infrared light to form the first light-emitting area and the second light-emitting area on the first surface.

3. The near-eye display device according to claim 2, characterized in that: The dots are convex dots; or, the shapes of the dots are cones or frustums.

4. The near-eye display device according to claim 2, characterized in that: The light guide structure further has a third surface, which is disposed at an end of the light guide structure, and is connected to the first surface and the second surface respectively, and forms the light intake area.

5. The near-eye display device according to claim 4, characterized in that: The light guide structure includes a first light guide segment and a second light guide segment, the first light guide segment is arranged on the peripheral side of the left lens, and a plurality of the first light exit areas are arranged on the first light guide segment; the second light guide segment is arranged on the peripheral side of the right lens, and a plurality of the second light exit areas are arranged on the second light guide segment; the infrared light emitter is arranged between the left lens and the right lens, and the third surface of the first light guide segment and the third surface of the second light guide segment face the infrared light emitter.

6. The near-eye display device according to any one of claims 1 to 5, characterized in that: It also includes a spectacles frame and a nose pad, wherein the nose pad is connected to the spectacles frame, the spectacles frame is connected to the lens assembly, and the infrared light receiver is arranged on the nose pad.

7. A control method, characterized in that: Applied to the near-eye display device according to any one of claims 1 to 6, the control method comprises: Acquire a virtual display screen, wherein the virtual display screen has a plurality of icons, each of which corresponds to an application program; Calculating the gaze point of the human eye according to the infrared reflected light; Determine a target icon among the multiple icons according to the gaze point; Get the trigger instruction; The application corresponding to the target icon is triggered according to the trigger instruction.

8. The control method according to claim 7, characterized in that: After determining the target icon among the multiple icons according to the gaze point, the method further includes: controlling the target icon to be in a highlighted state, a bold state or a protruding state.

9. An electronic device, characterized in that: include: A first acquisition module, the first acquisition module is used to acquire a virtual display screen, the virtual display screen has a plurality of icons, each of the icons corresponds to an application; A calculation module, the calculation module is used to calculate the gaze point of the human eye according to the infrared reflected light; a determination module, the determination module being used to determine a target icon among the multiple icons according to the gaze point; A second acquisition module, the second acquisition module is used to acquire a trigger instruction; A trigger module is used to trigger the application corresponding to the target icon according to the trigger instruction.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps in the control method according to claim 7 or 8 are implemented.

Images