Augmented reality device control method, augmented reality device, and storage medium
By acquiring the eye movement trajectory and gaze parameters of the wearer in the augmented reality device, multi-level dynamic power consumption switching based on gaze intention is achieved, solving the problem of insufficient battery life under all-day wear and improving the device's battery life and the accuracy of state switching.
Patent Information
- Application Number
- CN202611293628.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-24
- Publication Date
- 2026-11-10
AI Technical Summary
Existing augmented reality devices have insufficient battery life in all-day wear scenarios, and existing interaction methods consume a lot of power and are difficult to accurately identify the wearer's gaze intentions, resulting in wasted power consumption.
By acquiring the wearer's eye movement trajectory and gaze parameters when the control system is in deep sleep, switching to a pre-wake state according to preset conditions, and activating the display link when the gaze intent is determined to be the content gaze intent, multi-level dynamic power consumption switching is achieved.
It improves the battery life of augmented reality devices, reduces ineffective power consumption, and enhances the accuracy of state switching and the activation accuracy of the display link.
Smart Images

Figure CN123067718A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of device control technology, and in particular to an augmented reality device control method, an augmented reality device, and a storage medium. Background Technology
[0002] With the rapid popularization of augmented reality (AR) technology, all-day long battery life has become a core bottleneck for consumer-grade AR devices (especially everyday wearable AR glasses) to enter the mass market. In scenarios where AR devices are worn around the clock, users spend most of their time observing the physical world through the device's lenses, without needing the virtual display to provide information in real time. Therefore, reducing the power consumption of AR devices and improving their battery life is an urgent problem to be solved. Summary of the Invention
[0003] This invention provides an augmented reality device control method, an augmented reality device, and a storage medium, aiming to reduce the power consumption of the augmented reality device and improve its battery life.
[0004] In a first aspect, embodiments of the present invention provide an augmented reality device control method, the method comprising: The eye movement trajectory of the user wearing the augmented reality device is acquired when the control system of the augmented reality device is in a deep sleep state. If the eye movement trajectory meets the preset conditions, the control system is controlled to switch from the deep sleep state to the pre-wake state, and the gaze parameters of the wearer are obtained; If the user's gaze intent is determined to be content gaze intent based on the gaze parameters, the control system is controlled to switch from a pre-wake state to a full-function interactive state to activate the display link of the control system.
[0005] In one embodiment, after the step of controlling the control system to switch from the deep sleep state to the pre-wake state and obtaining the user's gaze parameters if the eye movement trajectory meets a preset condition, the method further includes: If the gaze intent of the user is determined to be a real viewing intent based on the gaze parameters, then the control system is controlled to switch from a pre-wake state to a deep sleep state.
[0006] In one embodiment, the line-of-sight parameter includes the spatial line-of-sight vector, and the method further includes: If the spatial gaze vector does not intersect with the area where the augmented reality device's display interface is located, or if the spatial gaze vector intersects with the area where the display interface is located, but the gaze depth corresponding to the spatial gaze vector is greater than the display depth of the display interface, then the gaze intention is determined to be a real viewing intention.
[0007] In one embodiment, the method further includes: If, within a first preset time period, the user's eyeballs produce two consecutive upward movement trajectories in the vertical direction, then the eye movement trajectory is determined to meet the preset conditions. and / or If the time difference between two blinks by the wearer is within a preset time range, then the eye movement trajectory is determined to meet the preset conditions.
[0008] In one embodiment, the line-of-sight parameter includes the spatial line-of-sight vector, and the method further includes: If the spatial gaze vector is within the area of the augmented reality device's display interface, and the duration of the gaze point corresponding to the spatial gaze vector on the display interface is greater than a second preset duration, then the gaze intent is determined to be a content gaze intent.
[0009] In one embodiment, after the step of controlling the control system to switch from a pre-wake state to a full-function interactive state to activate the display link of the control system if the user's gaze intent is determined to be a content gaze intent based on the gaze parameters, the method further includes: When it is determined that the user is in a closed-eye state, the frame image displayed on the current display interface is frozen, and / or the drive current of the display link is reduced.
[0010] In one embodiment, after the step of controlling the control system to switch from a pre-wake state to a full-function interactive state to activate the display link of the control system if the user's gaze intent is determined to be a content gaze intent based on the gaze parameters, the method further includes: If the duration of the user's closed eyes is greater than or equal to a first preset duration of closed eyes, but less than a second preset duration of closed eyes, then the power supply to the display link is disconnected, and / or the operating frequency of the application processor of the control system is reduced.
[0011] In one embodiment, after the step of controlling the control system to switch from a pre-wake state to a full-function interactive state to activate the display link of the control system if the user's gaze intent is determined to be a content gaze intent based on the gaze parameters, the method further includes: If the duration of the user's closed eyes is greater than or equal to the second preset duration, the application processor stores the context information in the non-volatile memory of the augmented reality device and disconnects the power supply to the application processor.
[0012] In a second aspect, an augmented reality device is provided by an embodiment of the present invention. The augmented reality device includes a control system, a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for implementing communication between the processor and the memory. When the computer program is executed by the processor, it implements the steps of the smart glasses control method as described in the first aspect.
[0013] Thirdly, embodiments of the present invention provide a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, which can be executed by one or more processors to implement the augmented reality device control method as described in the first aspect.
[0014] This invention provides an augmented reality (AR) device control method, an AR device, and a storage medium. The invention acquires the eye-tracking trajectory of the AR device wearer when the AR device's control system is in a deep sleep state. If the eye-tracking trajectory meets preset conditions, the control system switches from deep sleep to a pre-wake state and acquires the wearer's gaze parameters. If the gaze parameters determine that the wearer's gaze intent is content-focused, the control system switches from the pre-wake state to a full-function interactive state to activate the display link of the control system. This uses the wearer's actual content attention view as the activation condition for the display link. The control system only switches to the full-function interactive state when it is determined that the wearer has a content-focused gaze intent, thereby reducing the power consumption of the AR device and improving its battery life. Furthermore, this embodiment does not directly switch the control system from deep sleep to the full-function interactive state. Instead, it switches from deep sleep to a pre-wake state when the wearer's eye-tracking trajectory meets preset conditions, and then switches to normal operation when the wearer's gaze intent is determined to be content-focused. This hierarchical switching improves the accuracy of AR device state switching and the accuracy of display link activation. Attached Figure Description
[0015] Figure 1 This is a schematic flowchart of an augmented reality device control method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the spatial line-of-sight vector within the display interface of an augmented reality device in an embodiment of the present invention; Figure 3 This is a schematic diagram in an embodiment of the present invention where the spatial line of sight vector intersects with the area where the display interface is located, but the gaze depth corresponding to the spatial line of sight vector is greater than the display depth of the display interface; Figure 4 This is another flowchart illustrating an augmented reality device control method provided in an embodiment of the present invention; Figure 5 This is a schematic block diagram of the structure of an augmented reality device provided in an embodiment of the present invention. Detailed Implementation
[0016] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0017] The augmented reality device control method provided in this invention can be applied to augmented reality devices. Augmented reality devices may include AR (Augmented Reality) glasses, AR helmets, etc.
[0018] With the rapid popularization of augmented reality (AR) technology, lightweight design and all-day long battery life have become the core bottlenecks for AR devices (especially everyday wearable AR glasses) to enter the mass market. In order to achieve lightweight design, the battery capacity of AR devices is greatly limited, which requires AR devices to have extreme power consumption control capabilities during daily operation.
[0019] Augmented reality devices in related technologies typically employ interaction methods such as physical buttons, touch controls, or voice activation. These interaction methods either require users to perform additional physical movements, resulting in an unnatural interaction, or require the augmented reality device's microphone array or capacitive sensors to maintain high-frequency, full-function operation, leading to significant static standby power consumption. Some improved technologies attempt to introduce infrared proximity sensors to identify the wearing status of the augmented reality device, but these often only achieve a crude power management approach of "powering on when worn and powering off when removed."
[0020] In real-world all-day wear scenarios (i.e., all-day long battery life scenarios), users spend most of their time observing the physical world through the translucent lenses of the augmented reality device (AR), without needing information from the virtual display interface. During this time, the AR's display circuitry, application processor (AP), and complex anti-distortion image rendering algorithms continue to consume power, resulting in a significant waste of battery power. Furthermore, users also engage in normal physiological behaviors such as blinking, briefly closing their eyes to rest, or looking away during wear.
[0021] Therefore, how to accurately identify the wearer's true "gazing intention," filter out the wearer's unconscious daily eye movements without interfering with the wearer's conscious interactions, and achieve rapid, millisecond-level, multi-level dynamic power consumption switching when fine physiological behaviors such as blinking, closing the eyes, and looking away occur, is a technical problem that urgently needs to be solved in this field.
[0022] To address the aforementioned problems, embodiments of the present invention provide an augmented reality device control method, an augmented reality device, and a storage medium. For ease of explanation and understanding, the following detailed description, in conjunction with the accompanying drawings, focuses on an augmented reality device control method according to an embodiment of the present invention, while other aspects are extended and supplemented based on this description.
[0023] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0024] Please see Figure 1 , Figure 1 This is a flowchart illustrating an augmented reality device control method provided in an embodiment of the present invention.
[0025] like Figure 1 As shown, the augmented reality device control method includes steps S110, S120 and S130.
[0026] Step S110: Acquire the eye movement trajectory of the user wearing the augmented reality device while the control system of the augmented reality device is in a deep sleep state.
[0027] Augmented reality devices include a control system and an eye-tracking sensor. The control system includes a display link and an application processor (AP). In this embodiment, the display link is an optical display link, which can be a freeform surface optical link, an optical waveguide optical link, etc. If the display link is an optical waveguide optical link, it includes an optomechanical system and an optical waveguide. The optomechanical system includes a light source, an optomechanical display chip, and a projection lens assembly. The light emitted from the light source is modulated by the optomechanical display chip to form an image beam, which is then collimated and shaped by the projection lens assembly before being output. The light source can be an LED (Light-Emitting Diode) or a laser, etc. The optomechanical display chip can be an LCoS (Liquid Crystal on Silicon), a Micro-OLED (Micro Organic Light-Emitting Diode), or a DMD (Digital Micromirror Device), etc. The optical waveguide can be a geometrical waveguide or a diffractive waveguide. In one embodiment, the optical waveguide includes an insertion region, a transmission region, and an exit region. The image beam output from the optomechanical system enters the interior of the optical waveguide through the insertion region, propagates forward through total internal reflection in the transmission region, and finally extends and exits towards the user's eye through the exit region. Simultaneously, ambient light can directly enter the user's eye through the optical waveguide, thereby realizing a virtual image, i.e., the visual effect of superimposing the image displayed on the augmented reality device's display interface with the real background. The eye-tracking sensor is a device used to acquire images of the user's eyes.
[0028] When the control system of the augmented reality device is in a deep sleep state, the display link and application processor (AP) included in the control system are in a power-off state or suspended state, but the eye-tracking sensor is in an operating state. For example, the eye-tracking sensor is in an operating state at a preset sampling frequency, such as 30Hz, 40Hz, 45Hz, 53Hz or 60Hz.
[0029] Augmented reality devices also include an MCU (microcontroller unit). When the control system of the augmented reality device is in a deep sleep state, the device acquires eye images of the wearer through the eye-tracking sensor via the MCU and obtains the wearer's eye movement trajectory based on these images. The process of obtaining the eye movement trajectory based on the eye images can be found in relevant technologies and will not be described in detail here. The eye movement trajectory can be composed of the coordinates of the eyeball's movement relative to the center of the pupil.
[0030] Step S120: If the eye movement trajectory meets the preset conditions, control the control system to switch from the deep sleep state to the pre-wake state and obtain the gaze parameters of the wearer.
[0031] After the augmented reality device (AR) acquires the user's eye movement trajectory, it determines whether the acquired trajectory meets preset conditions. If the eye movement trajectory meets the preset conditions, the AR device controls its control system to switch from a deep sleep state to a pre-wake state. In the pre-wake state, the display link and application processor remain powered down or suspended, but the sampling frequency of the eye movement sensor in the pre-wake state is higher than that in the deep sleep state. It can be understood that when the control system switches from deep sleep to the pre-wake state, the AR device's MCU can increase the sampling frequency of the eye movement sensor and acquire the user's gaze parameters based on the increased sampling frequency. For example, the sampling frequency of the eye movement sensor in the pre-wake state can be determined as the sampling frequency when the eye movement sensor is in normal operating mode, while the sampling frequency in the deep sleep state can be determined as the sampling frequency when the eye movement sensor is in low-power mode. The sampling frequency in normal operating mode can be 50Hz, 56Hz, 100Hz, 112Hz, or 120Hz, etc. The gaze parameter is a parameter used to characterize the direction of the wearer's gaze and / or the position of the gaze point. For example, the gaze parameter can be a spatial gaze vector. The spatial gaze vector is a vector in a three-dimensional spatial coordinate system used to quantify the direction of the wearer's gaze and the position of the gaze. The starting point of this vector is usually the wearer's viewpoint (e.g., the center of the eyeball or the midpoint between the two eyes), and its direction is from the viewpoint to the wearer's point of interest (i.e., the gaze point) or gaze target in real or virtual space. The gaze position is the gaze point.
[0032] It is understandable that when the user's eye movement trajectory meets the preset conditions, the control system of the augmented reality device switches from a deep sleep state to a pre-wake state. This is the first identification of the power consumption control of the control system in this application embodiment. By identifying and shielding the user's unconscious eye movements or ordinary daydreaming in daily non-interactive states (the control system only switches states when the eye movement trajectory meets the preset conditions), some environmental noise and accidental touch behavior of the user are filtered out, thereby improving the accuracy of the control system's state switching.
[0033] In one embodiment, the method further includes: If, within a first preset time period, the user's eyeballs produce two consecutive upward movement trajectories in the vertical direction, then the eye movement trajectory is determined to meet the preset conditions. and / or If the time difference between two blinks by the wearer is within a preset time range, then the eye movement trajectory is determined to meet the preset conditions.
[0034] In one embodiment, the augmented reality device may use at least one of the following three methods to determine whether the wearer's eye movement trajectory meets preset conditions.
[0035] Method 1: When the control system is in deep sleep mode, if the user's eyeballs continuously move upwards twice in the vertical direction within a first preset duration, the eye movement trajectory is determined to meet the preset condition. The first preset duration can be set according to specific needs, such as 400ms, 500ms, or 550ms. In one embodiment, to improve the accuracy of determining whether the eye movement trajectory meets the preset condition, the speed and / or distance of the user's eyeballs moving upwards in the vertical direction can be set. If, within a 500ms sliding time window, the user's eyeballs are detected to continuously move upwards twice in the vertical direction with a speed (sweeping) greater than the preset speed, the eye movement trajectory is determined to meet the preset condition. This embodiment does not limit the magnitude of the preset speed; the magnitude of the preset speed can be set according to specific needs. If, within the first preset duration, the user's eyeballs do not continuously move upwards twice in the vertical direction, the eye movement trajectory is determined not to meet the preset condition. It is understandable that if the user's eyeball moves upward twice or more in the vertical direction, it is also considered that the user's eyeball moves upward twice in the vertical direction, and the eye movement trajectory is determined to meet the preset conditions.
[0036] Method 2: When the control system is in deep sleep mode, if at least two blinks are detected from the user, the blink time point of each blink is obtained. The time difference between two consecutive blinks is calculated, and it is determined whether this time difference is within a preset time range. This preset time range can be set according to specific needs, such as within 200ms, 250ms, or 300ms. If the time difference is within the preset time range, the eye movement trajectory is determined to meet the preset conditions; if the time difference is not within the preset time range, the eye movement trajectory is determined to not meet the preset conditions. In other words, when the control system is in deep sleep mode, if no blinks are detected from the user, or only one blink is detected, the eye movement trajectory is determined to not meet the preset conditions.
[0037] Method 3: If, within the first preset time period, the user's eyeballs continuously move upwards twice in the vertical direction, and the time difference between the two blinks is within the preset time range, then the eye movement trajectory is determined to meet the preset conditions. If, within the first preset time period, the user's eyeballs do not continuously move upwards twice in the vertical direction, and / or the time difference between the two blinks is not within the preset time range, then the eye movement trajectory is determined to not meet the preset conditions. The specific implementation process can be found in Methods 1 and 2, and will not be elaborated upon here.
[0038] Step S130: If the user's gaze intent is determined to be content gaze intent based on the gaze parameters, then the control system is controlled to switch from the pre-wake state to the full-function interactive state to activate the display link of the control system.
[0039] After the augmented reality device acquires the gaze parameters, it determines the user's gaze intent based on these parameters. In this embodiment, the gaze intent includes content gaze intent and real-world viewing intent. Content gaze intent indicates that the user's gaze is fixed on the augmented reality device's display interface, meaning the user is currently viewing virtual content displayed by the augmented reality device, such as notification messages or images displayed on the interface. Real-world viewing intent indicates that the user's gaze penetrates the display interface of the augmented reality device, meaning the user is currently looking at real-world objects through the augmented reality device, such as pedestrians or vehicles on the road.
[0040] If the augmented reality device (AR) determines that the user's gaze intent is content-focused based on gaze parameters, the AR device controls its control system to switch from a pre-wake state to a full-function interactive state. After the control system switches from the pre-wake state to the full-function interactive state, the AR device's MCU triggers a hardware interrupt signal via the high-impedance power management bus, raising the application processor's power rail (i.e., increasing the application processor's voltage) to activate the display link. It is understood that after activating the display link, the AR device lights up its display interface and displays relevant information for the user to view; this embodiment does not limit the information displayed on the interface. It is also understood that when the control system is in the full-function interactive state, the AR device is in normal operation, and all functions can be started and run normally.
[0041] Understandably, the control system switches from the pre-wake state to the full-function interactive state only when the gaze parameter determines that the wearer's gaze intent is content gaze intent. In other words, the gaze parameter performs a secondary isolation between "unconscious glance" (non-content gaze intent) and "conscious gaze" (content gaze intent), reducing the occurrence of frequent oscillations and restarts (full-function interactive state) or sleep (deep sleep state) of the control system caused by the wearer's gaze quickly and unintentionally sweeping across the display interface.
[0042] This embodiment acquires the eye movement trajectory of the user wearing the augmented reality device (ARD) while the ARD's control system is in a deep sleep state. If the eye movement trajectory meets preset conditions, the control system switches from deep sleep to a pre-wake state and acquires the user's gaze parameters. If the gaze parameters determine that the user's gaze intent is content-focused, the control system switches from the pre-wake state to a full-function interactive state to activate the display link of the control system. This uses the user's actual content attention view as the activation condition for the display link. The control system only switches to the full-function interactive state when it is determined that the user has a content-focused gaze intent, thereby reducing the power consumption of the ARD and improving its battery life. Furthermore, this embodiment does not directly switch the control system from deep sleep to the full-function interactive state. Instead, it switches from deep sleep to a pre-wake state when the user's eye movement trajectory meets preset conditions, and then switches to normal operation when the user's gaze intent is determined to be content-focused. This hierarchical switching improves the accuracy of ARD state switching and the accuracy of display link activation.
[0043] In one embodiment, the gaze parameter includes the spatial gaze vector, and the method further includes: if the spatial gaze vector is within the area of the display interface of the augmented reality device, and the duration of the gaze point corresponding to the spatial gaze vector on the display interface is greater than a second preset duration, then the gaze intent is determined to be a content gaze intent.
[0044] In one embodiment, the gaze parameters include a spatial gaze vector, and the area where the augmented reality device's display interface is located is a virtual display field of view (Display Box) determined by the projection lens group. The process by which the augmented reality device determines the user's gaze intent as a content gaze intent based on the gaze parameters can be as follows: determining whether the spatial gaze vector is within the area of the augmented reality device's display interface. (See reference...) Figure 2 , Figure 2This is a schematic diagram illustrating the spatial gaze vector within the display interface of an augmented reality (AR) device. It can be understood that if the spatial gaze vector penetrates the display interface, then the gaze position (i.e., the gaze point) of the spatial gaze vector is not within the display interface area of the AR device. In this embodiment, the gaze depth corresponding to the spatial gaze vector can be obtained, and this gaze depth is used to determine whether the spatial gaze vector penetrates the display interface. This gaze depth is the distance between the gaze point corresponding to the spatial gaze vector and the AR device. After obtaining the gaze depth, the AR device determines whether the gaze depth is equal to the display depth of the AR device's display interface, where the display depth is the distance between the display interface and the AR device. If the gaze depth is equal to the display depth, the AR device determines that the spatial gaze vector has not penetrated the display interface; if the gaze depth is greater than the display depth, the AR device determines that the spatial gaze vector has penetrated the display interface, meaning that the spatial gaze vector is not within the display interface area.
[0045] Once the spatial gaze vector is determined to be within the area of the augmented reality device's display interface, the augmented reality device acquires the duration of the gaze point corresponding to the spatial gaze vector on the display interface and determines whether this duration is greater than a second preset duration, which can be set to 200ms, 250ms, or 280ms, etc. The specific process of acquiring the gaze point's duration can be found in related technologies and will not be elaborated here. In this embodiment, the duration is a continuous duration. If the duration is greater than the second preset duration, the augmented reality device determines the gaze intent to be a content gaze intent. In one embodiment, if the duration is less than or equal to the second preset duration, the augmented reality device can determine the gaze intent to be a real-world viewing intent, or determine the gaze intent to be an uncertain intent. When the gaze intent is determined to be an uncertain intent, the augmented reality device maintains the current state of the control system and does not switch states until the gaze intent is determined to be a content gaze intent or a real-world viewing intent.
[0046] It is understandable that in augmented reality devices, the three-dimensional spatial coordinates of the virtual display field of view determined by the projection lens group can be established or pre-stored, and then the positional relationship between the spatial line of sight vector and the three-dimensional spatial coordinates can be used to determine whether the spatial line of sight vector is within the area of the augmented reality device display interface.
[0047] This embodiment improves the accuracy of content attention view determination by determining whether the user's attention view is a content attention view based on the positional relationship between the spatial gaze vector and the display interface of the augmented reality device, as well as the duration of the gaze point corresponding to the spatial gaze vector on the display interface.
[0048] In one embodiment, the method further includes: Step S140: If the gaze intent of the user is determined to be a real viewing intent based on the gaze parameters, then control the control system to switch from the pre-wake state to the deep sleep state.
[0049] In one embodiment, when the augmented reality device determines that the wearer's gaze intent is a real viewing intent based on the gaze parameters, it indicates that the wearer does not currently need to view the virtual information displayed on the augmented reality device's display interface, but rather needs to view objects in the real world. Therefore, in order to reduce the power consumption of the augmented reality device and improve its battery life, the augmented reality device controls its control system to switch from a pre-wake state to a deep sleep state.
[0050] In one embodiment, the gaze parameter includes the spatial gaze vector, and the method further includes: if the spatial gaze vector does not intersect with the area where the augmented reality device display interface is located, or if the spatial gaze vector intersects with the area where the display interface is located, but the gaze depth corresponding to the spatial gaze vector is greater than the display depth of the display interface, then the gaze intention is determined to be a real viewing intention.
[0051] Understandably, if the spatial gaze vector does not intersect with the area where the augmented reality device's display interface is located, that is, if the spatial gaze vector is not within the area of the display interface, it indicates that the wearer's current attention is not on the display interface. In this case, the augmented reality device can determine that the wearer's gaze view is the intended view of reality.
[0052] In one embodiment, if the spatial gaze vector intersects the area where the display interface is located, but the gaze depth corresponding to the spatial gaze vector is greater than the display depth of the display interface, this indicates that the wearer's gaze passes through the display interface. Therefore, it can be determined that the wearer's attention view is the actual viewing intention. (See also...) Figure 3 , Figure 3 This is a schematic diagram where the spatial line-of-sight vector intersects the area where the display interface is located, but the depth of gaze corresponding to the spatial line-of-sight vector is greater than the display depth of the display interface.
[0053] In this embodiment, when the user's gaze intent is determined to be a real-world viewing intent, the control system switches from a pre-wake state to a deep sleep state. Since the sampling frequency of the eye-tracking sensor in the deep sleep state is lower than that in the pre-wake state, the power consumption of the augmented reality device is reduced.
[0054] In one embodiment, reference is made to Figure 4 The method further includes: Step S140: When it is determined that the user is in a closed-eye state, freeze the frame image displayed on the current display interface and / or reduce the drive current of the display link.
[0055] After the display link is activated, the augmented reality device continues to acquire images of the user's eyes in real time or at regular intervals using an eye-tracking sensor. Based on these images, it determines whether the user's eyes are closed. This determination based on eye images can be found in related technologies, which will not be elaborated upon here. In full-function interactive mode, the augmented reality device determines whether the user's eyes are closed based on the acquired eye images. It is understood that when the eye images determine that the user's eyelids are closed, the user is considered to be in a closed-eye state. When the duration of the user's closed eyes is less than a first preset duration, it can be determined that the user's current closed-eye state is a normal physiological blink to maintain eye moisture; that is, the user's closed-eye state is a normal phenomenon. The first preset duration can be set according to specific needs. Therefore, when it is determined that the user is in a closed-eye state, the augmented reality device freezes the frame image displayed on its current display interface and / or reduces the drive current of the display link. The augmented reality device freezes the currently displayed frame image, meaning it does not refresh the image displayed on the screen. For example, the MCU of the augmented reality device instructs the optomechanical display chip in the display link to freeze the current frame image. In this embodiment, the freeze time of the currently displayed frame image can be equal to, less than, or greater than the blink time of a normal user, but the difference between the freeze time and the blink time should not be too large. The process of the augmented reality device reducing the drive current of the display link can be as follows: when it detects that the user's eyes are closed, it reduces the drive current of the light source of the optomechanical device, or directly stops supplying current to the light source, that is, reduces the current of the light source to 0.
[0056] This embodiment freezes the frame image displayed on the augmented reality device's screen immediately when the user's eyes are closed, pausing the refresh of the screen. Since the augmented reality device's screen is no longer scanned and updated, it does not overlap with the user's eyelid movements, thus avoiding the visual effect of screen tearing. In one embodiment, freezing the frame image reduces the drive current of the display link, thereby reducing peripheral light scattering received by the user's retina from the display link's optical engine, and thus reducing image ghosting. It is understood that if the drive current of the display link is reduced to 0 when freezing the frame image, peripheral light scattering received by the user's retina from the display link's optical engine can be avoided, thus preventing the problem of image ghosting.
[0057] In one embodiment, after freezing the frame image displayed on the current display interface, if the augmented reality device detects that the user has switched from a closed-eye state to an open-eye state based on the eye-tracking sensor, the augmented reality device refreshes the current display interface, such as by changing the frame image displayed on the display interface or changing the display parameters of the currently displayed frame image (i.e., display brightness or clarity, etc.). For example, the effect of refreshing the display interface can be achieved by increasing the display brightness or clarity of the frame image.
[0058] In one embodiment, after reducing the drive current of the display link, if the augmented reality device detects that the user has switched from a closed-eye state to an open-eye state based on the eye-tracking sensor, the augmented reality device restores the drive current of the display link, that is, increases the drive current to the current value before the reduction. This allows the display state of the augmented reality device's display interface to change with the change in the user's eye state, thereby reducing the power consumption of the augmented display device without affecting the user's ability to view the content displayed by the augmented reality device.
[0059] In one embodiment, after freezing the frame image displayed on the current display interface and reducing the drive current of the display link, if the augmented reality device detects that the user has switched from a closed-eye state to an open-eye state based on the eye-tracking sensor, the augmented reality device refreshes the current display interface and restores the drive current of the display link. For example, when the augmented reality device detects that the user has switched from a closed-eye state to an open-eye state, i.e., when it detects that the user's eyelids have switched from a closed state to an open state, the augmented reality device restores the current to the light source of the display link and notifies the optomechanical display chip in the display link to continue refreshing the frame image. This allows the display state of the augmented reality device's display interface to change according to the change in the user's eye state, reducing the power consumption of the augmented reality device without affecting the user's ability to view the content displayed by the augmented reality device.
[0060] In one embodiment, after step S130, the method further includes: if the duration of the user's closed eyes is greater than or equal to a first preset duration of closed eyes and less than a second preset duration of closed eyes, then disconnecting the power supply of the display link and / or reducing the operating frequency of the application processor of the control system.
[0061] In full-function interactive mode, when it is determined that the user is in a closed-eye state, the augmented reality device determines whether the duration of the user's closed eyes is greater than or equal to a first preset duration and less than a second preset duration. It is understood that the second preset duration is longer than the first preset duration. The first preset duration can be determined based on the user's physiological blinking duration, such as 400ms or 430ms. In this case, the second preset duration can be set to 2.5s, 3s, or 3.6s. When the user's closed-eye duration is greater than or equal to the first preset duration and less than the second preset duration, it can be determined that the user is currently resting with their eyes closed or has briefly left the augmented reality device's display interface. At this time, the augmented reality device can disconnect the power supply to the display link and / or reduce the operating frequency of the application processor in the control system. For example, the power supply to the optical engine of the display link can be cut off, putting the optical engine in a zero-power state, and the operating frequency of the application processor can be reduced, effectively suspending the application processor. In one embodiment, if the augmented reality device detects that the user has opened their eyes via an eye-tracking sensor after the power to the display link is disconnected and / or the operating frequency of the application processor of the control system is reduced, the augmented reality device refreshes the frame image displayed on the current display interface and restores the drive current of the display link within a short period of time.
[0062] This embodiment further reduces the power consumption of the augmented reality device and improves its battery life by disconnecting the power supply to the display link and / or reducing the operating frequency of the application processor of the control system when the user's eyes are closed for a duration greater than or equal to a first preset duration and less than a second preset duration.
[0063] In one embodiment, after step S130, the method further includes: if the duration of the user's closed eyes is greater than or equal to a second preset duration of closed eyes, then the application processor stores the context information in the non-volatile memory of the augmented reality device and disconnects the power supply to the application processor.
[0064] In one embodiment, when the duration of the user's closed eyes is greater than or equal to a second preset duration, it is determined that the user has entered a sleep state, or that the user is no longer wearing the augmented reality device. In this embodiment, if the eye-tracking sensor does not detect an image of the user's eyes, the user can also be defined as being in a closed-eye state, meaning the user is no longer wearing the augmented reality device, and the augmented reality device is in an unworn state. If the duration of the closed eyes is greater than or equal to the second preset duration, the augmented reality device stores the context information in non-volatile memory through the application processor, and disconnects the power supply to the application processor after successfully storing the context information. The context information is state information used by the augmented reality device to characterize the current instruction execution state during operation. When the application processor is powered back on, it can continue to execute relevant instructions based on this context information. In one embodiment, after disconnecting the power supply to the application processor, the eye-tracking sensor of the augmented reality device reduces its sampling frequency and enters a low-power mode.
[0065] This embodiment stores context information in the non-volatile memory of the augmented reality device (ARD) when the eye-closing duration is greater than or equal to a second preset eye-closing duration, and then disconnects the power supply to the application processor. It is understood that this embodiment, by combining the first and second preset eye-closing durations described in the previous embodiments, achieves a tiered reduction in AR device power consumption. This ensures that the degree of power consumption reduction is adapted to the user's eye-closing duration, i.e., whether the user needs to view the content displayed on the AR device's interface. This improves the intelligence of power consumption optimization while reducing AR device power consumption.
[0066] Reference Figure 5 In one embodiment, the augmented reality device 500 includes a processor 501, a memory 502, and a control system 504. The processor 501, the control system 504, and the memory 502 are connected via a bus 503, such as an I2C (Inter-integrated Circuit) bus.
[0067] Specifically, processor 501 provides computing and control capabilities to support the operation of the entire augmented reality device. Processor 501 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. In one embodiment, processor 501 may also include an MCU.
[0068] The memory 502 can be a Flash chip, a read-only memory (ROM), a disk, an optical disk, a USB flash drive, or a portable hard drive, etc.
[0069] The control system 504 includes a display link and an application processor. In one embodiment, the augmented reality device also includes an eye-tracking sensor (not shown).
[0070] Those skilled in the art will understand that Figure 5 The structures shown are merely block diagrams of some structures related to the embodiments of the present invention, and do not constitute a limitation on the augmented reality devices on which the embodiments of the present invention are applied. Specific augmented reality devices may include more or fewer components than those shown in the figures, or combine certain components, or have different component arrangements.
[0071] The processor 501 is used to run a computer program stored in the memory 502, and implements any of the augmented reality device control methods provided in the embodiments of the present invention when executing the computer program. That is, the processor 501 is used to control the computer program to implement any of the augmented reality device control methods provided in the embodiments of the present invention.
[0072] In one embodiment, the processor 501 is configured to run a computer program and, when executing the computer program, perform the following steps: The eye movement trajectory of the user wearing the augmented reality device is acquired when the control system of the augmented reality device is in a deep sleep state. If the eye movement trajectory meets the preset conditions, the control system is controlled to switch from the deep sleep state to the pre-wake state, and the gaze parameters of the wearer are obtained; If the user's gaze intent is determined to be content gaze intent based on the gaze parameters, the control system is controlled to switch from a pre-wake state to a full-function interactive state to activate the display link of the control system.
[0073] In some embodiments, when the processor 501 determines that the eye movement trajectory meets preset conditions, it controls the control system to switch from the deep sleep state to the pre-wake state, and after acquiring the user's gaze parameters, it is used to achieve the following: If the gaze intent of the user is determined to be a real viewing intent based on the gaze parameters, then the control system is controlled to switch from a pre-wake state to a deep sleep state.
[0074] In some embodiments, the line-of-sight parameter includes the spatial line-of-sight vector, and the processor 501 is further configured to implement: If the spatial gaze vector does not intersect with the area where the augmented reality device's display interface is located, or if the spatial gaze vector intersects with the area where the display interface is located, but the gaze depth corresponding to the spatial gaze vector is greater than the display depth of the display interface, then the gaze intention is determined to be a real viewing intention.
[0075] In some embodiments, the processor 501 is further configured to implement: If, within a first preset time period, the user's eyeballs produce two consecutive upward movement trajectories in the vertical direction, then the eye movement trajectory is determined to meet the preset conditions. and / or If the time difference between two blinks by the wearer is within a preset time range, then the eye movement trajectory is determined to meet the preset conditions.
[0076] In some embodiments, the line-of-sight parameter includes the spatial line-of-sight vector, and the processor 501 is further configured to implement: If the spatial gaze vector is within the area of the augmented reality device's display interface, and the duration of the gaze point corresponding to the spatial gaze vector on the display interface is greater than a second preset duration, then the gaze intent is determined to be a content gaze intent.
[0077] In some embodiments, when the processor 501 determines, based on the gaze parameters, that the user's gaze intent is a content gaze intent, and controls the control system to switch from a pre-wake state to a full-function interactive state to activate the display link of the control system, it is used to achieve the following: When it is determined that the user is in a closed-eye state, the frame image displayed on the current display interface is frozen, and / or the drive current of the display link is reduced.
[0078] In some embodiments, when the processor 501 determines, based on the gaze parameters, that the user's gaze intent is a content gaze intent, and controls the control system to switch from a pre-wake state to a full-function interactive state to activate the display link of the control system, it is used to achieve the following: If the duration of the user's closed eyes is greater than or equal to a first preset duration of closed eyes, but less than a second preset duration of closed eyes, then the power supply to the display link is disconnected, and / or the operating frequency of the application processor of the control system is reduced.
[0079] In some embodiments, when the processor 501 determines, based on the gaze parameters, that the user's gaze intent is a content gaze intent, and controls the control system to switch from a pre-wake state to a full-function interactive state to activate the display link of the control system, it is used to achieve the following: If the duration of the user's closed eyes is greater than or equal to the second preset duration, the application processor stores the context information in the non-volatile memory of the augmented reality device and disconnects the power supply to the application processor.
[0080] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the augmented reality device described above can be referred to the corresponding process in the aforementioned embodiments of the augmented reality device control method, and will not be repeated here.
[0081] This invention also provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs that can be executed by one or more processors to implement any of the augmented reality device control methods provided in the specification of this invention.
[0082] The storage medium can be volatile or non-volatile. It can be an internal storage unit of the augmented reality device described in the foregoing embodiments, such as the hard drive or memory of the augmented reality device. Alternatively, it can be an external storage device of the augmented reality device, such as a plug-in hard drive, smart media card (SMC), secure digital card (SD), flash card, etc. The processor can be the control unit in the foregoing embodiments, or a part of the control unit; or a separate processor in the augmented reality device. For example, the processor can be a central processing unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0083] Those skilled in the art will understand that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware embodiments, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.
[0084] The sequence numbers of the above embodiments of the present invention are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The above descriptions are only specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A method for controlling an augmented reality device, characterized in that, The method includes: The eye movement trajectory of the user wearing the augmented reality device is acquired when the control system of the augmented reality device is in a deep sleep state. If the eye movement trajectory meets the preset conditions, the control system is controlled to switch from the deep sleep state to the pre-wake state, and the gaze parameters of the wearer are obtained; If the user's gaze intent is determined to be content gaze intent based on the gaze parameters, the control system is controlled to switch from a pre-wake state to a full-function interactive state to activate the display link of the control system.
2. The augmented reality device control method according to claim 1, characterized in that, After the step of controlling the control system to switch from the deep sleep state to the pre-wake state and obtaining the user's gaze parameters if the eye movement trajectory meets the preset conditions, the method further includes: If the gaze intent of the user is determined to be a real viewing intent based on the gaze parameters, then the control system is controlled to switch from a pre-wake state to a deep sleep state.
3. The augmented reality device control method according to claim 2, characterized in that, The line-of-sight parameters include the spatial line-of-sight vector, and the method further includes: If the spatial gaze vector does not intersect with the area where the augmented reality device's display interface is located, or if the spatial gaze vector intersects with the area where the display interface is located, but the gaze depth corresponding to the spatial gaze vector is greater than the display depth of the display interface, then the gaze intention is determined to be a real viewing intention.
4. The augmented reality device control method according to claim 1, characterized in that, The method further includes: If, within a first preset time period, the user's eyeballs produce two consecutive upward movement trajectories in the vertical direction, then the eye movement trajectory is determined to meet the preset conditions. and / or If the time difference between two blinks by the wearer is within a preset time range, then the eye movement trajectory is determined to meet the preset conditions.
5. The augmented reality device control method according to claim 1, characterized in that, The line-of-sight parameters include the spatial line-of-sight vector, and the method further includes: If the spatial gaze vector is within the area of the augmented reality device's display interface, and the duration of the gaze point corresponding to the spatial gaze vector on the display interface is greater than a second preset duration, then the gaze intent is determined to be a content gaze intent.
6. The augmented reality device control method according to any one of claims 1 to 5, characterized in that, Following the step of controlling the control system to switch from a pre-wake state to a full-function interactive state to activate the display link of the control system if the user's gaze intent is determined to be a content gaze intent based on the gaze parameters, the method further includes: When it is determined that the user is in a closed-eye state, the frame image displayed on the current display interface is frozen, and / or the drive current of the display link is reduced.
7. The augmented reality device control method according to any one of claims 1 to 5, characterized in that, Following the step of controlling the control system to switch from a pre-wake state to a full-function interactive state to activate the display link of the control system if the user's gaze intent is determined to be a content gaze intent based on the gaze parameters, the method further includes: If the duration of the user's closed eyes is greater than or equal to a first preset duration of closed eyes, but less than a second preset duration of closed eyes, then the power supply to the display link is disconnected, and / or the operating frequency of the application processor of the control system is reduced.
8. The augmented reality device control method according to any one of claims 1 to 5, characterized in that, Following the step of controlling the control system to switch from a pre-wake state to a full-function interactive state to activate the display link of the control system if the user's gaze intent is determined to be a content gaze intent based on the gaze parameters, the method further includes: If the duration of the user's closed eyes is greater than or equal to the second preset duration, the application processor stores the context information in the non-volatile memory of the augmented reality device and disconnects the power supply to the application processor.
9. An augmented reality device, characterized in that, The augmented reality device includes a control system, a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for establishing communication between the processor and the memory, wherein when the computer program is executed by the processor, it implements the steps of the augmented reality device control method as described in any one of claims 1 to 8.
10. A storage medium for computer-readable storage, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the method for controlling an augmented reality device according to any one of claims 1 to 8.
