A device control method and electronic device

Abstract

This application discloses a device control method and an electronic device, relating to the field of electronic device technology, and applied to electronic devices. It includes: when the electronic device has its sleep aid function activated, acquiring environmental data. If the electronic device detects that the environmental data indicates a resting scenario, and detects that the user's eyes are focused on the display screen, the electronic device acquires the user's sleep aid data. Based on the sleep aid data, when the user is nearing sleep, the electronic device performs corresponding sleep aid operations. These operations include lowering the volume of the electronic device and / or dimming the brightness of the display screen. When it is determined that the user is nearing sleep, the electronic device can remind and assist the user to fall asleep by lowering its own volume or dimming the brightness of the display screen, thereby interrupting the user's continuous use of the electronic device as much as possible. This helps the user fall asleep more easily and quickly, thus improving the user's sleep quality and rest quality.

Description

Technical Field

[0001] This application relates to the field of electronic equipment technology, and in particular to a device control method and an electronic device. Background Technology

[0002] As electronic devices offer increasingly rich functionality, users are becoming more and more reliant on them. In some scenarios, such as during lunch breaks or before bedtime, users may continue to use electronic devices for a period of time even when they are already sleepy. Continuous use of electronic devices before falling asleep can make it difficult for users to fall asleep, resulting in sleep deprivation and poor sleep quality, which seriously affects users' physical health and mental state.

[0003] Existing technologies include solutions that use sleep aids to help users fall asleep. For example, a sleep aid device could be a smart sleep mask that users can manually activate. The smart sleep mask provides sleep-inducing functions such as heating to help users fall asleep.

[0004] However, sleep aids are designed for scenarios where users actively use them to help them fall asleep. In scenarios where users continuously use electronic devices and cannot fall asleep, sleep aids are unlikely to play their role in helping them fall asleep. Summary of the Invention

[0005] This application provides a device control method and an electronic device. When it is determined that a user is nearing sleep, the electronic device can remind and assist the user to fall asleep by adjusting its own volume or dimming the brightness of the display screen, thereby interrupting the user's addiction to the electronic device as much as possible. When the user is nearing sleep, it can help the user fall asleep more easily and quickly, thereby improving the user's sleep quality and rest quality.

[0006] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0007] Firstly, a device control method is provided for use in electronic devices, the method comprising:

[0008] When the sleep aid function is activated, electronic devices acquire environmental data.

[0009] If the electronic device detects that the environmental data indicates a resting environment, and the electronic device detects that the user's eyes are focused on the display screen, the electronic device acquires the user's sleep aid data.

[0010] Based on sleep aid data, the electronic device performs corresponding sleep aid actions when the user is close to falling asleep; these actions include lowering the volume of the electronic device and / or dimming the brightness of the display screen.

[0011] In this application, for scenarios where a user needs to sleep but is engrossed in electronic devices, the electronic device can first determine whether it is in a resting state based on environmental data. Once it is determined that the electronic device is in a resting state, and it detects that the user is looking at the screen, it can further determine whether the user is nearing sleep (or in a state of impending sleep). When it is determined that the user is nearing sleep, the electronic device can lower its volume or dim the screen brightness to remind and assist the user in falling asleep, thereby minimizing the user's engrossedness in the electronic device. Furthermore, when the user is nearing sleep, the electronic device can help the user fall asleep more easily and quickly, thus improving the user's sleep and rest quality.

[0012] In one possible implementation of the first aspect, the sleep aid data includes the number of blinks. The electronic device determines that the user is near sleep based on the sleep aid data by: if the electronic device detects that the user's blink count is greater than or equal to a blink count threshold within a preset detection period.

[0013] In this application, certain physiological characteristics can be used as sleep aid data to determine whether a user is nearing sleep. For example, when a user is sleepy, frequent blinking is a physiological characteristic; therefore, the number of blinks can be recorded to determine whether the user is nearing sleep. This method can simply and effectively determine whether a user is showing signs of wanting to sleep.

[0014] In another possible implementation of the first aspect, the sleep aid data also includes the number of yawns. The electronic device determines that the user is near sleep based on the sleep aid data, further including: the number of yawns by the user within a preset detection period is greater than or equal to a yawn count threshold.

[0015] In this application, certain physiological characteristics can be used as sleep aid data to determine whether a user is nearing sleep. For example, when a user is sleepy, the physiological characteristic is frequent yawning; therefore, the number of yawns can be collected to determine whether the user is nearing sleep. This method can simply and effectively determine whether a user is showing signs of wanting to sleep.

[0016] In another possible implementation of the first aspect, the sleep aid data also includes heart rate values. The electronic device determines that the user is nearing sleep based on the sleep aid data, further including: the user's heart rate remaining within a preset heart rate range for a preset detection period.

[0017] In this application, certain physiological characteristics can be used as sleep aid data to determine whether a user is nearing sleep. For example, when a user is sleepy, physiological characteristics include slow breathing and a low-frequency heart rate. Therefore, the heart rate value can be obtained to determine whether the user is nearing sleep. This method can simply and effectively determine whether a user is showing signs of wanting to sleep.

[0018] In another possible implementation of the first aspect, the sleep aid data also includes posture data. Based on the sleep aid data, the electronic device determines that the user is near sleep, and further includes: posture data within a preset detection period indicating whether the user is in a supine or lateral position.

[0019] In this application, certain physiological characteristics can be used as sleep aid data to determine whether a user is nearing sleep. For example, before a user prepares to fall asleep, their posture may be lying flat or on their side. Therefore, posture data can be acquired to determine whether a user is nearing sleep. This method can simply and effectively determine whether a user is preparing to sleep.

[0020] In another possible implementation of the first aspect, the environmental data includes the location of the electronic device and the system time. The electronic device detecting environmental data indicating a resting scenario includes:

[0021] The electronic device detects that its location is at a preset position, and the system time is within a specified time period corresponding to the preset position.

[0022] When the preset location is the user's home, the specified time period corresponding to the preset location is the nighttime rest period; when the preset location is the user's company, the specified time period corresponding to the preset location is the lunch break period.

[0023] In this application, the applicable scenario for the sleep aid operation is a scenario where the user should be in a sleep state (rest scenario). Therefore, before collecting sleep aid data and subsequent data processing, the rest scenario test is performed first to ensure that the timing of the sleep aid data collection and subsequent sleep aid operation is accurate and can effectively help the user fall asleep, thereby improving the user's experience of using the sleep aid function.

[0024] In another possible implementation of the first aspect, the electronic device includes a sleep aid application, a sensing platform, a smart sensor hub (SensorHub), and an always-on camera (ASC); the smart sensor hub includes a sleep aid detection algorithm module and an ASC camera driver; the sleep aid detection algorithm module communicates with the always-on camera (ASC) via the ASC camera driver; the sensing platform provides a communication interface for the sleep aid application to communicate with the smart sensor hub (SensorHub).

[0025] The method also includes:

[0026] The sleep aid app registers a gaze fence through a communication interface and sends the first instruction to the sleep aid detection algorithm module.

[0027] The sleep aid detection algorithm module responds to the first instruction and acquires the first user image captured by the always-sensing camera ASC through the ASC camera driver; the first user image includes multiple user images captured within the first acquisition time.

[0028] The sleep aid detection algorithm module extracts features from the first user image to obtain the eye features corresponding to the first user image.

[0029] The sleep aid detection algorithm module performs human eye gaze recognition based on the eye features corresponding to the first user image and obtains the human eye gaze recognition result.

[0030] The sleep aid detection algorithm module returns the human eye gaze recognition results to the sleep aid application through the communication interface.

[0031] So, when an electronic device detects that a person's eyes are fixed on the display screen, it acquires the user's sleep aid data, including:

[0032] If a sleep aid application determines that the human eye is focused on the display screen, the electronic device acquires the user's sleep aid data.

[0033] In this application, the sleep aid detection algorithm module can obtain gaze detection results based on image data acquired by an always-on camera. Deployed on the Sensorhub side, the algorithm module can operate in low-power mode. By acquiring user images through the always-on camera, it avoids the high power consumption problem caused by using front or rear main cameras to acquire image data, further realizing the detection of human eye gaze and the acquisition of sleep aid data in low-power mode.

[0034] In another possible implementation of the first aspect, when the sleep aid data includes the number of blinks, the method further includes:

[0035] The sleep aid app registers a blink fence through a communication interface and sends a second instruction to the sleep aid detection algorithm module.

[0036] The sleep aid detection algorithm module responds to the second instruction and acquires the second user image captured by the always-sensing camera ASC through the ASC camera driver; the second user image includes multiple user images captured within the second acquisition time.

[0037] The sleep aid detection algorithm module extracts features from the second user's image to obtain the corresponding eye features.

[0038] The sleep aid detection algorithm module performs blink recognition based on the eye features corresponding to the second user image to obtain blink detection results; the blink detection results include the number of blinks of the user within a preset detection time.

[0039] So, electronic devices acquire users' sleep aid data, including:

[0040] Sleep aid apps obtain blink detection results through a communication interface.

[0041] In this application, the sleep aid detection algorithm module can perform blink recognition based on image data acquired by an always-on camera to obtain blink detection results. The algorithm module is deployed on the Sensorhub side of the smart sensor hub, allowing it to operate in low-power mode. Furthermore, by acquiring user images through an always-on camera, it avoids the high power consumption issues caused by using front or rear main cameras to acquire image data, further realizing the acquisition of sleep aid data and blink detection recognition in low-power mode.

[0042] In another possible implementation of the first aspect, the sleep aid data also includes the number of yawns, and the method further includes:

[0043] The sleep aid app registers a yawning fence through a communication interface and sends a third instruction to the sleep aid detection algorithm module.

[0044] The sleep aid detection algorithm module responds to the third instruction and acquires the third user image captured by the always-sensing camera ASC through the ASC camera driver; the third user image includes multiple user images captured within the third acquisition time.

[0045] The sleep aid detection algorithm module extracts features from the third user's image and obtains the mouth features corresponding to the second user's image.

[0046] The sleep aid detection algorithm module identifies yawns based on the mouth features corresponding to the second user image and obtains the yawn detection results. The yawn detection results include the number of times the user yawns within a preset detection time.

[0047] So, electronic devices acquire users' sleep aid data, including:

[0048] Sleep aid apps obtain yawn detection results through a communication interface.

[0049] In this application, the sleep aid detection algorithm module can identify yawns based on image data collected by an always-on camera and obtain yawn detection results. The algorithm module is deployed on the Sensorhub side of the smart sensor hub and can operate in low-power mode. Furthermore, by collecting user data through an always-on camera, the high power consumption problem caused by using front or rear main cameras to collect image data can be avoided, further realizing the collection of sleep aid data and yawn detection and recognition in low-power mode.

[0050] In another possible implementation of the first aspect, the electronic device performs a corresponding sleep-aiding operation, including:

[0051] If the electronic device detects that the volume exceeds a preset decibel threshold, it outputs a volume adjustment prompt. Then, the electronic device lowers the volume to within the preset decibel range, or lowers the volume by the preset decibel value.

[0052] The volume adjustment prompt is used to indicate that the sleep aid function has been activated and that the volume is about to be reduced or has already been reduced.

[0053] In this application, electronic devices can provide users with a relatively quiet environment and adjust volume or sound effects to help users fall asleep. For example, electronic devices can help users fall asleep by lowering their own volume, thereby interrupting users' addiction to electronic devices; when users are close to falling asleep, they can help users fall asleep more easily and quickly, thereby improving the user's sleep quality and rest quality.

[0054] In another possible implementation of the first aspect, the electronic device performs a corresponding sleep-aiding operation, including:

[0055] If the electronic device detects that the brightness of its display screen exceeds a preset brightness threshold, the electronic device outputs a brightness adjustment prompt. Then, the electronic device adjusts the display screen brightness to the preset brightness range, or lowers the display screen brightness by a preset value.

[0056] The brightness adjustment prompt is used to indicate that the sleep aid function has been activated and the screen brightness has been dimmed.

[0057] In this application, electronic devices can provide users with a relatively dim environment to help them fall asleep. For example, the brightness of the electronic device's screen can be dimmed to help users fall asleep, thereby reducing the user's addiction to the electronic device; when the user is close to falling asleep, it can help the user fall asleep more easily and quickly, thereby improving the user's sleep quality and rest quality.

[0058] In another possible implementation of the first aspect, the electronic device performs a corresponding sleep-aiding operation, further including:

[0059] The electronic device outputs a sound effect playback prompt. Furthermore, the electronic device plays a sleep-aiding sound effect at a target decibel level.

[0060] The sound effect playback prompt message is used to indicate that the sleep aid function has been activated and that sleep aid sound effects are about to play or are currently playing.

[0061] In this application, the electronic device can provide a relatively quiet environment for the user and adjust the volume or sound effects to help the user fall asleep. For example, the electronic device can play sleep-inducing sounds at a target decibel level to help the user fall asleep, thereby interrupting the user's addiction to the electronic device as much as possible; when the user is close to falling asleep, it can help the user fall asleep better and faster, thereby improving the user's sleep quality and rest quality.

[0062] In another possible implementation of the first aspect, the electronic device performs a corresponding sleep-aiding operation, further including:

[0063] If the electronic device detects that the display screen is in normal display mode, it outputs a prompt to switch display modes. Then, the electronic device switches the display screen from normal display mode to eye-protection display mode.

[0064] The display mode switching prompt indicates that the sleep aid function has been activated and that the display is about to switch to eye-protection mode or has already switched to it. The display brightness and contrast in eye-protection mode are lower than in normal display mode.

[0065] In this application, the electronic device can provide a relatively dim environment for the user to help them fall asleep. For example, the electronic device can switch the display screen from normal display mode to eye-protection display mode to minimize the user's addiction to the electronic device; when the user is close to falling asleep, it can help the user fall asleep more easily and quickly, thereby improving the user's sleep quality and rest quality.

[0066] In a second aspect, an electronic device is provided, comprising a display screen, a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in any of the first aspects above.

[0067] Thirdly, a computer-readable storage medium is provided that stores instructions which, when executed by a processor, implement the steps of the method described in any of the first aspects above.

[0068] Fourthly, a computer program product including instructions is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method described in any of the first aspects above.

[0069] Fifthly, embodiments of this application provide a chip, the chip including a processor, the processor being configured to invoke a computer program in memory to perform the method as described in any one of the first aspects.

[0070] It is understood that the beneficial effects of the electronic device described in the second aspect, the computer-readable storage medium described in the third aspect, the computer program product described in the fourth aspect, and the chip described in the fifth aspect can be referred to the beneficial effects of the first aspect and any of its possible design embodiments, which will not be repeated here. Attached Figure Description

[0071] Figure 1 This application provides a schematic diagram of an interface for enabling the sleep aid function.

[0072] Figure 2 A schematic diagram illustrating the stages of a device control method provided in an embodiment of this application;

[0073] Figure 3 A schematic flowchart illustrating a device control method provided in an embodiment of this application;

[0074] Figure 4 A schematic diagram of a scenario for a device control method provided in an embodiment of this application;

[0075] Figure 5 This is a schematic diagram of a prompting information interface provided in an embodiment of this application;

[0076] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0077] Figure 7 A software structure block diagram of an electronic device provided in an embodiment of this application;

[0078] Figure 8 A schematic diagram illustrating the communication mechanism of an important module of an electronic device provided in an embodiment of this application;

[0079] Figure 9 A schematic diagram illustrating the execution timing of various modules of a device control method provided in an embodiment of this application;

[0080] Figure 10 This application provides an algorithm execution logic block diagram for obtaining blink detection results.

[0081] Figure 11This application provides an algorithm execution logic block diagram for obtaining yawn detection results.

[0082] Figure 12 A possible structural schematic diagram of the electronic device provided in the embodiments of this application;

[0083] Figure 13 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation

[0084] In the description of the embodiments of this application, the terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to be a limitation of this application. As used in the specification and appended claims of this application, the singular expressions "a," "the," "the," "the," and "this" are intended to also include expressions such as "one or more," unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, "at least one" and "one or more" refer to one or more (including two). The term "and / or" is used to describe the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can indicate: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0085] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. The term "connection" includes direct connections and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0086] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0087] With the continuous progress of society and the acceleration of life, people are experiencing increasing mental stress. This manifests itself in behaviors such as "revengeful" addiction to electronic devices before taking a short break after work or before going to bed at night. For example, even when feeling sleepy during lunch breaks or before bed, people may still immerse themselves in electronic devices for a period of time. This continuous use of electronic devices makes it harder to fall asleep or even leads to insomnia, resulting in sleep deprivation and poor sleep quality. Over time, this can seriously affect physical health and mental well-being.

[0088] Traditional technologies include sleep aids such as smart sleep masks, smart sleep speakers, and smart sleep lamps. Specifically, users can manually activate a smart sleep mask and enter sleep mode before falling asleep. Once worn, the smart sleep mask can use features like timer heating to help users fall asleep. Alternatively, users can manually activate smart sleep speakers or smart sleep lamps to enter sleep mode, allowing them to fall asleep with soothing sounds or light. Or, some sleep aids require connection to electronic devices; users can manually connect the device to their electronic device before falling asleep, switching it to sleep mode and using its features to fall asleep.

[0089] In many of the traditional technologies mentioned above, users need to manually activate or connect the sleep aid device to access its sleep-aid functions. In other words, the purpose of the sleep aid device is to provide sleep-aid features to help the user fall asleep when they actively want to. However, in some scenarios, such as when a user is engrossed in electronic devices but actually needs to sleep, the sleep aid device cannot function to help the user fall asleep unless the user actively uses it.

[0090] In scenarios where users are engrossed in electronic devices but actually need to sleep, such as preparing for a nap during lunch break or preparing to fall asleep at night, if the electronic device detects this and the user's eyes are fixed on the screen, it indicates that the user is continuing to use the device despite needing to sleep. Further detection can be performed to determine if the user is fatigued and nearing sleep. If so, the electronic device can trigger appropriate sleep-inducing actions. This effectively provides sleep aids to users who are continuously using electronic devices but need to sleep, thereby improving their rest and sleep quality.

[0091] Based on this approach, this application provides a device control method. For scenarios where a user needs to sleep but is engrossed in electronic devices, the electronic device can first determine whether it is in a resting state based on environmental data. Once it is determined that the electronic device is in a resting state, and when it detects that a person is looking at the screen, it can further determine whether the user looking at the screen is nearing sleep (or in a state of imminent sleep). When it is determined that the user is nearing sleep, the electronic device can lower its volume or dim the screen brightness to remind and assist the user in falling asleep, thereby interrupting the user's immersion in the electronic device as much as possible. When the user is nearing sleep, the method can help the user fall asleep more easily and quickly, thus improving the user's sleep and rest quality.

[0092] The sleep-aid function of electronic devices can be a sleep-aid function provided by the operating system; or it can be a sleep-aid function provided by a third-party application. For example, an application with a sleep-aid function can be a smart sleep-aid application.

[0093] For example, Figure 1 A schematic diagram of the interface for enabling the sleep aid function is provided.

[0094] Take the sleep aid function, a feature provided by the operating system of an electronic device, as an example. For instance... Figure 1 As shown in (a), the electronic device can display a system settings interface 100 on the screen in response to a user's operation of opening system settings. The system settings interface 100 may include function options such as applications, battery, storage, security, privacy, health and wellness, smart assistant, wallet, accessibility features, accounts, and system updates.

[0095] Electronic devices respond to the user's selection of a smart assistant, such as... Figure 1 As shown in (b), the smart assistant interface 200 is displayed on the screen. The interface 200 may include function options such as sleep aid function and eye protection mode.

[0096] Electronic devices respond to user selection of sleep aid functions, such as... Figure 1 As shown in (c), a details interface 300 for the sleep aid function is displayed on the screen. The details interface 300 includes an information box 301 and a function switch 302. The electronic device turns the sleep aid function on or off in response to the user's operation of the function switch 302. For example, when the electronic device does not have the sleep aid function enabled, it enables the sleep aid function in response to the user's operation of turning on the function switch 302. For example, Figure 1In section (c), the function switch button is positioned to the right, indicating that the sleep aid function is enabled. Correspondingly, to disable the sleep aid function, the function switch button can be moved to the left. A details interface 300 for the sleep aid function is displayed on the screen. The details interface 300 includes an information box 301 and a function switch 302.

[0097] In some embodiments, after the electronic device activates the sleep aid function, such as Figure 1 As shown in (c), the electronic device can also display an indicator 303 corresponding to the activation of the sleep aid function in the status bar at the top of the display screen. Indicator 303 can provide the user with a notification that the sleep aid function has been activated.

[0098] When the sleep aid function is activated, the electronic device can detect rest scenarios, detect eye gaze, collect and analyze sleep aid data, and execute sleep aid operations. For example, Figure 2 A schematic diagram of the stages of a device control method is provided.

[0099] The device control method provided in this embodiment includes a sleep aid function activation stage, a data acquisition stage, a user status recognition and judgment stage, a sleep aid operation execution stage, and a continuous monitoring stage.

[0100] The data acquisition phase includes collecting data for rest scenario detection and analysis, as well as sleep aid data. For example, data for rest scenario detection and analysis includes the location of the electronic device and system time. Sleep aid data includes the user's blink rate, yawn rate, heart rate, and posture data over a period of time. The user identification and judgment phase refers to identifying and judging the user's near-sleep state. If it is determined that the user is near sleep, it is determined that the user needs sleep aid operations, and the electronic device performs the corresponding sleep aid operations. These sleep aid operations may include dimming the screen brightness, lowering the volume of the electronic device, playing sleep aid sound effects, and switching to sleep aid display mode, etc.

[0101] The following specific embodiment illustrates the device control method provided in this application. For example, Figure 3 A flowchart of a device control method is given, and Figure 4 A scenario diagram of a device control method is provided, including:

[0102] S101, the electronic device confirms that the sleep aid function is activated.

[0103] The electronic device can obtain the parameter identifier corresponding to the sleep aid function from the system settings information and determine whether to enable the sleep aid function based on the parameter identifier. For example, a parameter identifier of the first value indicates that the sleep aid function is enabled; a parameter identifier of the second value indicates that the sleep aid function is not enabled. Alternatively, the electronic device can also determine whether to enable the sleep aid function based on whether the icon representing the sleep aid function is displayed in the status bar; if the icon is displayed, it indicates that the sleep aid function is enabled. Alternatively, the electronic device can also determine whether to enable the sleep aid function based on the state of the register corresponding to the sleep aid function. This embodiment does not limit how the sleep aid function is determined to be enabled.

[0104] S102, the electronic device starts data acquisition.

[0105] After the electronic device confirms that the sleep aid function is activated, the electronic device triggers a data collection operation.

[0106] For example, data collection includes collecting sleep aid data and rest scenario detection data. The rest scenario detection data includes one or more of the following: the location of the electronic device and the system time. Since the rest scenario detection data can determine whether the user is in a theoretically appropriate period of rest, in one example, the rest scenario detection data may include the location of the electronic device and the system time.

[0107] Sleep aid data can be used to determine whether a user is nearing sleep, that is, whether the user is sleepy and in a state close to sleep. For example, sleep aid data may include one or more of the following: the number of blinks, the number of yawns, heart rate, and posture data over a period of time. An electronic device can determine whether a user is nearing sleep based on one type of sleep aid data. Considering the accuracy of user state judgment in some cases, the electronic device can determine whether a user is nearing sleep based on multiple types of sleep aid data. For example, the electronic device can determine whether a user is nearing sleep based on the number of blinks and yawns over a period of time. Alternatively, the electronic device can determine whether a user is nearing sleep based on the number of blinks and heart rate over a period of time. Alternatively, the electronic device can determine whether a user is nearing sleep based on the number of yawns and heart rate over a period of time. Alternatively, the electronic device can determine whether a user is nearing sleep based on the number of blinks and posture data over a period of time. Alternatively, the electronic device can determine whether a user is nearing sleep based on the number of blinks, yawns, and posture data over a period of time.

[0108] Optionally, sleep aid data may also include other data that can characterize a user's near-sleep state, such as the user's breathing rate, ambient noise in the environment where the electronic device is located, etc.

[0109] In addition, data acquisition also includes gaze detection data. For example, gaze detection data can be facial images captured by electronic devices.

[0110] In this embodiment, the electronic device can collect different types of data at different acquisition frequencies.

[0111] In this embodiment, the electronic device can prioritize detecting rest scenarios. If the electronic device is not in the location and time period specified in the rest scenario, it is highly unlikely that the user will continuously use the electronic device and be in a near-sleep state. That is, when the electronic device is not in a rest scenario, sleep aid detection and sleep aid operations will not be performed. The electronic device may not collect sleep aid data and human eye gaze detection data, or it may collect sleep aid data and human eye gaze detection data at a lower frequency to reduce the power consumption of the electronic device.

[0112] S103, the electronic device recognizes rest scenarios.

[0113] In this embodiment, the rest scenario can represent the user's rest location and the corresponding rest time period. For example, if the user's location is home (first location), the user's rest time at home is most likely in the evening. The rest time period matching the location (home) can be the nighttime rest time period (first time period). For example, the nighttime rest time period can be 22:00-06:00, or 23:00-07:00. Similarly, if the user's location is at the company (second location), the user's rest time at the company is most likely at noon. The rest time period matching the location (company) can be the midday rest time period (second time period). For example, the midday rest time period can be 12:00-14:00, or 13:00-15:00.

[0114] In this embodiment, a geofence for the target location can be pre-set in the electronic device. For example, a geofence can be set for the company and a geofence for the home. The electronic device can obtain its location information through its positioning function, and based on the location of the electronic device and the geofence of the target location, it can determine whether it is within the specified location range of the rest scenario. Furthermore, the electronic device can determine whether it is within the rest time period that matches the location based on the system time. In this way, the electronic device can identify rest scenarios.

[0115] In some embodiments, the rest period time matched with a location is not a fixed preset value. Electronic devices can continuously collect user sleep profile data, for example, continuously collect data on the user's rest periods at a specified location, obtaining a large amount of data on the user's rest periods at that location. The appropriate rest period time for the user at a location is determined based on this large amount of sleep period data.

[0116] In this embodiment, the electronic device can determine the geofence of its target address based on at least one of the following data: cellular network location (cell), access network, and GPS data. For example, the information corresponding to a home geofence includes homecell, homeWiFi, and homeGPS. The information corresponding to a company geofence includes companycell, companyWiFi, and companyGPS. In this embodiment, the parameters used for geofencing may include other parameters for identifying the geographical location of the electronic device; this embodiment does not limit the parameters included in the geofence.

[0117] If the cellular network location of the electronic device is within the range indicated by homecell, and the access network is homeWiFi, and the GPS data is within the range indicated by homeGPS, then the electronic device is considered to be within the home fence, and its location is considered to be at home. If the cellular network location of the electronic device is within the range indicated by companycell, and the access network is companyWiFi, and the GPS data is within the range indicated by companyGPS, then the electronic device is considered to be within the company fence, and its location is considered to be at the company.

[0118] Electronic devices can collect location data, including cellular network positioning, access network data, and GPS data, to identify their location during specified time periods (00:00-06:00 and 10:00-17:00) and while the device is relatively stationary. "Relatively stationary" means the device is in a screen-off state and its location remains unchanged for a period of time, such as one hour. The access network of the electronic device can, to some extent, indicate its location. For example, if the access network is WiFi1 (indicated by "homeWiFi"), it can help determine the device's location as home. Similarly, if the access network is WiFi2 (indicated by "companyWiFi"), it can help determine the device's location as the company.

[0119] Electronic devices can continuously collect location data such as cellular network positioning, network access, and GPS data for a specified period of time for a week or longer, and dynamically learn and update location and matching rest time periods based on the collected location data.

[0120] Specifically, electronic devices can cluster location data collected over a specified time period. For example, an electronic device might continuously collect location data for a week (7 days) within a specified time period and then perform clustering on the collected location data. If the clustering results include location data representing at least 5 days of locations and rest periods—that is, if a certain proportion of the location data represents locations and rest periods that are similar—then that proportion of location data and rest periods is considered valid location data, and the location and rest period information is updated through learning. This certain proportion is greater than 50%, such as 60%, 70%, or 80%.

[0121] Let's illustrate this with specific data examples. For instance, location data collected by an electronic device over a week includes:

[0122] D1: Location data 11: Cellular network location 1 (within the range indicated by homecell), WiFi 1 (network indicated by homeWiFi), GPS 1 (within the range indicated by homeGPS), rest period is 00:00-06:00; Location data 21: Cellular network location 2 (within the range indicated by companycell), WiFi 2 (network indicated by companyWiFi), GPS 2 (within the range indicated by companyGPS), rest period is 13:00-14:00.

[0123] D2: Location data 12: Cellular network positioning 1, WiFi 1, GPS 1, rest period is 00:00-06:00.

[0124] D3: Location data 13: Cellular network positioning 1, WiFi 1, GPS 1, rest period is 00:00-06:00; Location data 23: Cellular network positioning 2, WiFi 2, GPS 2, rest period is 13:10-13:50.

[0125] D4: Location data 14: Cellular network positioning 1, WiFi 1, GPS 1, rest period is 01:00-06:00; Location data 24: Cellular network positioning 2, WiFi 2, GPS 2, rest period is 13:15-14:00.

[0126] D5: Location data 15: Cellular network positioning 1, WiFi 1, GPS 1, rest period is 00:00-06:00; Location data 25: Cellular network positioning 2, WiFi 2, GPS 2, rest period is 13:05-14:00.

[0127] D6: Location data 16: Cellular network positioning 1, WiFi 1, GPS 1, rest period is 00:00-06:00; Location data 26: Cellular network positioning 2, WiFi 2, GPS 2, rest period is 13:00-13:45.

[0128] D7: Location data 17: Cellular network positioning 1, WiFi 1, GPS 1, Rest period is 00:00-06:00.

[0129] After clustering the location data, locations 11, 12, 13, 15, 16, and 17 form a cluster that shows high similarity in both location and rest time period. This cluster includes 6 / 7 of the location data, satisfying a certain ratio, indicating that when the electronic device is located at home, the corresponding rest time period is 00:00-06:00. Locations 21, 23, 24, 25, and 26 form a cluster that shows high similarity in both location and rest time period. This cluster includes 5 / 7 of the location data, satisfying a certain ratio, indicating that when the electronic device is located at the office, the corresponding rest time period is 13:00-14:00.

[0130] Based on the clustering results of location data and the analysis results of effective location data, the system can dynamically learn and update location positions and corresponding rest periods, thereby updating the specified time period for collecting location data to more accurately update the data. For example, if the location in the above location data is "home," and the rest period is mostly 00:00-06:00, then the specified time period of 00:00-06:00 can be extended to 23:00-07:00 to further determine the rest period corresponding to being at home. Similarly, if the location in the above location data is "office," and the rest period is mostly 13:00-14:00, then the specified time period of 10:00-17:00 can be shortened to 12:00-15:00 to further determine the rest period corresponding to being at the office.

[0131] Continuously collecting location data over a specified time period and learning to update the time period based on clustering results makes the final location and matched rest time period more accurate. Electronic devices can more accurately identify the rest environment based on location and matched rest time period, thus making the triggering of subsequent sleep-aiding operations more precise. Furthermore, electronic devices obtain the user's corresponding location and matched rest time period based on the current user's profile data, making the identified rest environment more closely matched to the individual user. This personalized approach ensures that the identified rest environment is more realistic and accurate for each individual user.

[0132] In one example, such as Figure 4 As shown in (a), when the electronic device detects that the system time is 23:00 and the location is at the geofence corresponding to home, it determines that the user is in a resting environment.

[0133] S104, when the electronic device is determined to be in a resting scenario, recognizes the human eye looking at the display screen.

[0134] In a resting context, electronic devices can first detect and identify whether the user is continuously using the device. For example, an electronic device can determine whether the user is continuously using the device by detecting whether there is eye contact with the display screen.

[0135] It's understandable that in some scenarios, although the first image captured may contain the user's facial information, the user may not be looking at the phone at that moment. In scenarios where the user isn't looking at the phone, the probability of the user interacting with the electronic device is very low. Therefore, in such scenarios, even though a facial image of the user has been captured, the user may not be interacting with the electronic device at that time.

[0136] Therefore, in this embodiment, the electronic device can perform face detection on the acquired images using a face detection algorithm (e.g., an algorithm based on a Haar feature classifier). After determining that the acquired image contains a face, eye feature extraction can be performed, and based on the eye features, it can be further determined whether eye gaze is detected in the image. In one implementation, after obtaining an image containing face information, the image can be detected to determine whether eye gaze at the display screen is detected. This also filters out a large number of images containing faces acquired in scenarios where the user has not interacted with the electronic device. For these images, the current processing flow can be terminated directly, reducing the high power consumption generated by processing these images.

[0137] In one example, such as Figure 4 (b) and Figure 4 As shown in (c), the electronic device uses a constantly sensing camera to capture images of the user's face and recognizes whether the user's eyes are looking at the display screen.

[0138] For example, computer vision technology can be used to determine whether a person's eyes are looking at a display screen. This involves capturing a facial image with a camera, then using image processing algorithms to detect and track the position of the eyes. By analyzing feature points of the eyes, such as the corners of the eyes and the position of the pupils, it can be determined whether the person's eyes are looking at the display screen. However, it should be noted that there are various methods for determining eye gaze in actual implementation. The specific method chosen depends on the actual needs, and this embodiment does not limit this.

[0139] In some embodiments, if the electronic device detects that the location or system time is not at the location indicated by the rest scenario and the matching rest time period, the execution of S104 and subsequent steps may be omitted.

[0140] S105, after determining that a person's eyes are focused on the display screen, the electronic device identifies the state of near sleep based on sleep aid data.

[0141] In this embodiment, the electronic device can identify whether a user is nearing sleep based on one or more sleep-aiding data. If the electronic device detects that the user is nearing sleep, it can determine that it needs to provide sleep aid functionality and then execute the corresponding sleep-aiding operation. It is understood that being near sleep means the user is sleepy and exhibits physiological characteristics indicating a desire to sleep. Therefore, some physiological characteristics can be used as sleep-aiding data to determine whether a user is nearing sleep. For example, when a user is sleepy, physiological characteristics include frequent blinking, yawning, slow breathing, and a low-frequency heart rate; and before falling asleep, the user's posture is either lying flat or on their side.

[0142] For example, sleep aid data could be the number of blinks a user makes within a certain time period. Normally, users don't blink frequently, but they might blink frequently when trying to fall asleep. Therefore, the number of blinks within a certain time period can be used to determine if a user is nearing sleep. For instance, if the electronic device detects that the number of blinks within a certain time period is greater than or equal to a blink count threshold, it can be considered that the user is nearing sleep and needs assistance to fall asleep. This certain time period can be 30 seconds, 1 minute, 2 minutes, etc. Taking 1 minute as an example, the blink count threshold for 1 minute can be set to 8 times. If the electronic device detects that the user blinks more than or equal to 8 times within 1 minute, it considers the user to be nearing sleep and needs assistance to fall asleep. It is understood that the blink count threshold corresponding to a certain time period can be determined according to actual circumstances; this embodiment is merely an example.

[0143] For example, sleep aid data can also be the number of times a user yawns within a certain period of time. People yawn frequently when they want to sleep, so the number of yawns within a certain period can be used to determine whether a user is nearing sleep. Specifically, if the electronic device detects that the number of yawns within a certain period is greater than or equal to a yawn count threshold, it can be considered that the user is nearing sleep and needs assistance to fall asleep. This certain period can be 30 seconds, 1 minute, 2 minutes, etc. Taking a 1-minute period as an example, the yawn count threshold for 1 minute can be set to 1 yawn. If the electronic device detects that the user's yawn count threshold for 1 minute is greater than or equal to 1 yawn, it considers the user to be nearing sleep and needs assistance to fall asleep. It is understood that the yawn count threshold corresponding to a certain period can be determined according to actual circumstances; this embodiment is merely an example.

[0144] For example, sleep aid data can also be the user's heart rate value over a certain period of time. When a person is preparing to fall asleep or wants to sleep, their heart rate value will remain in a relatively low frequency range. Therefore, the heart rate value over a certain period of time can be used to determine whether the user is in a state of near sleep. Specifically, if the electronic device detects that the user's heart rate value remains in a low frequency range over a certain period of time, it can be considered that the user is in a state of near sleep and needs assistance to fall asleep. The certain period of time can be 30 seconds, 1 minute, 2 minutes, etc. The low frequency range is (50, 80) beats / minute. If the electronic device detects that the user's heart rate value remains in (50, 80) beats / minute for 1 minute, it is considered that the user is in a state of near sleep and needs assistance to fall asleep. It is understood that the low frequency range corresponding to the heart rate value over 1 minute can be determined according to the actual situation; this embodiment is only an example.

[0145] For example, sleep aid data can also include the user's posture. The user's posture can be determined through the posture of the electronic device. When the user is lying flat or on their side, holding the electronic device will result in the device being positioned sideways or with the screen facing down (or at a certain angle). When the user is lying flat using the electronic device, the device is in use and the screen is facing down. Therefore, the electronic device can acquire posture data to determine whether the user is lying flat or on their side. If the user is lying flat or on their side, it indicates that the user will rest or enter a sleep state within a certain period of time, which can also be considered as the user being close to sleep. Specifically, if the electronic device detects that the user is continuously lying flat or on their side, it can be considered that the user is close to sleep and needs assistance to fall asleep. For example, if the electronic device detects that the user is continuously lying flat or on their side for 10 minutes, 20 minutes, or 30 minutes, it considers the user to be close to sleep and needs assistance to fall asleep.

[0146] In some optional embodiments, the electronic device can identify the user's near-sleep state based on at least two types of sleep aid data. For example, if the electronic device detects that the number of blinks within a certain period is greater than or equal to a blink count threshold, and the electronic device detects that the number of yawns within a certain period is greater than or equal to a yawn count threshold, then the user is considered to be near sleep and needs sleep aid. Similarly, if the electronic device detects that the number of blinks within a certain period is greater than or equal to a blink count threshold, and the electronic device detects that the user is lying flat or on their side, then the user is considered to be near sleep and needs sleep aid. Furthermore, if the electronic device detects that the number of blinks within a certain period is greater than or equal to a blink count threshold, and the electronic device detects that the user's heart rate remains consistently within a low-frequency range, then the user is considered to be near sleep and needs sleep aid. For example, if an electronic device detects that the number of blinks within a certain period is greater than or equal to a blink count threshold, and the number of yawns within a certain period is greater than or equal to a yawn count threshold, and the electronic device detects that the user is lying flat or on their side, then it considers the user to be near sleep and requires assistance to fall asleep. Similarly, if an electronic device detects that the number of blinks within a certain period is greater than or equal to a blink count threshold, and the number of yawns within a certain period is greater than or equal to a yawn count threshold, and the electronic device detects that the user's heart rate remains consistently in a low-frequency range, then it considers the user to be near sleep and requires assistance to fall asleep.

[0147] In some embodiments, the electronic device acquires multiple sleep-aiding data. If more than 50% of the sleep-aiding data meets the corresponding judgment criteria, the device is considered to be close to sleep. For example, the electronic device acquires the user's blink count, yawn count, and posture data over a period of time. Specifically, if the user's blink count is greater than a blink count threshold (meeting the corresponding judgment criteria), the user's yawn count is less than a yawn count threshold (not meeting the corresponding judgment criteria), and the user's posture data indicates that the user is lying flat (meeting the corresponding judgment criteria), then the electronic device can determine that the user is close to sleep based on these three types of sleep-aiding data.

[0148] In this embodiment, the electronic device can combine one or more sleep aid data to detect the user's near-sleep state. Figure 3 The flowchart of the device control method shown provides an example of using the number of blinks as the primary sleep aid data for identifying the user's approach to sleep, and using the number of yawns, heart rate, and user posture as auxiliary sleep aid data for identifying the user's approach to sleep.

[0149] It is understood that in the identification schemes for one or more combinations of sleep aid data in this application, the priority of judging sleep aid data used for identifying the user's approach to sleep is not limited. For example, the number of yawns can be used as the primary sleep aid data for identifying the user's approach to sleep, with other sleep aid data as auxiliary data. Alternatively, the user's posture can be used as the primary sleep aid data for identifying the user's approach to sleep, with other sleep aid data as auxiliary data. In some embodiments, the sleep aid data may also include other data that can characterize the user's approach to sleep.

[0150] In one example, such as Figure 4 As shown in (d), the electronic device determines the user's state by the number of blinks within a certain time period. If the number of blinks is greater than or equal to a blink count threshold, it determines that the user is in a near-sleep state. Furthermore, as... Figure 4 As shown in (e), electronic devices can also use posture data to help determine whether a user is nearing sleep. Figure 4 As shown in (e), the user is lying flat, indicating that the user is close to falling asleep.

[0151] S106, when the electronic device determines that the user is close to falling asleep, it performs the corresponding sleep aid operation.

[0152] When an electronic device determines that a user is close to falling asleep, it can perform corresponding sleep-aiding operations from multiple dimensions.

[0153] For example, electronic devices can provide a quieter environment for users and adjust volume or sound effects to help them fall asleep. For instance, if an electronic device detects media playback, it can lower the volume. Alternatively, it can play sleep-inducing sounds or audio (such as sleep stories) to help users fall asleep.

[0154] For example, an electronic device can provide a relatively dim environment to help the user fall asleep. For instance, the electronic device can dim the brightness of its display screen. Alternatively, the electronic device can switch the display mode to a specified mode. For example, the specified display mode could be an eye-protection mode, a sleep-aid mode, etc. In the specified display mode, the color temperature is warmer, the brightness is lower than in the normal display mode, the contrast ratio is lower than in the normal display mode, and the color vividness is lower than in the normal display mode.

[0155] Alternatively, in some embodiments, the electronic device can perform various sleep-aid operations based on the aforementioned dimensions to help the user fall asleep. For example, when the volume of the electronic device is turned down, the brightness of the electronic device's screen is dimmed.

[0156] In some embodiments, when the electronic device determines that the user is nearing sleep, it displays a prompt message on the screen before, during, or after performing a corresponding sleep-aiding operation. For example, Figure 5 A schematic diagram of a notification interface is provided. For example, when the electronic device is a mobile phone, the phone detects that the user's location is home and the system time is 11:30 PM, determining that they are in a resting state. While the user is watching a video, the phone detects that the user's eyes are focused on the screen, and that the user is nearing sleep. In this case, the phone outputs a notification message on the screen, informing the user that the phone has entered sleep aid mode. This notification message can be displayed in the form of a pop-up window or a floating window anywhere on the screen.

[0157] Specifically, examples of sleep-aiding operations performed by electronic devices and their output prompts will be used to illustrate how electronic devices can perform sleep-aiding operations.

[0158] For example, the sleep aid operation can be volume adjustment. For instance, if the electronic device detects that its volume exceeds a preset decibel threshold, it can first output a volume adjustment prompt message, and then lower the volume to the target decibel value. The volume adjustment prompt message serves to inform the user that the sleep aid function has been activated and the volume will be lowered. Alternatively, the electronic device can output the volume adjustment prompt message while simultaneously lowering the volume to the target decibel value. Or, the electronic device can first lower the volume to the target decibel value, and then output the volume adjustment prompt message. The volume adjustment prompt message serves to inform the user that the sleep aid function has been activated and the volume has been lowered. There are no restrictions on the execution of the sleep aid operation or the order of the prompt message output.

[0159] For example, the sleep aid function can be brightness adjustment. For instance, if the electronic device detects that the brightness of its screen exceeds a preset brightness threshold, it can first output a brightness adjustment prompt, and then lower the screen brightness to the target brightness value. The brightness adjustment prompt informs the user that the sleep aid function has been activated and the screen brightness will be dimmed. Alternatively, the electronic device can simultaneously output the brightness adjustment prompt and lower the screen brightness to the target brightness value. The brightness adjustment prompt informs the user that the sleep aid function has been activated and the screen brightness has been dimmed. Or, the electronic device can first lower the screen brightness to the target brightness value, and then simultaneously output the brightness adjustment prompt. The brightness adjustment prompt informs the user that the sleep aid function has been activated and the screen brightness has been dimmed.

[0160] For example, the sleep aid operation could be playing a specified sound effect. For instance, the electronic device could first output a sound effect playback prompt message, and then play the sleep aid sound effect at a target decibel level. The sound effect playback prompt message serves to inform the user that the sleep aid function has been activated and the sleep aid sound effect will be played soon. Alternatively, the electronic device could output the sound effect playback prompt message while simultaneously playing the sleep aid sound effect at the target decibel level. The sound effect playback prompt message serves to inform the user that the sleep aid function has been activated. Or, the electronic device could first play the sleep aid sound effect at the target decibel level, and then output the sound effect playback prompt message. The sound effect playback prompt message serves to inform the user that the sleep aid function has been activated.

[0161] For example, the sleep aid operation can be switching display modes. For instance, if the electronic device detects that the screen is in normal display mode, it can first output a prompt to switch display modes, and then switch the screen from normal display mode to eye-protection display mode. The prompt message indicates that the sleep aid function has been activated and the screen is about to switch to eye-protection display mode. Alternatively, the electronic device can output the prompt message while simultaneously switching the screen from normal display mode to eye-protection display mode. The prompt message again indicates that the sleep aid function has been activated and the screen has switched to eye-protection display mode. Or, the electronic device can first switch the screen from normal display mode to eye-protection display mode, and then output the prompt message. The prompt message again indicates that the sleep aid function has been activated and the screen has switched to eye-protection display mode.

[0162] In one example, such as Figure 4 (f) and Figure 4 As shown in (g), the electronic device can perform sleep-aiding operations by lowering its volume. While lowering the volume, the electronic device can display a volume status bar indicating the intensity of the volume adjustment. It is understood that when the electronic device adjusts the screen brightness or switches the screen display mode, etc., it will also display the corresponding brightness adjustment status bar or the icon corresponding to the display mode on the screen; this will not be elaborated upon in this embodiment.

[0163] In addition, electronic devices can also be like Figure 5 As shown, prompts for sleep aid operations are displayed in the center or anywhere on the screen.

[0164] If the electronic device determines that the user is not in a state of near-sleep, for example, when judging based on one type of sleep aid data, the sleep aid data meets one of the following conditions: the number of blinks within a certain period of time is less than a blink count threshold, or the number of yawns within a certain period of time is less than a yawn count threshold, or the heart rate value is greater than the highest value in the low-frequency range, or the posture is not in a supine or side-lying position; or when the electronic device judges based on multiple types of sleep aid data, if at least half of the data meets the corresponding conditions, then it is determined that the user is not in a state of near-sleep. If the electronic device determines that there is still eye contact with the display screen, the electronic device will continue to collect sleep aid data, but will not perform any sleep aid operations temporarily.

[0165] This application provides a device control method. In scenarios where a user needs to sleep but is engrossed in electronic devices, the electronic device can first determine whether it is in a resting state based on environmental data. Once it is determined that the electronic device is in a resting state, and it detects that a person's eyes are focused on the screen, it can further determine whether the user is nearing sleep. When it is determined that the user is nearing sleep, the electronic device can lower its volume or dim the screen brightness to remind and assist the user in falling asleep, thereby minimizing the user's immersion in the electronic device. When the user is nearing sleep, the method can help them fall asleep more easily and quickly, thus improving the user's sleep and rest quality.

[0166] The device control method provided in this application can be applied to electronic devices that deploy sleep-aid functions. Exemplarily, the electronic device can be a portable computer (such as a mobile phone), tablet computer, laptop computer, etc., and the following embodiments do not impose any special limitations on the specific form of the electronic device. It is understood that the electronic device in this embodiment provides a low-power always-on camera function. The always-on camera can acquire image data with low power consumption for image data analysis in various scenarios. For example, applicable scenarios for the always-on camera may include smart code recognition, air gestures, intelligent screen rotation based on facial recognition, eye tracking, etc. In this embodiment, the image data acquired by the always-on camera can be used to recognize behaviors such as eye gaze, blinking, and yawning.

[0167] Figure 6A schematic diagram of the structure of electronic device 600 is shown. Electronic device 600 may include processor 610, external memory interface 620, internal memory 621, universal serial bus (USB) interface 630, charging management module 640, power management module 641, battery 642, antenna 1, antenna 2, mobile communication module 650, wireless communication module 660, audio module 670, speaker 670A, receiver 670B, microphone 670C, headphone jack 670D, sensor module 680, camera 693, display screen 694, intelligent sensor hub 695, etc.

[0168] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 600. In other embodiments of this application, the electronic device 600 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0169] The processor 610 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0170] The controller can be the nerve center and command center of the electronic device 600. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0171] The processor 610 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 610 is a cache memory. This memory can store instructions or data that the processor 610 has just used or that are used repeatedly. If the processor 610 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 610, and thus improves the efficiency of the system.

[0172] In this embodiment, the processor can act as the execution entity to perform the device control method provided in the embodiments of this application. In some embodiments, the processor can also be a designated chip that provides the ability to always sense a low-power camera.

[0173] In some embodiments, the processor 610 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0174] In this embodiment, the processor 610 can adjust the brightness of the display screen through the corresponding interface; or, the processor 610 can adjust the volume of the electronic device, etc., through the corresponding interface.

[0175] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 600. In other embodiments of this application, the electronic device 600 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0176] The power management module 641 connects the battery 642, the charging management module 640, and the processor 610. The power management module 641 receives input from the battery 642 and / or the charging management module 640, providing power to the processor 610, internal memory 621, external memory, display screen 694, camera 693, and wireless communication module 660. The power management module 641 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 641 may also be located within the processor 610. In other embodiments, the power management module 641 and the charging management module 640 may be located in the same device.

[0177] The wireless communication function of the electronic device 600 can be implemented through antenna 1, antenna 2, mobile communication module 650, wireless communication module 660, modem processor, and baseband processor.

[0178] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 600 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0179] Electronic device 600 implements display functions through a GPU, a display screen 694, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 694 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. Processor 610 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0180] Display screen 694 is used to display images, videos, etc. Display screen 694 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 600 may include one or N displays 694, where N is a positive integer greater than 1.

[0181] In this embodiment, the processor can adjust the brightness of the display screen; or, the processor can switch the display mode of the display screen. For example, the display mode of the display screen may include a normal display mode, an eye-protection display mode, a night display mode, etc. In the normal display mode, the brightness of the display screen is moderate, and the display style presents a white series. For example, the background color of components is white, the background color of the status bar and notification bar is white, etc. The brightness, contrast, and color vividness of the display screen in the eye-protection display mode are lower than those in the normal display mode. In the night display mode, the brightness of the display screen is low, and the display style presents a dark series. For example, the background color of components is dark (e.g., black or black with a certain degree of transparency), the background color of the status bar and notification bar is dark, etc.

[0182] In addition, when an electronic device determines that it is about to perform a sleep aid operation, it can display a prompt message on the screen to alert the user that the operation is about to begin. Performing the sleep aid operation can involve dimming the screen brightness, switching the display mode to an eye-friendly mode (or night mode), or playing sleep-aid animations or videos on the screen.

[0183] In some embodiments, after the sleep aid function is activated, an icon indicating that the sleep aid function is activated can also be displayed in the status bar at the top of the display screen.

[0184] Electronic device 600 can achieve shooting function through ISP, camera 693, video codec, GPU, display 694 and application processor.

[0185] The ISP (Image Signal Processor) is used to process data fed back from the camera 693. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 693.

[0186] Camera 693 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, electronic device 600 may include one or N cameras 693, where N is a positive integer greater than 1.

[0187] In this embodiment, the camera of the electronic device may include at least an Always Sensing Camera (ASC) disposed under the screen. The ASC is a low-power image acquisition device, and the processor can use the images acquired by the ASC to detect human eye gaze, blinking, yawning, and so on.

[0188] In some embodiments, the camera of the electronic device may include a front-facing camera positioned on the same plane as the main display screen. The ASC may be positioned near the front-facing camera.

[0189] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals. For example, when electronic device 600 is selecting a frequency, the DSP is used to perform Fourier transforms on the frequency energy.

[0190] Video codecs are used to compress or decompress digital video. Electronic device 600 can support one or more video codecs. Thus, electronic device 600 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0191] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs can enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0192] The external storage interface 620 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 600. The external memory card communicates with the processor 610 through the external storage interface 620 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0193] Internal memory 621 can be used to store computer executable program code, which includes instructions. Processor 610 executes various functional applications and data processing of electronic device 600 by running the instructions stored in internal memory 621. Internal memory 621 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 600 (such as audio data, phonebook, etc.). Furthermore, internal memory 621 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0194] In this embodiment, the images acquired by the ASC can be stored in a designated storage space for data analysis in different dimensions. For example, eye gaze detection can be performed based on multiple images containing facial information, blink count can be identified based on multiple images containing facial information, yawn count can be identified based on multiple images containing facial information, and so on.

[0195] Electronic device 600 can implement audio functions, such as music playback and recording, through audio module 670, speaker 670A, receiver 670B, microphone 670C, headphone jack 670D, and application processor.

[0196] The audio module 670 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 670 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 670 may be located in the processor 610, or some functional modules of the audio module 670 may be located in the processor 610.

[0197] The speaker 670A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. Electronic device 600 can listen to music or make hands-free calls through the speaker 670A.

[0198] In this embodiment, the electronic device performs a sleep-aid operation, which can play sleep-aid sounds, sleep-aid stories, sleep-aid videos, etc. The electronic device can play sleep-aid sounds, sleep-aid stories, sleep-aid videos, etc. through a speaker.

[0199] The receiver 670B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 600 answers a telephone call or voice message, the receiver 670B can be brought close to the ear to listen to the voice.

[0200] Microphone 670C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 670C, inputting the sound signal into microphone 670C. Electronic device 600 may have at least one microphone 670C. In some embodiments, electronic device 600 may have two microphones 670C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 600 may also have three, four, or more microphones 670C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0201] The headphone jack 670D is used to connect wired headphones. The headphone jack 670D can be a USB 630 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0202] In this embodiment, the sensor module 680 may include a gyroscope sensor 680A, a barometric pressure sensor 680B, an accelerometer sensor 680C, an ambient light sensor 680D, a bone conduction sensor 680E, etc.

[0203] The gyroscope sensor 680A can be used to determine the motion attitude of the electronic device 600. In some embodiments, the gyroscope sensor 680A can determine the angular velocity of the electronic device 600 about three axes (i.e., the x, y, and z axes). The gyroscope sensor 680A can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 680A detects the angle of the shake of the electronic device 600, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the electronic device 600 by moving in the opposite direction, thus achieving image stabilization. The gyroscope sensor 680A can also be used in navigation and motion-sensing game scenarios.

[0204] The barometric pressure sensor 180B is used to measure air pressure. In some embodiments, the electronic device 600 calculates altitude using the air pressure value measured by the barometric pressure sensor 180B to assist in positioning and navigation.

[0205] The accelerometer 180C can detect the magnitude of acceleration of electronic device 600 in various directions (typically three axes). When electronic device 600 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic device, and can be applied to applications such as screen orientation switching and pedometers.

[0206] In this embodiment, the electronic device can obtain attitude data through the gyroscope sensor 680A, the barometric pressure sensor 680B, and the accelerometer sensor 680C, thereby determining whether the user of the handheld electronic device is in a side-lying or flat-lying state.

[0207] The ambient light sensor 680D is used to sense the brightness of ambient light. The electronic device 600 can adaptively adjust the brightness of the display screen 694 based on the sensed ambient light level. The ambient light sensor 680D can also be used to automatically adjust the white balance when taking a picture. The ambient light sensor 680D can also work in conjunction with a proximity sensor (if present) to detect whether the electronic device 600 is in a pocket, preventing accidental touches.

[0208] In this embodiment, when the electronic device is in normal display mode, it can sense the ambient brightness through the ambient light sensor 680D and adaptively adjust the brightness of the display screen. If the electronic device switches the display mode to eye-protection display mode or night display mode, the function of adaptively adjusting the display screen brightness can be temporarily disabled by default.

[0209] In this embodiment, when the electronic device performs a sleep aid operation, the ambient light sensor 180D can sense the ambient light brightness and adjust the brightness of the display screen to an appropriate brightness (darkening the preset brightness or dimming the target brightness value).

[0210] The bone conduction sensor 680E can acquire vibration signals. In some embodiments, the bone conduction sensor 680E can acquire vibration signals from the vibrating bone segments of the human vocal cords. The bone conduction sensor 680E can also contact the human pulse to receive blood pressure signals. In some embodiments, the bone conduction sensor 680E can also be incorporated into headphones to form bone conduction headphones. The audio module 670 can parse the voice signals from the vibrating bone segments of the vocal cords acquired by the bone conduction sensor 680E to realize voice functionality. The application processor can parse heart rate information from the blood pressure signals acquired by the bone conduction sensor 680E to realize heart rate detection functionality.

[0211] In this embodiment, the electronic device can obtain the user's heart rate value through the bone conduction sensor 680E and identify the user's approaching sleep state based on the heart rate value.

[0212] In this embodiment, the electronic device includes a smart sensor hub 695 (SensorHub). The main function of the SensorHub is to connect and process data from various sensor modules. In this embodiment, the electronic device can acquire images collected by the ASC through the SensorHub and perform further detection and analysis on the images.

[0213] The software system of the electronic device 600 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses a layered architecture. Taking the system as an example, the software structure of electronic device 600 is illustrated.

[0214] Figure 7 This is a software structure block diagram of an electronic device 600 according to an embodiment of the present invention.

[0215] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, [the following is omitted as the text is incomplete and likely refers to a specific implementation or feature]. The system generally consists of three parts: the Application Processor (AP) side, the Sensor Hub side, and the hardware. The AP side includes the application layer (APPs), the framework layer (FWK), native libraries, and the hardware abstraction layer (HAL).

[0216] The apps can include multiple business applications. For example, a sleep aid application could be included. This sleep aid application can be a third-party application or a system application. Under certain scenarios, at certain times, and under certain human factors, the sleep aid application can decide whether the user's current state requires immediate sleep. If so, it will take certain measures to help the user quickly enter a sleep state, thus achieving the purpose of sleep aid. Specifically, the sleep aid application can identify rest scenarios, human eye gaze, and the state of near-sleep, and perform corresponding sleep aid operations.

[0217] The sleep aid app can collect both location and time data. Location data collection refers to capturing the location of the electronic device and the user's surroundings. In this embodiment, the location of the surroundings primarily includes identifying the user's location at home and at work. Time data collection refers to capturing the time periods of the user's daily activities, such as nighttime sleep from 9:00 PM to 6:00 AM and midday rest from 12:00 PM to 2:00 PM. Specifically, the electronic device determines whether a resting scenario is present based on the collected location and time data.

[0218] Among these features, sleep aid apps can collect gaze data. Specifically, sleep aid apps can register gaze fences (EGFs) and acquire images containing facial information collected by the ASC (Automatic Screen) to perform gaze recognition, identifying whether the user is looking at the screen. The acquired images containing facial information constitute the gaze data.

[0219] Among these applications, sleep aid apps can collect blink data. Specifically, sleep aid apps can register an eye blink fence (EBF) to obtain blink data. This blink data can include the number of blinks made by the user within a certain time period.

[0220] Among these applications, sleep aids can collect yawning data. Specifically, sleep aids can register yawn fences (YF) to obtain yawning data. This yawning data can include the number of times a user yawns within a certain time period.

[0221] Sleep aid apps can also collect other data. This other data can be auxiliary information that helps identify the user's approach to sleep. For example, a sleep aid app can collect the posture data of the electronic device to determine whether the user is lying flat or on their side, and then identify the user's state based on the posture data. Alternatively, a sleep aid app can collect the user's heart rate value and then identify the user's state based on the heart rate value. Finally, a sleep aid app can collect ambient noise data and then identify the user's state based on the ambient noise.

[0222] Sleep aid apps can identify the user's state. Specifically, after the electronic device collects at least one of the following data: blinking data, yawning data, and other data, it can determine whether the user is in a near-sleep state. For example, if the electronic device determines that the number of blinks is greater than or equal to a blinking threshold, it determines that the user is in a near-sleep state. Or, if the electronic device determines that the number of yawns is greater than or equal to a yawning threshold, it determines that the user is in a near-sleep state. Or, if the electronic device determines that both the number of blinks and yawns are greater than or equal to a blinking threshold, it determines that the user is in a near-sleep state.

[0223] Sleep aid apps can adjust brightness and / or volume. For example, a sleep aid app can communicate with the display screen to adjust the screen brightness during sleep aid operations. Alternatively, it can connect to the system backlight brightness to adjust the overall backlight brightness of the electronic device's display. Similarly, a sleep aid app can communicate with the speaker to adjust the volume during sleep aid operations. Or, it can connect to the system audio to adjust the overall audio output volume and sound effects of the electronic device.

[0224] A awareness platform is deployed in the FWK on the AP side. The awareness platform includes the awareness platform software framework (referred to as the awareness framework), gaze fence, blink fence, and yawn fence.

[0225] The sleep aid app can send a registration gaze fence command to the perception platform to register the gaze fence; the sleep aid app can send a registration blink fence command to the perception platform to register the blink fence; the sleep aid app can send a registration yawn fence command to the perception platform to register the yawn fence.

[0226] The sleep aid application can specify an interface to communicate with the perception platform for operations such as scene recognition and fence registration. The specified interface can be the software development kit (SDK) interface (AwarenessSDK) provided by the perception platform module.

[0227] The local library on the AP side deploys the AIDL interface (sleep aid interface) provided by the sleep aid function (same as the Hion service). The Hion AIDL interface provides control commands and data interfaces for registering, unregistering (removing) gaze fences, blink fences, and yawn fences. Simultaneously, the Hion AIDL interface can also be used to report gaze detection results, blink detection results, and yawn detection results. The Hion AIDL interface also provides capabilities for adding, deleting, and event callbacks for blink fences. Specifically, a separate local blink service can also be deployed in the local library to support the interface between the blink application and the HAL layer. A separate local yawning service can also be deployed in the local library to support the interface between the yawning application and the HAL layer.

[0228] The HAL layer on the AP side deploys a sleep-aid HAL interface (sleep-aid algorithm HAL interface, same as the Hiaon HAL interface). The Hiaon HAL interface enables connection and communication with the application-specific signal processing system on the SensorHub side via a designated interface (Qualcomm messaging interface, QMI). This supports control commands and data transmission capabilities for blink detection, gaze detection, and yawn detection.

[0229] The SensorHub side includes an application-specific digital signal processor (ADSP), also known as an ADSP system. The ADSP system can process instructions issued by the application and data uploaded by the hardware.

[0230] The ADSP system deploys a sleep aid algorithm driver (Hiaon driver, same as the sleep aid detection algorithm module). The Hiaon driver provides the ADSP system-side gaze fence detection framework and gaze algorithm (Gaze Algo), blink fence detection framework and blink algorithm (Blink Algo), and yawn fence detection framework and yawn algorithm (Yawn Algo).

[0231] The Hiaon driver is used to acquire images captured by the ASC camera driver for gaze, blink, and yawn event recognition, and to report the recognition results to the Hiaon HAL via the QMI interface.

[0232] The ASC camera driver is a low-power sensing camera driver provided for a specified platform. The ASC camera driver can acquire grayscale image data captured by ASC at a specified frame rate.

[0233] The gaze detection algorithm module (GAD) can analyze continuous image data containing facial information (e.g., 30 frames) collected by the ASC, combined with eye keypoint features, and perform a series of algorithmic calculations to output the detection result of whether a person is gazing, thus outputting the gaze event. For example, Gaze Algo can perform face detection on the collected images using face detection algorithms (e.g., algorithms based on Haar feature classifiers). After determining that a face is present in the collected images, it can extract eye features, obtain the gaze detection result based on the eye features, and output the gaze event.

[0234] The blink detection algorithm module (Blink Algo) can analyze continuous image data containing facial information (e.g., 30 frames) collected by ASC, combined with key eye features, and perform a series of algorithmic calculations to output the number of blinks and the effective detection duration (blink detection result), thus outputting a blink event. For example, Blink Algo can use face detection algorithms (such as algorithms based on Haar feature classifiers) to perform face detection on the collected images. After confirming that the collected images contain faces, it can extract eye features, obtain blink detection results based on the eye features, and output blink events. Simply put, eye features can include the distance between the upper and lower boundaries of the eyes. When the distance is less than a certain threshold, it is considered that the eyes are closed. When the distance between the upper and lower boundaries of the eyes is greater than or equal to a certain threshold, it is considered that the eyes are open. If an open-close-open sequence occurs in multiple consecutive images, a blink event is determined to have occurred. In some embodiments, after obtaining eye features, the blinking Algo can perform blink recognition using a machine learning model (such as a convolutional neural network) to obtain blink recognition results.

[0235] The Yawning Algo (yawn detection algorithm module) can analyze continuous image data containing facial information (e.g., 30 frames) collected by ASC, combined with mouth keypoint features, and perform a series of algorithmic calculations to output the number of yawns and the effective detection duration (yawn detection result), thus outputting a yawn event. For example, Yawning Algo can use face detection algorithms (such as algorithms based on HAAR feature classifiers) to perform face detection on the collected images. After confirming that the collected images contain faces, it can extract mouth features, obtain yawn detection results based on mouth features, and output a yawn event. Simply put, mouth features can include the distance between the upper and lower boundaries of the mouth. When the distance is greater than a certain threshold, it is considered that an open mouth behavior has occurred. When the distance between the upper and lower boundaries of the mouth is less than or equal to a certain threshold, it is considered that a closed mouth behavior has occurred. If a closed-open-closed behavior occurs in multiple consecutive images, then a yawn behavior is determined to have occurred. In some embodiments, after obtaining mouth features, Algo can perform yawn recognition using a machine learning model (such as a convolutional neural network) to obtain yawn recognition results.

[0236] The Hiaon driver can report eye gaze events, blink events, and / or yawning events to the HAL layer via QMI. This reporting then ascends through layers to the sleep aid application, which performs corresponding actions based on the received events. For example, upon receiving an eye gaze event, the sleep aid application determines that the user's eyes are focused on the display screen and acquires sleep aid data for user status identification. Similarly, upon receiving a blink event (including blink data), the application acquires the number of blinks and their effective duration. Based on a blink count threshold, it identifies the user's status; if the number of blinks within the effective duration is greater than or equal to the threshold, the user is determined to be near sleep. Likewise, upon receiving a yawn event (including yawn data), the application acquires the number of yawns and their effective duration. Based on a yawn count threshold, it identifies the user's status; if the number of yawns within the effective duration is greater than or equal to the threshold, the user is determined to be near sleep.

[0237] In this embodiment, the Hion driver on the SensorHub side deploys a gaze fence detection framework and gaze algorithm, enabling the sleep aid application to effectively identify continuous user use of electronic devices. The blink fence detection framework and blink algorithm, and the yawn fence detection framework and yawn algorithm in the Hion driver enable the sleep aid application to effectively identify whether the user is nearing sleep.

[0238] The hardware layer includes the ASC (Automatic Screen), display screen, and speakers. The ASC is used for image acquisition. The images acquired by the ASC can be grayscale images. This provides effective image data for the Hiaon driver to perform eye gaze detection, blink detection, and yawn detection.

[0239] In this embodiment, the various algorithm modules (Gaze Algo, Blink Algo, Yawning Algo) in the Hion driver are used to obtain gaze detection results, blink detection results, and yawn detection results based on image data collected by ASC. The algorithm modules are deployed on the Sensorhub side and can run in low-power mode, avoiding the high power consumption problem caused by using the front or rear main camera to collect image data. This enables the collection of sleep aid data and the detection of the user's approaching sleep state in low-power mode.

[0240] The sleep aid operations performed by the sleep aid application can also be extended according to the smart functions provided by the electronic device. In this embodiment, the form and specific content of the sleep aid operations provided by the sleep aid application are not limited.

[0241] In some embodiments, Figure 8 A schematic diagram of the communication mechanism of an important module of an electronic device is given.

[0242] Among them, the key modules of the electronic device include the sleep aid application in the application layer (APPs), the perception platform in the framework layer, Hiaon HAL in the HAL layer, Hiaon driver on the SensorHub side, and ASC camera driver on the SensorHub side.

[0243] The sleep aid application communicates with the perception platform via the AwarenessSDK interface to register different types of geofencing and issue relevant control commands in the device control method. The perception platform can report relevant events in the device control method to the sleep aid application through the AwarenessSDK interface. For example, events may include eye gaze events, blinking events, and yawning events, etc.

[0244] The perception platform communicates with Hion HAL via the Hion AIDL interface to issue relevant control commands in the device control method. Hion HAL can report events and relevant data from the device control method to the perception platform through the Hion AIDL interface.

[0245] Hiaon HAL communicates with Hiaon driver via the QMI interface to issue relevant control commands in the device control method. Hiaon driver can return relevant data in the device control method to Hiaon HAL through the QMI interface, or Hiaon driver can package relevant data into events and report them to Hiaon HAL through the QMI interface.

[0246] The Hiaon driver communicates with the ASC camera driver via the QMI interface to acquire image data captured by the ASC camera and to issue relevant control commands in the device control method. The ASC camera driver can return image data to the Hiaon driver via the QMI interface.

[0247] Combining the communication mechanisms between key modules of electronic devices, Figure 9 A timing diagram illustrating the execution of each module in a device control method is provided. Key modules include the sleep aid application, the perception platform, system services, the Hion driver (same as the sleep aid algorithm driver), and ASC. The system services include the Hion AIDL interface provided by the local library and the Hion HAL provided by the HAL layer. These system services enable the issuance of relevant commands and the reporting of relevant data / events.

[0248] Specifically, Figure 9 The device control methods shown include:

[0249] S201, the sleep aid application detects whether the sleep aid function is enabled.

[0250] The embodiments provided in S101 above can be referred to, and will not be repeated in this embodiment.

[0251] S202, when the sleep aid function is activated, the sleep aid application initiates location detection in the rest scene recognition.

[0252] The embodiments provided in S103 above can be referred to, and will not be repeated in this embodiment.

[0253] S203, if the sleep aid application determines that it is in a specified location within the rest scene, the sleep aid application initiates time detection in the rest scene recognition.

[0254] The embodiments provided in S103 above can be referred to, and will not be repeated in this embodiment.

[0255] When a sleep aid app determines a specific location and corresponding rest period, it can identify whether the user is continuously using electronic devices and the user's state.

[0256] In some feasible implementations, sleep aid apps can also initiate heart rate detection before performing eye gaze recognition. Based on this, the sleep aid app can perform:

[0257] S204, If the sleep aid application determines that the user is in the rest period corresponding to the specified location, start detecting the user's heart rate value.

[0258] In one scenario, if the user's heart rate is detected via other external devices (such as smart bracelets or smartwatches), the sleep aid app can initiate communication with these devices after determining the user's rest period corresponding to a specific location. It then acquires the user's heart rate data from these devices at a certain frequency and stores it in designated storage space for later use in user status identification. Alternatively, in another scenario, if the electronic device can directly collect the user's heart rate, the sleep aid app can call the appropriate interface to initiate the heart rate acquisition process, collecting the heart rate data at a certain frequency and storing it in designated storage space for later use in user status identification.

[0259] In some feasible implementations, sleep aid applications can also initiate user posture recognition before eye gaze recognition. Based on this, the sleep aid application can perform:

[0260] S205, If the sleep aid application determines that the user is in the rest period corresponding to the specified location, the user's posture detection is initiated.

[0261] In this embodiment, the sleep aid application can acquire the posture data of the electronic device through built-in sensors, such as gyroscope sensors and accelerometer sensors, thereby detecting the user's posture.

[0262] Understandably, if it is detected that the user is not lying flat or on their side, and it is determined that no sleep aid operation is needed, then the steps of human eye gaze recognition, user status recognition, and execution of sleep aid operation can be omitted.

[0263] The heart rate detection in S204 and user posture detection in S205 are performed before eye gaze recognition because these detections do not require data analysis of image data acquired via the ASCII interface. The sleep aid application can directly obtain heart rate or posture data through the corresponding communication interface. Therefore, the sleep aid application can execute S204 and / or S205 before eye gaze recognition. Alternatively, the sleep aid application can determine the presence of eye gaze behavior and obtain sleep aid data (e.g., blinking events, yawning events) before executing S204 and / or S205. The heart rate and / or user posture data are then synchronously used as sleep aid data for user status recognition. In this embodiment, the specific execution order of S204 and S205 is not strictly limited.

[0264] Sleep aid apps perform eye gaze recognition, including:

[0265] S206, the sleep aid application sends a registration gaze fence instruction to the Hion driver through the perception platform and system services.

[0266] Specifically, the sleep aid application can send a gaze fence registration command (addEGF) to the gaze fence detection framework driven by Hion through a communication interface, layer by layer.

[0267] S207, the Hion driver receives the register gaze fence instruction, acquires the first user image collected by ASC, and obtains the gaze detection result.

[0268] The gaze fence detection framework in the Hiaon driver responds to the gaze fence registration command by sending image acquisition commands to the ASC camera driver and acquiring the first user image acquired by the ASC at a certain frequency.

[0269] Here, the first user image refers to an image containing facial information. In this embodiment, the ASC is a low-power always-on camera, and the images acquired by the ASC are generally grayscale images. This embodiment can perform eye gaze recognition based on grayscale images.

[0270] Specifically, the Hiaon-driven gaze Algo can acquire first user images containing facial information collected by the ASC at a certain frequency. After acquiring a certain number of first user images, gaze Algo can perform human eye gaze recognition based on multiple first user images. This "certain number" can be 20 or 30 frames of first user images. Gazing Algo can extract eye keypoint features from each frame of the first user image, and based on the changes in eye keypoint features across multiple consecutive frames, determine whether human eye gaze behavior exists in those frames and obtain the gaze detection result.

[0271] The embodiments provided in S104 above can be referred to, and will not be repeated in this embodiment.

[0272] S208: After obtaining the gaze detection results, the Hiaon driver generates and reports the human eye gaze event.

[0273] The eye-gazing event can include gaze detection results and information indicating whether eye-gazing behavior exists. For example, the eye-gazing event includes an eye-gazing indicator. When the eye-gazing indicator is at a first value, it indicates that eye-gazing behavior exists; when the eye-gazing indicator is at a second value, it indicates that eye-gazing behavior does not exist.

[0274] Human eye gaze events can be reported up the chain of command to sleep aid applications through system services and a perception platform.

[0275] S209, Sleep aid applications determine the presence of eye-gazing behavior based on eye-gazing events.

[0276] In this embodiment, the sleep aid application can determine whether human eye gazing behavior exists based on the gaze detection results included in the human eye gazing event.

[0277] S210, if the sleep aid application determines that there is eye-gazing behavior, it sends a command to Hion driver to remove the gaze fence through the perception platform and system services.

[0278] In this embodiment, if the sleep aid application determines that there is eye-gazing behavior, it can pause the detection of eye-gazing and initiate user status recognition based on sleep aid data (such as the number of blinks and yawns). Pausing the detection of eye-gazing can be done by removing the gaze fence, thus preventing the execution of the eye-gazing detection operation. In this case, the sleep aid application can send a gaze fence removal command to the Hion driver through the perception platform and system services.

[0279] The Hiaon-driven gaze fence detection framework responds to the gaze fence removal command by removing the gaze fence and pausing the detection of human eye gaze by acquiring the first user image captured by ASC.

[0280] In some other embodiments, if the sleep aid application determines that there is no eye-gazing behavior, the sleep aid application may continue to detect eye-gazing, but temporarily does not trigger the collection of sleep aid data or the identification of user status.

[0281] When a sleep aid app detects eye-gazing behavior, it collects sleep aid data and identifies the user's state (near sleep state). This sleep aid data includes blink count, among other things.

[0282] S211, the sleep aid application sends a blink fence registration command to the Hion driver through the perception platform and system services.

[0283] Specifically, sleep aid applications can send registration blink fence commands (addEBF) to the blink fence detection framework driven by Hion through a communication interface, layer by layer.

[0284] In some embodiments, when the sleep aid data includes the number of blinks and yawns, the sleep aid application may simultaneously perform a yawn fence registration operation. Alternatively, when the sleep aid data includes the number of yawns, it may include:

[0285] S212, the sleep aid application sends a yawning fence registration command to the Hion driver through the perception platform and system services.

[0286] Specifically, the sleep aid application can send the yawn fence registration command (addYF) to the yawn fence detection framework driven by Hion through the communication interface, layer by layer.

[0287] S213, the Hion driver receives the registered blink fence instruction / register yawn fence instruction, continuously collects image data, and obtains the detection results.

[0288] In one scenario, the blink fence detection framework in the Hiaon driver responds to the registered blink fence command by sending an image acquisition command to the ASC via the ASC camera driver, acquiring second user images captured by the ASC within a certain effective duration at a certain frequency. For example, the effective duration can be 1 minute. That is, it acquires second user images containing facial information acquired within 1 minute. In this embodiment, the ASC is a low-power always-on camera, and the second user images acquired by the ASC are grayscale images. This embodiment can perform blink recognition and blink count statistics based on grayscale images.

[0289] Specifically, the Hiaon-driven blinking Algo can acquire second user images containing facial information collected by the ASC within a valid timeframe. The gaze Algo can perform blink recognition and count the number of blinks based on multiple second user images. Specifically, the gaze Algo can extract key eye features from each frame of the second user image, and based on the changes in these key eye features across multiple consecutive frames, determine whether blinking occurs in those frames, count the number of blinks, and obtain blink detection results.

[0290] For example, the execution logic block diagram of the blink detection algorithm Algo (same as the blink algorithm) for obtaining blink detection results can be found in the following example. Figure 10 As shown, it includes:

[0291] S11, the blink algorithm acquires the second user image captured by the constantly sensing camera.

[0292] S12, the blinking algorithm performs face detection on the second user image to obtain the target second user image containing facial information.

[0293] S13, the blinking algorithm extracts key eye features from multiple target second user images to obtain the eye features corresponding to each target second user image.

[0294] S14, the blinking algorithm performs blinking recognition based on the eye features of the target second user images that are adjacent in time to determine whether blinking behavior exists.

[0295] S15, if the blinking algorithm determines that blinking behavior exists, count the number of blinks to obtain the blinking detection result.

[0296] In this embodiment, the blinking algorithm can employ computer vision technology to determine whether blinking behavior exists. This can be understood as analyzing eye features, such as changes in pixel values ​​within the eye, to determine if blinking behavior exists. However, it should be noted that in actual implementation, there are multiple methods for blink detection. The specific method chosen depends on actual needs, and this embodiment does not limit this selection.

[0297] In another scenario, the yawning fence detection framework in the Hion driver responds to the yawning fence registration command by sending an image acquisition command to the ASC via the ASC camera driver. It then acquires third-user images captured by the ASC within a certain effective timeframe at a certain frequency. For example, the effective timeframe can be one minute. That is, it acquires third-user images containing facial information captured within one minute. In this embodiment, the ASC is a low-power, always-on camera, and the third-user images acquired by the ASC are grayscale images. This embodiment can perform blink recognition and blink count statistics based on the grayscale images.

[0298] Specifically, the Hiaon-driven Yawning Algo can acquire third-party user images containing facial information collected by ASC within a reasonable timeframe. The Yawning Algo can perform yawn recognition and count the number of yawns based on multiple third-party user images. Specifically, the Yawning Algo can extract key mouth features from each frame of the third-party user image. Based on the changes in these key mouth features across multiple consecutive frames, it determines whether yawning occurs in those frames and counts the number of yawns to obtain the yawn detection results.

[0299] For example, the algorithm execution logic block diagram of Yawning Algo (same as the yawning algorithm) for obtaining yawn detection results can be referred to as follows. Figure 11 As shown, it includes:

[0300] S21, the yawning algorithm obtains the third user image captured by the constantly sensing camera.

[0301] S22, the yawning algorithm performs face detection on the third user image to obtain the target third user image containing facial information.

[0302] S23, the yawning algorithm extracts key eye features from multiple target third-user images to obtain mouth features corresponding to each target third-user image.

[0303] S24, the yawning algorithm performs blink recognition based on the mouth features of the target second user images that are adjacent in time to determine whether yawning behavior exists.

[0304] One instance of yawning is defined as opening the mouth, pausing for a certain period of time, and then closing it once.

[0305] S25. If the yawning algorithm determines that yawning behavior exists, count the number of yawns to obtain the yawning detection result.

[0306] In this embodiment, Algo can use computer vision technology to determine whether a yawning behavior exists. It can be understood that this is achieved by analyzing mouth features, such as changes in the position of the upper lip, the lower lip, and the shape of the mouth. However, it should be noted that there are various methods for yawn detection in actual implementation. The specific method chosen depends on the actual needs, and this embodiment does not limit this.

[0307] After obtaining the blink detection results in the Hiaon driver:

[0308] S214, the Hiaon driver generates and reports blink events based on blink detection results.

[0309] Blinking events can be reported up the chain of command to the sleep aid application through system services and the perception platform.

[0310] The blink event can include blink detection results, which can include the number of blinks.

[0311] Alternatively, in other feasible embodiments, the blink detection result may not include the number of blinks, but it may include the detected blinking behavior. The Hiaon driver can report blink events to the perception platform, which then counts the blink behaviors included in the blink detection result of the blink event to obtain the blink count. Based on the blink count, a new blink event is generated and reported to the sleep aid application.

[0312] After the Hiaon driver obtains the yawn detection results:

[0313] S215, the Hion driver generates and reports yawning events based on yawning detection results.

[0314] Yawning events can be reported up the chain of command to sleep aid applications through system services and a perception platform.

[0315] The yawning event can include yawning detection results, which can include the number of yawns.

[0316] Alternatively, in other feasible embodiments, the yawn detection result may not include the number of yawns, but it may include the detected yawning behavior. The Hiaon driver can report yawning events to the perception platform, which then counts the number of yawning behaviors included in the yawn detection result of the yawning event to obtain the yawn count. Based on the yawn count, a new yawning event is generated and reported to the sleep aid application.

[0317] S216, The sleep aid application uses sleep aid data to identify the user's state and determine whether the user is close to falling asleep.

[0318] Among them, sleep aid data refers to the number of blinks included in blink events (the number of blinks within the effective duration) and / or the number of yawns included in yawn events (the number of yawns within the effective duration).

[0319] The sleep aid application identifies the user's state based on sleep aid data to determine whether the user is close to falling asleep. Specifically, please refer to the embodiment provided in S105 above, which will not be repeated in this embodiment.

[0320] S217, the sleep aid application determines that the user is close to falling asleep and performs the corresponding sleep aid operation.

[0321] The sleep aid application performs the corresponding sleep aid operation. Specifically, please refer to the embodiment provided in S106 above, which will not be repeated in this embodiment.

[0322] After performing the corresponding sleep aid operations, to avoid the resource consumption and performance waste of electronic devices caused by blinking and yawning fences, and to avoid performance waste and redundant image data storage caused by continuous ASC image data acquisition, the sleep aid application can issue commands to remove the blinking fence and yawning fence. Specifically:

[0323] S218, the sleep aid application sends a command to Hion driver to remove the blink fence through the perception platform and system services.

[0324] S219, the sleep aid application sends a command to Hion driver to remove the yawning fence through the perception platform and system services.

[0325] In this embodiment, once the sleep aid application determines that it will perform a sleep aid operation, it no longer needs to collect sleep aid data or identify the user's state. In this case, the sleep aid application can send commands to remove the blink fence and remove the yawn fence to the Hion driver through the perception platform and system services, thereby stopping the collection of sleep aid data (number of blinks and yawn data).

[0326] After receiving the commands to remove the blink fence and the yawn fence, the Hion driver sends a stop image acquisition command to the ASC camera driver via the S220 to make the ASC stop acquiring images.

[0327] In this embodiment, after determining that it is unnecessary to collect sleep aid data (especially blink counts and yawn counts obtained from image data), the Hion driver sends a stop image acquisition command to the ASC via the ASC camera driver, causing the ASC to enter a state of paused image data acquisition. This reduces the power consumption caused by the ASC acquiring image data.

[0328] In this embodiment, for a user who needs to sleep but is engrossed in an electronic device, the electronic device can first determine whether it is in a resting state based on environmental data. Once it is determined that the electronic device is in a resting state, and it detects that the user is looking at the screen, it can further determine whether the user is nearing sleep. When it is determined that the user is nearing sleep, the electronic device can lower its volume or dim the screen brightness to remind and assist the user to fall asleep, thus minimizing the user's immersion in the electronic device. Being near sleep can help the user fall asleep more easily and quickly, thereby improving the user's sleep and rest quality. Furthermore, the electronic device adds a sleep aid function to its user-friendly features, enhancing the user experience.

[0329] Figure 12 A possible structural schematic diagram of the electronic device involved in the above embodiments is shown. Figure 12 The electronic device 1200 shown includes a processing module 1201, an image acquisition module 1202, a display module 1203, and a storage module 1204.

[0330] The processing module 1201 may be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The processor may include an application processor and a baseband processor. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0331] For example, the processing module 1201 can be as follows: Figure 6 The processor 610 shown; the image acquisition module 1202 can be as follows: Figure 6 The ASCII display module 1203 shown can be as follows: Figure 6 The display screen 694 shown; the storage module 1204 can be as follows Figure 6 The internal memory 621 shown. The electronic device provided in this application embodiment can be Figure 6 The electronic device shown is 600.

[0332] This application also provides a chip system (e.g., a system-on-a-chip (SoC), or a specified chip), such as... Figure 13 As shown, the chip system includes at least one processor 1301 and at least one interface circuit 1302. The processor 1301 and the interface circuit 1302 are interconnected via lines. For example, the interface circuit 1302 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 1302 can be used to send signals to other devices (e.g., the processor 1301 or the camera of an electronic device). Exemplarily, the interface circuit 1302 can read instructions stored in the memory and send those instructions to the processor 1301. When the instructions are executed by the processor 1301, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete components, and this application embodiment does not specifically limit this.

[0333] This application also provides a computer-readable storage medium including computer instructions that, when executed on the electronic device, cause the electronic device to perform various functions or steps performed by the electronic device 600 in the above method embodiment.

[0334] This application also provides a computer program product that, when run on a computer, causes the computer to perform the various functions or steps performed by the electronic device 600 in the above method embodiments. For example, the computer may be the aforementioned electronic device 600.

[0335] It should be noted that the personal information used in the technical solution of this application (such as image data, sleep aid data, location information, etc.) is limited to information with individual consent, including but not limited to notifying and reminding users to read the relevant user agreement (notification) and sign the agreement (authorization) which includes the authorization of relevant user information before users use the function.

[0336] The technical solutions disclosed in this application involve the collection, storage, use, processing, transmission, provision, and disclosure of users' personal information, all of which comply with relevant laws and regulations and do not violate public order and good morals.

[0337] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0338] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0339] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0340] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0341] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0342] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A device control method, characterized in that, Applied to electronic devices, the method includes: When the sleep aid function is activated, the electronic device acquires environmental data; If the electronic device detects that the environmental data indicates a resting scenario, and the electronic device detects that the user's eyes are focused on the display screen, the electronic device acquires the user's sleep aid data; Based on the sleep aid data, when the user is nearing sleep, the electronic device performs a corresponding sleep aid operation; the sleep aid operation includes lowering the volume of the electronic device and / or dimming the brightness of the display screen.

2. The method according to claim 1, characterized in that, The sleep aid data includes the number of blinks. The electronic device determines that the user is in a near-sleep state based on the sleep aid data, including: If the electronic device detects that the number of blinks by the user is greater than or equal to the blink count threshold within a preset detection period.

3. The method according to claim 2, characterized in that, The sleep aid data also includes the number of yawns. The electronic device, based on the sleep aid data, determines that the user is in a near-sleep state, and further includes: The number of times the user yawns within the preset detection time is greater than or equal to the yawning frequency threshold.

4. The method according to claim 2 or 3, characterized in that, The sleep aid data also includes heart rate values. The electronic device, based on the sleep aid data, determines that the user is in a near-sleep state, and further includes: During the preset detection period, the user's heart rate value remains within the preset heart rate range.

5. The method according to any one of claims 2-4, characterized in that, The sleep aid data also includes posture data. The electronic device, based on the sleep aid data, determines that the user is in a near-sleep state, and further includes: Within the preset detection time, the posture data indicates that the user is in a supine or side-lying posture.

6. The method according to any one of claims 1-5, characterized in that, The environmental data includes the location and system time of the electronic device; The electronic device detects environmental data indicating a resting scenario, including: The electronic device detects that its location is at a preset position, and the system time is within a specified time period corresponding to the preset position; Wherein, when the preset location is the user's home, the specified time period corresponding to the preset location is the nighttime rest period; when the preset location is the user's company, the specified time period corresponding to the preset location is the midday rest period.

7. The method according to any one of claims 1-6, characterized in that, The electronic device includes a sleep aid application, a sensing platform, a smart sensor hub (SensorHub), and an always-on camera (ASC). The smart sensor hub includes a sleep aid detection algorithm module and an ASC camera driver. The sleep aid detection algorithm module communicates with the always-on camera (ASC) through the ASC camera driver. The perception platform provides a communication interface for the sleep aid application to communicate with the intelligent sensor hub (SensorHub). The method further includes: The sleep aid application registers a gaze fence through the communication interface, and the sleep aid application sends a first instruction to the sleep aid detection algorithm module; The sleep aid detection algorithm module responds to the first instruction and acquires a first user image captured by the always-sensing camera ASC through the ASC camera driver; the first user image includes multiple user images captured within a first acquisition time. The sleep aid detection algorithm module extracts features from the first user image to obtain the eye features corresponding to the first user image. The sleep aid detection algorithm module performs human eye gaze recognition based on the eye features corresponding to the first user image and obtains the human eye gaze recognition result. The sleep aid detection algorithm module returns the human eye gaze recognition result to the sleep aid application through the communication interface; When the electronic device detects that a person's eyes are gazing at the display screen, the electronic device acquires the user's sleep aid data, including: If the sleep aid application determines that the eye fixation result indicates that the eye is fixating on the display screen, the electronic device acquires the user's sleep aid data.

8. The method according to claim 7, characterized in that, The sleep aid data includes the number of blinks. The method further includes: The sleep aid application registers a blink fence through the communication interface, and the sleep aid application sends a second instruction to the sleep aid detection algorithm module; The sleep aid detection algorithm module responds to the second instruction and acquires a second user image captured by the always-sensing camera ASC through the ASC camera driver; the second user image includes multiple user images captured within the second acquisition time. The sleep aid detection algorithm module extracts features from the second user image to obtain the eye features corresponding to the second user image. The sleep aid detection algorithm module performs blink recognition based on the eye features corresponding to the second user image to obtain blink detection results; the blink detection results include the number of blinks of the user within a preset detection time. The electronic device acquires the user's sleep aid data, including: The sleep aid application obtains the blink detection results through the communication interface.

9. The method according to claim 8, characterized in that, The sleep aid data also includes the number of times yawns occur. The method further includes: The sleep aid application registers a yawning fence through the communication interface, and the sleep aid application sends a third instruction to the sleep aid detection algorithm module. The sleep aid detection algorithm module responds to the third instruction and acquires a third user image captured by the always-sensing camera ASC through the ASC camera driver; the third user image includes multiple user images captured within the third acquisition time period; The sleep aid detection algorithm module extracts features from the third user image to obtain the mouth features corresponding to the second user image; The sleep aid detection algorithm module performs yawn recognition based on the mouth features corresponding to the second user image and obtains yawn detection results; the yawn detection results include the number of times the user yawns within a preset detection time. The electronic device acquires the user's sleep aid data, including: The sleep aid application obtains the yawning detection results through the communication interface.

10. The method according to any one of claims 1-9, characterized in that, The electronic device performs corresponding sleep-aid operations, including: If the electronic device detects that the volume of the electronic device is greater than a preset decibel threshold, the electronic device outputs a volume adjustment prompt message; and the electronic device lowers the volume to the preset decibel range, or the electronic device lowers the volume by a preset decibel value; The volume adjustment prompt is used to indicate that the sleep aid function has been activated and the volume is about to be reduced or has already been reduced.

11. The method according to any one of claims 1-10, characterized in that, The electronic device performs corresponding sleep-aid operations, including: If the electronic device detects that the brightness of its display screen is greater than a preset brightness threshold, the electronic device outputs a brightness adjustment prompt; and the electronic device lowers the brightness of the display screen to a preset brightness range, or the electronic device lowers the brightness of the display screen by a preset brightness value. The brightness adjustment prompt is used to indicate that the sleep aid function has been activated, and the brightness of the display screen has been dimmed.

12. The method according to any one of claims 1-11, characterized in that, The electronic device performs corresponding sleep-aid operations, and further includes: The electronic device outputs sound effect playback prompts; and the electronic device plays sleep-aid sound effects at a target decibel value. The sound effect playback prompt information is used to indicate that the sleep aid function has been activated and that sleep aid sound effects are about to play or are currently playing.

13. The method according to any one of claims 1-12, characterized in that, The electronic device performs corresponding sleep-aid operations, and further includes: If the electronic device detects that the display screen is in normal display mode, the electronic device outputs a prompt message to switch display modes; and the electronic device switches the display screen from normal display mode to eye-protection display mode. The display mode switching prompt message indicates that the sleep aid function has been activated and that the display is about to switch to the eye protection display mode. The display brightness in the eye protection display mode is lower than that in the normal display mode, and the display contrast in the eye protection display mode is lower than that in the normal display mode.

14. An electronic device comprising a display screen, a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-13.

15. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-13.

16. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-13.

Images