Terminal control method, smart watch, head-mounted device and terminal control system

Abstract

The invention relates to the technical field of terminal control, in particular to a terminal control method, a smart watch, head-mounted equipment and a terminal control system. The terminal control method applied to the smart watch comprises the following steps: determining current pose information of the smart watch; and in response to the current pose information of the smart watch representing that the action of the user of the smart watch is a target action, sending a target notification to a head-mounted device in communication connection with the smart watch, so that the head-mounted device sends the sight direction information of the user to the smart watch in response to the target notification, the head-mounted device is worn on the head of a user; receiving sight line direction information; and controlling the execution of the screen-on operation based on the sight direction information and the current pose information of the smart watch. The power consumption caused by mistakenly executing the screen-on operation can be greatly reduced while the requirement of a user for checking the content of the dial plate can be guaranteed, so that the service life of the intelligent watch can be prolonged.

Description

Technical Field

[0001] This disclosure relates to the field of terminal control technology, specifically to a terminal control method, a smartwatch, a head-mounted device, and a terminal control system. Background Technology

[0002] Smartwatches are becoming increasingly widespread as wearable devices. To conserve power, smartwatch screens cannot remain constantly lit; they only illuminate when the user needs to view the watch. Therefore, in related technologies, raise-to-wake has become one of the trigger conditions for smartwatches to activate their screens. Raise-to-wake, also known as wrist-raise-to-wake, refers to the smartwatch automatically lighting up its screen when the user naturally raises their wrist and turns the watch face towards themselves, allowing the user to instantly check the time and notifications without pressing buttons or touching anything.

[0003] However, in daily life, when users perform other actions, such as raising their wrists, the smartwatch may accidentally turn on the screen, thus increasing the power consumption of the smartwatch. Summary of the Invention

[0004] To overcome the problems existing in related technologies, an exemplary embodiment of this disclosure provides a terminal control method applied to a smartwatch. The method includes: determining the current pose information of the smartwatch; in response to the current pose information of the smartwatch indicating that the user's action is a target action, sending a target notification to a head-mounted device communicatively connected to the smartwatch, so that the head-mounted device responds to the target notification by sending the user's gaze direction information to the smartwatch, wherein the head-mounted device is worn on the user's head; receiving the gaze direction information; and controlling the execution of a screen-on operation based on the gaze direction information and the current pose information of the smartwatch.

[0005] In some embodiments, the terminal control method further includes: determining the confidence level that the user's action is a target action based on the current pose information of the smartwatch; and determining that the current pose information of the smartwatch represents the user's action as a target action in response to the confidence level being greater than or equal to a first threshold.

[0006] In some embodiments, the terminal control method further includes: performing a screen-on operation in response to a confidence level greater than or equal to a second threshold, wherein the second threshold is greater than a first threshold.

[0007] In some embodiments, determining the confidence level that a user's action is a target action based on the current pose information of the smartwatch includes: detecting the current pose information of the smartwatch through a built-in target neural network model to determine the confidence level that the user's action is a target action.

[0008] In some embodiments, the gaze direction information includes the gaze direction. Based on the gaze direction information and the current pose information of the smartwatch, the execution of the screen-on operation is controlled, including: in response to the gaze direction being within the target range and the current pose information of the smartwatch indicating that the user's action is the target action, the screen-on operation is executed.

[0009] In some embodiments, the terminal control method further includes: determining the time of receiving gaze direction information; and controlling the execution of the screen-on operation based on the gaze direction information and the current pose information of the smartwatch, including: controlling the execution of the screen-on operation based on the gaze direction information and the current pose information of the smartwatch at the corresponding receiving time.

[0010] In some embodiments, the gaze direction information includes the gaze direction. Based on the gaze direction information and the current pose information of the smartwatch at the corresponding receiving time, the execution of the screen-on operation is controlled, including: in response to the gaze direction being within the target range and the current pose information of the smartwatch at the corresponding receiving time indicating that the user's action is the target action, the screen-on operation is executed.

[0011] Secondly, this disclosure also provides a terminal control method applied to a head-mounted device. The method includes: receiving a target notification based on a communication connection with a smartwatch, wherein the target notification is sent by the smartwatch in response to the smartwatch's current pose information indicating that the user's action is a target action; and sending the user's gaze direction information in response to the target notification, so that the smartwatch controls the execution of a screen-on operation based on the gaze direction information and the smartwatch's current pose information, wherein the head-mounted device is worn by the user on their head.

[0012] In some embodiments, the terminal control method further includes: acquiring multiple consecutive frames of eye images of the user; and determining the user's gaze direction information based on the multiple consecutive frames of eye images.

[0013] Thirdly, this disclosure also provides a smartwatch, including: a pose determination module for determining the current pose information of the smartwatch; a first sending module for sending a target notification to a head-mounted device communicatively connected to the smartwatch in response to the current pose information of the smartwatch indicating that the user's action is a target action, so that the head-mounted device responds to the target notification and sends the user's gaze direction information to the smartwatch, wherein the head-mounted device is worn on the user's head; a first receiving module for receiving the gaze direction information; and a control module for controlling the execution of a screen-on operation based on the gaze direction information and the current pose information of the smartwatch.

[0014] Fourthly, this disclosure also provides a head-mounted device, including: a second receiving module, configured to receive a target notification based on a communication connection with a smartwatch, wherein the target notification is sent by the smartwatch in response to the smartwatch's current pose information indicating that the user's action is a target action; and a second sending module, configured to send the user's gaze direction information in response to the target notification, so that the smartwatch controls the execution of a screen-on operation based on the gaze direction information and the smartwatch's current pose information, wherein the head-mounted device is worn by the user on their head.

[0015] Fifthly, this disclosure also provides a terminal control system, including: a smartwatch and a head-mounted device communicatively connected to the smartwatch, wherein the head-mounted device is worn on the head by a user of the smartwatch; wherein the smartwatch is used to determine the current pose information of the smartwatch, and in response to the current pose information of the smartwatch indicating that the user's action is a target action, sends a target notification to the head-mounted device; the head-mounted device is used to send the user's gaze direction information in response to the target notification; the smartwatch is used to receive the gaze direction information, and based on the gaze direction information and the current pose information of the smartwatch, controls the execution of the screen-on operation.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure.

[0017] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: According to the terminal control method provided by this disclosure, when the smartwatch detects that the user's action is the target action, it obtains the user's gaze direction information through the head-mounted device, and determines whether to perform a screen-on operation based on the gaze direction information and the current posture information of the smartwatch. This can greatly reduce the power consumption caused by the accidental execution of screen-on operation while ensuring the user's need to view the watch face content, thereby helping to extend the usage time of the smartwatch. Attached Figure Description

[0018] This disclosure can be better understood by describing exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, in which:

[0019] Figure 1 This is a structural block diagram of a terminal control system according to an exemplary embodiment disclosed in a publication;

[0020] Figure 2 This is a flowchart illustrating a terminal control method according to an exemplary embodiment of a published document;

[0021] Figure 3 This is a flowchart illustrating another terminal control method according to an exemplary embodiment of a published document;

[0022] Figure 4 This is a flowchart illustrating yet another terminal control method according to an exemplary embodiment disclosed in a book.

[0023] Figure 5 This is a flowchart illustrating another terminal control method according to an exemplary embodiment of a published document;

[0024] Figure 6 This is a schematic diagram of terminal interaction according to an exemplary embodiment disclosed in a book;

[0025] Figure 7 This is another terminal interaction diagram illustrated according to an exemplary embodiment of a published document;

[0026] Figure 8 This is a structural block diagram of a smartwatch according to an exemplary embodiment disclosed in a book;

[0027] Figure 9 This is a structural block diagram of a head-mounted device according to an exemplary embodiment disclosed in a publication;

[0028] Figure 10 This is a structural block diagram of another terminal control system illustrated in an exemplary embodiment of a publication. Detailed Implementation

[0029] The following describes specific embodiments of this disclosure. It should be noted that, in order to provide a concise description, this specification cannot exhaustively describe all features of the actual embodiments. It should be understood that, in the actual implementation of any embodiment, just as in any engineering or design project, various specific decisions are often made to achieve the developer's specific goals and to meet system-related or business-related constraints, and this can change from one embodiment to another. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this disclosure, changes in design, manufacturing, or production based on the technical content disclosed in this disclosure are merely conventional technical means and should not be construed as insufficient content of this disclosure.

[0030] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. The terms “a” or “one,” etc., do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “comprising,” “including,” etc., mean that the element or object preceding “comprising” or “including” encompasses the elements or objects listed following “comprising” or “including” and their equivalents, and do not exclude other elements or objects. The terms “connected,” “linked,” etc., are not limited to physical or mechanical connections, nor are they limited to direct or indirect connections.

[0031] In related technologies, smartwatches trigger a screen-on operation when they detect a user raising their wrist and the watch face turning towards the user, allowing the user to instantly view the time and notifications without pressing buttons or touching anything. The wrist-raising motion and the screen's orientation can be detected by the smartwatch's internal accelerometer and gyroscope.

[0032] However, in daily life, when users perform other actions, such as raising their wrist, the smartwatch may accidentally turn on the screen, thereby increasing the power consumption of the smartwatch.

[0033] Other behaviors may include, but are not limited to, any of the following: running, brisk walking, swinging arms, skipping rope, swinging a racket, swinging a bat, throwing, clapping, waving, turning the steering wheel while driving, turning on the lights, shifting gears, cycling, skateboarding, holding the handlebars of an electric vehicle; typing, using a mouse, writing, turning pages in a book, looking at the screen of a mobile phone / taking a selfie, making or receiving phone calls, drinking water, raising a glass while eating, raising chopsticks, washing face, brushing teeth, rinsing mouth, combing hair / hair dryer, cooking, stir-frying and flipping the pan, wiping windows, mopping the floor, sweeping the floor, lifting dumbbells (bice-curl / press), bench press, push-ups, rowing machine, swinging a tennis racket, swinging a badminton racket, swinging a golf racket, shaking hands, saluting, raising a hand to signal, turning over at night, stretching, or raising an arm to adjust the blanket.

[0034] To address the aforementioned problems, this disclosure provides a terminal control system. For example... Figure 1As shown, the terminal control system 100 includes a smartwatch 110 and a head-mounted device 120 that communicates with the smartwatch 110. The head-mounted device 120 is worn on the head by the user of the smartwatch 110. That is, for the same user, the user wears the smartwatch 110 on their wrist and the head-mounted device 120 that can communicate with the smartwatch 110 on their head. While the user is wearing the smartwatch 110, the head-mounted device 120 can assist the smartwatch 110 in determining whether to perform a screen-on operation based on the communication connection with the smartwatch 110. This ensures that the user's need to view the smartwatch 110's screen is met while greatly reducing the occurrence of accidental screen-on operations, thereby saving the power consumption of the smartwatch 110 and extending its usage time. The head-mounted device 120 may include, but is not limited to, smart glasses, AI glasses, headphones, true wireless earbuds, etc. The communication connection between the smart glasses 110 and the head-mounted device 120 can include, but is not limited to, Bluetooth, WiFi, or Ultra Wide Band (UWB) connections. The specific connection method depends on the internally deployed communication module and the corresponding communication protocol. Bluetooth connections can include classic Bluetooth or Bluetooth Low Energy (BLE) connections. For example, the communication module can include a Bluetooth module (BT), a Bluetooth Low Energy (BLE) module, a WiFi module, an Ultra Wide Band (UWB) module, or other communication modules. If the communication module is a Bluetooth module, the corresponding communication protocol is the Bluetooth protocol, and therefore the communication connection between the smart glasses 110 and the head-mounted device 120 can be a classic Bluetooth connection or a BLE connection. If the communication module is a WiFi module, the corresponding communication protocol is the WiFi protocol, and therefore the communication connection between the smart glasses 110 and the head-mounted device 120 can be a WiFi connection. If the communication module is a UWB module, the corresponding communication protocol is the UWB protocol, and therefore the communication connection between the smart glasses 110 and the head-mounted device 120 can be a UWB connection.

[0035] Based on the same inventive concept, this disclosure provides a terminal control method for smartwatches. For example... Figure 2 As shown, the terminal control method may include the following steps:

[0036] Step S210: Determine the current pose information of the smartwatch.

[0037] The smartwatch's pose information changes according to the user's behavior. For example, when the user is standing naturally with their arms hanging down, the smartwatch is located in the sagittal plane of the body (the vertical plane that divides the body into left and right parts). At this time, the smartwatch's pose information is: the watch face is parallel to the sagittal plane, and the normal direction (i.e., the direction perpendicular to the screen) points inward towards the body. When the user raises their wrist to check the watch face, they need to bend their elbow and rotate their forearm to bring the watch to chest level or within their line of sight. At this time, the smartwatch's pose information is: the watch face is nearly horizontal or at an acute angle to the horizontal plane, and the screen is facing the user's face.

[0038] The pose information of a smartwatch can be obtained through a built-in Inertial Measurement Unit (IMU). This IMU can include, but is not limited to, accelerometers and gyroscopes. The accelerometer detects acceleration displacement and the direction of gravity to determine the linear acceleration of the smartwatch in spatial pose, thus sensing whether the smartwatch is vertical or tilted. The gyroscope determines the angular velocity of the smartwatch in spatial pose, thus sensing its rotational state. The IMU can be a system of multiple independent sensors or an integrated unit, depending on the specific requirements. When the IMU is a system, the accelerometer and gyroscope are two independent devices. When the IMU is an integrated unit, the accelerometer and gyroscope are integrated on the same chip. The choice of IMU can be determined based on the hardware layout requirements of the smartwatch and is not limited here.

[0039] In some examples, pose information can be the output of the IMU within the smartwatch. In other examples, pose information can be transformation parameters output by the IMU within the smartwatch, such as pose trajectory, attitude change process, velocity at multiple moments, acceleration at multiple moments, etc., which can be obtained according to actual needs.

[0040] Therefore, in order to determine whether the user needs to view the watch face, the current pose information of the smartwatch is determined, and the user's need to use the smartwatch is judged based on the current pose information.

[0041] In step S220, in response to the smartwatch's current pose information indicating that the smartwatch user's action is a target action, a target notification is sent to the head-mounted device that is communicatively connected to the smartwatch, so that the head-mounted device responds to the target notification and sends the user's gaze direction information to the smartwatch.

[0042] The target action can be considered predetermined, representing the action that the user is expected to view the watch face. The action representing the user's current posture information in response to the smartwatch is the target action, indicating that the user may currently need to view the watch face. However, since the target action may also occur while the user is performing other activities, it's easy for the smartwatch to mistakenly execute the screen-on operation. For example, the target action might be raising the wrist to approximately horizontal or within a specified acute angle range with the horizontal. However, when a user swings their racket while playing ball, waves, jumps rope, or engages in other wrist-raising actions, a wrist-raising action will occur, which the smartwatch may mistakenly interpret as a need to view the watch face.

[0043] Therefore, to prevent accidental screen-on operations, the smartwatch sends a target notification to the connected head-mounted device. This allows the head-mounted device to respond by sending the user's gaze direction information back to the smartwatch, informing the smartwatch whether the user genuinely needs to view the watch face. The head-mounted device is worn on the user's head, and the target notification informs it to send the user's gaze direction information to the smartwatch. The communication connection between the smart glasses and the head-mounted device can include, but is not limited to, Bluetooth, Wi-Fi, or Ultra Wideband (UWB) connections, depending on the internal communication module and corresponding communication protocol.

[0044] Step S230: Receive line-of-sight direction information.

[0045] The system receives gaze direction information from a head-mounted device to determine the user's gaze direction. This gaze direction information can be represented as a three-dimensional direction vector in either the head coordinate system or the head-mounted device coordinate system. The angle between this three-dimensional direction vector and a reference plane (e.g., a horizontal plane) allows for a more precise determination of the user's gaze direction.

[0046] For example, if the gaze direction information is represented as horizontal and the angle between it and the reference plane is zero, it can be determined that the user's gaze direction is looking straight ahead. If the gaze direction information is represented as looking down and the angle between it and the reference plane is acute, it indicates that the user is looking down. The angle size reflects the degree of downward gaze. If the gaze direction information is represented as looking up and the angle between it and the reference plane is acute, it indicates that the user is looking up. The angle size reflects the degree of upward gaze. If the gaze direction information is represented as horizontal deflection and the angle between it and the reference plane is acute, it indicates that the user is looking left or right.

[0047] For example, gaze direction information can be represented by the angle between the gaze and the front of the user's body, indicating the left or right direction and the degree of deviation to the left or right. For instance, when the user's gaze coincides with the front of their body, the gaze direction information can be defined as 0° to show no left or right deviation. When the user's gaze deviates to the left, the corresponding angle is either positive or negative (the labeling rules can be standardized). For example, if the gaze is looking to the left and forms a 30° angle with the baseline, the gaze direction information can represent a 30° deviation to the left; similarly, if the gaze is looking to the right and forms a 30° angle with the baseline, the gaze direction information can represent a 30° deviation to the right.

[0048] Step S240: Based on the gaze direction information and the current pose information of the smartwatch, control the execution of the screen-on operation.

[0049] By analyzing the user's gaze direction, the user's gaze direction can be clearly identified. By analyzing the smartwatch's current posture information, it can be determined whether the user is currently performing a target action. Therefore, by combining gaze direction information with the smartwatch's current posture information to jointly determine whether a screen-on operation is necessary, and then performing targeted control, the accuracy of screen-on operation can be ensured, effectively preventing erroneous execution. This, in turn, can effectively reduce the smartwatch's power consumption and help extend its usage time.

[0050] For example, if the gaze direction information indicates that the user's gaze is looking down, and the left or right lateral movement is in the direction the smartwatch might be located, and the smartwatch's current pose information indicates that the user's action is the target action, then it can be assumed that the user needs to view the watch face content, and thus the screen can be turned on to meet the user's needs. If the gaze direction information indicates that the user's gaze is looking up or the horizontal lateral movement does not overlap with the direction the smartwatch might be located, then it can be assumed that the user does not need to view the watch face content. Therefore, regardless of whether the smartwatch's current pose information indicates that the user's action is the target action, it can be assumed that the screen does not need to be turned on, thus effectively avoiding accidental touches and reducing the smartwatch's power consumption.

[0051] According to the terminal control method provided in this disclosure, when the smartwatch detects that the user's action is the target action, it obtains the user's gaze direction information through the head-mounted device, and determines whether to perform a screen-on operation based on the gaze direction information and the current posture information of the smartwatch. This can greatly reduce the power consumption caused by accidental screen-on operation while ensuring that the user can view the content of the watch face, thereby helping to extend the usage time of the smartwatch.

[0052] In some embodiments, such as Figure 3As shown, the above-mentioned terminal control method may further include the following steps:

[0053] Step S250: Based on the current pose information of the smartwatch, determine the confidence level that the user's action is the target action.

[0054] To determine the user's action, the current pose information of the smartwatch is detected to determine whether the user's action corresponding to the current pose information is the target action, and the confidence level of whether it is the target action. The closer the confidence level is to 1, the higher the reliability of considering the user's action as the target action; the closer the confidence level is to 0, the lower the reliability of considering the user's action as the target action.

[0055] In some examples, the current pose information of the smartwatch can be detected by the built-in target neural network model to determine the confidence level that the user's action is the target action.

[0056] The target neural network model is a pre-trained artificial intelligence (AI) model. Its input is the current pose information of the smartwatch, and its output is whether the user's action is the target action and the confidence level of that action. Based on its internal neurons, the target neural network model can identify and detect actions corresponding to the smartwatch's current pose information, thereby obtaining the output result.

[0057] By using a target neural network model to detect the current pose information of a smartwatch, it is possible to quickly determine whether the user's action is the target action and the confidence level of whether it is the target action. This can improve the determination efficiency, speed up the process of determining whether to perform a screen-on operation, and thus help improve the performance of the smartwatch.

[0058] In other examples, a correspondence between target pose information and target action can be pre-established. Then, based on the degree of matching between the smartwatch's pose information and the target pose information, as well as the degree of matching between changes in the smartwatch's pose information and changes in the target pose information, the confidence level that the user's action is the target action can be determined. This helps simplify the determination process and improve efficiency. To avoid missed recognition, the target pose information can be a range of pose information that characterizes the target action, thereby improving the robustness of the smartwatch in recognizing the target action.

[0059] In step S260, in response to the confidence level being greater than or equal to the first threshold, it is determined that the current pose information of the smartwatch represents the user's action as the target action.

[0060] The first threshold can be considered as the minimum confidence level that the user's action is the target action. For example, the first threshold can be 0.85, 0.9, or other values ​​less than 1, and can be configured according to requirements.

[0061] When the confidence level is greater than or equal to the first threshold, the user's action can be considered to be the target action, and thus the current pose information of the smartwatch can be used to determine that the user's action is the target action.

[0062] In some examples, in response to a confidence level less than a first threshold, it can be determined that the smartwatch's current pose information indicates that the user's action is not the target action. It can be assumed that the user has no need to view the watch face content, and therefore there is no need to perform the step of sending a target notification to the head-mounted device.

[0063] By determining whether the current pose information of the smartwatch can represent the user's action as the target action through confidence level, the occurrence of false recognition can be effectively reduced, and the reliability of determining that the user's action is the target action can be guaranteed.

[0064] In other embodiments, such as Figure 3 As shown, the above-mentioned terminal control method may further include the following steps:

[0065] In step S270, in response to a confidence level greater than or equal to the second threshold, a screen-on operation is performed.

[0066] The second threshold can be considered as the minimum confidence level required to determine that the user's action is the target action. For example, the second threshold can be 0.95, 0.99, or other values ​​close to 1, which can be configured according to requirements. The second threshold is greater than the first threshold.

[0067] If the confidence level is greater than or equal to the second threshold, it can be assumed that the user's action is the target action, meaning the user needs to view the watch face. Therefore, the screen-on operation can be performed directly to improve the smartwatch's responsiveness and enhance the user experience. Furthermore, by directly performing the screen-on operation, the power consumption for communication with the head-mounted device can be reduced, which in turn helps save power consumption for the head-mounted device's response.

[0068] In some embodiments, the gaze direction information may include the gaze direction, and step S240 may include: in response to the gaze direction being within the target range and the smartwatch's current pose information indicating that the user's action is the target action, performing a screen-on operation.

[0069] The target area can be considered as a three-dimensional spatial range based on a reference plane, used to determine whether the user is looking down and potentially looking at the smartwatch. If the detected gaze direction is within the target area, it can be assumed that the user is likely looking at the smartwatch. If the detected gaze direction is not within the target area, it can be assumed that the user is not looking at the smartwatch.

[0070] Therefore, in response to the gaze direction being within the target range, it can be assumed that the user is looking at the smartwatch. Furthermore, if the current pose information of the smartwatch indicates that the user's action is the target action, it can be determined that the user needs to view the watch face content, and thus the screen-on operation is performed to meet the user's usage needs.

[0071] For example, taking a horizontal plane as the reference plane, and the target range as the angle between the line of sight and the horizontal plane being within -30° to -60° (where "-" indicates the line of sight is downward), and deflected less than 30° to the left or right relative to the user's body. If the detected line of sight has an angle of -45° with the horizontal plane in the vertical direction and deflected 15° to the left in the horizontal direction, it can be assumed that the user is looking at the watch face. Therefore, when the smartwatch's current pose information indicates that the user's action is the target action, the screen can be turned on to meet the user's needs. If the detected line of sight has an angle greater than +15° with the horizontal plane (where "+" indicates the line of sight is upward), or deflected more than 50° to the left or right relative to the user's body, it can be assumed that the user is not looking at the watch face. Therefore, when the smartwatch's current pose information indicates that the user's action is the target action, it can be assumed that the user does not want to view the watch face content, and thus the screen does not need to be turned on. The specific numerical range within the target range can be obtained based on prior knowledge or experimental testing, and is not limited here.

[0072] In some embodiments, such as Figure 4 As shown, the terminal control method may include the following steps:

[0073] Step S210: Determine the current pose information of the smartwatch.

[0074] In step S220, in response to the smartwatch's current pose information indicating that the smartwatch user's action is a target action, a target notification is sent to the head-mounted device that is communicatively connected to the smartwatch, so that the head-mounted device responds to the target notification and sends the user's gaze direction information to the smartwatch.

[0075] The system determines the initial moment of the target action by identifying the user's action on the smartwatch and triggers a target notification at that initial moment. This allows the head-mounted device to assist in determining whether the user needs to view the watch face content, ensuring the timely execution of screen-on operations.

[0076] Step S230: Receive line-of-sight direction information.

[0077] Step S280: Determine the time of receiving the line-of-sight direction information.

[0078] Because the head-mounted device receives a target notification and responds to it, there is a certain time interval in the process of transmitting the gaze direction information back to the smartwatch. For example, this time interval may be tens of milliseconds (ms), hundreds of milliseconds, or even thousands of milliseconds.

[0079] During this time interval, the user's actions may constantly change, leading to continuous changes in the detected smartwatch's pose information. Therefore, to ensure the reliability of screen-on operations and increase the probability that the user's action based on the detected smartwatch's current pose information is the target action, the timing of receiving gaze direction information is determined. Based on the pose information prior to this receiving moment, the user's action is further determined to be the target action, effectively reducing the latency of screen-on control and improving the smartwatch's response performance. Therefore, the initial timing is earlier than the timing of receiving gaze direction information, which effectively reduces the latency of subsequent screen-on control and helps improve the reliability of determining the target action.

[0080] In some examples, to ensure the accuracy of the screen-on operation, the confidence level for determining whether the user's action is the target action based on the pose information before the reception time is higher than the confidence level based on the initial moment. For example, if the minimum confidence level for determining that the user's action is the target action at the initial moment is 90%, and the minimum confidence level for determining that the user's action is the target action before the reception time is 95%, then if the confidence level for determining that the user's action is the target action at the initial moment is 92%, a target notification is sent to the head-mounted device connected to the smartwatch, and the confidence level for determining that the user's action is the target action is determined based on the smartwatch's pose information before the reception time. If the confidence level for determining that the user's action is the target action based on the smartwatch's pose information before the reception time is 96%, then the user's action can be considered the target action. If the confidence level of determining that the user's action is the target action based on the smartwatch's pose information before the receiving time is less than 95%, for example, 92%, then the user's action is not considered to be the required target action.

[0081] In other embodiments, the target action includes at least two types, and they have a sequential order. For example, the target action may include raising the arm and bending the forearm. The target action to be determined at the initial moment is raising the arm, and the target action to be determined before the receiving moment is bending the forearm. If the user's action is determined to be raising the arm at the initial moment, and bending the forearm is determined to be the user's action before the receiving moment, then the user's action can be considered the required target action.

[0082] Step S240, based on the gaze direction information and the current pose information of the smartwatch, controls the execution of the screen-on operation, including:

[0083] Step S241: Based on the gaze direction information and the current pose information of the smartwatch at the corresponding receiving time, control the execution of the screen-on operation.

[0084] By analyzing gaze direction information, it can be determined whether the user's gaze is directed towards the smartwatch, thus predicting whether the user needs to view the watch face. By analyzing the smartwatch's current posture information at the moment of reception, it can be determined whether the user's action upon receiving gaze direction information is the target action.

[0085] Based on the direction of the user's gaze and the current pose information of the smartwatch at the corresponding receiving time, the system jointly determines whether the user needs to view the watch face content. At the same time, it notifies the head-mounted device to obtain the direction of the user's gaze at the initial moment, which can effectively avoid the situation where the user sees the watch face but the watch face has not yet been lit up. This can greatly shorten the latency of the smartwatch to control and execute the screen-on operation, improve the response efficiency of the smartwatch, and thus help to enhance the user experience.

[0086] In some examples, in response to the gaze direction being within the target range and the smartwatch's current pose information at the moment of reception indicating that the user's action is the target action, it can be assumed that the user has a need to view the watch face content, and thus the screen-on operation is performed.

[0087] In other examples, if the user's gaze is not within the target range, it can be assumed that the user has no need to view the watch face content. Therefore, regardless of whether the current pose information received by the smartwatch at the corresponding moment indicates that the user's action is the target action, the screen-on operation can be avoided to save power consumption.

[0088] In some other examples, in response to the gaze being within the target range, but the smartwatch's current pose information at the moment of reception indicating that the user's action is not the target action, it is assumed that the user has no need to view the watch face content, and therefore the screen-on operation is not performed to save power.

[0089] In some other examples, once the smartwatch's current pose information is first detected, indicating that the user's action is the target action, the smartwatch remains in a state of detecting its pose information until the moment the user's gaze direction information is received from the head-mounted device. This is to avoid missed detections and increase the probability of detecting that the user's action is the target action.

[0090] In some application scenarios, a pre-trained neural network model can be used to determine whether a screen-on operation needs to be performed. That is, the input to this trained neural network model can be gaze direction information and the current pose information of the smartwatch at the corresponding receiving time, and the output can be whether the target action is included or whether a screen-on operation should be performed. During the training of the neural network model, multiple test subjects wearing the head-mounted device (some looking at the watch, some not looking at the watch, including actions that are prone to misjudgment, such as the aforementioned "running, brisk walking, large arm swings, skipping rope," etc.) are involved. The neural network model is trained based on pose information, gaze direction information, and the corresponding markers indicating whether the watch was looked at. In one example, to improve recognition accuracy, the training parameters can also include training data on whether the wearer wears the watch on their left or right hand.

[0091] By training the neural network model in this way, the probability of the user looking at the smartwatch from different viewing directions can be fully utilized, thereby better integrating the viewing direction with the posture information, more reliably determining whether the user has raised their hand to look at the watch, and thus more reliably turning on the screen.

[0092] Based on the same inventive concept, this disclosure also provides a terminal control method for head-mounted devices. For example... Figure 5 As shown, the terminal control method may include the following steps:

[0093] Step S310: Receive target notification based on the communication connection with the smartwatch.

[0094] Target notification is sent by the smartwatch in response to the user's current pose information, indicating that the user's action is a target action. The communication connection between the head-mounted device and the smart glasses can include, but is not limited to, Bluetooth, WiFi, or Ultra Wide Band (UWB) connections, depending on the internally deployed communication module and the corresponding communication protocol.

[0095] In step S320, in response to the target notification, the user's gaze direction information is sent so that the smartwatch can control the execution of the screen-on operation based on the gaze direction information and the current pose information of the smartwatch.

[0096] In this system, the head-mounted device is worn by the user, and the changes in the posture information of the head-mounted device can be approximated as changes in the posture information of the user's head. This ensures that the determined gaze direction information is closer to the user's actual gaze direction. As a result, when the smartwatch uses the gaze direction information and the current posture information of the smartwatch to jointly determine whether the user needs to view the watch face content, the reliability of the judgment can be improved, making the execution process of controlling the screen to turn on more accurate, effectively avoiding the occurrence of erroneous execution, and reducing the power consumption of the smartwatch.

[0097] According to the terminal control method provided in this disclosure, by sending the user's gaze direction information through a head-mounted device, the smartwatch can combine multi-dimensional information to jointly determine whether the user needs to view the watch face content. This can effectively avoid the occurrence of accidental screen-on operations, ensure the accuracy and reliability of the judgment, thereby helping to save the power consumption of the smartwatch and extend its usage time.

[0098] Moreover, target notifications and gaze direction information are transmitted based on the communication connection between the head-mounted device and the smartwatch, which helps to enhance the interaction capabilities between devices, break down information silos between devices, and thus more accurately understand the user's intentions, achieve more natural and intelligent interaction, and enhance the user experience.

[0099] In some embodiments, the terminal control method described above may further include the following steps:

[0100] Step a1: Acquire multiple consecutive frames of eye images of the user.

[0101] To facilitate the determination of the user's gaze direction, the camera and image acquisition module of the head-mounted device acquire multiple consecutive frames of eye images of the user. Based on these multiple frames of eye images, the movement of the user's eyeballs is tracked, which facilitates the subsequent determination of the user's gaze direction information.

[0102] In some optional application scenarios, in order to accurately track eye movements, the head-mounted device can use a near-infrared light source for illumination, and the image acquisition module with the camera as the core is responsible for acquiring multiple consecutive frames of eye images of the user, so as to effectively improve the recognition of eye features in the image, thereby improving the accuracy of determining the direction of gaze information.

[0103] In other optional application scenarios, the target acquisition range relative to the user's eyes can be predetermined. Then, for this target acquisition range, the camera and image acquisition module of the head-mounted device can only acquire image data within this target acquisition range (for example, through digital zoom or sensor windowing readout technology). While reducing the amount of data, it ensures that each frame captures high-resolution and complete eye features, thereby obtaining continuous multi-frame eye images of the user, which can significantly improve the accuracy and real-time performance of gaze tracking.

[0104] Step a2: Determine the user's gaze direction information based on multiple consecutive frames of eye images.

[0105] Eye features, such as the relative pose vector between the pupil center and the corneal reflector, are extracted from each frame of the eye image. Then, a time-varying gaze direction vector sequence is generated according to the order in which the eye images were acquired. By performing temporal analysis and filtering on this vector sequence, a stable and reliable user gaze direction information is finally determined.

[0106] In some optional application scenarios, the user's gaze direction information can be obtained periodically. This can not only be used to inform the smartwatch user of their gaze in a timely manner, reducing the interaction latency between the head-mounted device and the smartwatch, but also allow the head-mounted device to provide functions adapted to the user's needs based on the user's gaze direction, thereby meeting the user's usage requirements for the head-mounted device.

[0107] In other optional application scenarios, gaze direction information can be obtained in response to a received target notification, thereby ensuring the accuracy of gaze direction information acquisition. This allows the smartwatch to control the screen-on operation based on the gaze direction information and the smartwatch's current posture, ensuring the precision of control, effectively avoiding erroneous execution, and effectively reducing the power consumption of the smartwatch.

[0108] In some optional application scenarios, combined Figure 1 The terminal control system shown illustrates an interaction process between a smartwatch and a head-mounted device for terminal control, which can be described as follows: Figure 6 As shown. The specific interaction process can be as follows:

[0109] The smartwatch determines its current pose information. Responding to this pose information, and recognizing the user's action as a target action, the smartwatch sends a target notification to the head-mounted device.

[0110] The head-mounted device periodically detects the user's gaze direction.

[0111] In response to receiving a target notification, the head-mounted device sends the user's gaze direction information to the smartwatch.

[0112] The smartwatch receives gaze direction information and, in response to the gaze direction being within the target range and the smartwatch's current pose information indicating that the user's action is the target action, performs a screen-on operation; if the smartwatch responds to the gaze direction not being within the target range, or the smartwatch's current pose information indicating that the user's action is not the target action, it does not perform a screen-on operation.

[0113] Controlling the screen-on operation in this way can effectively prevent accidental execution, reduce the power consumption of the smartwatch, and thus help extend the usage time of the smartwatch.

[0114] In other alternative application scenarios, combined with Figure 1 The terminal control system shown illustrates an interaction process between a smartwatch and a head-mounted device for terminal control, which can be described as follows: Figure 7 As shown. The specific interaction process can be as follows:

[0115] The smartwatch determines its current pose information. Responding to this pose information, and recognizing the user's action as a target action, the smartwatch sends a target notification to the head-mounted device.

[0116] In response to receiving a target notification, the head-mounted device acquires multiple consecutive frames of eye images of the user and determines the user's gaze direction information based on these multiple frames of eye images.

[0117] The head-mounted device sends information about the user's gaze direction to the smartwatch.

[0118] The smartwatch receives gaze direction information and determines the timing of receiving the gaze direction information.

[0119] The smartwatch will turn on the screen if the user's gaze is within the target range and the smartwatch's current pose information at the time of reception indicates that the user's action is the target action; otherwise, it will not turn on the screen if the user's gaze is not within the target range or the smartwatch's current pose information at the time of reception indicates that the user's action is not the target action.

[0120] Controlling the screen-on operation in this way avoids situations where the user's eyes are on the surface but it is not yet lit. This not only reduces the latency of the screen-on operation and the probability of false detection, but also saves power consumption of the head-mounted device. As a result, it can extend the usage time of both smartwatches and head-mounted devices.

[0121] Based on the same inventive concept, this disclosure also provides a smartwatch. For example... Figure 8As shown, the smartwatch 400 may include:

[0122] The pose determination module 410 is used to determine the current pose information of the smartwatch;

[0123] The first sending module 420 is used to respond to the current pose information of the smartwatch, which indicates that the user's action is a target action, and send a target notification to the head-mounted device that is connected to the smartwatch, so that the head-mounted device responds to the target notification and sends the user's gaze direction information to the smartwatch. The head-mounted device is worn by the user on the head.

[0124] The first receiving module 430 is used to receive line-of-sight direction information;

[0125] The control module 440 is used to control the execution of the screen-on operation based on the gaze direction information and the current pose information of the smartwatch.

[0126] In some embodiments, the smartwatch 400 may further include: a detection module, configured to determine the confidence level of a user's action as a target action based on the current pose information of the smartwatch; and a judgment module, configured to determine that the current pose information of the smartwatch represents the user's action as a target action in response to the confidence level being greater than or equal to a first threshold.

[0127] In some embodiments, the control module 440 is further configured to perform a screen-on operation in response to a confidence level greater than or equal to a second threshold, wherein the second threshold is greater than the first threshold.

[0128] In some embodiments, the detection module is used to detect the current pose information of the smartwatch through a built-in target neural network model, and determine the confidence level that the user's action is the target action.

[0129] In some embodiments, the gaze direction information includes the gaze direction. The control module 440 is configured to perform a screen-on operation in response to the gaze direction being within the target range and the smartwatch's current pose information indicating that the user's action is the target action.

[0130] In some embodiments, the smartwatch 400 further includes: a time determination module for determining the time of receiving gaze direction information; and a control module 440 for controlling the execution of the screen-on operation based on the gaze direction information and the current pose information of the smartwatch at the corresponding receiving time.

[0131] In some embodiments, the gaze direction information includes the gaze direction. The control module 440 is used to perform a screen-on operation in response to the gaze direction being within the target range and the smartwatch's current pose information at the time of reception indicating that the user's action is the target action.

[0132] Regarding the smartwatch in the above embodiments, the specific ways in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0133] Based on the same inventive concept, this disclosure also provides a head-mounted device. For example... Figure 9 As shown, the head-mounted device 500 may include:

[0134] The second receiving module 510 is used to receive a target notification based on the communication connection with the smartwatch, wherein the target notification is sent by the smartwatch in response to the smartwatch's current pose information indicating that the user's action is a target action.

[0135] The second sending module 520 is used to send the user's gaze direction information in response to the target notification, so that the smartwatch can control the execution of the screen-on operation based on the gaze direction information and the current posture information of the smartwatch, wherein the head-mounted device is worn by the user on the head.

[0136] In some embodiments, the head-mounted device 500 may further include: an acquisition module for acquiring multiple consecutive frames of eye images of the user; and an information determination module for determining the user's gaze direction information based on the multiple consecutive frames of eye images.

[0137] Regarding the head-mounted device in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0138] Based on the same inventive concept, this disclosure also provides a terminal control system, including as follows: Figure 10 The smartwatch 400 shown and the device connected to the smartwatch 400 are as follows: Figure 10 The diagram shows a head-mounted device 500, which is worn on the head by a user of a smartwatch 400. For illustrative purposes, the head-mounted device 500 is represented by smart glasses in the diagram. It should be noted that in practical applications, the head-mounted device 500 is not limited to the smart glasses shown in the diagram; it can also be smart glasses, headphones, true wireless earbuds, or other devices that can be worn on the head.

[0139] The smartwatch 500 determines the current pose information of the smartwatch 400 and, in response to the smartwatch 400's current pose information indicating that the user's action is a target action, sends a target notification to the head-mounted device 500; the head-mounted device 500, in response to the target notification, sends the user's gaze direction information; the smartwatch 400 receives the gaze direction information and, based on the gaze direction information and the smartwatch 400's current pose information, controls the execution of the screen-on operation.

[0140] Regarding the smartwatches and head-mounted devices in the above embodiments, the specific ways in which each module performs its operations have been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0141] Based on the same inventive concept, this disclosure also provides a computer-readable storage medium storing a program for performing the device interaction method of any of the foregoing embodiments.

[0142] This disclosure uses specific terms to describe embodiments of the present disclosure. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of the present disclosure. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, certain features, structures, or characteristics in one or more embodiments of the present disclosure can be appropriately combined.

[0143] In the context of this disclosure, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of expressly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0144] Similarly, it should be noted that, in order to simplify the description of this disclosure and thus aid in the understanding of one or more embodiments, the foregoing description of embodiments of this disclosure may sometimes combine multiple features into a single embodiment, drawing, or description thereof. However, this disclosure method does not imply that the subject matter of this disclosure requires more features than the features claimed. In fact, the embodiments contain fewer features than all the features of the single embodiments disclosed above.

[0145] The basic concepts have been described above. It is obvious that the above disclosure is merely illustrative and does not constitute a limitation of this disclosure. Although not explicitly stated herein, various modifications, improvements, and corrections may be made to this disclosure by those skilled in the art. Such modifications, improvements, and corrections are suggested in this disclosure and therefore remain within the spirit and scope of the embodiments of this disclosure.

Claims

1. A terminal control method applied to a smartwatch, the method comprising: Determine the current pose information of the smartwatch; In response to the current pose information of the smartwatch indicating that the user's action is a target action, a target notification is sent to a head-mounted device that is communicatively connected to the smartwatch, so that the head-mounted device responds to the target notification and sends the user's gaze direction information to the smartwatch, wherein the head-mounted device is worn on the user's head. Receive the line-of-sight direction information; The screen-on operation is controlled based on the gaze direction information and the current pose information of the smartwatch.

2. The terminal control method according to claim 1, wherein, The method further includes: Based on the current pose information of the smartwatch, determine the confidence level that the user's action is the target action; In response to the confidence level being greater than or equal to a first threshold, the current pose information of the smartwatch is determined to indicate that the user's action is the target action.

3. The terminal control method according to claim 2, wherein, The method further includes: The screen-on operation is performed in response to the confidence level being greater than or equal to the second threshold, wherein the second threshold is greater than the first threshold.

4. The terminal control method according to claim 2 or 3, wherein, The determination of the confidence level that the user's action is a target action based on the current pose information of the smartwatch includes: The smartwatch's current pose information is detected by a built-in target neural network model to determine the confidence level that the user's action is the target action.

5. The terminal control method according to claim 1, wherein, The gaze direction information includes the gaze direction, and the step of controlling the screen-on operation based on the gaze direction information and the current pose information of the smartwatch includes: In response to the gaze direction being within the target range, and the smartwatch's current pose information indicating that the user's action is the target action, the screen-on operation is executed.

6. The terminal control method according to claim 1, wherein, The method further includes: Determine the timing of receiving the line-of-sight direction information; The step of controlling the screen-on operation based on the gaze direction information and the current pose information of the smartwatch includes: The screen-on operation is controlled based on the gaze direction information and the current pose information of the smartwatch at the time of reception.

7. The terminal control method according to claim 6, wherein, The gaze direction information includes the gaze direction. Controlling the screen-on operation based on the gaze direction information and the current pose information of the smartwatch corresponding to the receiving time includes: In response to the gaze direction being within the target range, and the smartwatch indicating that the user's action is the target action based on the current pose information at the time of reception, the screen-on operation is executed.

8. A terminal control method applied to a head-mounted device, the method comprising: Based on the communication connection with the smartwatch, a target notification is received, wherein the target notification is sent by the smartwatch in response to the smartwatch's current pose information indicating that the user's action is a target action; In response to the target notification, the user's gaze direction information is sent so that the smartwatch controls the execution of the screen-on operation based on the gaze direction information and the smartwatch's current posture information, wherein the head-mounted device is worn on the user's head.

9. The terminal control method according to claim 8, wherein, The method further includes: Acquire consecutive multi-frame images of the user's eyes; Based on the continuous multiple frames of eye images, the user's gaze direction information is determined.

10. A smartwatch, comprising: The pose determination module is used to determine the current pose information of the smartwatch; The first sending module is used to send a target notification to a head-mounted device that is communicatively connected to the smartwatch in response to the current pose information of the smartwatch indicating that the user's action is a target action. This causes the head-mounted device to send the user's gaze direction information to the smartwatch in response to the target notification. The head-mounted device is worn on the user's head. The first receiving module is used to receive the line-of-sight direction information; The control module is used to control the execution of the screen-on operation based on the gaze direction information and the current pose information of the smartwatch.

11. A head-mounted device, comprising: The second receiving module is used to receive a target notification based on the communication connection with the smartwatch, wherein the target notification is sent by the smartwatch in response to the smartwatch's current pose information indicating that the user's action is a target action; The second sending module is used to send the user's gaze direction information in response to the target notification, so that the smartwatch can control the execution of the screen-on operation based on the gaze direction information and the current posture information of the smartwatch, wherein the head-mounted device is worn on the user's head.

12. A terminal control system, comprising: A smartwatch and a head-mounted device that communicates with the smartwatch, wherein the head-mounted device is worn on the head by the user of the smartwatch; in, The smartwatch is used to determine the current pose information of the smartwatch, and in response to the current pose information of the smartwatch indicating that the user's action is a target action, sends a target notification to the head-mounted device. The head-mounted device is used to send the user's gaze direction information in response to the target notification; The smartwatch is used to receive the gaze direction information and, based on the gaze direction information and the current pose information of the smartwatch, control the execution of the screen-on operation.

Images