Head-mounted device, wrist device, and bright screen identification method and system

Abstract

The invention provides a head-mounted device, a wrist device, and a bright screen identification method and system. The head-mounted device comprises a first wireless module, a camera shooting assembly and a first gesture recognition module. The camera shooting assembly is used for collecting image data; the first wireless module is used for being connected with wrist equipment and receiving a first instruction sent by the wrist equipment; the first gesture recognition module is used for acquiring image data acquired by the camera component under the condition that the first wireless module receives the first instruction, and judging whether the image data comprises a target hand lifting or not; and under the condition that the target hand raising action is recognized, sending a second instruction for indicating the wrist equipment to turn on the screen to the wrist equipment. According to the head-mounted equipment, the power consumption of the wrist equipment can be reduced under the condition that the screen of the wrist equipment is turned on mistakenly.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of wearable devices, and in particular, provides a head-mounted device, a wrist device, a screen-on identification method and system. BACKGROUND

[0002] A wrist device such as a smart watch is configured with a raise-to-wake screen-on (or raise-to-wrist screen-on) function. This function controls the screen-on based on the reason why the user needs to raise the hand to view the watch by detecting whether the user has a raise hand action (or watch action). This way can effectively reduce the power consumption of the smart watch and avoid the smart watch being on for a long time.

[0003] Currently, the raise-to-wrist screen-on function of the smart watch is usually based on the inertial data collected by the inertial sensor to determine the raise hand action. However, there are some raise hand actions similar to watching the watch, such as clapping, waving, answering the phone, etc. The smart watch is prone to misjudgment and screen-on, thereby causing additional power consumption. SUMMARY

[0004] Therefore, the present application aims to provide a head-mounted device, a wrist device, a screen-on identification method and system to reduce the additional power consumption caused by the false screen-on of the wrist device.

[0005] Firstly, the present application provides a head-mounted device, comprising: a first wireless module, a camera assembly and a first gesture recognition module; the first gesture recognition module is connected with the first wireless module and the camera assembly respectively; the camera assembly is configured to collect image data; the first wireless module is configured to connect with a wrist device and receive a first instruction sent by the wrist device; the first instruction represents that the wrist device recognizes an initial raise hand action; the first gesture recognition module is configured to, in the case that the first wireless module receives the first instruction, acquire the image data collected by the camera assembly, and identify whether the image data includes a target raise hand action; the first gesture recognition module is further configured to, in the case that it is determined that the target raise hand action is included, send a second instruction to the wrist device, the second instruction instructing the wrist device to turn on the screen.

[0006] Secondly, the present application provides a wrist device, comprising: a second wireless module, an inertial sensor and a second gesture recognition module; the second gesture recognition module is connected with the second wireless module and the inertial sensor respectively; the second wireless module is configured to be communicatively connected with the head-mounted device of any one of the first aspect; the inertial sensor is configured to collect inertial data; the second gesture recognition module is configured to identify a raise hand action based on the inertial data, and feed back a first instruction to the head-mounted device in the case that the initial raise hand action is identified; and the wrist device is further configured to turn on the screen in the case that the second instruction from the head-mounted device is received.

[0007] In a third aspect, the embodiments of the present application provide a screen-on identification system, comprising: a wrist device and a head-mounted device; the head-mounted device comprises: a first wireless module, a camera assembly and a first gesture identification module; the wrist device comprises: a second wireless module, an inertial sensor and a second gesture identification module; the first wireless module is in communication connection with the second wireless module; the wrist device collects inertial data through the inertial sensor, and the second gesture identification module is configured to identify an initial hand-raising action based on the inertial data, and feed back a first instruction to the head-mounted device through the second wireless module in a case where the initial hand-raising action is identified; the head-mounted device receives the first instruction through the first wireless module, the first gesture identification module acquires image data collected by the camera assembly, and identifies a target hand-raising action based on the image data, and sends a second instruction to the wrist device through the second wireless module in a case where the target hand-raising action is identified based on the image data; the wrist device turns on the screen in a case where the second instruction is received through the first wireless module.

[0008] In a fourth aspect, the embodiments of the present application provide a screen-on identification method, applied to the head-mounted device of any one of the first aspect, the head-mounted device being connected with a wrist device, the screen-on identification method comprising: acquiring image data collected by the camera assembly in a case where a first instruction sent by the wrist device is received; the first instruction indicating that the wrist device identifies an initial hand-raising action; identifying whether the target hand-raising action is included based on the image data; in a case where it is determined that the target hand-raising action is included, sending a second instruction indicating the wrist device to turn on the screen to the wrist device.

[0009] In a fifth aspect, the embodiments of the present application provide a screen-on identification method, applied to the wrist device of the second aspect, the wrist device being connected with a head-mounted device, the screen-on identification method comprising: collecting inertial data; identifying an initial hand-raising action based on the inertial data; feeding back a first instruction to the head-mounted device in a case where the initial hand-raising action is identified; the head-mounted device collects image data to identify whether the target hand-raising action is included in a case where the first instruction is received, and sends a second instruction to the wrist device in a case where it is determined that the target hand-raising action is included; turning on the screen in a case where the second instruction of the head-mounted device is received.

[0010] In a sixth aspect, the embodiments of the present application provide a screen-on identification method applied to the screen-on identification system of the third aspect; the screen-on identification method comprises: the wrist device acquires inertial data; the wrist device identifies an initial hand-raising action based on the inertial data; in the case where the wrist device identifies the initial hand-raising action, the head-mounted device is fed back a first instruction; the head-mounted device acquires image data in the case where the first instruction is received; the head-mounted device identifies a target hand-raising action based on the image data; in the case where the target hand-raising action is identified based on the image data, the head-mounted device sends a second instruction to the wrist device; and the wrist device turns on the screen in the case where the second instruction is received.

[0011] In the embodiments of the present application, when the wrist device identifies the initial hand-raising action, it is indicated that the user may need to view the wrist device, at this time, the first instruction can be sent to the head-mounted device, the head-mounted device identifies the image captured by the camera group after receiving the first instruction, and the hand-raising action is identified again through the image data, and when the second hand-raising action is identified through the image data, it is indicated that the user is likely to view the wrist device, at this time, the second instruction can be sent to the wrist device to control the wrist device to turn on the screen. This way, through twice confirmation of whether the hand-raising action is valid, the situation of false screen-on caused by the wrist device identifying other actions similar to the hand-raising action is effectively reduced, the effectiveness of screen-on is improved, and the power consumption caused by false screen-on of the wrist device is reduced. At the same time, the head-mounted device performs twice confirmation through the image captured by the camera, which can further improve the accuracy of identification compared with single inertial data. In addition, the head-mounted device such as smart glasses is equipped with a camera and other camera components, so it is not necessary to adjust the structure of the wrist device, thereby reducing the cost. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0013] Figure 1 A schematic diagram of a head-mounted device provided by an embodiment of the present application; Figure 2 A schematic diagram of a wrist device provided by an embodiment of the present application; Figure 3 A flowchart of a screen-on identification method applied to a head-mounted device provided by an embodiment of the present application; Figure 4A flowchart of a screen-on identification method applied to a wrist device is provided in an embodiment of the present application.

[0014] Icon: head-mounted device 100; first wireless module 110; camera assembly 120; first gesture recognition module 130; wrist device 200; second wireless module 210; inertial sensor 220; second gesture recognition module 230. DETAILED DESCRIPTION

[0015] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0016] The embodiments of the present application provide a wrist device and a head-mounted device. The wrist device can be a watch-type device with a display screen, such as a smart watch. When the wrist device detects a lifting hand action that can require a screen-on, the head-mounted device is used to perform secondary identification on the lifting hand action. The head-mounted device includes, but is not limited to, smart glasses, a headset, a true wireless earphone, or other head-mounted devices with the same structure provided later.

[0017] Next, the screen-on of the wrist device will be described in combination with the functions of the wrist device and the head-mounted device.

[0018] Please refer to Figure 1 , Figure 1 A structural schematic diagram of a head-mounted device 100 is provided in an embodiment of the present application. The head-mounted device 100 includes a first wireless module 110, a camera assembly 120, and a first gesture recognition module 130.

[0019] The first wireless module 110 is configured to be communicatively connected with the wrist device 200. In the embodiments of the present application, the first wireless module 110 includes, but is not limited to, a Bluetooth module, a Bluetooth Low Energy module, a Wi-Fi module, an Ultra Wide Band (UWB) module, and the like, which are not limited herein.

[0020] In the embodiments of the present application, the first gesture recognition module 130 is connected with the first wireless module 110 and the camera assembly 120, respectively. The first gesture recognition module 130 includes, but is not limited to, a CPU, an NPU, an MCU, an AI module, and the like.

[0021] The camera assembly 120 includes a camera and its supporting circuit, such as a collection circuit, a conversion circuit, and the like, which are not described herein in excess.

[0022] In embodiments of the present application, the camera assembly 120 can include a first camera connected with the first gesture recognition module 130. The first camera is configured to capture outward image data, which is image data in a preset direction other than the user's face. For example, when the head-mounted device 100 is a smart glasses, the first camera can be a camera for capturing the front of the user.

[0023] In some other embodiments of the present application, the camera assembly 120 can include a second camera configured to capture inward image data, which is image data in a preset direction of the user's eyes. For example, the second camera can be disposed inside the frame of the smart glasses close to the side of the user's eyes. The second camera can be an infrared camera, or also referred to as an infrared sensor.

[0024] In embodiments of the present application, the head-mounted device 100 is configured to connect with a wrist device and receive a first instruction sent by the wrist device. The first instruction indicates that the wrist device has recognized a hand-raising action.

[0025] In embodiments of the present application, the wrist device can include an inertial sensor, and the hand-raising action can be detected based on inertial data collected by the inertial sensor. When the hand-raising action is recognized based on the inertial data, the wrist device sends the first instruction to the head-mounted device 100.

[0026] The inertial sensor can include, but is not limited to, an accelerometer and a gyroscope. The accelerometer can be used to detect acceleration displacement and gravity direction, and the gyroscope can be used to detect angular velocity to depict the rotation speed and amplitude of the wrist, thereby improving real-time performance and robustness. In addition, the wrist device can be configured with a neural network module to recognize the hand-raising action based on the inertial data through the neural network. Details can be referred to the prior art, which will not be described here.

[0027] However, the accuracy of recognizing the hand-raising action based on the inertial data alone is low, and some similar actions can be misjudged, such as running, fast walking, swinging arms, jumping rope, swinging a racket / club, throwing action, clapping, waving, waving to say hello, turning the steering wheel while driving, turning on the light, shifting gears, holding the handle while riding / skating / electric vehicle, typing, using a mouse, writing, turning pages, holding a mobile phone to watch the screen / selfie, answering a phone call, holding a cup while drinking / eating, holding chopsticks while eating, washing face / brushing teeth / mouth, combing hair / blow dryer, cooking / stir-frying turning the pot, wiping windows / mopping / sweeping, holding dumbbells (bending / raising), bench press, push-ups, rowing machine, tennis / badminton / golf racket, shaking hands, saluting, waving to say hello, turning over at night, stretching, raising arms to adjust the quilt, etc.

[0028] Therefore, in the embodiments of the present application, the head-mounted device 100 can be used for secondary judgment, the first gesture recognition module 130 is used for acquiring image data collected by the camera assembly 120 when the first wireless module 110 receives the first instruction, and identifying a target hand-raising action based on the image data to determine whether the initial hand-raising action recognized by the wrist device is valid.

[0029] In addition, the first gesture recognition module 130 is further used for sending a second instruction indicating that the wrist device turns on the screen to the wrist device when the target hand-raising action is identified, so that the wrist device turns on the screen after receiving the second instruction.

[0030] In the embodiments of the present application, when the wrist device identifies the initial hand-raising action, the wrist device does not directly turn on the screen, but further judges whether the initial hand-raising action is valid, so as to effectively reduce the possibility of misjudgment, improve the effectiveness of screen turning on, and further reduce the power consumption of false screen turning on. The initial hand-raising action is the hand-raising action recognized by the wrist device, and the target hand-raising action is the hand-raising action recognized by the head-mounted device. The two can be the same hand-raising action of the user.

[0031] In the embodiments of the present application, the first instruction can be sent to the head-mounted device 100, and the head-mounted device 100 performs secondary judgment by shooting image data.

[0032] On the one hand, compared with inertial data, image data can more intuitively determine whether the user needs to view the wrist device, so as to reduce misjudgment and improve the effectiveness of screen turning on. For example, the camera assembly 120 of the head-mounted device 100 can shoot images in a relatively large range in front of the user. Therefore, when the head-mounted device 100 shoots image data including the wrist device, the motion track, orientation of the wrist device, and other actions, postures of the user, and the situation around the user can be further observed, so as to determine whether the user is likely to view the wrist device.

[0033] Therefore, the wrist device can only represent a hand-raising action, but does not necessarily represent that the user needs to view the wrist device. In many scenarios, the user can only have a hand-raising action or an approximate hand-raising action, and does not need to view the watch. Therefore, the head-mounted device 100 further judges the image data on the basis of the inertial data, which can help to improve the recognition accuracy of the hand-raising action.

[0034] On the other hand, devices such as smart glasses already have a camera structure for shooting image data. On this basis, without making corresponding improvements to the structure of the smart glasses and the wrist device, only the program needs to be updated. This way can help to reduce costs and popularize on existing wrist devices and head-mounted devices 100.

[0035] Meanwhile, in the present application, the wrist device will only send the first instruction to the head-mounted device 100 in the case of identifying the hand-raising action, and the head-mounted device 100 will only work after receiving the first instruction. Therefore, the wrist device does not need to send data to the head-mounted device 100 at all times, and the head-mounted device 100 also does not need to collect images and identify at all times, which will not produce more additional power consumption. Compared with reducing the power consumption of the bright screen, the power consumption will be lower, and the overall power consumption will be reduced.

[0036] In addition, although the wrist device has gone through the process of sending the first instruction, the head-mounted device 100 collecting images and identifying, and feeding back the second instruction, these processes will increase a certain delay, but the delay is small, generally less than 1s, and even can be less than 200ms, 100ms, etc., at the same time, the user's hand-raising action itself has a certain action time, and the delay caused by the above process is generally less than the user's action time, therefore, the process of the head-mounted device 100 secondary judgment will not affect the user's use experience.

[0037] In the embodiments of the present application, the head-mounted device 100 can include cameras for collecting different types of images, i.e. the first camera and the second camera. For the case that the head-mounted device 100 includes the first camera, the first gesture recognition module 130 is configured to acquire outward image data collected by the first camera, and identify whether the outward image data includes a target hand-raising action, so as to determine whether the initial hand-raising action identified by the wrist device is valid.

[0038] In this embodiment, the first camera is configured to collect outward image data, and the outward image data is image data in a preset direction other than the user's face, for example, an image 180° in front of the user. Based on this, it can be determined whether the wrist device is included in the outward image data, and then it is determined whether the user exists in the case of looking at the wrist device, and in the case of determining that the wrist device is included, it is determined that the hand-raising action is valid, so as to feed back the second instruction to the wrist device.

[0039] In some embodiments of the present application, the first camera is generally arranged at a fixed position on the head-mounted device 100, and the image direction collected by the first camera is also fixed, while the user's head direction is generally fixed when looking at the watch, accordingly, this will make the wrist device generally appear in a specific area in the outward image data, therefore, when identifying the hand-raising action by using the image data, it can also be determined whether the wrist device is in the specific area in the outward image data.

[0040] The above is only an example, and there are many ways to identify a specific object (wrist device) by using image data to determine whether there is a hand-raising and watch action, which will not be expanded here.

[0041] In the embodiments of the present application, the neural network module can be configured in the first gesture recognition module 130 to recognize the hand-raising action in the outward image data through the neural network module. Accordingly, the first gesture recognition module 130 can be a module including a processor such as a CPU, a GPU, an NPU, etc.

[0042] In this embodiment, the first gesture recognition module 130 includes a pre-trained second gesture recognition model, which is used to recognize whether the target hand-raising action is included based on the outward image data. The second gesture recognition model is trained based on labeled outward image training data. The labeled outward image training data includes a plurality of outward image data and whether each outward image data corresponds to a hand-raising action. By using the trained second gesture recognition model, the complexity of recognition can be reduced while the accuracy of recognition can be improved.

[0043] As mentioned before, in the embodiments of the present application, the camera assembly 120 further includes a second camera, which is used to collect inward image data, i.e. image data of a preset user eye direction. That is, after the head-mounted device 100 is worn, the second camera can collect images including the user's eyes.

[0044] Accordingly, in some embodiments of the present application, the first gesture recognition module 130 can be used to, in the case that the first wireless module 110 receives the first instruction, acquire the inward image data collected by the second camera, and recognize whether the target hand-raising action is included based on the inward image data, and then determine whether the initial hand-raising action recognized by the wrist device is valid.

[0045] In the embodiments of the present application, the inward image data includes the user's eyes, and therefore, the line-of-sight direction of the user can be recognized based on the inward image data to determine whether the line-of-sight direction of the user points to the direction where the wrist device is located. If it points to the direction where the wrist device is located, it indicates that the user is looking at the watch, and in this case, the second instruction can be fed back. Conversely, if it does not point to the direction where the wrist device is located, it indicates that the user is not looking at the watch, and the wrist device does not brighten the screen.

[0046] There can be various ways to recognize the user's line-of-sight, for example, in one embodiment, the second camera is an infrared camera; the first gesture recognition module 130 is used to perform eye tracking based on the inward image data to determine the line-of-sight direction; and in the case that the line-of-sight direction points to the preset watch direction, it is determined that the hand-raising action is valid. That is, the existing technology for eye tracking on infrared images can be used to determine the line-of-sight direction, and then it is determined whether the target hand-raising action is included to determine whether the initial hand-raising action is valid.

[0047] It should be noted that some wrist devices can be equipped with an infrared camera for eye recognition, but in such wrist devices, there are structural defects of complexity and redundancy. In addition to eye tracking, the infrared camera usually has no other functions, which leads to a small role of the infrared camera and occupies the volume of the wrist device, resulting in additional power consumption and cost.

[0048] In addition, except for some special cases, when the user watches the watch, the eyes are usually at a certain distance from the watch. If the infrared camera is too far from the eyes, the image accuracy of the captured image will be low, and the recognition accuracy will be low. At the same time, the shooting angle of the camera on the watch has a certain range, which is small. In many cases, if the user's wrist has a certain angle offset, the camera on the watch may not be able to capture the user's eyes, and thus cannot be recognized, which requires the user to be in a specific posture to be recognized, affecting the user's use comfort.

[0049] Compared with the watch, the head-mounted device 100 is closer to the eyeball, so that the second camera based on the head-mounted device 100 can collect inward image data including an eye image with higher accuracy, and thus the recognition can have higher accuracy. At the same time, the head-mounted device 100 is worn on the user's head, so that in the case that the camera angle is set, the image captured by the camera can effectively identify the user's line of sight, further improving the accuracy of the line of sight recognition and the accuracy of the screen-on control.

[0050] In the embodiments of the present application, the above eye tracking based on the inward image data to determine the line of sight direction can also be configured to be executed by a neural network model, and the specific implementation manner is not described here. Specifically, existing neural network technology can be used.

[0051] In some embodiments of the present application, the first gesture recognition module 130 is also configured to, in the case that the first wireless module 110 receives the first instruction, acquire outward image data and inward image data, and recognize whether the target hand-raising action is included based on the outward image data and the inward image data.

[0052] In this embodiment, the inward image data and the outward image data collected by the camera assembly 120 have a corresponding relationship, for example, can be collected at the same time, have corresponding identification information, etc.

[0053] In this embodiment, one of the outward image data and the inward image data can be recognized first, and in the case that the hand-raising action is recognized, the other image data is used to recognize the hand-raising action. In the case that both of them recognize the hand-raising action, it is determined that the target hand-raising action is included, and thus it can be determined that the initial hand-raising action recognized by the wrist device is valid.

[0054] In some embodiments, the outward image data and the inward image data can also be identified by using a neural network model. For example, the first gesture recognition module 130 includes a pre-trained first gesture recognition model, which is used to determine whether the target lifting hand action is included based on the outward image data and the inward image data. The first gesture recognition model is trained based on first training data, and the first training data includes labeled outward image data and inward image data and the association relationship therebetween. The label of each outward image data and inward image data represents whether the corresponding image data has the target lifting hand action.

[0055] The outward image data and the inward image data, and whether the corresponding image data has the lifting hand action. The training data can come from a plurality of testers wearing the head-mounted device, in which there are various lifting hand actions and various other actions, especially similar actions that are easily mistaken for lifting hand actions, such as running, fast walking, swinging arms, jumping rope, swinging a racket / club, throwing actions, clapping, waving, waving to say hello, turning the steering wheel while driving, turning on the light, shifting gears, holding the handle while riding / skating / scooter, typing, using a mouse, writing, turning the book / page, holding the phone to watch the screen / taking a selfie, answering a phone call, holding a cup while drinking / eating, washing face / brushing teeth / mouth, combing hair / blow dryer, cooking / stir-frying turning the pot, wiping the window / mopping / sweeping, holding dumbbells (bending / raising), bench press, push-ups, rowing machine, tennis / badminton / golf racket, shaking hands, saluting, waving to say hello, turning over at night, stretching, lifting arms to adjust the quilt, and the like.

[0056] In some embodiments, the training data provided by the embodiments of the present application can also include image data in which a standard lifting hand action exists but the user of the head-mounted device does not view the wrist device, and can also include image data in which the user views the wrist device but the lifting hand action is not standard or even does not have a lifting hand action.

[0057] Compared with single inward image data and outward image data, the neural network can combine the association between the inward image data and the outward image data when identifying the lifting hand action, so as to further improve the accuracy of the lifting hand action recognition and reduce the false screen lightening of the wrist device.

[0058] In the embodiments of the present application, the first gesture recognition model and the second gesture recognition model in the first gesture recognition module 130 can be a lightweight model, for example, a model distilled by a knowledge distillation technology. In this way, the size of the model can be reduced on the basis of ensuring a certain recognition accuracy, and the demand for device performance can be reduced.

[0059] In the embodiments of the present application, the first gesture recognition model and the second gesture recognition model can be different models that implement different functions in the first gesture recognition module 130, or different functions of the same model, which is not limited herein.

[0060] In the embodiments of the present application, the three ways in which the head-mounted device 100 performs after receiving the first instruction can be configured according to the device type, configured hardware, software, and other aspects of the head-mounted device 100.

[0061] For example, for smart glasses, it usually includes a camera for collecting outward image data, and even some smart glasses can be configured with an infrared camera to collect eye images. Therefore, for such smart glasses that are configured with a camera for collecting outward image data and an infrared camera, only outward head image data recognition, only inward image data recognition, or a combination of inward and outward image data recognition can be performed.

[0062] For another example, currently, true wireless earphones are not suitable for being configured with an infrared camera, so the true wireless earphones can be configured only with the first camera and use the outward image data for hand-raising action recognition.

[0063] The above is only an example, and the specific configuration can be selected according to actual needs, which will not be expanded herein.

[0064] Correspondingly, the embodiments of the present application also provide a wrist device that can be connected with the head-mounted device 100 provided in the foregoing embodiments to send the first instruction to the head-mounted device 100 and receive the second instruction.

[0065] Please refer to Figure 2 , Figure 2 A schematic diagram of a wrist device provided by an embodiment of the present application. In the wrist device 210 provided by the embodiments of the present application, it includes a second wireless module 210, an inertial sensor 220, and a second gesture recognition module 230.

[0066] The second wireless module 210 is configured to be connected with the head-mounted device 100, and specifically can be connected with the first wireless module 110. Wherein, the first and second wireless modules are used to distinguish different devices configured, and the first wireless module 110 and the second wireless module can use the same type of wireless module.

[0067] The inertial sensor 220 includes but is not limited to an accelerometer and a gyroscope. The accelerometer can be used to detect acceleration displacement and gravity direction, and the gyroscope can be used to detect angular velocity to depict the wrist rotation speed and amplitude, and improve real-time performance and robustness. For details, please refer to the prior art, which will not be expanded herein.

[0068] The second gesture recognition module 230 is connected with the second wireless module 210 and the inertial sensor 220 respectively, and is configured to recognize the initial hand-lifting action based on the inertial data, and feed back the first instruction to the head-mounted device 100 in the case that the hand-lifting action is recognized. The way of recognizing the hand-lifting action based on the inertial data can refer to the prior art.

[0069] In the embodiments of the present application, the wrist device 200 includes a display screen, a power supply and other structures, and the specific structures can refer to various wrist devices in the prior art, which will not be described here.

[0070] In some embodiments of the present application, the second gesture recognition module 230 can include a neural network model configured, which can recognize the initial hand-lifting action based on the inertial data, and the second gesture recognition module 230 can feed back the first instruction to the head-mounted device 100 after recognizing the initial hand-lifting action. In addition, the wrist device turns on the screen in the case that the second instruction of the head-mounted device 100 is received through the second wireless module. The second gesture recognition module 230 can be a module including a processor such as CPU, NPU, etc.

[0071] In some embodiments of the present application, when the second gesture recognition module 230 recognizes the hand-lifting action based on the inertial data, it determines whether the action corresponding to the inertial data is the initial hand-lifting action by confidence, for example, when the confidence of the hand-lifting action corresponding to the inertial data is greater than 90%, it is determined as the initial hand-lifting action.

[0072] Based on this, in some embodiments of the present application, the second gesture recognition module 230 can also be configured to determine that the confidence of the initial hand-lifting action recognized based on the inertial data is within a preset range before feeding back the first instruction to the head-mounted device 100. For example, the preset range can be greater than 90%.

[0073] In another embodiment of the present application, the preset range has an upper limit, which is less than 1, for example, 98%, and in this embodiment, the second gesture recognition module 230 is further configured to turn on the screen in the case that the confidence exceeds the upper limit of the preset range.

[0074] For example, the preset range can be 90% to 98%, when the confidence of the initial hand-lifting action determined based on the inertial data is 99%, the wrist device directly turns on the screen, and when the confidence of the hand-lifting action is determined to be 97%, the first instruction is sent to the head-mounted device 100 to instruct the head-mounted device 100 to make a second judgment.

[0075] That is, in this embodiment, the wrist device 200 directly turns on the screen in the case that the confidence of the hand-lifting action is high enough, without the need for a second judgment by the head-mounted playback device, which can reduce power consumption on the basis of ensuring the accuracy of turning on the screen.

[0076] For the convenience of understanding, an example is provided to illustrate the interaction process between the wrist device 2000 and the head-mounted device 100.

[0077] Firstly, when the user performs an action, the wrist device collects inertia data when performing the action through the inertia data collection, and identifies whether the inertia data includes a hand-lifting action through the second gesture recognition module, and in the case of determining that the hand-lifting action is included, determines whether to send a first instruction or turn on the screen according to the confidence of the hand-lifting action. For example, the preset range is 90% to 98%, the confidence of identifying that the inertia data includes the hand-lifting action is 80%, and the hand-lifting action is not included. When the confidence of including the hand-lifting action is 99%, the wrist device directly turns on the screen, and if the confidence of the hand-lifting action is 96%, the first instruction is sent to the head-mounted device 100.

[0078] That is, in the embodiment of the present application, if the confidence of the initial hand-lifting action is less than the lower limit of the preset range, it is determined that there is no initial hand-lifting action; if the confidence of the initial hand-lifting action is within the preset range, the first instruction is sent to the head-mounted device 100; if the confidence of the initial hand-lifting action is greater than the upper limit of the preset range, the wrist device directly turns on the screen, and the first instruction is not sent to the head-mounted device 100.

[0079] The specific numerical values mentioned in the above embodiments are only examples and should not be considered as a limitation of the present application. In the embodiment of the present application, the confidence of the initial hand-lifting action recognized by the second gesture recognition module can be configured according to actual needs.

[0080] After receiving the first instruction, the head-mounted device 100 obtains image data captured by the camera assembly 120, and identifies a target hand-lifting action based on the image data. For example, the first gesture recognition module 130 of the head-mounted device 100 can obtain outward image data captured by the first camera, and identify the target hand-lifting action based on the outward image data; or the first gesture recognition module 130 of the head-mounted device 100 can obtain inward image data captured by the second camera, and identify the hand-lifting action based on the inward image data; or the first gesture recognition module 130 of the head-mounted device 100 can obtain outward image data captured by the first camera and inward image data captured by the second camera, and identify the target hand-lifting action based on the outward image data and the inward image data. The first gesture recognition module 130 is configured with a neural network model to identify whether the inward image data and / or the outward image data includes the target hand-lifting action through the neural network model.

[0081] Finally, the first gesture recognition module 130 feeds back a second instruction to the wrist device in the case of recognizing the hand lifting action, and the wrist device turns on the screen in the case of receiving the second instruction. Alternatively, the first gesture recognition module 130 does not turn on the screen in the case of not recognizing the target hand lifting action.

[0082] Based on the same inventive concept, the embodiments of the present application also provide a screen-on identification system, which can include the wrist device and the head-mounted device 100 provided in the foregoing embodiments.

[0083] The head-mounted device 100 includes a first wireless module 110, a camera assembly 120, and a first gesture recognition module 130. The wrist device includes a second wireless module, an inertial sensor, and a second gesture recognition module. The first wireless module 110 is in communication connection with the second wireless module.

[0084] The wrist device collects inertial data through the inertial sensor, and the second gesture recognition module is configured to recognize an initial hand lifting action based on the inertial data, and feed back a first instruction to the head-mounted device 100 through the second wireless module in the case of recognizing the hand lifting action. The head-mounted device 100 receives the first instruction through the first wireless module 110, the first gesture recognition module 130 acquires image data collected by the camera assembly 120, and recognizes a target hand lifting action based on the image data, and sends a second instruction to the wrist device through the second wireless module in the case of recognizing the target hand lifting action based on the image data. Finally, the wrist device turns on the screen in the case of receiving the second instruction through the second wireless module.

[0085] The head-mounted device 100 and the wrist device in the screen-on identification system can refer to the foregoing embodiments, which will not be repeated here.

[0086] Based on the same inventive concept, the present application also provides a screen-on identification method, which can be applied to the head-mounted device 100 provided in the foregoing embodiments, wherein the head-mounted device 100 is connected with a wrist device.

[0087] Please refer to Figure 3 , Figure 3 A flowchart of a screen-on identification method for a head-mounted device is provided in an embodiment of the present application. The screen-on identification method includes the following steps. S310, in the case of receiving a first instruction sent by the wrist device, acquiring image data collected by a camera assembly.

[0088] The first instruction represents that the wrist device recognizes an initial hand lifting action.

[0089] S320, recognizing whether the image data includes a target hand lifting action.

[0090] S330, in a case where it is determined that the target hand-raising action is included, sending a second instruction indicating that the wrist device turns on the screen to the wrist device.

[0091] Based on the same inventive concept, the embodiment of the present application further provides a screen-on identification method, which can be applied to the wrist device provided by the foregoing embodiment, wherein the head-mounted device is connected with the wrist device.

[0092] Please refer to Figure 4 , Figure 4 A flow chart of a screen-on identification method applied to a wrist device is provided by an embodiment of the present application, and the screen-on identification method comprises: S410, collecting inertial data.

[0093] S420, identifying an initial hand-raising action based on the inertial data.

[0094] S430, in a case where the initial hand-raising action is identified, feeding back a first instruction to the head-mounted device.

[0095] Wherein, the head-mounted device identifies whether a target hand-raising action is included in a case where the first instruction is received, and sends a second instruction to the wrist device in a case where it is determined that the target hand-raising action is included.

[0096] S440, turning on the screen in a case where the second instruction of the head-mounted device is received.

[0097] The screen-on identification method provided by the foregoing embodiments has the same function as the foregoing head-mounted device, wrist device and screen-on identification system of the two, and specific reference can be made to the foregoing content, which will not be described here. In the embodiments provided by the present application, it should be understood that the disclosed method and device can also be implemented in other ways. The device embodiments described above are only illustrative. The function modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0098] The above embodiments can be freely combined without conflict, and the embodiments obtained by combination are covered by the protection scope of the present application.

[0099] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be limited by the protection scope of the claims.

[0100] It has to be noted that, as used herein, the terms "includes" and / or "contains", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. No limitation is implied by such statements expressing the inclusion of an element, unless more limitation are explicitly described in connection with the statement that the element is included.

Claims

1. A head-mounted device, characterized in that, include: A first wireless module, a camera component, and a first gesture recognition module; The first gesture recognition module is connected to both the first wireless module and the camera component. The camera component is used to acquire image data; The first wireless module is used to connect to the wrist device and receive a first instruction sent by the wrist device; the first instruction indicates that the wrist device has recognized an initial hand-raising motion; The first gesture recognition module is used to acquire image data collected by the camera component when the first wireless module receives the first instruction, and to identify whether the image data includes a target raising hand action. The first gesture recognition module is also used to send a second instruction to the wrist device to instruct the wrist device to turn on its screen when the target raises its hand.

2. The head-mounted device according to claim 1, characterized in that, The camera component includes a first camera, which is connected to the first gesture recognition module; The first camera is used to collect outward image data, which is image data in a preset direction other than the user's face. The first gesture recognition module is used to acquire outward image data captured by the first camera, and to identify whether the outward image data includes the target's hand-raising action.

3. The head-mounted device according to claim 2, characterized in that, The camera assembly also includes a second camera, which is used to acquire inward image data, the inward image data being image data of a preset user eye direction; The second camera is connected to the first gesture recognition module; The first gesture recognition module is further configured to, when the first wireless module receives the first instruction, acquire the outward image data and the inward image data, and determine whether the outward image data and the inward image data include the target raising hand action.

4. The head-mounted device according to claim 3, characterized in that, The first gesture recognition module includes a pre-trained first gesture recognition model, which is used to determine whether the outward image data and the inward image data include the target raising hand action. The first gesture recognition model is trained based on the first training data, which includes labeled outward image data and inward image data and the relationship between them.

5. The head-mounted device according to claim 2, characterized in that, The first gesture recognition module includes a pre-trained second gesture recognition model, which is used to identify whether the outward image data includes the target raising hand action. The second gesture recognition model is trained based on labeled outward-facing image training data.

6. The head-mounted device according to claim 1, characterized in that, The camera assembly also includes a second camera, which is used to acquire inward image data, the inward image data being image data of a preset user eye direction; The second camera is connected to the first gesture recognition module; The first gesture recognition module is used to acquire inward image data collected by the second camera when the first wireless module receives the first instruction, and to identify whether the target's hand-raising action is included based on the inward image data.

7. The head-mounted device according to claim 6, characterized in that, The second camera is an infrared camera; the first gesture recognition module is used for: Eye tracking is performed based on the inward image data to determine the direction of gaze; When the line of sight points in the direction of the preset watch, the action of raising the target's hand is determined.

8. A wrist device, characterized in that, include: The second wireless module, the inertial sensor, and the second gesture recognition module; The second gesture recognition module is connected to the second wireless module and the inertial sensor, respectively; The second wireless module is used for communication connection with the head-mounted device as described in any one of claims 1-7; The inertial sensor is used to collect inertial data; The second gesture recognition module is used to recognize the initial hand-raising action based on the inertial data, and to send a first command to the head-mounted device when the initial hand-raising action is recognized; Furthermore, the wrist device is also used to light up the screen upon receiving a second instruction from the head-mounted device via the second wireless module.

9. The wrist device according to claim 8, characterized in that, The second gesture recognition module is also used for: Before sending the first instruction to the head-mounted device, it is determined that the confidence level corresponding to the initial hand-raising action identified based on the inertial data is within a preset range.

10. The wrist device according to claim 9, characterized in that, The second gesture recognition module is also used to: turn on the screen when it is determined that the confidence level exceeds the upper limit of the preset range.

11. A screen-on recognition system, characterized in that, include: Wrist-worn devices and head-mounted devices; The head-mounted device includes: a first wireless module, a camera component, and a first gesture recognition module; The wrist device includes: a second wireless module, an inertial sensor, and a second gesture recognition module; The first wireless module is communicatively connected to the second wireless module; The wrist device collects inertial data through the inertial sensor, and the second gesture recognition module is used to recognize the initial hand raising action based on the inertial data, and when the initial hand raising action is recognized, it feeds back a first command to the head-mounted device through the second wireless module. The head-mounted device receives the first instruction through the first wireless module, the first gesture recognition module acquires the image data collected by the camera component, and recognizes the target's hand raising action based on the image data. When the target's hand raising action is recognized based on the image data, the second instruction is sent to the wrist device through the second wireless module. The wrist device lights up when it receives the second instruction via the second wireless module.

12. A screen-on recognition method, characterized in that, Applied to a head-mounted device as described in any one of claims 1-7, wherein the head-mounted device is connected to a wrist device, the screen-on recognition method includes: Upon receiving a first instruction from the wrist device, the image data captured by the camera component is acquired; the first instruction indicates that the wrist device has recognized an initial hand-raising motion. Based on the image data, determine whether the target's hand-raising action is included; If the target arm raises, a second instruction is sent to the wrist device to instruct the wrist device to turn on its screen.

13. A screen-on recognition method, characterized in that, Applied to a wrist device as described in any one of claims 8-10, wherein the wrist device is connected to a head-mounted device, the screen-on recognition method includes: Collect inertial data; The initial hand-raising motion is identified based on the inertial data; Upon recognizing the initial hand-raising motion, a first instruction is sent to the head-mounted device; upon receiving the first instruction, the head-mounted device acquires image data to identify whether the target hand-raising motion is included, and if the target hand-raising motion is determined to be included, sends a second instruction to the wrist device. And, upon receiving a second instruction from the head-mounted device, the screen lights up.

Images