Wearable device control method and device, wearable device and medium

Abstract

The invention relates to a control method and device of wearable equipment, the wearable equipment and a medium. The control method comprises the steps that a target image and auxiliary information are acquired; determining body movement information based on a pre-stored first judgment model and the target image; determining auxiliary action information based on a pre-stored second judgment model and the auxiliary information; and determining a control instruction based on the limb action information and the auxiliary action information. The auxiliary action information can accurately judge whether the body action is touched or not, the body action information determined based on the target image is combined with the auxiliary action information to determine the body action of the user, even if the user quickly makes the body action, the action instruction made by the user can be accurately judged, and the user experience is improved. The interaction efficiency of the user and the wearable device is improved, the probability of misoperation is reduced, and the use experience of the user is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of electronic equipment, and in particular to a control method and device for a wearable device, a wearable device, and a medium. Background Art

[0002] With the development of technology, wearable devices such as smart watches and smart bracelets are widely used in daily life, providing users with convenient and intelligent functions to help users manage time, health and communication. During use, users can interact with wearable devices using gestures, and wearable devices perform operations corresponding to gestures based on the recognition of gestures. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a control method and apparatus for a wearable device, a wearable device and a medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a control method for a wearable device is provided, the control method comprising:

[0005] Acquire a target image, where the target image is used to represent a limb movement;

[0006] Acquiring auxiliary information, wherein the auxiliary information includes at least one of a touch signal, a bioelectric signal, and an inertial signal;

[0007] Determine limb motion information based on a pre-stored first judgment model and the target image, wherein the limb motion information is used to characterize a position change of a limb;

[0008] Determine auxiliary action information based on the pre-stored second judgment model and the auxiliary information, wherein the auxiliary action information is used to indicate whether physical contact occurs;

[0009] Based on the limb motion information and the auxiliary motion information, a control instruction is determined.

[0010] In some exemplary embodiments of the present disclosure, the method for forming the second judgment model includes:

[0011] Training the first neural network model based on the labeled sample information to obtain the second judgment model;

[0012] The sample information includes at least one of a touch sample signal, a bioelectric signal sample signal and an inertial sample signal. If the sample signals included in the sample information are different, the obtained second judgment model is different.

[0013] The mark includes a first mark and a second mark, the first mark is used to indicate that the sample information is a sample in which physical contact occurs, and the second mark is used to indicate that the sample information is a sample in which physical contact does not occur.

[0014] In some exemplary embodiments of the present disclosure, the bioelectric signal is generated based on action potential and / or resting potential; and / or,

[0015] The inertial signal includes at least one of acceleration and angular acceleration.

[0016] In some exemplary embodiments of the present disclosure, the method for forming the first judgment model includes:

[0017] Training a second neural network model based on a limb sample image with action identification to obtain the first judgment model, wherein the target area in the limb sample image has a preset mark;

[0018] The action identifier is used to indicate the limb action feature represented by the preset mark in each limb sample image.

[0019] In some exemplary embodiments of the present disclosure, determining the body motion information based on the pre-stored first judgment model and the target image includes:

[0020] Based on the target image, obtaining motion features of a target area in the target image;

[0021] The limb movement information is determined based on the movement feature and the first judgment model.

[0022] In some exemplary embodiments of the present disclosure, the auxiliary action information includes first auxiliary information and second auxiliary information, the first auxiliary information is used to characterize the occurrence of physical contact, and the second auxiliary information is used to characterize the absence of physical contact, the physical action information includes hand action information, and determining the control instruction based on the physical action information and the auxiliary action information includes:

[0023] If the hand motion information represents a target hand motion and the auxiliary motion information is the first auxiliary information, it is determined that the control instruction is a confirmation instruction.

[0024] In some exemplary embodiments of the present disclosure, the determining the motion control instruction based on the limb motion information and the auxiliary motion information further includes:

[0025] If the hand motion information represents a target hand motion and the auxiliary motion information is the second auxiliary information, it is determined that the control instruction is invalid.

[0026] According to a second aspect of an embodiment of the present disclosure, a control device for a wearable device is provided, the control device for the wearable device comprising:

[0027] An acquisition module, used for acquiring a target image, wherein the target image is used for representing a limb movement;

[0028] The acquisition module is further used to acquire auxiliary information, wherein the auxiliary information includes at least one of a touch signal, a bioelectric signal and an inertial signal;

[0029] A determination module, used to determine limb motion information based on a pre-stored first judgment model and the target image, wherein the limb motion information is used to characterize a position change of a limb;

[0030] The determination module is further used to determine auxiliary action information based on the pre-stored second judgment model and the auxiliary information, wherein the auxiliary action information is used to indicate whether physical contact occurs;

[0031] The determination module is further used to determine a control instruction based on the limb movement information and the auxiliary movement information.

[0032] According to a third aspect of an embodiment of the present disclosure, a wearable device is provided, the wearable device comprising:

[0033] processor;

[0034] a memory for storing processor-executable instructions;

[0035] Among them, the processor is configured to execute the executable instructions in the memory to implement the control method of the wearable device provided in the first aspect of the present disclosure.

[0036] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided, on which executable instructions are stored. When the executable instructions are executed by a processor, the control method of the wearable device provided in the first aspect of the present disclosure is implemented.

[0037] The above method disclosed in the present invention has the following beneficial effects: the auxiliary action information in the present invention can accurately determine whether a touch occurs in the limb movement, and the limb movement information determined based on the target image can be combined with the auxiliary action information to determine the user's limb movement. Even if the user makes a limb movement quickly, the action command made by the user can be accurately determined, thereby improving the interaction efficiency between the user and the wearable device, reducing the probability of misoperation, and improving the user's experience.

[0038] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0040] Figure 1 The figure is a flowchart of a method for controlling a wearable device according to an exemplary embodiment.

[0041] Figure 2 The figure is a flowchart of a method for controlling a wearable device according to an exemplary embodiment.

[0042] Figure 3 The invention is a block diagram of a control device for a wearable device according to an exemplary embodiment.

[0043] Figure 4 is a block diagram of a wearable device according to an exemplary embodiment. DETAILED DESCRIPTION

[0044] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0045] In the related art, wearable devices use cameras to recognize the user's hand image, recognize the user's gesture based on the recognition and tracking of the user's hand key points, combined with gesture classification, and perform the corresponding operation of the gesture. However, due to the low accuracy of the hand key point detection algorithm, when using the hand key points for recognition, it is impossible to accurately determine whether the gesture has finger contact in a short time, and thus it is impossible to accurately determine the user's gesture command, which is easy to cause misjudgment. If you want to improve the reliability of gesture commands, users need to increase the dwell time of the gesture or combine multiple gestures for comprehensive judgment, which affects the efficiency of users using gestures to interact with wearable devices, and the user experience is not good.

[0046] In order to solve the above problems, the present disclosure provides a control method for a wearable device, which uses a camera to obtain a target image and a sensor to obtain auxiliary information; based on the pre-stored first judgment model and the second judgment model, the body movement information and the auxiliary movement information are respectively determined. By adopting the method in the present disclosure, there is no need for the user to increase the stay time of the body movement in front of the camera, and there is no need to increase the number of cameras, so there is no need to increase the hardware cost; at the same time, based on the body movement information collected by the camera, combined with the auxiliary movement information, it is possible to accurately judge whether a body touch occurs, thereby improving the accuracy of the judgment of the body movement and improving the accuracy and reliability of the control process. In addition, the control instructions issued by the user are comprehensively determined using the body movement information and the auxiliary movement information, which improves the interaction efficiency between the user and the wearable device, reduces the probability of misjudgment, and improves the user experience.

[0047] The exemplary embodiments of the present disclosure provide a control method for a wearable device, which is applied to a wearable device having a camera and a sensor. The wearable device may be a smart watch, a smart bracelet, a smart ring, etc. Figure 1 As shown, the control method of the wearable device shown in the present disclosure includes:

[0048] S101, obtaining a target image, where the target image is used to represent a body movement;

[0049] S102, obtaining auxiliary information, where the auxiliary information includes at least one of a touch signal, a bioelectric signal, and an inertial signal;

[0050] S103, determining limb motion information based on a pre-stored first judgment model and a target image, where the limb motion information is used to characterize a position change of a limb;

[0051] S104, determining auxiliary action information based on the pre-stored second judgment model and auxiliary information, where the auxiliary action information is used to indicate whether physical contact occurs;

[0052] S105. Determine control instructions based on the limb movement information and the auxiliary movement information.

[0053] In step S101, the wearable device uses a camera to obtain a target image representing the user's body movements. In order to ensure that the target image can be obtained at any time, the camera can be in a normally open state to obtain the user's target image in real time. In addition, in order to reduce the power consumption caused by the camera, when the user uses the wearable device to view messages, reply to messages, and other operations that do not require the camera, the camera can be temporarily turned off and turned on again after the user has finished using the camera; the camera can also be turned on at a fixed interval.

[0054] Among them, body movements may include user's hand movements, and may also include user's hand and arm movements, etc. The content in the target image is related to the field of view of the camera and the distance between the user's body and the camera. If the field of view of the camera is large and the user's body is at a certain distance from the camera, the camera can capture the user's body movements in a wider range. For example, the captured target image includes the movements of the user's arm and fingers, and the target image reflects the user bending his arm and sliding his index finger upward. If the field of view of the camera is small and the user's body is close to the camera, the camera can only capture the user's body movements in a smaller range. For example, the captured target image only includes the movements of the user's fingers and palms, and the target image reflects the user's palm facing the camera with five fingers together.

[0055] It should be noted that since the action instructions made by the user are generally continuous actions, the target image is actually a continuous animation containing multiple frames of images, so as to more accurately determine the user's action instructions. When collecting the user's actions, key point tracking technology can be used to track the positions of multiple key positions of the user's hands, so as to identify the user's body movements.

[0056] In step S102, the wearable device acquires auxiliary information collected by various sensors, the auxiliary information being non-image information, including at least one of touch signals, bioelectric signals, and inertial signals. The content of the auxiliary information collected varies depending on the type of sensor set on the wearable device.

[0057] For touch signals, touch sensors can be built into wearable devices to collect touch signals. The touch sensor can be a capacitive sensor or a resistive sensor. When the user's finger touches the screen of the wearable device, the touch sensor collects the pressure information acting on the screen and converts the pressure information into a touch signal. The touch signal can reflect whether the user's limbs are in contact with the screen and perform touch operations. When the wearable device collects the target image through the camera and determines the user's limb movements based on the target image, it may not be able to accurately determine whether the user has touched the display screen of the wearable device due to the influence of the algorithm accuracy. By adding a touch sensor, the touch signal of the user's limbs acting on the wearable device is collected as auxiliary information to improve the accuracy and reliability of the control instructions determined based on the user's operation.

[0058] Inertial signals can be obtained from the built-in motion sensors of wearable devices. Motion sensors may include IMU (Inertial Measurement Unit), acceleration sensors, gravity sensors, gyroscopes, etc. Motion sensors generate inertial signals based on the state of the wearable device. Inertial signals can reflect the acceleration, angular velocity, tilt angle and other information of the wearable device. Inertial signals include at least one of acceleration and angular velocity. For example, when the user's thumb and index finger move to touch, the acceleration and angular velocity of the two fingers change due to the movement of the two fingers. The acceleration signal or angular acceleration signal can be captured by the motion sensor. The acceleration or angular velocity collected by the motion sensor is used as auxiliary information. The wearable device uses the auxiliary information combined with the target image to improve the accuracy of judging the user's body movements.

[0059] The wearable device also has a built-in bioelectric signal sensor, which is used to collect the user's bioelectric signal. The generation of bioelectric signals is based on changes in cell membrane potential or polarity state. When the cell is at rest, its bioelectric signal is at resting potential; when the cell is excited, its bioelectric signal is at action potential. Among them, bioelectric signals include skin electrical signals, muscle electrical signals, etc. As an electrical signal, bioelectric signals can be directly collected and recorded by bioelectric signal sensors. Based on bioelectric signals, the user's behavior and state can be identified and predicted. For example, when the user's two fingers touch together and there is a certain squeezing action, the user's muscle electrical signal will change. The changes in muscle electrical signals collected by the bioelectric signal sensor are used as auxiliary information. The auxiliary information is combined with the target image, and the user's control instructions can be clearly determined in the subsequent judgment process, thereby improving the control accuracy.

[0060] In addition, it should be noted that the more types of signals the auxiliary information contains, the higher the reliability of the auxiliary information in the subsequent judgment process. The auxiliary action information obtained based on the auxiliary information (described in detail later) combined with the limb movement information obtained based on the target image (described in detail later) can more accurately judge the user's control commands and improve the reliability of the control commands.

[0061] In step S103, the first judgment model is a neural network model pre-stored in the wearable device after training. The neural network can be a convolutional neural network, a fully connected neural network, a generative adversarial network, etc., and can be selected based on the memory and performance of the wearable device. During the training of the first judgment model, the target image when the user uses body movements to control the wearable device is collected and marked as a learning sample and input into the neural network for training. The trained neural network model can output body movement information based on the target image obtained by the wearable device. The body movement information is used to determine the position change of the user's body, that is, the first judgment model can recognize the user's body movement. For example, the wearable device obtains a target image of the user's five fingers open, inputs the target image into the first judgment model, and determines that the output body movement information is the user's five fingers open.

[0062] In step S104, the second judgment model is a neural network model pre-stored in the wearable device after training. The neural network used in the second judgment model can be the same as the neural network type of the first judgment model, or it can be different. For example, the first judgment model and the second judgment model are both obtained by training with a convolutional neural network; for another example, the first judgment model is obtained by training with a convolutional neural network, and the second judgment model is obtained by training with a generative adversarial network. During the training process of the second judgment model, the auxiliary information when the user uses body movements to control the wearable device is collected and marked as a learning sample and input into the neural network for training. After the training, the neural network model can output auxiliary action information based on the auxiliary information obtained by the wearable device. The auxiliary action information is used to characterize whether a body touch occurs, that is, the second judgment model can identify more detailed touch actions. For example, the wearable device obtains a bioelectric signal generated based on an action potential, inputs the auxiliary information into the second judgment model, and the auxiliary action information output by the second judgment model determines the occurrence of a body touch.

[0063] In step S105, due to the specifications and performance limitations of the wearable device, the target image captured by the camera has limited content, and the image quality such as clarity and brightness is low. In addition, due to the accuracy of the image algorithm, the target image may only reflect the approximate content of the user's body movements. The wearable device cannot use the first judgment model and the target image to judge the user's complex body movements. For example, the body movement information can only reflect that the user's thumb and index finger are close, but it cannot identify whether the thumb and index finger are touching. Therefore, the wearable device needs to use the body movement information and the auxiliary movement information to jointly judge the user's body movement, that is, it needs to identify the user's body movement and confirm whether there is a body touch, so as to improve the recognition accuracy of the body movement and ensure that the user can use the body movement to accurately control the wearable device.

[0064] The wearable device pre-stores control instructions corresponding to the limb motion information and the auxiliary motion information. When the wearable device determines the limb motion information based on the first judgment model and combines the auxiliary motion information determined based on the second judgment model, the control instruction can be determined, and the wearable device implements the operation corresponding to the control instruction. In one example, the wearable device pre-stores the correspondence between the control instruction and the user's action, and each control instruction has corresponding limb motion information and auxiliary motion information. For example, the touch of the thumb and the index finger is a page turning instruction. When the acquired limb motion information reflects that the first knuckle of the thumb and the first knuckle of the index finger are close, and the auxiliary motion information reflects that the first knuckle of the thumb and the first knuckle of the index finger touch, it is determined that the user triggers the page turning instruction. It should be noted that when there is no limb touch in the limb motion corresponding to the control instruction, the wearable device can determine the control instruction based only on the limb motion information, without using the auxiliary motion information. For example, if the five fingers are opened as a cancel or return instruction, the control instruction can be determined based only on the target image and the limb motion information generated by the first judgment model.

[0065] In the present disclosure, the auxiliary action information can accurately determine whether the limb action involves touch. Therefore, the limb action information is combined with the auxiliary action information to determine the user's limb action, which can improve the accuracy of limb action judgment. The user does not need to increase the dwell time of the limb action, thereby improving the interaction efficiency between the user and the wearable device, reducing the possibility of accidental touch, and improving the user's experience.

[0066] According to an exemplary embodiment, Figure 2 As shown, the control method of the wearable device in this embodiment includes:

[0067] S201, training a first neural network model based on sample information with labels to obtain a second judgment model;

[0068] S202, training a second neural network model based on the limb sample images with action identifiers to obtain a first judgment model;

[0069] S203, obtaining a target image, where the target image is used to represent the body movement;

[0070] S204, obtaining auxiliary information, where the auxiliary information includes at least one of a touch signal, a bioelectric signal, and an inertial signal;

[0071] S205, based on the target image, obtaining the motion features of the target area in the target image;

[0072] S206, determining body movement information based on the movement feature and the first judgment model;

[0073] S207, determining auxiliary action information based on the pre-stored second judgment model and auxiliary information, where the auxiliary action information is used to indicate whether physical contact occurs;

[0074] S208: if the hand motion information represents the target hand motion and the auxiliary motion information is the first auxiliary information, determine that the control instruction is a confirmation instruction;

[0075] S209: If the hand motion information represents the target hand motion and the auxiliary motion information is the second auxiliary information, determine that the control instruction is invalid.

[0076] Among them, steps S203, S204, and S207 are implemented in the same way as steps S101, S102, and S104 in the above embodiment, and are not repeated here.

[0077] Step S201 is a method for forming a second judgment model, and the sample information includes at least one of a touch sample signal, a bioelectric signal sample signal, and an inertial sample signal. The selection of sample information is determined based on the type of sensor set on the wearable device. When multiple sensors are set on the wearable device, multiple sample signals need to be trained to ensure reliability and accuracy when the second judgment model is used for subsequent judgment.

[0078] Since the sample information includes multiple types, when the sample information contains different sample signals, the second judgment model obtained by training is different. If the first neural network model is trained using the touch sample signal, a second judgment model based on the touch signal is obtained to judge whether a physical touch occurs; if the first neural network model is trained using the bioelectric signal sample signal, a second judgment model based on the bioelectric signal is obtained to judge whether a physical touch occurs; if the first neural network model is trained using the inertia sample signal, a second judgment model based on the inertia signal is obtained to judge whether a physical touch occurs.

[0079] When the sample information includes two or three sample signals, the weight values ​​of the two or three sample signals can be determined based on the influence of various signals during physical touch, so that the trained first judgment model can integrate two or three auxiliary information, output auxiliary action information, and judge whether physical touch occurs. For example, the sample information includes touch sample signals, bioelectric signal sample signals, and inertial sample signals. When training the first neural network model, the weight values ​​are determined according to the influence of the three sample information on judging whether physical touch occurs. Since the relationship between the influence of the three signals is that the touch sample signal is greater than the inertial sample signal and greater than the bioelectric signal sample signal, the weight value of the touch sample signal in the first neural network model can be assigned to be 0.4, the weight value of the inertial sample signal is 0.35, and the weight value of the bioelectric signal sample signal is 0.25. The first neural network model processes the sample information according to the three weight values ​​for training.

[0080] Each sample information will be marked as a first mark or a second mark. The first mark indicates that the sample information is a sample with physical contact, and the second mark indicates that the sample information is a sample without physical contact. For example, the first mark is 1 and the second mark is 0, that is, all sample information with physical contact is marked as 1, and all sample information without physical contact is marked as 0. The sample information marked with 1 or 0 is input into the first neural network model for training. After the training, the first neural network model can output auxiliary action information based on the input auxiliary information. The first neural network model after the training is the second judgment model. The auxiliary action information includes the first auxiliary information and the second auxiliary information. The first auxiliary information is used to represent the occurrence of physical contact, and the second auxiliary information is used to represent the absence of physical contact. For example, the first auxiliary information is marked as 1 and the second auxiliary information is marked as 0. When the auxiliary information is a bioelectric signal based on action potential, the second judgment model outputs auxiliary action information with a result of 1 based on the input auxiliary information, indicating that the user's body movement has physical touch; when the auxiliary information is a touch signal with a large and dispersed touch area, the second judgment model outputs auxiliary action information with a result of 0 based on the input auxiliary information, indicating that the user's body movement has not caused physical touch.

[0081] In step S202, the second neural network model is trained using sample images of limbs with action labels. When the second neural network model can output results consistent with the action labels, the training of the second neural network model is completed and can be used as the first judgment model.

[0082] Since the image information is relatively complex, in order to facilitate the learning of the second neural network, the target area in the limb sample image is marked with a preset mark to represent the key points in the limb sample image. The preset mark can be a black dot, a black triangle, a black square, etc. The target area in the limb sample image is the hand area, that is, each finger in the hand area is marked with a preset mark. The more preset marks there are, the more limb movement features the limb sample image can reflect. Among them, the number of preset marks is determined based on the magnitude of the second neural network model. If the second neural network model is a small lightweight neural network, the preset mark can be selected to mark the joint part of each finger in the target area; if the second neural network model is a large neural network, the preset mark can be selected to mark the joint part of each finger in the target area and between the joints. The action mark is used to indicate the limb movement feature represented by the preset mark in each limb sample image, that is, the action mark is the target that the second neural network model needs to learn. For example, a limb sample image is five fingers open, and the joints of the five fingers of the hand are marked with preset marks of black dots. The preset marks are arranged in a divergent shape, and the limb sample image is marked with the action mark of "five fingers open".

[0083] In step S205, the positions of the key points of the target area in the target image are determined by analyzing the target image, for example, the coordinate information corresponding to the key points is determined. As the user's body movements change, the positions of the key points of the target area in the target image change synchronously, that is, there are multiple positions and combinations of key points, each of which corresponds to an action feature, so the action feature can be determined based on the relationship between the coordinate information corresponding to each key point. For example, the wearable device obtains a target image of an "OK" gesture, and the key points corresponding to the middle finger, ring finger, and little finger in the target image are arranged in a divergent shape, and the key points corresponding to the index finger and thumb are arranged in a circle. The action feature of "OK" can be determined based on the coordinate information corresponding to each key point on the five fingers.

[0084] In step S206, since the first judgment model is trained based on sample images of limbs with action identifiers, the first judgment model can determine the limb movement information based on the action features of the target image, that is, determine the user's specific limb movement based on the action features corresponding to the key point positions of the target image.

[0085] In step S208, since the field of view of the camera of the wearable device is limited and the fingers can perform a wide range of movements, in actual use, hand movements can be used to interact with the wearable device, that is, the hand movement information in the body movement information is used to determine the control instruction. If the hand movement information represents the target hand movement and the auxiliary movement information is the first auxiliary information, that is, the wearable device can determine the specific hand movement of the user and clearly indicates that there is a finger touch on the hand, then the control instruction is determined to be a confirmation instruction.

[0086] In step S209, if the hand motion information represents the target hand motion and the auxiliary motion information is the second auxiliary information, that is, the wearable device can determine the specific hand motion of the user and it is clear that there is no finger touch on the hand, then it is determined that the control instruction is invalid.

[0087] It should be noted that the hand movement information and auxiliary movement information corresponding to the control instructions are not fixed. Users can change the hand movements when interacting with the wearable device according to their personal usage habits. For example, the control instructions corresponding to all hand movements whose auxiliary movement information is the first auxiliary information can be set to invalid instructions, and the control instructions corresponding to all hand movements whose auxiliary movement information is the second auxiliary information can be set to confirmed instructions.

[0088] In the present disclosure, the second neural network model is trained using sample images to obtain the first judgment model, thereby improving the accuracy of determining the body movement information; the first neural network model is trained using sample information to obtain the second judgment model, thereby improving the accuracy of judging whether a body touch occurs. Thus, the body movement information and auxiliary movement information can be used to accurately identify the user's specific body movements and movement details, thereby improving the accuracy and speed of the user's use of body movements to control the wearable device, thereby improving the user's experience.

[0089] An exemplary embodiment of the present disclosure provides a control device for a wearable device. Figure 3 As shown, a block diagram of a control device for a wearable device shown in the present disclosure.

[0090] The block diagram includes: an acquisition module 31 and a determination module 32. The acquisition module 31 is used to acquire a target image, which is used to characterize body movements; the acquisition module 31 is also used to acquire auxiliary information, which includes at least one of a touch signal, a bioelectric signal, and an inertial signal; the determination module 32 is used to determine body movement information based on a pre-stored first judgment model and a target image, which is used to characterize a position change of a body; the determination module 32 is also used to determine auxiliary movement information based on a pre-stored second judgment model and auxiliary information, which is used to characterize whether a body touch occurs; the determination module 32 is also used to determine a control instruction based on the body movement information and the auxiliary movement information.

[0091] In an exemplary embodiment of the present disclosure, the acquisition module 31 is also used to: train the first neural network model based on sample information with a label to obtain a second judgment model; wherein the sample information includes at least one of a touch sample signal, a bioelectric signal sample signal and an inertial sample signal, and if the sample information includes different sample signals, the obtained second judgment model is different; the label includes a first label and a second label, the first label is used to characterize the sample information as a sample in which physical touch has occurred, and the second label is used to characterize the sample information as a sample in which no physical touch has occurred.

[0092] In an exemplary embodiment of the present disclosure, the bioelectric signal is generated based on an action potential and / or a resting potential; and / or the inertial signal includes at least one of acceleration and angular acceleration.

[0093] In an exemplary embodiment of the present disclosure, the acquisition module 31 is also used to: train a second neural network model based on limb sample images with action identifiers to obtain a first judgment model, wherein the target area in the limb sample images has a preset mark; wherein the action identifier is used to indicate the limb movement characteristics represented by the preset mark in each limb sample image.

[0094] In an exemplary embodiment of the present disclosure, the determination module 32 is further used to: obtain motion features of a target area in the target image based on the target image; and determine limb motion information based on the motion features and the first judgment model.

[0095] In an exemplary embodiment of the present disclosure, the auxiliary action information includes first auxiliary information and second auxiliary information, the first auxiliary information is used to characterize the occurrence of physical contact, and the second auxiliary information is used to characterize the absence of physical contact, the physical action information includes hand action information, and the determination module 32 is also used to: if the hand action information characterizes the target hand action and the auxiliary action information is the first auxiliary information, determine that the control instruction is a confirmation instruction.

[0096] In an exemplary embodiment of the present disclosure, the determination module 32 is further used to: if the hand motion information represents a target hand motion and the auxiliary motion information is second auxiliary information, determine that the control instruction is invalid.

[0097] Regarding the control device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0098] Figure 4 4 is a block diagram of a wearable device 400 according to an exemplary embodiment. For example, the wearable device 400 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0099] Reference Figure 4 The wearable device 400 may include one or more of the following components: a processing component 402 , a memory 404 , a power component 406 , a multimedia component 408 , an audio component 410 , an input / output (I / O) interface 412 , a sensor component 414 , and a communication component 416 .

[0100] The processing component 402 generally controls the overall operation of the wearable device 400, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 402 may include one or more processors 420 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 402 may include one or more modules to facilitate the interaction between the processing component 402 and other components. For example, the processing component 402 may include a multimedia module to facilitate the interaction between the multimedia component 408 and the processing component 402.

[0101] The memory 404 is configured to store various types of data to support operations on the wearable device 400. Examples of such data include instructions for any application or method operating on the wearable device 400, contact data, phone book data, messages, pictures, videos, etc. The memory 404 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0102] The power supply component 406 provides power to various components of the wearable device 400. The power supply component 406 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the wearable device 400.

[0103] The multimedia component 408 includes a screen that provides an output interface between the wearable device 400 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 408 includes a front camera and / or a rear camera. When the wearable device 400 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0104] The audio component 410 is configured to output and / or input audio signals. For example, the audio component 410 includes a microphone (MIC), and when the wearable device 400 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 404 or sent via the communication component 416. In some embodiments, the audio component 410 also includes a speaker for outputting audio signals.

[0105] I / O interface 412 provides an interface between processing component 402 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0106] The sensor assembly 414 includes one or more sensors for providing various aspects of status assessment for the wearable device 400. For example, the sensor assembly 414 can detect the open / closed state of the wearable device 400, the relative positioning of the components, such as the display and keypad of the wearable device 400, and the sensor assembly 414 can also detect the position change of the wearable device 400 or a component of the wearable device 400, the presence or absence of contact between the user and the wearable device 400, the orientation or acceleration / deceleration of the wearable device 400, and the temperature change of the wearable device 400. The sensor assembly 414 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 414 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 414 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0107] The communication component 416 is configured to facilitate wired or wireless communication between the wearable device 400 and other devices. The wearable device 400 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 416 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 416 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0108] In an exemplary embodiment, the wearable device 400 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.

[0109] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions, and the instructions can be executed by a processor 420 of a wearable device 400 to complete the control method of the wearable device. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0110] A non-temporary computer-readable storage medium, when instructions in the storage medium are executed by a processor of a wearable device, enables a processing device of the wearable device to execute a control method for the wearable device provided in an exemplary embodiment of the present disclosure.

[0111] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0112] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A control method for a wearable device, characterized in that: The control method comprises: Acquire a target image, where the target image is used to represent a limb movement; Acquiring auxiliary information, wherein the auxiliary information includes at least one of a touch signal, a bioelectric signal, and an inertial signal; Determine limb motion information based on a pre-stored first judgment model and the target image, wherein the limb motion information is used to characterize a position change of a limb; Determine auxiliary action information based on the pre-stored second judgment model and the auxiliary information, wherein the auxiliary action information is used to indicate whether physical contact occurs; Based on the limb motion information and the auxiliary motion information, a control instruction is determined.

2. The control method of the wearable device according to claim 1, characterized in that: The method for forming the second judgment model includes: Training the first neural network model based on the labeled sample information to obtain the second judgment model; The sample information includes at least one of a touch sample signal, a bioelectric signal sample signal and an inertial sample signal. If the sample signals included in the sample information are different, the obtained second judgment model is different. The mark includes a first mark and a second mark, the first mark is used to indicate that the sample information is a sample in which physical contact occurs, and the second mark is used to indicate that the sample information is a sample in which physical contact does not occur.

3. The control method of the wearable device according to claim 1, characterized in that: The bioelectric signal is generated based on action potential and / or resting potential; and / or, The inertial signal includes at least one of acceleration and angular acceleration.

4. The control method of the wearable device according to claim 1, characterized in that: The method for forming the first judgment model includes: Training a second neural network model based on a limb sample image with action identification to obtain the first judgment model, wherein the target area in the limb sample image has a preset mark; The action identifier is used to indicate the limb action feature represented by the preset mark in each limb sample image.

5. The control method of the wearable device according to claim 4, characterized in that: The determining of the body motion information based on the pre-stored first judgment model and the target image includes: Based on the target image, obtaining motion features of a target area in the target image; The limb movement information is determined based on the movement feature and the first judgment model.

6. The control method of the wearable device according to claim 4, characterized in that: The auxiliary action information includes first auxiliary information and second auxiliary information, the first auxiliary information is used to indicate that a physical contact occurs, and the second auxiliary information is used to indicate that a physical contact does not occur, the physical action information includes hand action information, and determining a control instruction based on the physical action information and the auxiliary action information includes: If the hand motion information represents a target hand motion and the auxiliary motion information is the first auxiliary information, it is determined that the control instruction is a confirmation instruction.

7. The control method of the wearable device according to claim 6, characterized in that: The determining of the action control instruction based on the limb action information and the auxiliary action information further includes: If the hand motion information represents a target hand motion and the auxiliary motion information is the second auxiliary information, it is determined that the control instruction is invalid.

8. A control device for a wearable device, characterized in that: The control device of the wearable device comprises: An acquisition module, used for acquiring a target image, wherein the target image is used for representing a limb movement; The acquisition module is further used to acquire auxiliary information, wherein the auxiliary information includes at least one of a touch signal, a bioelectric signal and an inertial signal; A determination module, used to determine limb motion information based on a pre-stored first judgment model and the target image, wherein the limb motion information is used to characterize a position change of a limb; The determination module is further used to determine auxiliary action information based on the pre-stored second judgment model and the auxiliary information, wherein the auxiliary action information is used to indicate whether physical contact occurs; The determination module is further used to determine a control instruction based on the limb movement information and the auxiliary movement information.

9. A wearable device, characterized in that: The wearable device comprises: processor; a memory for storing processor-executable instructions; The processor is configured to execute executable instructions in the memory to implement the control method of the wearable device as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having executable instructions stored thereon, characterized in that: When the executable instruction is executed by the processor, the control method of the wearable device according to any one of claims 1 to 7 is implemented.

Images