Man-machine interaction method and device, intelligent glasses, intelligent wearing system and medium

Abstract

The invention relates to the field of man-machine interaction, in particular to a man-machine interaction method and device, intelligent glasses, an intelligent wearing system and a medium. Calling a sensor to detect a current illumination condition, and receiving arm motion parameters and wrist motion parameters sent by the smart watch; and responding to the gesture instruction according to the illumination condition, the arm motion parameter and the wrist motion parameter, and performing gesture recognition. By adopting the scheme, the accuracy and reliability of gesture recognition during man-machine interaction are improved.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction of smart glasses, and in particular to a human-computer interaction method, device, smart glasses, smart wearable system and medium. Background Art

[0002] With the continuous development of smart wearable device technology, gesture recognition technology has become an important research direction in the field of human-computer interaction. Through gesture recognition, users can interact with smart devices more naturally and intuitively, greatly improving the user experience.

[0003] However, the research found that most of the existing gesture recognition technologies rely on single image information or sensor data, which often leads to decreased accuracy and reliability of gesture recognition during human-computer interaction in complex environments (such as insufficient lighting, intense movement, etc.). Summary of the invention

[0004] In view of this, an object of the present invention is to provide a human-computer interaction method, device, smart glasses, smart wearable system and medium to improve the accuracy and reliability of gesture recognition during human-computer interaction.

[0005] In a first aspect, an embodiment of the present application provides a human-computer interaction method, the method comprising: Receive gesture commands sent by the user; Call the sensor to detect the current lighting conditions and receive the arm motion parameters and wrist motion parameters sent by the smart watch; The gesture recognition is performed in response to the gesture instruction according to the lighting condition, the arm movement parameter, and the wrist movement parameter.

[0006] Optionally, responding to the gesture instruction according to the illumination condition, the arm movement parameter, and the wrist movement parameter to perform gesture recognition includes: In response to the lighting condition, the arm movement parameter, and the wrist movement parameter all satisfying preset conditions, gesture recognition is performed through an image sensor of the smart glasses.

[0007] Optionally, in response to the illumination condition, the arm motion parameter, and the wrist motion parameter all satisfying a preset condition, performing gesture recognition through an image sensor of the smart glasses includes: Receive infrared light information emitted by the smart watch through an image sensor; Determining relative position information of the smart watch according to the infrared light information; Correcting the image acquired by the image sensor using the relative position information; Gesture recognition is performed using the rectified image.

[0008] Optionally, in response to the illumination condition, the arm motion parameter, and the wrist motion parameter all satisfying a preset condition, performing gesture recognition through an image acquired by an image sensor of the smart glasses includes: Determining that the gesture command is a fine gesture command, obtaining the electromyographic information sent by the smart watch; Gesture recognition is performed using the image acquired by the image sensor of the smart glasses, the arm motion parameters, the wrist motion parameters, and the electromyographic information.

[0009] Optionally, performing gesture recognition in response to the gesture instruction based on the lighting condition, the arm motion parameter, and the wrist motion parameter includes: In response to the lighting condition not satisfying a preset condition, acquiring IMU information and electromyography information from a smart watch; Gesture recognition is performed through the IMU information and the electromyography information.

[0010] Optionally, performing gesture recognition in response to the gesture instruction based on the lighting condition, the arm motion parameter, and the wrist motion parameter includes: In response to the illumination condition satisfying a preset condition, the arm motion parameter and the wrist motion parameter do not satisfy the preset condition; Determining relative position information of the smart watch according to the infrared light information; Correcting the position of the hand in the image acquired by the image sensor based on the relative position information; Gesture recognition is performed based on images after correcting the hand position, IMU information, and electromyography information.

[0011] In a second aspect, an embodiment of the present application provides a human-computer interaction method, which is applied to a smart wearable system, wherein the system includes smart glasses and a smart watch that are connected to each other for communication; the method includes: The smart glasses receive gesture commands sent by the user; Smart glasses use sensors to detect current lighting conditions; The smart watch detects the user's arm motion parameters and wrist motion parameters, and sends the arm motion parameters and wrist motion parameters to the smart glasses; The smart glasses respond to the gesture instruction according to the lighting conditions, the arm movement parameters, and the wrist movement parameters to perform gesture recognition.

[0012] In a third aspect, an embodiment of the present application provides a human-computer interaction device, which is applied to smart glasses, and the device includes: A receiving module, used for receiving gesture instructions sent by the user; A calling module is used to call the sensor to detect the current lighting conditions and receive the arm motion parameters and wrist motion parameters sent by the smart watch; The recognition module is used to respond to the gesture instruction and perform gesture recognition according to the lighting condition, the arm movement parameter, and the wrist movement parameter.

[0013] In a fourth aspect, an embodiment of the present application provides a pair of smart glasses, comprising: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions are executed by the processor to perform the steps of the voltage sampling method described in any optional implementation of the first and second aspects above.

[0014] In a fifth aspect, an embodiment of the present application provides a smart wearable system, including: smart glasses and a smart watch, wherein the system includes the smart glasses and the smart watch that are communicatively connected to each other; Smart glasses, used to receive gesture commands sent by users; Smart glasses, which use sensors to detect current lighting conditions; A smart watch, used for detecting arm motion parameters and wrist motion parameters of a user, and sending the arm motion parameters and wrist motion parameters to smart glasses; The smart glasses are used to respond to the gesture instruction and perform gesture recognition according to the lighting conditions, the arm movement parameters, and the wrist movement parameters.

[0015] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the human-computer interaction method described in any optional implementation manner of the first aspect and the second aspect are executed.

[0016] The technical solution provided by this application includes but is not limited to the following beneficial effects: When receiving a gesture command sent by a user, the present application calls a sensor to detect the current lighting conditions, and receives arm motion parameters and wrist motion parameters sent by a smart watch, and finally responds to the gesture command according to the lighting conditions, arm motion parameters and wrist motion parameters, and selects the optimal gesture recognition method under the current conditions to perform gesture recognition. The present application can make full use of the complementarity of multiple sensor data in complex environments (such as insufficient lighting, intense exercise, etc.), effectively deal with the failure of a single information source, enhance the robustness of the gesture recognition system, and at the same time, through the effective fusion of multi-source information, significantly improve the accuracy and reliability of gesture recognition in the process of human-computer interaction.

[0017] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A flow chart of a human-computer interaction method provided by an embodiment of the present invention is shown; Figure 2 A flowchart of a gesture recognition method provided by an embodiment of the present invention is shown; Figure 3 A flowchart of a second gesture recognition method provided by an embodiment of the present invention is shown; Figure 4 A flowchart of a third gesture recognition method provided by an embodiment of the present invention is shown; Figure 5 A flowchart of a fourth gesture recognition method provided by an embodiment of the present invention is shown; Figure 6 A schematic diagram of a human-computer interaction process based on smart glasses and a watch provided by an embodiment of the present invention is shown; Figure 7 A flow chart of a second human-computer interaction method provided by an embodiment of the present invention is shown; Figure 8 A flow chart of a human-computer interaction device provided by an embodiment of the present invention is shown; Fig. 9 A schematic diagram of the structure of smart glasses provided by an embodiment of the present invention is shown; Fig.10 A schematic structural diagram of a smart wearable system provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0021] To facilitate understanding of this application, Figure 1 The flowchart of a human-computer interaction method provided by an embodiment of the present invention is shown to describe the contents of the embodiment of the present application in detail.

[0022] See also Figure 1 As shown, Figure 1 A flow chart of a human-computer interaction method provided by an embodiment of the present invention is shown, which is applied to smart glasses, wherein the method comprises steps S101 to S103: S101: receiving a gesture instruction sent by a user.

[0023] Specifically, gesture commands include commands created by users through specific finger movements, arm swings, or wrist rotations. Smart glasses have built-in sensors such as cameras and infrared sensors, which are responsible for capturing and identifying user gestures to obtain gesture commands.

[0024] S102: calling the sensor to detect the current lighting conditions, and receiving the arm motion parameters and wrist motion parameters sent by the smart watch.

[0025] Specifically, since lighting conditions are crucial to the accuracy of gesture recognition based on image recognition, and different light intensities may affect the device's ability to capture and interpret gestures, the smart glasses call on their built-in photosensors to detect the current ambient lighting conditions, so as to subsequently determine the gesture recognition strategy based on different ambient lighting conditions.

[0026] At the same time, smart glasses establish communication with other wearable devices such as smart watches, and receive arm motion parameters and wrist motion parameters sent by these devices. These parameters are used to describe the motion status of the user's arm and wrist, including acceleration, angular velocity, position information, etc. The arm motion parameters and wrist motion parameters are calibrated and preprocessed to eliminate sensor deviations and errors and improve data accuracy.

[0027] S103: Respond to the gesture instruction and perform gesture recognition according to the lighting condition, the arm movement parameter, and the wrist movement parameter.

[0028] Specifically, the gesture instructions are parsed and recognized based on the lighting conditions, arm motion parameters, and wrist motion parameters. It is understandable that the weights of the three input parameters in the gesture recognition algorithm can be dynamically adjusted through the intervals of the three input parameters, namely, the lighting conditions, the arm motion parameters, and the wrist motion parameters, so as to ensure the strong robustness of the gesture recognition algorithm under different input parameter intervals.

[0029] In some embodiments, an image recognition algorithm is used to identify hand movements corresponding to user gesture instructions, arm motion parameters and wrist motion parameters detected by a smart watch are received, the user's hand movements are matched with gestures in a preset gesture library, and the corresponding preliminary gesture instructions are determined. Then, the motion state of the user's arm and wrist is determined based on the arm motion parameters and wrist motion parameters. Based on the preliminary gesture instructions, arm motion parameters and wrist motion parameters, the human-computer interaction that the user wants to perform is determined, that is, the execution subject smart glasses determine the gesture recognition result.

[0030] After the gesture recognition result is determined, the smart glasses generate corresponding control instructions.

[0031] For example, when a grabbing gesture is recognized, a control instruction is generated to make the virtual hand in the AR environment perform a grabbing action, and the joint movement and posture change of the virtual hand are controlled to realize the grabbing operation of the virtual object. In this scenario, the smart glasses can also update the status of the virtual hand and virtual object in the AR scene according to the control instruction, and present it to the user through its display unit, so as to realize the interaction between the user and the AR environment.

[0032] This application, through steps S101 to S103, calls the sensor to detect the current lighting conditions and receives the arm motion parameters and wrist motion parameters sent by the smart watch when receiving the gesture command sent by the user. Finally, according to the lighting conditions, the arm motion parameters and the wrist motion parameters, the gesture command is responded to, and the optimal gesture recognition method is selected under the current conditions to perform gesture recognition. This application can make full use of the complementarity of multiple sensor data in complex environments (such as insufficient lighting, intense exercise, etc.), effectively deal with the failure of a single information source, enhance the robustness of the gesture recognition system, and at the same time, through the effective fusion of multi-source information, significantly improve the accuracy and reliability of gesture recognition in the human-computer interaction process.

[0033] In an optional embodiment, responding to the gesture instruction according to the lighting condition, the arm movement parameter, and the wrist movement parameter to perform gesture recognition includes: In response to the lighting condition, the arm movement parameter, and the wrist movement parameter all satisfying preset conditions, gesture recognition is performed through an image sensor of the smart glasses.

[0034] Specifically, the photosensors built into smart glasses measure the ambient light intensity, color balance, and other lighting-related parameters. To ensure the accuracy of gesture recognition, these lighting conditions need to be within a specific range, neither too bright nor too dim, to avoid overexposure or underexposure of the image, thereby affecting the effect of gesture recognition. Arm motion parameters include information about the movement of the user's arm, including speed, direction, and acceleration. These parameters help the device to more accurately identify specific parts of the user's gestures, such as the opening or closing of fingers, which can be collected through sensors on smart watches or other wearable devices (such as accelerometers and gyroscopes). Wrist motion parameters include subtle wrist movement information, such as rotation angle, flip angle, etc. These parameters help the device to more accurately identify specific parts of the user's gestures, such as the opening or closing of fingers, which can also be collected through sensors on smart watches or other wearable devices.

[0035] When all of the above conditions meet the preset threshold, the smart glasses capture images of the user's hands through their image sensors and use image processing algorithms to parse these images to identify the specific gestures performed by the user. This application does not impose any particular restrictions on the gesture recognition algorithm based on image processing, which can be a convolutional neural network, a support vector machine, and a gesture recognition algorithm based on color thresholds.

[0036] Through the above steps, the present application adaptively adjusts the gesture recognition strategy according to different lighting conditions, arm and wrist movement parameters, so that the system can operate stably in various complex environments, improving the adaptability and flexibility of the system.

[0037] In an alternative embodiment, see Figure 2 As shown, Figure 2 A flowchart of a gesture recognition method provided by an embodiment of the present invention is shown, wherein, in response to the illumination condition, the arm motion parameter, and the wrist motion parameter all satisfying a preset condition, gesture recognition is performed through an image sensor of the smart glasses, including steps S201 to S204: S201: Receive infrared light information emitted by the smart watch through an image sensor.

[0038] Specifically, the smart watch is equipped with one or more infrared LED lights, which emit infrared light, and the infrared light information can help the smart glasses determine the precise location of the smart watch (and the user's wrist and arm). The image sensor of the smart glasses captures the infrared light information from the smart watch.

[0039] S202: Determine the relative position information of the smart watch according to the infrared light information.

[0040] Specifically, after the grayscale image sensor of the smart glasses receives infrared light information, it uses the infrared light information to calculate the position of the smart watch relative to the glasses. This usually involves image processing algorithms that can analyze the intensity and distribution of infrared light to determine the exact location of the light source (i.e., the smart watch). This position information is crucial for image correction in subsequent steps.

[0041] S203: Correcting the image acquired by the image sensor according to the relative position information.

[0042] Specifically, since the user's hand may move due to various factors (such as arm swing, wrist rotation, etc.), and the gesture action is an operation instruction in three-dimensional space, the two-dimensional image information obtained by the image sensor will have a certain impact when processing the operation instruction in three-dimensional space. By using the infrared light information of the smart watch to determine the position of the hand and make corresponding corrections to the image, the smart glasses can ensure the accuracy of gesture recognition.

[0043] S204: Perform gesture recognition using the corrected image.

[0044] Specifically, the corrected image is subjected to image preprocessing, including grayscale conversion, edge detection, etc., to extract gesture features, and then the smart glasses use image processing algorithms and machine learning models to analyze the corrected image to identify the specific gestures performed by the user.

[0045] In this application, the infrared light information emitted by the smart watch in steps S201 to S204 enables the smart glasses to more accurately determine the position of the watch (as well as the user's wrist and arm), which helps to accurately correct the hand position in the subsequent gesture recognition process, thereby improving the accuracy of gesture recognition. At the same time, the participation of infrared light information can also reduce the impact of ambient light on gesture recognition to a certain extent, so that the system can maintain a high recognition accuracy rate under various lighting conditions.

[0046] In an alternative embodiment, see Figure 3 As shown, Figure 3 A flowchart of a second gesture recognition method provided by an embodiment of the present invention is shown, wherein, in response to the illumination condition, the arm motion parameter, and the wrist motion parameter all satisfying a preset condition, gesture recognition is performed through an image acquired by an image sensor of the smart glasses, including steps S301-S302: S301: If it is determined that the gesture instruction is a fine gesture instruction, then the electromyographic information sent by the smart watch is obtained.

[0047] Specifically, fine gesture commands usually refer to those that require high precision and fine muscle control, such as dragging interactive objects with fingers. These gestures are often difficult to accurately identify only through external motion information captured by image sensors (such as arm and wrist movements), so it is necessary to judge the type of gesture command to determine whether the gesture command is a fine gesture command and whether fine recognition is required.

[0048] When the gesture instruction is determined to be a fine gesture, the smart glasses obtain the user's electromyographic information from the smart watch. The electromyographic information is captured by the electromyographic sensors (EMG sensors) on the smart watch, which can detect the weak electrical signals generated by the arm muscles. These electrical signals are closely related to the contraction and relaxation of the muscles, and can therefore provide important data about the fine movements of the hands.

[0049] S302: Perform gesture recognition using the image acquired by the image sensor of the smart glasses, the arm motion parameters, the wrist motion parameters, and the electromyography information.

[0050] Specifically, after obtaining the electromyographic information, the collected electromyographic signals are preprocessed, including filtering to remove noise interference, such as using a bandpass filter to filter out 50Hz power frequency interference and high-frequency noise. Then, amplification and analog-to-digital conversion are performed to convert the analog signal into a digital signal, and the electromyographic information is combined with the image captured by the image sensor of the smart glasses and the arm and wrist motion parameters obtained by other sensors (such as accelerometers, gyroscopes, etc.), and gesture recognition is performed using a pre-trained gesture recognition model.

[0051] This application introduces the electromyographic information of the smart watch through steps S301~S302, which can capture the weak electrical signals generated by the arm muscles. These signals are closely related to the contraction and relaxation of the muscles, thereby providing important data about fine hand movements and significantly improving the accuracy of gesture recognition.

[0052] In an alternative embodiment, see Figure 4 As shown, Figure 4 A flowchart of a third gesture recognition method provided by an embodiment of the present invention is shown, wherein gesture recognition is performed based on the lighting condition, the arm movement parameter, and the wrist movement parameter in response to the gesture instruction, including steps S401-S402: S401: In response to the lighting condition not satisfying a preset condition, obtaining IMU information and electromyography information from a smart watch.

[0053] Specifically, if there is insufficient light or overexposure, the accuracy of gesture recognition through image sensors may be greatly reduced. Therefore, when it is detected that the lighting conditions do not meet the preset conditions, the smart glasses will send a request to the smart watch to obtain IMU information and electromyographic information. IMU information usually includes motion parameters such as acceleration and angular velocity, which are collected through sensors such as accelerometers and gyroscopes built into smart watches. Electromyographic information is captured by the electromyographic sensor on the smart watch to reflect the activity status of the arm muscles.

[0054] In addition, the smart watch can be used to remind the user to go to a place with suitable lighting or to remind the user to stay away from obstructions. The smart watch can also be used to remind the user to place the hand in the appropriate gesture recognition area or to adjust the gesture recognition area.

[0055] S402: Perform gesture recognition using the IMU information and the electromyography information.

[0056] Specifically, the IMU information and electromyography information are combined and a pre-trained gesture recognition model is used to recognize user gestures. This method not only improves the accuracy of gesture recognition, but also enhances the robustness and adaptability of the system, enabling it to work under different lighting conditions.

[0057] In some specific embodiments, the surface electromyography sensor of the smart watch is used to sample the skin to obtain the sEMG signal of the gesture, that is, the surface electromyography signal. The collected sEMG signal is filtered, feature extracted, and other processes are performed to remove noise and extract effective features. The processed sEMG signal is classified using a classification algorithm to achieve gesture recognition.

[0058] The present application integrates the IMU information and electromyography information of the smart watch through steps S401-S402, and can still perform accurate gesture recognition when the lighting conditions do not meet the preset conditions, thereby improving the accuracy and effectiveness of gesture recognition.

[0059] In an alternative embodiment, see Figure 5 As shown, Figure 5 A flowchart of a fourth gesture recognition method provided by an embodiment of the present invention is shown, wherein gesture recognition is performed based on the lighting condition, the arm movement parameter, and the wrist movement parameter in response to the gesture instruction, including steps S501 to S504: S501: In response to the lighting condition satisfying a preset condition, the arm motion parameter and the wrist motion parameter do not satisfy the preset condition.

[0060] Specifically, if the lighting conditions meet the preset conditions, but the arm motion parameters and wrist motion parameters do not meet the preset conditions, such as the motion is too violent or unstable, then the gesture may not be accurately recognized relying solely on image information.

[0061] S502: Determine the relative position information of the smart watch according to the infrared light information.

[0062] Specifically, the infrared light information emitted by the smart watch is used to determine its relative position information, and the position and orientation of the smart watch are determined through an image processing algorithm.

[0063] S503: Correcting the position of the hand in the image acquired by the image sensor based on the relative position information.

[0064] Specifically, since smart watches are usually worn on the user's wrist, their position information can indirectly reflect the position of the hand in the image. By correcting the hand position, the system can ensure that the subsequent gesture recognition process is more accurate.

[0065] S504: Perform gesture recognition based on the image after the hand position is corrected, the IMU information, and the electromyography information.

[0066] Specifically, gesture recognition is performed using corrected images, IMU information, and electromyographic information. Data fusion algorithms, such as Kalman filtering or particle filtering, are used to integrate images, IMU information, and electromyographic information after correcting the hand position. During the integration process, infrared signals are used to assist in determining the approximate position of the hand, electromyographic signals are used to identify muscle states and subtle gestures, and IMU data provides dynamic information of the hand. Through this multimodal data fusion, the user's gestures can be captured and understood more accurately.

[0067] Through steps S501 to S504, when the lighting conditions meet the preset conditions, even if the motion parameters of the arm and wrist do not meet the preset conditions (such as the movement is too intense or unstable), the relative position of the watch can still be determined through the infrared light information of the smart watch, and then the position of the hand in the image can be corrected to ensure the accuracy of gesture recognition. In addition, by fusing images, IMU information and electromyography information, the dynamic information, muscle state and subtle gestures of the hand can be captured, so as to more accurately recognize the user's gestures.

[0068] Using deep learning technology, such as 3D convolutional neural networks (3D CNNs) or skeleton key point recognition algorithms (ST-GCN), gesture recognition models are constructed. The gesture recognition model can extract features from the fused data and learn gesture patterns. The gesture recognition model is trained with a large number of labeled gesture samples, enabling it to recognize and distinguish various gestures, such as grabbing, releasing, rotating, clicking, etc.

[0069] Smart glasses run gesture recognition models in real time and analyze the fused data stream to recognize the user's gestures in real time. At the same time, smart glasses can dynamically adjust the data fusion strategy and recognition model parameters based on the recognition results to adapt to different users and environmental conditions, thereby improving the accuracy and robustness of recognition.

[0070] Smart glasses feed back the recognition results to smart watches, which can provide users with confirmation of successful operations through vibration, etc. At the same time, based on user feedback and recognition accuracy, the data fusion algorithm and gesture recognition model are learned and optimized to continuously improve the interactive performance of the system.

[0071] In order to better understand the human-computer interaction method provided by this application, the process of human-computer interaction based on smart glasses and watches is described here. Figure 6 As shown, Figure 6 A schematic diagram of a human-computer interaction process based on smart glasses and a watch provided by an embodiment of the present invention is shown, wherein, after a user wears smart glasses and a smart watch, the user inputs a gesture command through the smart glasses, and the smart glasses obtain the arm or wrist movement parameters detected by the watch sensor, and at the same time call the sensor to detect the current light condition parameters. Then, based on the above parameters, the user's gesture command is responded to, the user's gesture operation is guided, or the gesture recognition area is adjusted.

[0072] After the smart glasses guide the user to change the lighting conditions, if they detect that the lighting conditions meet expectations, they perform gesture recognition based on the image sensor; if the lighting conditions do not meet the expectations, they obtain IMU and electromyography information through the watch. After the watch sends the IMU information and electromyography information to the glasses, the smart glasses perform gesture recognition based on the IMU information and electromyography information.

[0073] After the smart glasses guide the user to change the position of the hand relative to the glasses and the user moves the arm and wrist, the smart glasses detect the infrared rays emitted by the smart watch and make corrections based on the infrared rays and the hand position in the image sensor. Then the IMU and electromyography information are obtained through the watch, and gesture recognition is performed based on the image information, IMU information and electromyography information after the hand position is corrected.

[0074] The human-computer interaction method provided in this application can be applied to multiple scenarios, including but not limited to augmented reality (AR) game scenarios, simulation training scenarios, remote operation scenarios, robotics technology scenarios, smart home control scenarios, virtual design scenarios, creative industry scenarios, and education and training scenarios.

[0075] In augmented reality (AR) game scenarios, players can interact with the game world through natural gesture control, such as grabbing virtual items, casting magic, or interacting with game characters through gestures. This interaction method not only improves the immersion of the game, but also provides players with a more intuitive and interesting gaming experience.

[0076] In simulation training scenarios, in the field of simulation training, such as military training or surgical simulation, the present invention can provide accurate hand motion tracking to make training more realistic and effective. For example, surgeons can improve surgical skills by simulating surgical gestures, while military personnel can simulate various tactical gestures to improve teamwork capabilities.

[0077] In remote operation scenarios, in the field of remote operation, such as remote medical care or industrial maintenance, doctors or engineers can accurately control the robot arm through gestures to perform delicate operations. This gesture recognition technology based on electromyographic sensors can reduce operation delays and improve the accuracy and safety of operations.

[0078] In robotics scenarios, through the linkage of smart watches and smart glasses, robot operators can interact with robots more naturally, such as using gestures to guide the robot's movement or grasp objects, thereby improving the efficiency of human-machine collaboration.

[0079] In the smart home control scenario, in a smart home environment, users can use gestures to control various smart devices in their home, such as adjusting light brightness, controlling temperature, or switching TV channels. This control method is not only convenient for users, but also provides a new smart home interactive experience. For example, users can turn on or off smart bulbs with simple gestures, or adjust the brightness of lights with rotation gestures. This gesture-based control method makes the smart home system more intuitive and easy to use.

[0080] In the virtual design scene, designers can use gestures to manipulate virtual models for 3D design and editing. This gesture recognition technology based on electromyographic sensors can provide more precise control and a more natural way of interaction, thereby improving design efficiency and creativity.

[0081] In creative industry scenarios, such as film production or animation design, artists can use gestures to control virtual characters, perform motion capture and simulate expressions. This interactive method not only improves production efficiency, but also provides artists with greater creative freedom.

[0082] In education and training scenarios, teachers and students can interact through gestures, such as writing on a virtual blackboard, operating a teaching model, or conducting scientific experiments. This gesture-based interaction method can improve the interactivity and fun of teaching and stimulate students' interest in learning.

[0083] In training scenarios, such as emergency rescue or firefighting training, trainees can use gestures to simulate actual operations, such as performing cardiopulmonary resuscitation or operating firefighting equipment. This gesture recognition technology based on electromyographic sensors can provide a more realistic training experience and improve training effectiveness.

[0084] The human-computer interaction method provided in the present application comprehensively considers lighting conditions, arm motion parameters and wrist motion parameters, and flexibly adjusts the gesture recognition strategy according to different lighting conditions and the motion states of the arms and wrists, thereby ensuring accurate gesture recognition in a variety of environments, improving the accuracy of gesture recognition, and enhancing the ability to adapt to complex environments during human-computer interaction.

[0085] Based on the same application concept, the embodiment of the present invention also provides another human-computer interaction method, which is applied to a smart wearable system, wherein the system includes smart glasses and smart watches that are connected to each other for communication; see Figure 7 As shown, Figure 7 A flowchart of a second human-computer interaction method provided by an embodiment of the present invention is shown, wherein the method comprises steps S701 to S704: S701: The smart glasses receive a gesture command sent by a user; S702: The smart glasses call sensors to detect current lighting conditions; S703: The smart watch detects the arm motion parameters and wrist motion parameters of the user, and sends the arm motion parameters and wrist motion parameters to the smart glasses; S704: The smart glasses respond to the gesture instruction according to the lighting condition, the arm movement parameter, and the wrist movement parameter, and perform gesture recognition.

[0086] Based on the same application concept, the embodiment of the present invention also provides a human-computer interaction device applied to smart glasses, see Figure 8 As shown, Figure 8 A flow chart of a human-computer interaction device provided by an embodiment of the present invention is shown, wherein the device comprises: The receiving module 801 is used to receive the gesture command sent by the user; The calling module 802 is used to call the sensor to detect the current lighting conditions and receive the arm motion parameters and wrist motion parameters sent by the smart watch; The recognition module 803 is used to respond to the gesture instruction and perform gesture recognition according to the lighting condition, the arm movement parameter, and the wrist movement parameter.

[0087] Based on the same application concept, see Fig. 9 As shown, Fig. 9FIG. 1 shows a schematic diagram of the structure of a smart glasses provided by an embodiment of the present invention, wherein Fig. 9 As shown, a smart glasses 900 provided in an embodiment of the present application includes: A processor 901 and a memory 902, wherein the memory 902 stores machine-readable instructions executable by the processor 901, and the machine-readable instructions are executed by the processor 901 to perform the steps of the human-computer interaction method described in any one of the above embodiments when executed.

[0088] Based on the same application concept, the embodiment of the present invention also provides a smart wearable system, see Fig.10 As shown, Fig.10 A schematic diagram of the structure of a smart wearable system provided by an embodiment of the present invention is shown, wherein the system includes smart glasses 1001 and a smart watch 1002 that are communicatively connected to each other; Smart glasses, used to receive gesture commands sent by users; Smart glasses, which use sensors to detect current lighting conditions; A smart watch, used for detecting arm motion parameters and wrist motion parameters of a user, and sending the arm motion parameters and wrist motion parameters to smart glasses; The smart glasses are used to respond to the gesture instruction and perform gesture recognition according to the lighting conditions, the arm movement parameters, and the wrist movement parameters.

[0089] Based on the same application concept, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the human-computer interaction method described in any one of the above embodiments are executed.

[0090] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0091] The computer program product for human-computer interaction provided in the embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be found in the method embodiment and will not be repeated here.

[0092] The human-computer interaction device provided in the embodiment of the present invention can be specific hardware on the device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in the embodiment of the present invention are the same as those of the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the device embodiment can refer to the corresponding contents in the aforementioned method embodiment. It can be clearly understood by technicians in the relevant field that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0093] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0094] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0095] In addition, each functional unit in the embodiment provided by the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0096] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0097] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0098] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or perform equivalent replacements on some of the technical features thereof; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. They should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A human-computer interaction method, applied to smart glasses, wherein: The method comprises: Receive gesture commands sent by the user; Call the sensor to detect the current lighting conditions and receive the arm motion parameters and wrist motion parameters sent by the smart watch; The gesture recognition is performed in response to the gesture instruction according to the lighting condition, the arm movement parameter, and the wrist movement parameter.

2. The method according to claim 1, wherein: Responding to the gesture instruction according to the illumination condition, the arm motion parameter, and the wrist motion parameter to perform gesture recognition includes: In response to the lighting condition, the arm movement parameter, and the wrist movement parameter all satisfying preset conditions, gesture recognition is performed through an image sensor of the smart glasses.

3. The method according to claim 2, wherein: In response to the illumination condition, the arm motion parameter, and the wrist motion parameter all satisfying a preset condition, performing gesture recognition through an image sensor of the smart glasses, including: Receive infrared light information emitted by the smart watch through an image sensor; Determining relative position information of the smart watch according to the infrared light information; Correcting the image acquired by the image sensor using the relative position information; Gesture recognition is performed using the rectified image.

4. The method according to claim 2, wherein: In response to the illumination condition, the arm motion parameter, and the wrist motion parameter all satisfying a preset condition, performing gesture recognition through an image acquired by an image sensor of the smart glasses, including: Determining that the gesture command is a fine gesture command, obtaining the electromyographic information sent by the smart watch; Gesture recognition is performed using the image acquired by the image sensor of the smart glasses, the arm motion parameters, the wrist motion parameters, and the electromyographic information.

5. The method according to claim 1, characterized in that Responding to the gesture instruction based on the illumination condition, the arm motion parameter, and the wrist motion parameter to perform gesture recognition includes: In response to the lighting condition not satisfying a preset condition, acquiring IMU information and electromyography information from a smart watch; Gesture recognition is performed through the IMU information and the electromyography information.

6. The method according to claim 1, characterized in that Responding to the gesture instruction based on the illumination condition, the arm motion parameter, and the wrist motion parameter to perform gesture recognition includes: In response to the illumination condition satisfying a preset condition, the arm motion parameter and the wrist motion parameter do not satisfy the preset condition; Determine the relative position information of the smart watch based on the infrared light information; Correcting the position of the hand in the image acquired by the image sensor based on the relative position information; Gesture recognition is performed based on images after correcting the hand position, IMU information, and electromyography information.

7. A human-computer interaction method, applied to a smart wearable system, wherein: The system comprises smart glasses and a smart watch which are communicatively connected to each other; the method comprises: The smart glasses receive gesture commands sent by the user; Smart glasses use sensors to detect current lighting conditions; The smart watch detects the user's arm motion parameters and wrist motion parameters, and sends the arm motion parameters and wrist motion parameters to the smart glasses; The smart glasses respond to the gesture instruction according to the lighting conditions, the arm movement parameters, and the wrist movement parameters to perform gesture recognition.

8. A human-computer interaction device, applied to smart glasses, wherein: The device comprises: A receiving module, used for receiving gesture instructions sent by the user; A calling module is used to call the sensor to detect the current lighting conditions and receive the arm motion parameters and wrist motion parameters sent by the smart watch; The recognition module is used to respond to the gesture instruction and perform gesture recognition according to the lighting condition, the arm movement parameter, and the wrist movement parameter.

9. A smart glasses comprising: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is performed.

10. A smart wearable system, comprising: Smart glasses and smart watches, wherein the system comprises smart glasses and smart watches that are communicatively connected to each other; Smart glasses, used to receive gesture commands sent by users; Smart glasses, which use sensors to detect current lighting conditions; A smart watch, used for detecting arm motion parameters and wrist motion parameters of a user, and sending the arm motion parameters and wrist motion parameters to smart glasses; The smart glasses are used to respond to the gesture instruction and perform gesture recognition according to the lighting conditions, the arm movement parameters, and the wrist movement parameters.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the human-computer interaction method according to any one of claims 1 to 7 are executed.

Images