MR head display intelligent control system

Abstract

The invention discloses an MR head-mounted display intelligent control system, which relates to the technical field of MR head-mounted displays and comprises a gesture interaction control module, a voice interaction control module, an eye movement interaction control module and a collaborative linkage module. The gesture control module is used for configuring the high-definition camera and the depth sensor to automatically adjust parameters in a complex environment, providing real-time gesture prompt and feedback, and carrying out recognition and instruction control on complex gestures; the high-definition camera is used for collecting hand actions; the depth sensor is used for embedding a gesture recognition algorithm. According to the invention, accurate complex gesture recognition and instruction control interaction recognition are realized through the gesture interaction control module, so that a user can interact with a virtual and real fused environment more naturally and smoothly. Meanwhile, when the voice interaction control module and the eye movement interaction control module achieve accurate interaction recognition, the attention of the user can be more focused on the experience content instead of being disturbed by frequent recognition errors.

Description

Technical Field

[0001] The present invention relates to the technical field of MR head display, and in particular to an MR head display intelligent control system. Background Art

[0002] MR (Mixed Reality) headset is an advanced head-mounted display device that combines the features of virtual reality (VR) and augmented reality (AR), enabling users to simultaneously perceive and interact with information in both the real and virtual worlds.

[0003] MR headsets are usually equipped with a variety of sensors to sense the position and movement of the user's head, as well as the three-dimensional spatial information of the surrounding environment. For example, the IMU can obtain the rotation and translation data of the head, and the camera and depth sensor can scan the surrounding environment and build a spatial model of the environment to determine the position and presentation of virtual objects in the real space. At the same time, it uses special display technology to project virtual images into the user's field of view while ensuring that the user can see the real world. Some MR headsets achieve this function through translucent optical lenses, so that the light of the real world can pass through the lenses, and the virtual image is superimposed on the real scene seen by the user through optical reflection or projection. In addition, some use video perspective technology, that is, the real-world picture is captured by the camera, and the virtual content is synthesized with the picture before being displayed to the user.

[0004] At present, the interactive control method of MR head display has the following disadvantages:

[0005] 1) Disadvantages of gesture interaction control: In actual use, the recognition system of the headset may misjudge some more complex or similar gestures. For example, when the user makes some hand movements quickly, or the amplitude and angle of the hand movements deviate from the preset standard gestures, the system may not be able to accurately recognize the user's intentions, resulting in operational errors. At the same time, factors such as ambient light and object occlusion may affect the accuracy of gesture recognition. For example, in an environment with too strong or too dark light, the camera may not be able to clearly capture the user's hand movements; if the user's hands are blocked by other objects, the system cannot correctly recognize the gestures.

[0006] 2) Disadvantages of voice interaction control: In a noisy environment, the headset's microphone may pick up a lot of background noise, which will affect the accuracy of voice recognition. For example, in public places, factory workshops and other environments, the surrounding human voices, machine sounds and other noises may cause the system to be unable to correctly recognize the user's voice commands. At the same time, differences in accents and language habits: Users in different regions may have different accents and language habits, which may cause the voice recognition system to inaccurately recognize the voice commands of some users. For example, the pronunciation of dialects in some places may be quite different from the standard Mandarin pronunciation, and the system may not be able to correctly recognize these dialect voice commands.

[0007] 3) Disadvantages of eye-movement interaction control: If the user uses the eye tracking function for a long time, the user's eyes may feel tired and uncomfortable. Because the user needs to constantly look at a specific area or object in the headset to interact, this will put the eye muscles in a tense state. Long-term use may cause problems such as dry eyes and pain. At the same time, eye tracking technology may be affected by the user's unconscious eye movements, blinking and other actions, resulting in false triggering or misoperation. For example, when the user is thinking or inadvertently looking in a certain direction, the system may misjudge it as the user's operation instruction, thereby performing some unnecessary operations.

[0008] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0009] In view of the problems in the related technology, the present invention proposes an MR head display intelligent control system to overcome the above technical problems existing in the existing related technology.

[0010] The technical solution of the present invention is achieved in this way:

[0011] An intelligent control system for an MR head display, comprising: a gesture interaction control module, a voice interaction control module, an eye movement interaction control module and a collaborative linkage module, wherein;

[0012] The gesture interaction control module is used to configure the high-definition camera and the depth sensor to automatically adjust parameters in a complex environment and provide real-time gesture prompts and feedback, so as to recognize and control complex gestures; wherein the high-definition camera is used to collect hand movements; the depth sensor is used to embed a gesture recognition algorithm to adapt to different hand movements, speeds and angles to meet the recognition of complex gestures;

[0013] The voice interaction control module is used to filter and suppress the noise in the environment, highlight the user's voice signal, and automatically adjust and optimize according to the user's voice characteristics through the embedded voice recognition engine to improve the recognition of voice control instructions in a noisy environment. It includes: a noise elimination module and a voice enhancement optimization module, wherein the noise elimination module is used to filter and suppress the noise in the environment by using a microphone array and digital signal processing to highlight the user's voice signal; the voice enhancement optimization module is used to embed the voice recognition engine, and can automatically adjust and optimize according to the user's voice characteristics;

[0014] The eye movement interaction control module is used to track the user's eye movement and perform filtering and command control during the recognition process of blinking actions, including: a calibration tracking module and a blink detection filtering module, wherein the calibration tracking module is used to perform eye movement calibration according to the eye movement tracking algorithm before using the head display; the blink detection filtering module is used to determine whether it is a blinking action by analyzing the frequency and amplitude characteristics of the eye movement, and perform filtering during the recognition process;

[0015] Furthermore, it also includes:

[0016] The collaborative linkage module is used for the gesture interaction control module, the voice interaction control module and the eye movement interaction control module to work together, and when the gesture interaction control module, the voice interaction control module or the eye movement interaction control module performs priority control, the gesture interaction control module, the voice interaction control module or the eye movement interaction control module is linked for collaborative control.

[0017] Furthermore, the gesture interaction control module further includes: an environment calibration template and a gesture guidance module, wherein;

[0018] The environmental calibration template is used to enable the head display to automatically adjust parameters in different usage environments to adapt to environmental changes, which at least includes: automatically adjusting the exposure parameters of the high-definition camera in an environment with too strong or too dark light to ensure that hand movements can be clearly captured;

[0019] The gesture guidance module is used to provide real-time gesture prompts and feedback in the head-mounted display interface to help users perform gesture operations correctly.

[0020] Furthermore, the voice interaction control module further includes: a voice setting module and a voice feedback module, wherein;

[0021] The voice setting module is used to provide voice templates and shortcut command settings, allowing users to customize voice commands;

[0022] The voice feedback module is used to provide voice feedback after recognizing the user's voice command.

[0023] Furthermore, the blink detection filtering module comprises the following steps:

[0024] A differential algorithm is used in advance to calculate the first-order differential velocity and the second-order differential acceleration of the eye movement position data to obtain the velocity and acceleration of the eye movement;

[0025] Use the Canny edge detection image recognition algorithm to determine the pupil boundary, calculate the pupil area, and obtain the pupil area change;

[0026] Eye movement velocity, acceleration, and pupil area change are used as feature inputs, and a support vector machine is used to determine whether it is a blink action;

[0027] Calibration is used in eye tracking to select preset scenes. If it is judged as a blink action, the blink action is filtered and feedback is prompted.

[0028] Beneficial effects of the present invention:

[0029] The present invention realizes accurate recognition of complex gestures and interactive recognition of command control through the gesture interaction control module, which enables users to interact with the virtual and real environment more naturally and smoothly. At the same time, when the voice interaction control module and the eye movement interaction control module realize accurate interactive recognition, the user's attention can be more focused on the experience content itself, rather than being disturbed by frequent recognition errors.

[0030] The precise interactive recognition of the present invention reduces the fatigue of users caused by repeated operations or correcting wrong operations. With the recognition of complex gestures and command control, users do not need to repeat the same action many times to achieve their goals, and the burden on arm and hand muscles is reduced. Similarly, in improving the recognition of voice control commands in noisy environments, accurate recognition means that users do not need to speak loudly or slowly repeatedly to make the system understand their intentions, thereby reducing fatigue caused by voice interaction. For eye movement interactive control, filtering and command control are performed during the recognition process of blinking actions. Accurate recognition can avoid the need for users to constantly adjust their line of sight due to frequent false triggers, reducing eye tension and fatigue. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 It is a principle block diagram of an MR head display intelligent control system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0034] According to an embodiment of the present invention, an MR head display intelligent control system is provided.

[0035] like Figure 1 As shown, the MR head display intelligent control system according to an embodiment of the present invention includes: a gesture interaction control module 1, a voice interaction control module 2, an eye movement interaction control module 3 and a collaborative linkage module 4, wherein;

[0036] The gesture interaction control module 1 is used to configure the high-definition camera 11 and the depth sensor 12 to automatically adjust parameters in a complex environment and provide real-time gesture prompts and feedback, and perform recognition and command control of complex gestures, wherein;

[0037] A high-definition camera 11, used for collecting hand movements;

[0038] The depth sensor 12 is used for the built-in gesture recognition algorithm, so that it can better adapt to different hand movements, speeds and angles, and meet the recognition capability of complex gestures.

[0039] This technical solution uses a high-definition camera 11 and a depth sensor 12 to more accurately capture the position, shape and movement trajectory of the hand, thereby improving its accuracy and sensitivity. At the same time, the depth sensor 12 can accurately measure the distance between the object and the head display under different lighting conditions, thereby improving the accuracy of gesture recognition.

[0040] The environmental calibration template 13 is used to enable the head display to automatically adjust parameters in different usage environments to adapt to environmental changes.

[0041] Specifically, the technical solution automatically adjusts the exposure parameters of the high-definition camera in an environment with too strong or too dark light to ensure that hand movements can be clearly captured.

[0042] The gesture guidance module 14 is used to provide real-time gesture prompts and feedback in the head-mounted display interface to help users perform gesture operations correctly.

[0043] In this technical solution, when the user's gesture is not recognized, the system can display a prompt message to tell the user how to adjust the gesture. At the same time, when applied, detailed gesture operation guides and training tutorials can be provided to users in advance to let users understand the correct gesture operation methods and precautions. For example, through video tutorials, graphic descriptions, etc., users can be shown how to make standard gesture movements and how to avoid some common incorrect operations.

[0044] The voice interaction control module 2 is used to filter and suppress the noise in the environment, highlight the user's voice signal, and automatically adjust and optimize according to the user's voice characteristics through the embedded voice recognition engine to improve the recognition of voice control commands in a noisy environment, including: a noise elimination module 21 and a voice enhancement optimization module 22, wherein;

[0045] The noise elimination module 21 is used to filter and suppress the noise in the environment by using a microphone array and digital signal processing, so as to highlight the user's voice signal.

[0046] The speech enhancement and optimization module 22 is used for the embedded speech recognition engine, and can automatically adjust and optimize according to the user's speech characteristics.

[0047] This technical solution can improve the adaptability to different accents, speaking speeds and language habits through the speech recognition engine, and improve the accuracy of speech recognition in noisy environments in combination with the noise elimination module.

[0048] The voice setting module 23 is used to provide voice templates and shortcut command settings, allowing users to customize voice commands;

[0049] This technical solution can facilitate users to make personalized voice settings through the voice setting module, so that the system can better adapt to the user's voice characteristics. For example, users can record their own voice samples according to the provided voice templates and shortcut settings, allowing the system to learn and optimize, improve the recognition accuracy of user-specific voices, and improve interaction efficiency. For example, users can set shortcuts such as "open pictures" and "switch scenes" to quickly operate during use.

[0050] The voice feedback module 24 is used to provide voice feedback after recognizing the user's voice command.

[0051] This technical solution introduces a voice feedback module to let users know whether the system correctly understands their instructions. Specifically, for example, when a user issues a command to "enlarge the image", the system can reply "enlarging the image" to let the user confirm whether the operation is correct.

[0052] In addition, users are asked to confirm some important operations or potentially ambiguous instructions. For example, when the system recognizes that the user's voice instruction may have multiple interpretations, it can ask the user "Do you want to open file A or file B?" and ask the user to confirm before executing the operation.

[0053] The eye movement interaction control module 3 is used to track the user's eye movement and perform filtering and command control during the recognition process of blinking actions, including: a calibration tracking module 31 and a blink detection filtering module 32, wherein;

[0054] The calibration tracking module 31 is used to perform eye movement calibration according to the eye movement tracking algorithm before using the head display.

[0055] This technical solution ensures that the system can accurately track the user's eye movements through the eye movement interaction control module 3. For example, the user is asked to gaze at a series of specific points or patterns, and the system is calibrated according to the user's gaze position to improve the accuracy of eye movement tracking.

[0056] At the same time, by introducing the eye tracking algorithm, the tracking accuracy and stability are improved, allowing the system to automatically learn the user's eye movement habits, thereby better predicting the user's gaze direction and intention.

[0057] The blink detection and filtering module 32 is used to determine whether it is a blink action by analyzing the frequency and amplitude characteristics of eye movements, and to perform filtering during the recognition process.

[0058] In this technical solution, the eye tracking sensor in the MR headset is responsible for collecting the raw data of the user's eyes. These data include information such as the position of the eyes, changes in the size of the pupils, and the movement trajectory of the eyeballs, and are collected at a certain frequency, for example: 120 to 240 frames per second. Blinking is usually accompanied by the closing and opening of the eyes, and specific changes in parameters such as the eye position and pupil size will occur. For example, when blinking, the pupil will be temporarily blocked by the eyelids, and the eye tracking sensor will detect that the pupil area has decreased sharply or even disappeared, and the eye position will be relatively fixed at the position where the eyelids are closed. At the same time, the frequency and amplitude of blinking also have certain characteristics. The normal blinking frequency is generally about 10 to 20 times per minute, and the blinking amplitude is relatively stable. In some special cases, such as when the eyes are tired or stimulated, the blinking frequency may increase and the amplitude may become larger.

[0059] Specifically, the following steps are included:

[0060] A differential algorithm is used in advance to calculate the first-order differential velocity and second-order differential acceleration of the eye movement position data to obtain the velocity and acceleration of the eye movement.

[0061] In this technical solution, when a blink occurs, the eye movement velocity will first drop rapidly (eyes closed) and then rise rapidly (eyes open), and the acceleration will also have a corresponding peak. These characteristics can be captured by calculating the eye movement velocity and acceleration.

[0062] Use the Canny edge detection image recognition algorithm to determine the pupil boundary, calculate the pupil area, and obtain the pupil area change;

[0063] This technical solution, through the recognition of pupil boundaries and area calculation, indicates a blinking action when the pupil area drops below a certain threshold in a short period of time and then recovers.

[0064] Eye movement velocity, acceleration, and pupil area change are used as feature inputs, and a support vector machine is used to determine whether it is a blink action.

[0065] Calibration is used in eye tracking to select preset scenes. If it is judged as a blink action, the blink action is filtered and feedback is prompted;

[0066] Specifically, in the scenario where eye tracking is used to select a virtual object, if a blink action is determined to be an eye movement, the system will not treat it as a selection operation for the object, thereby avoiding false triggering.

[0067] At the same time, when the system filters out a blink action, a small icon is displayed at the edge of the headset's field of view, indicating that the blink action just now was recognized and filtered, so that the user can understand the working status of the system and is used to indicate prompt feedback.

[0068] With the help of the above solution, through effective blink detection and filtering mechanism, unconscious eye movements such as blinking can be prevented from being misidentified as operation instructions, and the impact of environmental factors on eye tracking can be reduced.

[0069] The collaborative linkage module 4 is used for the gesture interaction control module 1, the voice interaction control module 2 and the eye movement interaction control module 3 to work together, and when the gesture interaction control module 1 or the voice interaction control module 2 or the eye movement interaction control module 3 performs priority control, the gesture interaction control module 1 or the voice interaction control module 2 or the eye movement interaction control module 3 is linked for collaborative control.

[0070] Specifically, in the present technical solution, when a user touches a virtual object, the system detects information such as the position and strength of the touch action through sensors in the MR headset, such as a hand position sensor and a force sensor, and performs gesture recognition through the gesture interaction control module 1, and then performs coordinated control with the voice interaction control module 2 and the eye movement interaction control module 3. For the voice interaction control module 2, a corresponding sound is generated according to the touch action and the characteristics of the object, and command control is performed according to the user's voice signal. At the same time, for the eye movement interaction control module 3, it is used to track the user's eye movement according to the touch action and the characteristics of the object for coordinated control.

[0071] In summary, with the help of the above technical solution of the present invention, the following effects can be achieved:

[0072] The present invention realizes accurate recognition of complex gestures and interactive recognition of command control through the gesture interaction control module 1, which enables users to interact with the virtual and real fusion environment more naturally and smoothly. For example, in an MR game scene, if the gesture recognition is accurate, the player can easily pick up virtual weapons, open virtual treasure chests, and perform other operations as easily as in the real world. This seamless interactive experience will make the user feel that he is truly part of the virtual world, and the sense of immersion will be greatly enhanced. At the same time, when the voice interaction control module 2 and the eye movement interaction control module 3 realize accurate interactive recognition, the user's attention can be more focused on the experience content itself, rather than being disturbed by frequent recognition errors. For example, the user can select an object by natural eye gaze, or quickly obtain information with voice commands, as if the virtual environment can read his mind. The accurate interactive recognition of the present invention reduces the fatigue caused by repeated operations or correcting wrong operations. The recognition and command control of complex gestures do not require the user to repeat the same action many times to achieve the purpose, and the burden on the arm and hand muscles will be reduced. Similarly, in improving the recognition of voice control commands in noisy environments, accurate recognition means that users do not need to speak loudly or slowly repeatedly to make the system understand their intentions, thereby reducing fatigue caused by voice interaction. For eye movement interaction control, filtering and command control are performed during the recognition process of blinking movements. Accurate recognition can avoid users having to constantly adjust their line of sight due to frequent false triggers, reducing eye tension and fatigue. For example, when using a headset to read virtual documents or observe virtual scenes for a long time, accurate eye tracking can allow users to browse content more comfortably.

[0073] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. After considering the disclosure of the specification and the examples, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and the examples are only regarded as exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.

[0074] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An intelligent control system for a MR head display, characterized in that: include: A gesture interaction control module (1), a voice interaction control module (2) and an eye movement interaction control module (3), wherein; The gesture interaction control module (1) is used to configure a high-definition camera (11) and a depth sensor (12) to automatically adjust parameters in a complex environment and provide real-time gesture prompts and feedback, so as to perform recognition and command control of complex gestures; wherein the high-definition camera (11) is used to collect hand movements; and the depth sensor (12) is used to embed a gesture recognition algorithm to adapt to different hand movements, speeds and angles, so as to meet the requirements for recognition of complex gestures; The voice interaction control module (2) is used to filter and suppress noise in the environment, highlight the user's voice signal, and automatically adjust and optimize according to the user's voice characteristics through an embedded voice recognition engine to improve the recognition of voice control instructions in a noisy environment. It includes: a noise elimination module (21) and a voice enhancement optimization module (22), wherein the noise elimination module (21) is used to filter and suppress noise in the environment by using a microphone array and digital signal processing to highlight the user's voice signal; the voice enhancement optimization module (22) is used to embed a voice recognition engine and can automatically adjust and optimize according to the user's voice characteristics; The eye movement interaction control module (3) is used to track the eye movement of the user and to filter and control commands during the recognition process of blinking actions, and comprises: a calibration tracking module (31) and a blink detection filtering module (32), wherein the calibration tracking module (31) is used to perform eye movement calibration according to an eye movement tracking algorithm before using the head display; and the blink detection filtering module (32) is used to determine whether it is a blinking action by analyzing the frequency and amplitude characteristics of the eye movement, and to filter during the recognition process.

2. The MR head display intelligent control system according to claim 1, characterized in that: Also includes: The collaborative linkage module (4) is used for the gesture interaction control module (1), the voice interaction control module (2) and the eye movement interaction control module (3) to work in collaboration, and when the gesture interaction control module (1) or the voice interaction control module (2) or the eye movement interaction control module (3) performs priority control, the gesture interaction control module (1) or the voice interaction control module (2) or the eye movement interaction control module (3) is linked to perform collaborative control.

3. The MR head display intelligent control system according to claim 1, characterized in that: The gesture interaction control module (1) further comprises: an environment calibration template (13) and a gesture guidance module (14), wherein; The environmental calibration template (13) is used to enable the head display to automatically adjust parameters in different usage environments to adapt to environmental changes, which at least includes: in an environment with too strong or too dark light, automatically adjusting the exposure parameters of the high-definition camera to ensure that hand movements can be clearly captured; The gesture guidance module (14) is used to provide real-time gesture prompts and feedback in the head display interface to help the user perform gesture operations correctly.

4. The MR head display intelligent control system according to claim 1, characterized in that: The voice interaction control module (2) further comprises: a voice setting module (23) and a voice feedback module (24), wherein; The voice setting module (23) is used to provide voice templates and shortcut command settings, allowing users to customize voice commands; The voice feedback module (24) is used to provide voice feedback after recognizing the user's voice command.

5. The MR head display intelligent control system according to claim 1, characterized in that: The blink detection filtering module (32) comprises the following steps: A differential algorithm is used in advance to calculate the first-order differential velocity and the second-order differential acceleration of the eye movement position data to obtain the velocity and acceleration of the eye movement; Use the Canny edge detection image recognition algorithm to determine the pupil boundary, calculate the pupil area, and obtain the pupil area change; Eye movement velocity, acceleration, and pupil area change are used as feature inputs, and a support vector machine is used to determine whether it is a blink action; Calibration is used in eye tracking to select preset scenes. If it is judged as a blink action, the blink action is filtered and feedback is prompted.

Images