Mixed reality eye-tracking interaction system and method based on dense map semantic segmentation

Abstract

A mixed reality eye-movement interaction system and method based on dense map semantic segmentation relates to the field of human-computer interaction and includes a head-mounted device that integrates a communication module, a mixed reality display module, an eye-tracking module, and a three-dimensional environment perception module. The three-dimensional environment perception module acquires image and depth data of the surrounding environment and performs three-dimensional reconstruction to obtain a dense map, which is then divided into different blocks and the semantic information of the blocks is annotated. The eye-tracking module emits rays into the dense map along the gaze direction and determines the gaze target based on the physical information of the collision point between the ray and the dense map. The communication module communicates with the physical object of the gaze target based on the annotation of the gaze target and physical information. The eye-tracking module obtains control instructions based on eye movements and preset control rules, and sends the control instructions to the physical object of the gaze target. This solves the problem of insufficient immersion caused by the inability to conduct natural eye-movement interaction with physical objects in existing mixed reality scenarios.

Description

Technical Field

[0001] The present invention relates to an eye movement interaction method, and in particular to a mixed reality eye movement interaction system and method based on dense map semantic segmentation, belonging to the field of human-computer interaction technology. Background Art

[0002] With the development of near-eye display technology, the difference between virtual objects rendered by mixed reality headsets and real objects has gradually decreased, and the effect of virtual-reality fusion has been improved, providing users with an immersive experience. More and more mixed reality headsets have been launched and are gaining popularity. When using mixed reality headsets, users are no longer fixed in space. Therefore, traditional human-computer interaction methods such as mice and keyboards are no longer applicable. There is an urgent need to explore new interaction methods as input for mixed reality scenes. Among them, eye movement interaction technology, with its advantages of fast response, high accuracy, and convenient operation, is gradually becoming a new paradigm for human-computer interaction in mixed reality.

[0003] Currently, eye movement interaction methods in mixed reality scenes are mainly divided into two categories: one is to use eye tracking equipment to obtain the gaze direction of the human eye, and emit a ray from the center of the pupil in the gaze direction, detect whether the ray collides with the virtual object, and thus determine whether to interact with the object; the other is to obtain the gaze direction of the human eye, calculate the coordinates of the gaze point on the plane based on the distance between the virtual plane and the human eye, and then complete the interaction with the object on the virtual plane. However, both methods interact with virtual objects and cannot interact with real objects, thereby limiting the interactive content when the user uses a mixed reality helmet and reducing the user's immersion. A close-range physical object eye movement interaction method based on an augmented reality helmet (publication number CN113961068A) uses simultaneous positioning and mapping technology to convert the gaze point to a fixed screen in the real scene, thereby completing the interaction with the object on the screen. This method can only interact with fixed objects and can only be used in scenes where mapping is completed, and its scope of use is limited. A mixed reality-based human-computer interaction method and system (publication number CN117762260A) uses target recognition in the user's field of view to determine the coordinate range of real objects in the image. The system then determines the physical object the user intends to interact with based on whether the gaze point falls within the recognized coordinate range. This method can only interact with pre-defined, recognizable physical objects, and can result in recognition failure or errors if the field of view is obstructed or the object is not fully displayed. Summary of the Invention

[0004] The technical problem solved by this application is to overcome the shortcomings of the existing technology and provide a flexible and accurate virtual-reality fusion eye movement interaction method to solve the problem of insufficient immersion caused by unnatural eye movement interaction in existing mixed reality scenes.

[0005] The technical solutions provided in this application are as follows:

[0006] A mixed reality eye-tracking interaction system based on dense map semantic segmentation, comprising a head-mounted device, wherein the head-mounted device includes a communication module, a mixed reality display module, an eye-tracking module, and a three-dimensional environment perception module;

[0007] Mixed reality display module to observe the real world;

[0008] A 3D environment perception module acquires image and depth data of the surrounding environment in the real world, reconstructs the surrounding environment in 3D, obtains a dense map, divides the dense map into different blocks using a 3D semantic segmentation algorithm, and annotates the semantic information of the blocks with physical information; and aligns the position of the dense map with the real world observed by the mixed reality display module;

[0009] The eye tracking module captures the user's eye movements, uses the center of the user's eyes as a starting point, and emits rays into the dense map along the calculated gaze direction. The ray is then detected for collision with different blocks of the dense map, and the gaze target is determined based on the physical object information at the user's gaze location. The eye tracking module determines the interaction intention based on the eye movement, obtains the control instruction based on the interaction intention and preset control rules, and sends the control instruction to the physical object in the real world corresponding to the gaze target through the communication module. The physical object then executes according to the instruction information.

[0010] The communication module communicates with the physical object of the gaze target according to the marking of the gaze target and the physical object information.

[0011] The eye tracking module includes an infrared emitter and a near-infrared sensor. The infrared emitter emits infrared light toward the eyes, the eyes reflect the infrared light, and the near-infrared sensor receives the reflected infrared light. The eye tracking algorithm is used to calculate the pupil position and gaze direction of the eyeball.

[0012] The three-dimensional environment perception module is composed of multiple cameras and depth sensors, and is used to obtain images and depth data of the surrounding environment.

[0013] The mixed reality display module, eye tracking module and three-dimensional environment perception module are calibrated through coordinate systems.

[0014] The labeling of the semantic information of the block with physical information includes: constructing a labeling dataset of physical objects and physical information covering various common indoor and outdoor scenes, training a deep convolutional neural network model so that it can accurately distinguish different types of physical objects and assign corresponding physical information labels.

[0015] A data enhancement algorithm is introduced into the training process to simulate the effects of different lighting conditions and perspective changes on real objects, so as to improve the generalization ability and robustness of the deep convolutional neural network model.

[0016] The eye tracking module has initial calibration and dynamic correction. The initial calibration is started synchronously when the user wears the device. The user needs to look at several fixed points on the screen as instructed, and the system establishes an eye parameter model based on this; if the user's head displacement is detected, dynamic correction will be triggered, and the eye parameter model will be adjusted according to the dynamic correction process.

[0017] A method for mixed reality eye movement interaction based on dense map semantic segmentation is based on any of the above-mentioned mixed reality eye movement interaction systems based on dense map semantic segmentation, wherein a user manipulates physical objects in the surrounding environment through a head-mounted device, the head-mounted device including a communication module, a mixed reality display module, an eye tracking module, and a three-dimensional environment perception module; and comprises the following steps:

[0018] The mixed reality display module displays the virtual model, and the user observes the real world through the mixed reality display module;

[0019] The 3D environment perception module acquires image and depth data of the surrounding environment in the real world, and performs 3D reconstruction of the surrounding environment to obtain a dense map. The dense map is divided into different blocks using a 3D semantic segmentation algorithm, and the semantic information of the blocks is annotated with physical information; and the dense map is aligned with the real world observed by the mixed reality display module.

[0020] The eye tracking module captures the user's eye movements and, starting from the center of the user's eyes, emits a ray into the dense map along the calculated gaze direction. It then performs collision detection on the ray with different blocks of the dense map and determines the gaze target based on the physical information of the user's gaze location.

[0021] The communication module communicates with a physical object in the real world corresponding to the gaze target according to the annotation of the gaze target and the physical object information;

[0022] The eye tracking module determines the interaction intention based on the eye movement, obtains the control instruction based on the interaction intention and the preset control rules, and sends the control instruction to the physical object of the gaze target through the communication module, and the physical object executes according to the instruction information.

[0023] The head-mounted device supports custom preset control rules.

[0024] In summary, this application has at least the following beneficial technical effects:

[0025] (1) The present invention introduces an eye-movement interaction method into the mixed reality scene, which allows the human eye to quickly select the physical interaction target, making the human-computer interaction response faster and the interaction more natural;

[0026] (2) The present invention introduces depth information into the physical eye movement interaction through the ray collision detection method, which enables interaction with real objects at different depths and effectively solves the occlusion problem between objects at different depths;

[0027] (3) The present invention reconstructs real objects in the environment in three dimensions, and can accurately project virtual information onto real objects during virtual-real fusion, thereby achieving a virtual-real fusion effect and enhancing the immersiveness of mixed reality through eye-movement interaction with real objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is an implementation flow chart of the present invention. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments disclosed in the present invention will be further described in detail below with reference to the accompanying drawings.

[0030] The embodiments of the present application disclose a mixed reality eye-movement interaction system based on dense map semantic segmentation, which aims to achieve a natural, intuitive and efficient human-computer interaction experience with objects in a real physical environment through a head-mounted device that integrates high-precision eye tracking, three-dimensional environmental perception and mixed reality display functions.

[0031] like Figure 1 As shown, it includes a head-mounted device. The head-mounted device involved in the present invention must be designed to balance lightness and comfort to ensure that the user does not feel uncomfortable during long-term wear. The device integrates a communication module, a mixed reality display module, a high-precision eye tracking module, and a three-dimensional environment perception module. The mixed reality display module uses a high-resolution display screen and supports transparent display, allowing users to clearly see the real world while observing the virtual content generated by the computer. The eye tracking module includes an infrared emitter and a near-infrared sensor. The infrared emitter emits infrared light towards the eyes, which reflects the infrared light. The near-infrared sensor receives the reflected infrared light and calculates pupil position, gaze direction, and blink information based on the eye tracking algorithm. The eye tracking module uses a near-infrared sensor with a high sampling rate (at least 250Hz) and low latency (less than 1ms) to ensure that it can accurately capture the user's eye movements in real time, including but not limited to information such as pupil position and gaze direction. The three-dimensional environment perception module consists of multiple cameras and a depth sensor to obtain images and depth data of the surrounding environment. The modules of the head-mounted device need to undergo precise coordinate system calibration to ensure the accuracy of the selection of interactive objects during human-computer interaction.

[0032] After the user wears the device, the environment modeling is started synchronously. The process first captures a color image through the camera and obtains depth information using the depth sensor. Subsequently, the SLAM algorithm (Simultaneous Localization and Mapping) or other advanced 3D reconstruction technologies are used to process the collected data to construct high-precision point cloud data. In this process, not only the precise position of static objects must be recorded, but also the motion trajectory of dynamic objects must be tracked in real time to ensure the timeliness and accuracy of the map. Next, the dense map generation algorithm is used to convert the initially obtained sparse point cloud data into a continuous and detailed surface representation to form a dense map. The dense map not only contains the geometric information of the object, but also retains the texture details, providing rich data support for subsequent semantic segmentation.

[0033] Based on the generated dense map, the present invention uses a deep learning algorithm to carefully classify each part of the environment and assign corresponding labels. Specifically, it is first necessary to build a large-scale labeled data set covering various common indoor and outdoor scenes. Then train the deep convolutional neural network model of the U-Net architecture so that it can accurately distinguish different types of objects such as walls, floors, furniture, etc., and assign corresponding labels. In order to improve the generalization ability and robustness of the model, data enhancement technology will be introduced during the training process to simulate the influence of factors such as different lighting conditions and perspective changes. After sufficient training, the model can quickly and accurately complete the semantic segmentation task in complex and changeable actual scenes, dividing the dense map of environmental modeling into multiple areas or objects with clear semantic information labels, providing a basis for eye movement interaction.

[0034] The eye tracking module continuously monitors the user's eye movement trajectory and calculates the current gaze direction through the eye tracking algorithm. In order to improve the positioning accuracy, the present invention has designed a complete calibration process, including two stages: initial calibration and dynamic correction. The initial calibration is started synchronously when the user wears the device. The user needs to look at several fixed points on the screen as instructed, and the system establishes an eye parameter model based on this. During use, if a large displacement of the user's head is detected, dynamic correction will be automatically triggered to fine-tune the eye parameter model to ensure that even when the user is free to move, a high gaze point positioning accuracy can be maintained. In addition, considering that the human eye may have slight drift, the present invention uses smoothing filtering technologies such as Kalman filters to further optimize the stability of the gaze point and reduce the probability of misjudgment.

[0035] After determining the user's gaze point, the system will use the center position of the user's eyes as the starting point and emit rays into the modeled environment along the calculated gaze direction. The intersection of the ray and the dense map is the user's gaze point. At the same time, based on the results of semantic segmentation in step 3, the semantic information of the gaze point location area or object is clarified, and the gaze target is determined based on the physical information of the gaze point. The communication module communicates with the physical object of the gaze target based on the annotation of the gaze target and physical information, and uses the gaze target as the input of eye movement interaction.

[0036] Based on the detected gaze target, according to the preset control rules and the gaze target, the control instructions are obtained, and the control instructions are sent to the physical object of the gaze target. The physical object of the gaze target executes according to the instruction information, thereby realizing eye movement interaction with objects in the real world. For example, when a user stares at a specific object (such as a lamp) for a long time, the system can automatically recognize the intention and perform corresponding operations (such as turning the lamp on or off). In addition, users can also convey more complex commands through specific eye movements (such as rapid blinking, double-clicking, etc.), such as adjusting the opening and closing degree of curtains, adjusting the volume, etc. In order to enhance the user experience, the present invention also supports customized preset control rules, allowing users to set different trigger conditions and response behaviors according to personal preferences.

[0037] In summary, the mixed reality eye-tracking interaction method and system based on dense map semantic segmentation proposed in this paper integrates advanced hardware technologies and software algorithms to achieve full coverage from environmental perception and semantic understanding to precise interaction, providing users with a new ability to interact with objects in the real world. Real objects (such as air conditioners, washing machines, and lamps) can be controlled through head-mounted devices.

[0038] This embodiment also discloses a mixed reality eye movement interaction method based on dense map semantic segmentation. Based on the above-mentioned mixed reality eye movement interaction system based on dense map semantic segmentation, a user controls physical objects in the surrounding environment through a head-mounted device, and the head-mounted device includes a communication module, a mixed reality display module, an eye tracking module, and a three-dimensional environment perception module. The method includes the following steps:

[0039] The mixed reality display module displays the virtual model, and the user observes the real world through the mixed reality display module;

[0040] The 3D environment perception module acquires image and depth data of the surrounding environment in the real world, and performs 3D reconstruction of the surrounding environment to obtain a dense map. The dense map is divided into different blocks using a 3D semantic segmentation algorithm, and the semantic information of the blocks is annotated with physical information; and the dense map is aligned with the real world observed by the mixed reality display module.

[0041] The eye tracking module captures the user's eye movements and, starting from the center of the user's eyes, emits a ray into the dense map along the calculated gaze direction. It then performs collision detection on the ray with different blocks of the dense map and determines the gaze target based on the physical information of the user's gaze location.

[0042] The communication module communicates with a physical object in the real world corresponding to the gaze target according to the annotation of the gaze target and the physical object information;

[0043] The eye tracking module determines the interaction intention based on the eye movement, obtains the control instruction based on the interaction intention and the preset control rules, and sends the control instruction to the physical object of the gaze target through the communication module, and the physical object executes according to the instruction information.

[0044] The contents not described in detail in this application specification are common knowledge to those skilled in the art.

[0045] The present application has been described in detail above with reference to specific embodiments and exemplary examples. However, these descriptions should not be construed as limiting the present application. Those skilled in the art will appreciate that, without departing from the spirit and scope of the present application, various equivalent substitutions, modifications, or improvements may be made to the technical solutions and implementations of the present application, all of which fall within the scope of the present application. The scope of protection of the present application shall be determined by the appended claims.

Claims

1. A mixed reality eye-tracking interaction system based on dense map semantic segmentation, characterized by: The head-mounted device includes a communication module, a mixed reality display module, an eye tracking module, and a three-dimensional environment perception module; Mixed reality display module to observe the real world; A 3D environment perception module acquires image and depth data of the surrounding environment in the real world, reconstructs the surrounding environment in 3D, obtains a dense map, divides the dense map into different blocks using a 3D semantic segmentation algorithm, and annotates the semantic information of the blocks with physical information; and aligns the position of the dense map with the real world observed by the mixed reality display module; The eye tracking module captures the user's eye movements, uses the center of the user's eyes as a starting point, and emits rays into the dense map along the calculated gaze direction. The ray is then detected for collision with different blocks of the dense map, and the gaze target is determined based on the physical object information at the user's gaze location. The eye tracking module determines the interaction intention based on the eye movement, obtains the control instruction based on the interaction intention and preset control rules, and sends the control instruction to the physical object in the real world corresponding to the gaze target through the communication module. The physical object then executes according to the instruction information. The communication module communicates with the physical object of the gaze target according to the marking of the gaze target and the physical object information.

2. The mixed reality eye-tracking interaction system based on dense map semantic segmentation according to claim 1, characterized in that: The eye tracking module includes an infrared emitter and a near-infrared sensor. The infrared emitter emits infrared light toward the eyes, the eyes reflect the infrared light, and the near-infrared sensor receives the reflected infrared light. The eye tracking algorithm is used to calculate the pupil position and gaze direction of the eyeball.

3. The mixed reality eye-tracking interaction system based on dense map semantic segmentation according to claim 1, characterized in that: The three-dimensional environment perception module includes multiple cameras and depth sensors for acquiring images and depth data of the surrounding environment.

4. The mixed reality eye-tracking interaction system based on dense map semantic segmentation according to claim 1, characterized in that: The mixed reality display module, eye tracking module and three-dimensional environment perception module are calibrated through coordinate systems.

5. The mixed reality eye-tracking interaction system based on dense map semantic segmentation according to claim 1, characterized in that: The labeling of the semantic information of the block with physical information includes: constructing a labeling dataset of physical objects and physical information covering various common indoor and outdoor scenes, training a deep convolutional neural network model so that it can accurately distinguish different types of physical objects and assign corresponding physical information labels.

6. The mixed reality eye-tracking interaction system based on dense map semantic segmentation according to claim 5, characterized in that: A data enhancement algorithm is introduced into the training process to simulate the effects of different lighting conditions and perspective changes on real objects, so as to improve the generalization ability and robustness of the deep convolutional neural network model.

7. The mixed reality eye-tracking interaction system based on dense map semantic segmentation according to claim 1, characterized in that: The eye tracking module includes initial calibration and dynamic correction steps. The initial calibration starts synchronously when the user wears the device. The user needs to follow the instructions to look at several fixed points on the screen, and the system establishes an eye parameter model based on this. If the user's head displacement is detected, dynamic correction will be triggered, and the eye parameter model will be adjusted according to the dynamic correction process.

8. The mixed reality eye-tracking interaction system based on dense map semantic segmentation according to claim 1, characterized in that: The head-mounted device supports custom preset control rules.

9. A mixed reality eye movement interaction method based on dense map semantic segmentation, characterized in that: A mixed reality eye-tracking interaction system based on dense map semantic segmentation according to any one of claims 1 to 8 is performed, wherein a user manipulates physical objects in the surrounding environment through a head-mounted device, the head-mounted device including a communication module, a mixed reality display module, an eye-tracking module, and a three-dimensional environment perception module; and the system comprises the following steps: The mixed reality display module displays the virtual model, and the user observes the real world through the mixed reality display module; The 3D environment perception module acquires image and depth data of the surrounding environment in the real world, and performs 3D reconstruction of the surrounding environment to obtain a dense map. The dense map is divided into different blocks using a 3D semantic segmentation algorithm, and the semantic information of the blocks is annotated with physical information; and the dense map is aligned with the real world observed by the mixed reality display module. The eye tracking module captures the user's eye movements and, starting from the center of the user's eyes, emits a ray into the dense map along the calculated gaze direction. It then performs collision detection on the ray with different blocks of the dense map and determines the gaze target based on the physical information of the user's gaze location. The communication module communicates with a physical object in the real world corresponding to the gaze target according to the annotation of the gaze target and the physical object information; The eye tracking module determines the interaction intention based on the eye movement, obtains the control instruction based on the interaction intention and the preset control rules, and sends the control instruction to the physical object of the gaze target through the communication module, and the physical object executes according to the instruction information.

Images