Mixed reality eye movement interaction system and method based on dense map semantic segmentation

Abstract

The invention discloses a mixed reality eye movement interaction system and method based on dense map semantic segmentation, and relates to the field of man-machine interaction, and the system comprises a head-mounted device which integrates a communication module, a mixed reality display module, an eye movement tracking module and a three-dimensional environment perception module. The three-dimensional environment perception module obtains images and depth data of the surrounding environment and carries out three-dimensional reconstruction to obtain a dense map, the dense map is divided into different blocks, and semantic information of the blocks is marked; the eye movement tracking module emits rays to the dense map in the gazing direction, and a gazing target is determined according to the real object information of the collision point position of the rays and the dense map; the communication module communicates with the real object of the watching target according to the watching target and the label of the real object information; and the eye movement tracking module obtains a control instruction according to the eyeball movement and a preset control rule, and sends the control instruction to a real object watching the target. The problem that the immersion feeling is insufficient due to the fact that natural eye movement interaction with a real object cannot be carried out in an existing mixed reality scene is solved.

Description

Technical Field

[0001] The present invention relates to an eye movement interaction method, and in particular to a mixed reality eye movement interaction system and method based on dense map semantic segmentation, belonging to the technical field of human-computer interaction. Background Art

[0002] With the development of near-eye display technology, the difference between virtual objects rendered by mixed reality helmets and real objects has gradually decreased, and the effect of virtual-reality fusion has been improved, providing users with an immersive experience. More and more mixed reality helmet products have been launched and favored by users. When using mixed reality helmets, users are no longer in a fixed position in space at all times, so traditional human-computer interaction methods, such as mouse and keyboard, are no longer applicable. It is urgent to explore new interaction methods as input for mixed reality scenes. Among them, eye movement interaction technology has gradually become a new paradigm of human-computer interaction in mixed reality with its advantages of fast response, high accuracy, and convenient operation.

[0003] At present, the eye movement interaction methods in mixed reality scenes are mainly divided into two categories: one is to use eye tracking equipment to obtain the gaze direction of the human eye, and emit a ray from the center of the pupil to the gaze direction, detect whether the ray collides with the virtual object to determine whether to interact with the object; the second is to obtain the gaze direction of the human eye, calculate the coordinates of the gaze point on the plane according to the distance between the virtual plane and the human eye, and then complete the interaction with the object on the virtual plane. However, both methods interact with virtual objects and cannot interact with real objects, thereby limiting the interactive content when the user uses a mixed reality helmet and reducing the user's immersion. A close-range physical eye movement interaction method based on an augmented reality helmet (publication number CN113961068A) uses simultaneous positioning and mapping technology to convert the gaze point to a fixed screen in the real scene, thereby completing the interaction with the object on the screen. This method can only interact with fixed objects and can only be used in scenes where mapping is completed, and its scope of use is limited. A human-computer two-way interaction method and system based on mixed reality technology (publication number CN117762260A) obtains the coordinate range of the real object in the image by performing target recognition on the user's visual field image, and determines the physical target that the user intends to interact with based on whether the gaze point is within the recognized coordinate range. This method can only interact with set identifiable physical targets, and may cause recognition failure or errors when there is visual obstruction or incomplete display of the target. Summary of the invention

[0004] The technical problem solved by the present application is to overcome the deficiencies of the prior art and provide a flexible and accurate virtual-reality fusion eye movement interaction method to solve the problem of insufficient immersion caused by unnatural eye movement interaction in existing mixed reality scenes.

[0005] The technical solutions provided by this application are as follows:

[0006] A mixed reality eye-tracking interaction system based on dense map semantic segmentation, comprising a head-mounted device, wherein the head-mounted device comprises a communication module, a mixed reality display module, an eye-tracking module and a three-dimensional environment perception module;

[0007] Mixed reality display module to observe the real world;

[0008] A three-dimensional environment perception module obtains images and depth data of the surrounding environment in the real world, and performs three-dimensional reconstruction of the surrounding environment to obtain a dense map. The dense map is divided into different blocks using a three-dimensional semantic segmentation algorithm, and the semantic information of the blocks is annotated with physical information; and the dense map is made to correspond in position with the real world observed by the mixed reality display module;

[0009] The eye tracking module captures the user's eye movements, takes the center position of the user's eyes as the starting point, emits rays into the dense map along the calculated gaze direction, performs collision detection on the rays and different blocks of the dense map, and determines the gaze target according to the physical information of the user's gaze position; the eye tracking module determines the interaction intention according to the eye movement, obtains the control instruction according to the interaction intention and the preset control rules, and sends the control instruction to the physical object in the real world corresponding to the gaze target through the communication module, and the physical object executes according to the instruction information;

[0010] The communication module communicates with the physical object of the gaze target according to the marking of the gaze target and the physical object information.

[0011] The eye tracking module includes an infrared transmitter and a near-infrared sensor. The infrared transmitter transmits infrared light to the eyes, the eyes reflect the infrared light, and the near-infrared sensor receives the reflected infrared light. The pupil position and gaze direction of the eyeball are calculated according to the eye tracking algorithm.

[0012] The three-dimensional environment perception module is composed of multiple cameras and depth sensors, and is used to obtain images and depth data of the surrounding environment.

[0013] The mixed reality display module, eye tracking module and three-dimensional environment perception module are calibrated through a coordinate system.

[0014] The labeling of the semantic information of the block with physical information includes: constructing a labeling dataset of physical objects and physical information covering various common indoor and outdoor scenes, training a deep convolutional neural network model so that it can accurately distinguish different types of physical objects and assign corresponding physical information labels.

[0015] A data enhancement algorithm is introduced in the training process to simulate the effects of different lighting conditions and perspective changes on physical objects, so as to improve the generalization ability and robustness of the deep convolutional neural network model.

[0016] The eye tracking module has initial calibration and dynamic correction. The initial calibration is started synchronously when the user wears the device. The user needs to look at several fixed points on the screen as instructed, and the system establishes an eye parameter model based on this. If displacement of the user's head is detected, dynamic correction will be triggered, and the eye parameter model will be adjusted according to the dynamic correction process.

[0017] A mixed reality eye movement interaction method based on dense map semantic segmentation is based on any of the above-mentioned mixed reality eye movement interaction systems based on dense map semantic segmentation, and a user manipulates physical objects in the surrounding environment through a head-mounted device, and the head-mounted device includes a communication module, a mixed reality display module, an eye movement tracking module and a three-dimensional environment perception module; and includes the following steps:

[0018] The mixed reality display module displays the virtual model, and the user observes the real world through the mixed reality display module;

[0019] The 3D environment perception module obtains the image and depth data of the surrounding environment of the real world, and performs 3D reconstruction of the surrounding environment to obtain a dense map, divides the dense map into different blocks using a 3D semantic segmentation algorithm, and annotates the semantic information of the blocks with physical information; and makes the dense map correspond to the position of the real world observed by the mixed reality display module;

[0020] The eye tracking module captures the user's eye movements, and uses the center position of the user's eyes as the starting point to emit rays into the dense map along the calculated gaze direction, performs collision detection between the rays and different blocks of the dense map, and determines the gaze target based on the physical information of the user's gaze position;

[0021] The communication module communicates with a physical object in the real world corresponding to the gaze target according to the annotation of the gaze target and the physical object information;

[0022] The eye tracking module determines the interaction intention according to the eye movement, obtains the control instruction according to the interaction intention and the preset control rules, and sends the control instruction to the physical object of the gaze target through the communication module, and the physical object executes according to the instruction information.

[0023] The head mounted device supports custom preset control rules.

[0024] In summary, this application at least includes the following beneficial technical effects:

[0025] (1) The present invention introduces an eye movement interaction method in a mixed reality scene, which allows the human eye to quickly select a physical interaction target, making the human-computer interaction response faster and the interaction more natural;

[0026] (2) The present invention introduces depth information into the physical eye movement interaction through the ray collision detection method, which enables interaction with real objects at different depths and effectively solves the occlusion problem between objects at different depths;

[0027] (3) The present invention reconstructs real objects in the environment in three dimensions. When the virtual and the real are fused, the virtual information can be accurately projected onto the real objects, thereby achieving a virtual and the real fusion effect. The immersive feeling of mixed reality is enhanced by eye movement interaction with the real objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is an implementation flow chart of the present invention. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments disclosed in the present invention will be further described in detail below with reference to the accompanying drawings.

[0030] The embodiment of the present application discloses a mixed reality eye movement interaction system based on dense map semantic segmentation, which aims to achieve a natural, intuitive and efficient human-computer interaction experience with objects in a real physical environment through a head-mounted device that integrates high-precision eye tracking, three-dimensional environment perception and mixed reality display functions.

[0031] like Figure 1 As shown, it includes a head-mounted device. The head-mounted device involved in the present invention needs to be designed with both lightness and comfort in mind to ensure that the user does not feel uncomfortable during long-term wearing. The device integrates a communication module, a mixed reality display module, a high-precision eye tracking module and a three-dimensional environment perception module. Among them, the mixed reality display module uses a high-resolution display screen and supports transparent display. The user can still clearly see the real world while observing the virtual content generated by the calculation. The eye tracking module includes an infrared transmitter and a near-infrared sensor. The infrared transmitter emits infrared light to the eyes, the eyes reflect the infrared light, and the near-infrared sensor receives the reflected infrared light. According to the eye tracking algorithm, the pupil position, gaze direction and blinking information are calculated. The eye tracking module uses a near-infrared sensor with a high sampling rate (at least 250Hz) and low latency (less than 1ms) to ensure that the user's eye movement can be captured in real time and accurately, including but not limited to information such as pupil position and line of sight direction. The three-dimensional environment perception module is composed of multiple cameras and depth sensors, which are used to obtain images and depth data of the surrounding environment. The modules of the head-mounted device need to be precisely calibrated with coordinate systems to ensure the accuracy of the selection of interactive objects during human-computer interaction.

[0032] After the user wears the device, the environment modeling is started synchronously. The process first captures a color image through the camera and obtains depth information using a depth sensor. Subsequently, the collected data is processed using the SLAM algorithm (Simultaneous Localization and Mapping) or other advanced 3D reconstruction technology to construct high-precision point cloud data. In this process, not only the precise position of static objects must be recorded, but also the motion trajectory of dynamic objects must be tracked in real time to ensure the timeliness and accuracy of the map. Next, the dense map generation algorithm is used to convert the initially obtained sparse point cloud data into a continuous and detailed surface representation to form a dense map. The dense map not only contains the geometric information of the object, but also retains the texture details, providing rich data support for subsequent semantic segmentation.

[0033] Based on the generated dense map, the present invention uses a deep learning algorithm to carefully classify each part of the environment and assign corresponding labels. Specifically, it is first necessary to build a large-scale annotated data set covering various common indoor and outdoor scenes. Then train the deep convolutional neural network model of the U-Net architecture so that it can accurately distinguish different types of objects such as walls, floors, furniture, etc., and assign corresponding labels. In order to improve the generalization ability and robustness of the model, data enhancement technology will be introduced during the training process to simulate the influence of factors such as different lighting conditions and perspective changes. After sufficient training, the model can quickly and accurately complete the semantic segmentation task in complex and changeable actual scenes, and divide the dense map of environmental modeling into multiple areas or objects with clear semantic information labels, providing a basis for eye movement interaction.

[0034] The eye tracking module continuously monitors the user's eye movement trajectory and calculates the current gaze direction through the eye tracking algorithm. In order to improve the positioning accuracy, the present invention has designed a complete calibration process, including two stages: initial calibration and dynamic correction. The initial calibration is started synchronously when the user wears the device. The user needs to look at several fixed points on the screen according to the instructions, and the system establishes an eye parameter model based on this. During use, if a large displacement of the user's head is detected, dynamic correction will be automatically triggered to fine-tune the eye parameter model to ensure that even when the user is free to move, a high gaze point positioning accuracy can be maintained. In addition, considering that the human eye may have a slight drift phenomenon, the present invention uses smoothing filtering technologies such as Kalman filters to further optimize the stability of the gaze point and reduce the probability of misjudgment.

[0035] After determining the user's gaze point, the system will use the center position of the user's eyes as the starting point and emit rays into the modeled environment along the calculated gaze direction. The intersection of the ray and the dense map is the user's gaze point. At the same time, based on the result of semantic segmentation in step 3, the semantic information of the gaze point location area or object is clarified, and the gaze target is determined according to the physical information of the gaze point. The communication module communicates with the physical object of the gaze target based on the annotation of the gaze target and the physical information, and uses the gaze target as the input of eye movement interaction.

[0036] Based on the detected gaze target, according to the preset control rules and the gaze target, the control instructions are obtained, and the control instructions are sent to the physical object of the gaze target. The physical object of the gaze target executes according to the instruction information to achieve eye movement interaction with objects in the real world. For example, when a user stares at a specific object (such as a lamp) for a long time, the system can automatically recognize the intention and perform corresponding operations (such as turning on or off the lamp). In addition, users can also convey more complex commands through specific eye movements (such as rapid blinking, double-clicking, etc.), such as adjusting the opening and closing degree of curtains, adjusting the volume, etc. In order to enhance the user experience, the present invention also supports custom preset control rules, allowing users to set different trigger conditions and response behaviors according to personal preferences.

[0037] In summary, the mixed reality eye movement interaction method and system based on dense map semantic segmentation proposed in the present invention realizes the full process coverage from environmental perception, semantic understanding to precise interaction by integrating advanced hardware technology and software algorithms, providing users with a new ability to interact with objects in the real world. The control of real objects (air conditioners, washing machines, lamps, etc.) is realized through head-mounted devices.

[0038] This embodiment also discloses a mixed reality eye movement interaction method based on dense map semantic segmentation, which is based on the above-mentioned mixed reality eye movement interaction system based on dense map semantic segmentation, and the user controls the physical objects in the surrounding environment through a head-mounted device, and the head-mounted device includes a communication module, a mixed reality display module, an eye movement tracking module and a three-dimensional environment perception module; and includes the following steps:

[0039] The mixed reality display module displays the virtual model, and the user observes the real world through the mixed reality display module;

[0040] The 3D environment perception module obtains the image and depth data of the surrounding environment of the real world, and performs 3D reconstruction of the surrounding environment to obtain a dense map, divides the dense map into different blocks using a 3D semantic segmentation algorithm, and annotates the semantic information of the blocks with physical information; and makes the dense map correspond to the position of the real world observed by the mixed reality display module;

[0041] The eye tracking module captures the user's eye movements, and uses the center position of the user's eyes as the starting point to emit rays into the dense map along the calculated gaze direction, performs collision detection between the rays and different blocks of the dense map, and determines the gaze target based on the physical information of the user's gaze position;

[0042] The communication module communicates with a physical object in the real world corresponding to the gaze target according to the annotation of the gaze target and the physical object information;

[0043] The eye tracking module determines the interaction intention according to the eye movement, obtains the control instruction according to the interaction intention and the preset control rules, and sends the control instruction to the physical object of the gaze target through the communication module, and the physical object executes according to the instruction information.

[0044] The contents not described in detail in this application specification belong to the common knowledge of those skilled in the art.

[0045] The present application is described in detail above in conjunction with specific implementation methods and exemplary examples, but these descriptions cannot be understood as limiting the present application. Those skilled in the art understand that, without departing from the spirit and scope of the present application, a variety of equivalent replacements, modifications or improvements can be made to the technical solution of the present application and its implementation methods, all of which fall within the scope of the present application. The scope of protection of the present application shall be subject to the attached claims.

Claims

1. A mixed reality eye-tracking interaction system based on dense map semantic segmentation, characterized in that: The head-mounted device includes a communication module, a mixed reality display module, an eye tracking module and a three-dimensional environment perception module; Mixed reality display module to observe the real world; A three-dimensional environment perception module obtains images and depth data of the surrounding environment in the real world, and performs three-dimensional reconstruction of the surrounding environment to obtain a dense map. The dense map is divided into different blocks using a three-dimensional semantic segmentation algorithm, and the semantic information of the blocks is annotated with physical information; and the dense map is made to correspond in position with the real world observed by the mixed reality display module; The eye tracking module captures the user's eye movements, takes the center position of the user's eyes as the starting point, emits rays into the dense map along the calculated gaze direction, performs collision detection between the rays and different blocks of the dense map, and determines the gaze target according to the physical information of the user's gaze position; the eye tracking module determines the interaction intention according to the eye movement, obtains the control instruction according to the interaction intention and the preset control rules, and sends the control instruction to the physical object in the real world corresponding to the gaze target through the communication module, and the physical object executes according to the instruction information; The communication module communicates with the physical object of the gaze target according to the marking of the gaze target and the physical object information.

2. The mixed reality eye-movement interaction system based on dense map semantic segmentation according to claim 1, characterized in that: The eye tracking module includes an infrared transmitter and a near-infrared sensor. The infrared transmitter transmits infrared light to the eyes, the eyes reflect the infrared light, and the near-infrared sensor receives the reflected infrared light. The pupil position and gaze direction of the eyeball are calculated according to the eye tracking algorithm.

3. The mixed reality eye-movement interaction system based on dense map semantic segmentation according to claim 1, characterized in that: The three-dimensional environment perception module includes multiple cameras and depth sensors, which are used to obtain images and depth data of the surrounding environment.

4. The mixed reality eye-movement interaction system based on dense map semantic segmentation according to claim 1, characterized in that: The mixed reality display module, eye tracking module and three-dimensional environment perception module are calibrated through a coordinate system.

5. The mixed reality eye-movement interaction system based on dense map semantic segmentation according to claim 1, characterized in that: The labeling of the semantic information of the block with physical information includes: constructing a labeling dataset of physical objects and physical information covering various common indoor and outdoor scenes, training a deep convolutional neural network model so that it can accurately distinguish different types of physical objects and assign corresponding physical information labels.

6. The mixed reality eye-movement interaction system based on dense map semantic segmentation according to claim 5, characterized in that: A data enhancement algorithm is introduced in the training process to simulate the effects of different lighting conditions and perspective changes on physical objects, so as to improve the generalization ability and robustness of the deep convolutional neural network model.

7. The mixed reality eye-movement interaction system based on dense map semantic segmentation according to claim 1, characterized in that: The eye tracking module includes initial calibration and dynamic correction steps. The initial calibration is started synchronously when the user wears the device. The user needs to look at several fixed points on the screen as instructed, and the system establishes an eye parameter model based on this. If displacement of the user's head is detected, dynamic correction will be triggered, and the eye parameter model will be adjusted according to the dynamic correction process.

8. The mixed reality eye-movement interaction system based on dense map semantic segmentation according to claim 1, characterized in that: The head mounted device supports custom preset control rules.

9. A mixed reality eye movement interaction method based on dense map semantic segmentation, characterized in that: A mixed reality eye-tracking interaction system based on dense map semantic segmentation according to any one of claims 1 to 8 is performed, wherein a user manipulates physical objects in the surrounding environment through a head-mounted device, and the head-mounted device includes a communication module, a mixed reality display module, an eye-tracking module, and a three-dimensional environment perception module; and the following steps are included: The mixed reality display module displays the virtual model, and the user observes the real world through the mixed reality display module; The 3D environment perception module obtains the image and depth data of the surrounding environment of the real world, and performs 3D reconstruction of the surrounding environment to obtain a dense map, divides the dense map into different blocks using a 3D semantic segmentation algorithm, and annotates the semantic information of the blocks with physical information; and makes the dense map correspond to the position of the real world observed by the mixed reality display module; The eye tracking module captures the user's eye movements, and uses the center position of the user's eyes as the starting point to emit rays into the dense map along the calculated gaze direction, performs collision detection between the rays and different blocks of the dense map, and determines the gaze target based on the physical information of the user's gaze position; The communication module communicates with a physical object in the real world corresponding to the gaze target according to the annotation of the gaze target and the physical object information; The eye tracking module determines the interaction intention according to the eye movement, obtains the control instruction according to the interaction intention and the preset control rules, and sends the control instruction to the physical object of the gaze target through the communication module, and the physical object executes according to the instruction information.

Images