Sight tracking method and system based on 3D attitude and 6DoF positioning

Abstract

The invention relates to the technical field of VR recognition, and particularly discloses a sight tracking method and system based on 3D attitude and 6DoF localization, and the method comprises the steps: obtaining light spot images of infrared light emitted by infrared lamps at a plurality of fixed positions in a cabin through a plurality of first cameras, and carrying out the reverse calculation to obtain first coordinate information of a helmet relative to the cabin; acquiring 6DoF inertial information of the helmet in real time, and when the first coordinate information of the helmet cannot be obtained based on the light spot image, combining the recently acquired first coordinate information with the corresponding 6DoF inertial information, and calculating to obtain the latest second coordinate information of the helmet relative to the cabin; shooting the eyes of the user by using a second camera in the helmet, and calculating to obtain third coordinate information of the sight line of the eyes of the user relative to the helmet; fourth coordinate information of the eye sight of the user relative to the cabin is obtained through calculation. The problem that 3D posture recognition is lost due to the fact that a user breaks away from a recognition area can be solved.

Description

Technical Field

[0001] The present invention relates to the field of VR recognition technology, and in particular to a sight tracking method and system based on 3D gesture and 6DoF positioning. Background Art

[0002] Eye tracking is a process of identifying what someone is looking at and how they are looking at it. It has been widely used in many fields such as human-computer interaction, virtual reality, vehicle assisted driving, human factors analysis and psychological research. From the physiological structure of the eyeball, humans mainly obtain visual data through the fovea, which can only provide about 1-2 degrees of visual angle. Although this area only occupies a very small part of the visual field, the information recorded by this area contains 50% of the effective visual information transmitted to the brain through the optic nerve. Therefore, the human visual and attention systems work around one main goal: to focus the optical imaging of the target of interest on the fovea. This is the most fundamental and main reason for eye movement behavior.

[0003] According to the usage scenario, eye tracking devices can be mainly divided into two types: screen-based eye trackers and wearable eye trackers. Screen-based eye trackers are placed at a certain distance from the user to track the user's eye movements. Wearable eye trackers collect the user's eye movement behavior in a real environment by integrating the eye tracking system and scene camera on a lightweight frame, such as glasses or helmets. The user can move freely to maximize the ecological validity of the experiment.

[0004] Based on this, if the wearable eye tracker is combined with the game console, many interesting experiences can be achieved. The console can accurately identify the user's 3D posture, and the wearable eye tracker can identify the user's line of sight. The combination of the two can achieve more virtual operations. However, if the user's movements are too large and out of the recognition area, the 3D posture recognition will be lost. Summary of the invention

[0005] In view of the above technical problems, the present invention provides a gaze tracking method and system based on 3D gesture and 6DoF positioning to solve the problem of losing 3D gesture recognition due to the user leaving the recognition area.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present invention, a gaze tracking method based on 3D gesture and 6DoF positioning is disclosed, the method comprising: A plurality of first cameras are used to obtain a spot image of infrared light emitted by infrared lamps at a plurality of fixed positions in the cabin. The first camera is arranged on a head-mounted helmet. The position of the infrared lamp is calculated based on the spot image. Based on the position of the infrared lamp and the principle of multi-view geometry, a reverse calculation is performed to obtain first coordinate information of the helmet relative to the cabin. The first coordinate information includes a three-dimensional coordinate and a posture. The first camera is an infrared camera. acquiring 6DoF inertial information of the helmet in real time, and when the first coordinate information of the helmet cannot be obtained based on the light spot image, combining the most recently obtained first coordinate information with the corresponding 6DoF inertial information to calculate the latest second coordinate information of the helmet relative to the cabin; Based on eye tracking, a second camera in the helmet is used to photograph the user's eyes, and third coordinate information of the user's eye sight relative to the helmet is calculated; The third coordinate information is associated with the first coordinate information or the third coordinate information is associated with the second coordinate information to obtain fourth coordinate information of the user's eye sight relative to the cabin.

[0008] Furthermore, when the first coordinate information of the helmet is obtained by reverse calculation, it specifically includes: Based on an image processing algorithm, the pixel coordinates of the light spot are extracted from the light spot image; Matching the pixel coordinates captured by the plurality of the first cameras to identify the projection positions of the same infrared light in different first cameras; Based on the multi-view geometry principle, the three-dimensional coordinates of the infrared light relative to the first camera are calculated by performing triangulation through the pixel coordinates of the matched light spot and the relative position of the first camera; The first coordinate information is calculated based on the three-dimensional coordinates of the infrared light relative to the first camera and the three-dimensional coordinates relative to the cabin, and based on different positions of the first camera on the helmet.

[0009] Furthermore, after calculating and obtaining the latest second coordinate information of the helmet relative to the cabin, the method further includes: Calculating a position error between the first coordinate information and the second coordinate information in real time according to the first coordinate information of the helmet obtained from the light spot image; Based on the fuzzy prediction algorithm, the 6DoF inertial information is used as input, the position error is used as output, and a fuzzy learning model is trained and constructed; When the first coordinate information of the helmet cannot be obtained based on the light spot image, using the fuzzy learning model to predict the position error; Based on the 6DoF inertial information and the first coordinate information, the second coordinate information is derived, and the predicted position error is compensated to the second coordinate information.

[0010] Furthermore, when training and constructing the fuzzy learning model, it includes: Defining a structure of the fuzzy learning model, wherein the structure of the fuzzy learning model includes n fuzzy subsystems and m enhancement nodes; defining a plurality of fuzzy rules for each of the fuzzy subsystems, wherein the fuzzy rules are used to describe the relationship between the 6DoF inertial information and the position error; A Gaussian membership function is defined for each fuzzy set, and its center and width are determined based on the K-means clustering algorithm; calculating the activation strength of each of the fuzzy rules as a product of the Gaussian membership function of the 6DoF inertial information; Based on the activation strength and the parameters of the fuzzy subsystem, construct an output function of each of the fuzzy subsystems; The 6DoF inertial information is organized into a training data set and represented by a matrix, and the parameters of the fuzzy subsystem are trained and adjusted to obtain the fuzzy learning model.

[0011] Further, when calculating the third coordinate information of the user's eye sight relative to the helmet, it specifically includes: Based on at least two second cameras continuously capturing pupil images of the same eye of the user, a bounding box of the pupil is calculated based on a deep learning algorithm to detect the center of the pupil; Based on the principle of multi-view geometry and the internal and external parameters of different cameras, different second cameras are projected to the corresponding pupil center into the three-dimensional space to obtain rays, and the intersection point or the closest point of different rays is used as the three-dimensional position of the pupil center. Multiple three-dimensional positions of the pupil center are fitted, and the user's eyeball model is obtained by combining the empirical constant of the eyeball radius; The pupil center of the user is continuously tracked, and the vector line from the eyeball model to the pupil center is used as the eye sight line of the user, and the third coordinate information of the eye sight line relative to the first camera is obtained according to the coordinate relationship between the second camera and the first camera.

[0012] Furthermore, the multiple second cameras arranged in the helmet include at least one infrared camera and one RGB camera.

[0013] Furthermore, after the eyeball model is established, at least one infrared camera is used to continuously track the pupil center of the user, and at least one RGB camera is used to periodically obtain the pupil center of the user to verify the accuracy of the infrared camera.

[0014] According to a second aspect of the present disclosure, a sight tracking device based on 3D gesture and 6DoF positioning is provided, comprising: an optical tracking module, for acquiring, through a plurality of first cameras, a spot image of infrared light emitted by infrared lamps at a plurality of fixed positions in a cabin, the first camera being arranged on a head-mounted helmet, calculating the position of the infrared lamp based on the spot image, and obtaining first coordinate information of the helmet relative to the cabin by reverse calculation based on the position of the infrared lamp and the principle of multi-view geometry, the first coordinate information comprising a three-dimensional coordinate and a gesture, and the first camera being an infrared camera; an inertial tracking module, configured to obtain the 6DoF inertial information of the helmet in real time, and when the first coordinate information of the helmet cannot be obtained based on the light spot image, combine the most recently obtained first coordinate information with the corresponding 6DoF inertial information to calculate the latest second coordinate information of the helmet relative to the cabin; An eye tracking module, configured to capture the user's eyes using a second camera in the helmet based on eye tracking, and calculate third coordinate information of the user's eye sight line relative to the helmet; A calculation module is used to associate the third coordinate information with the first coordinate information or the third coordinate information with the second coordinate information to obtain fourth coordinate information of the user's eye line of sight relative to the cabin.

[0015] The technical solution disclosed in this disclosure has the following beneficial effects: Combining the helmet's 6DoF inertial information with optical tracking allows the helmet to deduce its current position and posture through 6DoF inertial information even when it cannot be identified by the first camera, ensuring that the tracking process of the helmet will not be interrupted, improving the operating experience and tracking accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of a sight tracking method based on 3D posture and 6DoF positioning in an embodiment of this specification; Figure 2 This is a structural block diagram of a gaze tracking device based on 3D posture and 6DoF positioning in an embodiment of this specification. DETAILED DESCRIPTION

[0017] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0018] In addition, the accompanying drawings are only schematic illustrations of the present disclosure. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0019] like Figure 1 As shown, the embodiment of this specification provides a gaze tracking method based on 3D posture and 6DoF positioning, and the execution subject of the method can be a computer, a server, a wearable smart device, etc. The method can specifically include the following steps S101~S104: In step S101, a plurality of first cameras are used to obtain a spot image of infrared light emitted by infrared lamps at a plurality of fixed positions in a cabin. The first camera is disposed on a head-mounted helmet. The position of the infrared lamp is calculated based on the spot image. Based on the position of the infrared lamp and the principle of multi-view geometry, a reverse calculation is performed to obtain first coordinate information of the helmet relative to the cabin. The first coordinate information includes a three-dimensional coordinate and a posture. The first camera is an infrared camera.

[0020] Specifically, when the first coordinate information of the helmet is obtained by reverse calculation, it specifically includes: based on the image processing algorithm, extracting the pixel coordinates of the light spot from the light spot image; matching the pixel coordinates captured by multiple first cameras to identify the projection positions of the same infrared lamp in different first cameras; based on the multi-view geometry principle, triangulating and calculating the three-dimensional coordinates of the infrared lamp relative to the first camera through the matched pixel coordinates of the light spot and the relative position of the first camera; based on the three-dimensional coordinates of the infrared lamp relative to the first camera and the three-dimensional coordinates relative to the cabin, as well as based on the positions of different first cameras on the helmet, calculate the first coordinate information.

[0021] Among them, the first camera is an infrared camera, and the image processing algorithm may include threshold segmentation, morphological operations and edge detection. Threshold segmentation can be achieved by setting a grayscale threshold to binarize the image, so that the light spot part becomes white and the background becomes black. Morphological operations can be open operations and closed operations to remove noise in the image and retain the light spot area. Edge detection can be to perform contour detection on the processed image, find the center position of the light spot, and record its pixel coordinates. In the images captured by multiple first cameras (infrared cameras), the projection positions of the light spot images are different under different viewing angles. In order to calculate the three-dimensional coordinates of the infrared lamp in space, it is necessary to identify the corresponding light spots of the same infrared lamp in different camera images. The specific process includes feature matching, that is, using the shape, size and brightness features of the light spot to match the same light spot in different camera images, which can be expressed as: ;

[0022] The multi-view geometry principle includes epipolar geometry constraints, which further verify the accuracy of matching through epipolar geometry. For two cameras, the corresponding point of any point must be on the epipolar line of the other camera, expressed as: ; in, are homogeneous coordinates on the image plane and F is the fundamental matrix.

[0023] Triangulation is the process of calculating the three-dimensional coordinates of the infrared light relative to each camera by matching the pixel coordinates of the light spot in different cameras and using multi-view geometry. That is, the three-dimensional coordinates of the object are inferred from the two-dimensional image plane projection from the perspective of two or more cameras.

[0024] In step S102, the 6DoF inertial information of the helmet is acquired in real time. When the first coordinate information of the helmet cannot be obtained based on the light spot image, the most recently acquired first coordinate information is combined with the corresponding 6DoF inertial information to calculate the latest second coordinate information of the helmet relative to the cabin.

[0025] Among them, in the helmet positioning and tracking system, in order to ensure the continuity and accuracy of the user's helmet position and posture (i.e. 6DoF - six degrees of freedom, including position and posture angle in three-dimensional space) information under different environmental conditions, the system not only relies on the infrared spot data captured by the camera, but also combines the real-time data of the inertial measurement unit (IMU). Specifically, the inertial measurement unit (IMU) usually integrates accelerometers, gyroscopes and sometimes magnetometers, which can provide the acceleration, angular velocity and direction change information of the helmet in real time. These data can be used to derive the instantaneous position and posture of the helmet through integration and filtering. Accelerometer: Measures the linear acceleration of the helmet in space. Gyroscope: Measures the angular velocity of the helmet around three axes. Magnetometer: Provides a direction reference, usually used to compensate for errors caused by gyroscope drift. Due to ambient light interference, poor infrared light reflection or other external factors, the camera may sometimes fail to reliably capture the infrared spot image, and the system cannot calculate the first coordinate information of the helmet (i.e., accurate spatial positioning data based on the spot) through the spot. In this case, we fall back to the first coordinate information obtained by the most recent spot calculation (i.e., the accurate position and posture of the helmet successfully captured last time), and combine it with the 6DoF inertial information currently acquired in real time to infer the current position and posture of the helmet (second coordinate information). The last first coordinate information is fused with the acceleration and angular velocity data provided by the IMU, and algorithms such as Kalman filter or extended Kalman filter (EKF) are used to predict the current position and posture of the helmet. IMU data can provide high-frequency posture and position updates, thereby compensating for the impact of missing optical data in a short time. Kalman filter can be applied, which is a recursive algorithm that can predict the current state based on the system model and correct the prediction error based on the sensor data (IMU information). Specific process: In the prediction stage, the next position and posture of the helmet are predicted based on the real-time data of the IMU; in the update stage, once the spot data is restored, the system will use the latest optical information to correct the helmet position and posture predicted by the IMU. Finally, the latest helmet position and attitude are calculated: the second coordinate information is calculated. By fusing and processing the IMU data with the most recent first coordinate information, the system can estimate the latest position and attitude of the helmet. The result is the second coordinate information; position estimation. The displacement information of the helmet is derived through double integration of the IMU data; attitude estimation. The rotation angle change of the helmet is derived through the gyroscope data of the IMU, and the current attitude is calculated in combination with the previous attitude information; error compensation. In order to reduce the drift and cumulative errors inherent in inertial sensors, the system may adopt error correction algorithms, such as real-time correction in communication with the ground station, vision-based error feedback and other methods.

[0026] In step S103, based on eye tracking, the user's eyes are photographed by using the second camera in the helmet, and third coordinate information of the user's eye sight line relative to the helmet is calculated.

[0027] Among them, when calculating the third coordinate information of the user's eye sight relative to the helmet, it specifically includes: based on at least two second cameras continuously capturing the pupil image of the same eye of the user, based on the deep learning algorithm, calculating the bounding box of the pupil to detect the pupil center; based on the multi-view geometry principle and the internal and external parameters of different cameras, projecting different second cameras to their corresponding pupil centers into three-dimensional space to obtain rays, using the intersection point or the closest point of different rays as the three-dimensional position of the pupil center, fitting multiple three-dimensional positions of the pupil center, and combining the empirical constant of the eyeball radius to obtain the user's eyeball model; continuously tracking the user's pupil center, using the vector line from the eyeball model to the pupil center as the user's eye sight, and according to the coordinate relationship between the second camera and the first camera, obtaining the third coordinate information of the eye sight relative to the first camera.

[0028] Among them, the second camera includes at least one infrared camera and one RGB camera. After the eyeball model is established, at least one infrared camera is used to continuously track the pupil center of the user, and at least one RGB camera is used to periodically obtain the pupil center of the user to verify the accuracy of the infrared camera.

[0029] In step S104, the third coordinate information is associated with the first coordinate information or the third coordinate information is associated with the second coordinate information to obtain fourth coordinate information of the user's eye sight relative to the cabin.

[0030] In one embodiment, after calculating the latest second coordinate information of the helmet relative to the cabin, it also includes: calculating the position error between the first coordinate information and the second coordinate information based on the first coordinate information of the helmet obtained in real time according to the light spot image; based on a fuzzy prediction algorithm, taking the 6DoF inertial information as input and the position error as output, training and constructing a fuzzy learning model; when the first coordinate information of the helmet cannot be obtained based on the light spot image, using the fuzzy learning model to predict the position error; based on the 6DoF inertial information and the first coordinate information, deriving the second coordinate information, and compensating the predicted position error to the second coordinate information.

[0031] Among them, when training and constructing the fuzzy learning model, it includes: defining the structure of the fuzzy learning model, the structure of the fuzzy learning model includes n fuzzy subsystems and m enhanced nodes; defining multiple fuzzy rules for each of the fuzzy subsystems, the fuzzy rules are used to describe the relationship between the 6DoF inertial information and the position error; defining a Gaussian membership function for each fuzzy set, and determining its center and width based on the K-means clustering algorithm; calculating the activation strength of each of the fuzzy rules as the product of the Gaussian membership function of the 6DoF inertial information; based on the activation strength and the parameters of the fuzzy subsystem, constructing the output function of each fuzzy subsystem; organizing the 6DoF inertial information into a training data set, representing it in a matrix, training and adjusting the parameters of the fuzzy subsystem, and obtaining the fuzzy learning model.

[0032] Based on the same idea, Figure 2 As shown, an exemplary embodiment of the present disclosure further provides a gaze tracking system based on 3D gesture and 6DoF positioning, comprising: The optical tracking module 201 is used to obtain, through a plurality of first cameras, a spot image of infrared light emitted by infrared lamps at a plurality of fixed positions in the cabin, wherein the first camera is arranged on a head-mounted helmet, and the position of the infrared lamp is calculated based on the spot image, and the first coordinate information of the helmet relative to the cabin is obtained by reverse calculation based on the position of the infrared lamp and the principle of multi-view geometry, wherein the first coordinate information includes a three-dimensional coordinate and a posture, and the first camera is an infrared camera; an inertial tracking module 202, configured to obtain the 6DoF inertial information of the helmet in real time, and when the first coordinate information of the helmet cannot be obtained based on the light spot image, combine the most recently obtained first coordinate information with the corresponding 6DoF inertial information to calculate the latest second coordinate information of the helmet relative to the cabin; The eye tracking module 203 is used to use the second camera in the helmet to shoot the user's eyes based on eye tracking, and calculate the third coordinate information of the user's eye sight relative to the helmet; The calculation module 204 is used to associate the third coordinate information with the first coordinate information or the third coordinate information with the second coordinate information to obtain fourth coordinate information of the user's eye sight relative to the cabin.

[0033] The above system combines the helmet's 6DoF inertial information with optical tracking, so that when the helmet cannot be identified by the first camera, it can also deduce the current position and posture of the helmet through the 6DoF inertial information, ensuring that the tracking process of the helmet will not be interrupted, thereby improving the operating experience and tracking accuracy.

[0034] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the exemplary implementation of the present disclosure.

[0035] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiment of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0036] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0037] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A gaze tracking method based on 3D gesture and 6DoF positioning, characterized in that: The method comprises: A plurality of first cameras are used to obtain a spot image of infrared light emitted by infrared lamps at a plurality of fixed positions in the cabin. The first camera is arranged on a head-mounted helmet. The position of the infrared lamp is calculated based on the spot image. Based on the position of the infrared lamp and the principle of multi-view geometry, a reverse calculation is performed to obtain first coordinate information of the helmet relative to the cabin. The first coordinate information includes a three-dimensional coordinate and a posture. The first camera is an infrared camera. acquiring 6DoF inertial information of the helmet in real time, and when the first coordinate information of the helmet cannot be obtained based on the light spot image, combining the most recently obtained first coordinate information with the corresponding 6DoF inertial information to calculate the latest second coordinate information of the helmet relative to the cabin; Based on eye tracking, a second camera in the helmet is used to photograph the user's eyes, and third coordinate information of the user's eye sight relative to the helmet is calculated; The third coordinate information is associated with the first coordinate information or the third coordinate information is associated with the second coordinate information to obtain fourth coordinate information of the user's eye sight relative to the cabin.

2. The sight tracking method based on 3D posture and 6DoF positioning according to claim 1, characterized in that: When the first coordinate information of the helmet is obtained by reverse calculation, the method specifically includes: Based on an image processing algorithm, the pixel coordinates of the light spot are extracted from the light spot image; Matching the pixel coordinates captured by the plurality of the first cameras to identify the projection positions of the same infrared light in different first cameras; Based on the multi-view geometry principle, the three-dimensional coordinates of the infrared light relative to the first camera are calculated by performing triangulation through the pixel coordinates of the matched light spot and the relative position of the first camera; The first coordinate information is calculated based on the three-dimensional coordinates of the infrared light relative to the first camera and the three-dimensional coordinates relative to the cabin, and based on different positions of the first camera on the helmet.

3. The sight tracking method based on 3D posture and 6DoF positioning according to claim 1, characterized in that: After calculating and obtaining the latest second coordinate information of the helmet relative to the cabin, the method further includes: Calculating a position error between the first coordinate information and the second coordinate information in real time according to the first coordinate information of the helmet obtained from the light spot image; Based on the fuzzy prediction algorithm, the 6DoF inertial information is used as input, the position error is used as output, and a fuzzy learning model is trained and constructed; When the first coordinate information of the helmet cannot be obtained based on the light spot image, using the fuzzy learning model to predict the position error; Based on the 6DoF inertial information and the first coordinate information, the second coordinate information is derived, and the predicted position error is compensated to the second coordinate information.

4. The sight tracking method based on 3D posture and 6DoF positioning according to claim 3, characterized in that: When training and constructing the fuzzy learning model, it includes: Defining a structure of the fuzzy learning model, wherein the structure of the fuzzy learning model includes n fuzzy subsystems and m enhancement nodes; defining a plurality of fuzzy rules for each of the fuzzy subsystems, wherein the fuzzy rules are used to describe the relationship between the 6DoF inertial information and the position error; A Gaussian membership function is defined for each fuzzy set, and its center and width are determined based on the K-means clustering algorithm; calculating the activation strength of each of the fuzzy rules as a product of the Gaussian membership function of the 6DoF inertial information; Based on the activation strength and the parameters of the fuzzy subsystem, construct an output function of each of the fuzzy subsystems; The 6DoF inertial information is organized into a training data set and represented by a matrix, and the parameters of the fuzzy subsystem are trained and adjusted to obtain the fuzzy learning model.

5. The sight tracking method based on 3D posture and 6DoF positioning according to claim 1, characterized in that: When calculating the third coordinate information of the user's eye sight relative to the helmet, it specifically includes: Based on at least two second cameras continuously capturing pupil images of the same eye of the user, a bounding box of the pupil is calculated based on a deep learning algorithm to detect the center of the pupil; Based on the principle of multi-view geometry and the internal and external parameters of different cameras, different second cameras are projected to the corresponding pupil center into the three-dimensional space to obtain rays, and the intersection point or the closest point of different rays is used as the three-dimensional position of the pupil center. Multiple three-dimensional positions of the pupil center are fitted, and the user's eyeball model is obtained by combining the empirical constant of the eyeball radius; The pupil center of the user is continuously tracked, and the vector line from the eyeball model to the pupil center is used as the eye sight line of the user, and the third coordinate information of the eye sight line relative to the first camera is obtained according to the coordinate relationship between the second camera and the first camera.

6. The sight tracking method based on 3D gesture and 6DoF positioning according to claim 5, characterized in that: The multiple second cameras arranged in the helmet include at least one infrared camera and one RGB camera.

7. The sight tracking method based on 3D gesture and 6DoF positioning according to claim 6, characterized in that: After the eyeball model is established, at least one infrared camera is used to continuously track the pupil center of the user, and at least one RGB camera is used to periodically obtain the pupil center of the user to verify the accuracy of the infrared camera.

8. A gaze tracking system based on 3D gesture and 6DoF positioning, characterized in that: include: an optical tracking module, for acquiring, through a plurality of first cameras, a spot image of infrared light emitted by infrared lamps at a plurality of fixed positions in the cabin, wherein the first camera is arranged on a head-mounted helmet, and the position of the infrared lamp is calculated based on the spot image, and based on the position of the infrared lamp and the principle of multi-view geometry, first coordinate information of the helmet relative to the cabin is obtained by reverse calculation, wherein the first coordinate information includes a three-dimensional coordinate and a posture, and the first camera is an infrared camera; an inertial tracking module, configured to obtain the 6DoF inertial information of the helmet in real time, and when the first coordinate information of the helmet cannot be obtained based on the light spot image, combine the most recently obtained first coordinate information with the corresponding 6DoF inertial information to calculate the latest second coordinate information of the helmet relative to the cabin; An eye tracking module, configured to capture the user's eyes using a second camera in the helmet based on eye tracking, and calculate third coordinate information of the user's eye sight line relative to the helmet; A calculation module is used to associate the third coordinate information with the first coordinate information or the third coordinate information with the second coordinate information to obtain fourth coordinate information of the user's eye line of sight relative to the cabin.

Images