A method and system for assisted understanding based on line-of-sight estimation

Abstract

The application discloses a kind of based on line-of-sight estimation auxiliary understanding method and system, comprising: through the near-eye infrared camera of AR / VR / MR equipment, the eye image of the user wearing AR / VR / MR equipment is collected;The three-dimensional line-of-sight vector is determined by processing eye image;The three-dimensional line-of-sight vector of the left and right eyes of user is mapped to virtual screen as the fixation point of user;The fixation time of fixation point is acquired, when fixation point and fixation time satisfy preset condition, the category of target that user fixates is determined according to the scene picture of user fixation and pre-trained DETR algorithm and control AR / VR / MR equipment executes and displays the category of the target, category includes object shape, color and mutual position relationship between object.In AR / VR / MR and so on equipment, object in scene image is described, help baby, old people and adult who is mentally impaired due to disease carry out man-machine interaction through eye movement, auxiliary understanding person or thing in scene image.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, gaze estimation, and eye-tracking interaction technology, and relates to an auxiliary understanding method and system based on gaze estimation. Background Technology

[0002] In recent years, gaze estimation has been widely applied in fields such as human-computer interaction, virtual reality (VR), augmented reality (AR), mixed reality (MR), medical diagnosis, and psychoanalysis. It analyzes and judges the current user's attention based on gaze point estimation. Meanwhile, object detection, which finds objects of interest to the user in an image, is a multi-classification task. However, unlike other classification tasks, this task requires determining the size and position of each object. Therefore, object detection has wide applications in autonomous driving, human-computer interaction, object tracking, and instance segmentation.

[0003] Furthermore, most current gaze estimation methods are appearance-based, feeding binocular eye images into a network model for calculation and ultimately returning the estimated gaze point location. This method is simple but lacks accuracy and requires restriction of head movement. Existing techniques combining gaze estimation and object detection only perform a simple fusion; when the gaze falls on a target object, only the name and confidence score of the detected object are displayed, without further description of objects of interest to the user, such as shape, color, or relative positions between objects. Current technologies typically use gaze estimation to select targets and interact manually, but this is too restrictive for infants, the elderly, and people with upper limb disabilities. Therefore, integrating eye-tracking interaction technology can overcome this limitation. To address these issues, gaze estimation, object detection, and eye-tracking interaction technologies are integrated for use in AR / VR / MR devices to provide text descriptions of objects in scene images. Summary of the Invention

[0004] The purpose of this invention is to address the aforementioned shortcomings in existing technologies, namely that most existing gaze estimation methods are based on appearance, which involves feeding binocular eye images into a network model, which then calculates and returns the estimated gaze point location. Existing methods are relatively simple but have low accuracy and require restriction of head movement. To address these shortcomings, this invention proposes an auxiliary understanding method and system based on gaze estimation.

[0005] To achieve the above objectives, the present invention adopts the following technical solution.

[0006] On the one hand, the present invention provides an auxiliary understanding method based on gaze estimation, comprising:

[0007] Images of the eyes of users wearing AR / VR / MR devices are captured using a near-eye infrared camera.

[0008] The eye image is processed to determine the three-dimensional gaze vector;

[0009] The three-dimensional gaze vectors of the user's left and right eyes are mapped onto the virtual screen as the user's gaze points;

[0010] The gaze duration of the gaze point is obtained. When the gaze point and the gaze duration meet preset conditions, the category information corresponding to the target being gazed at is determined based on the scene image being gazed at by the user and the pre-trained DETR algorithm. The AR / VR / MR device is then controlled to perform the operation of displaying the category information corresponding to the target. The category information includes the object shape, color, and the relative positions of the objects.

[0011] In some embodiments, processing the eye image to determine a three-dimensional gaze vector includes:

[0012] The eye image is input into the RITnet network segmentation model to segment the eye image and obtain segmentation maps of the edge information of the four parts: background, sclera, iris, and pupil.

[0013] The obtained segmentation map is fitted with an edge detector and the least squares method to obtain the pupil ellipse and determine the parameter information of the pupil ellipse;

[0014] Determine the camera parameters, and then use these parameters to establish the camera coordinate system in three-dimensional space.

[0015] In the camera coordinate system, a 3D eye model is fitted based on the pupil ellipse of multiple eye images of the user. The fitted pupil ellipse is then triangulated to obtain a pupil circle tangent to the 3D eye model. The parameters of the 3D eye model are determined based on multiple eye images, the pupil circle information corresponding to each eye image, and the least squares method. The three-dimensional gaze vector is also determined.

[0016] In some embodiments, the obtained segmentation map is fitted to the pupil ellipse using an edge detector and least squares method, and the parameter information of the pupil ellipse is determined, including:

[0017] The segmentation map, consisting of the background, sclera, iris, and pupil, obtained through the RITnet network segmentation model, is used to extract the pupil boundary after thresholding using an edge detector. The boundary information is then fitted using the least squares method to obtain the parameter information of the pupil ellipse. The pupil ellipse is then drawn, and the mathematical expression of its general equation is determined as follows:

[0018] G(x, y) = Ax2 +Bxy+Cy 2 +Dx+Ey+F

[0019]

[0020]

[0021] Where: the parameters of the general equation for an elliptical pupil are (A, B, C, D, E, F); the center of the pupil ellipse is (X... e Y e ).

[0022] In some embodiments, in a camera coordinate system, a 3D eyeball model is fitted based on the pupil ellipse of multiple eye images of the user, and the fitted pupil ellipse is triangulated to obtain a pupil circle tangent to the 3D eyeball model, including:

[0023] Using triangulation geometry, a three-dimensional cone is constructed with the camera as the focal point and the camera as the vertex. The pupil ellipse segmented from the eye image is the intersection line between the eye image and the cone.

[0024] In a fully perspective projection model, the pupil ellipse is back-projected onto a 3D eyeball model, ensuring that the back-projected pupil circle is tangent to the 3D eyeball model. The pupil circle projected onto the surface of the 3D eyeball model is considered as a cross-section of a cone with the camera focus as its vertex, and the boundary of the pupil ellipse in the eye image is the intersection point of the two-dimensional plane intersecting the cone. Once the pupil circle is determined, the pupil circle information is obtained:

[0025] Pupile circle = (p, m, r)

[0026] Where: p represents the three-dimensional pupil coordinates; m represents the normal vector of the pupil; r represents the radius of the pupil circle;

[0027] The process of determining 3D eyeball model parameters and three-dimensional gaze vectors based on multiple eye images, pupil circle information corresponding to each eye image, and least squares method includes:

[0028] Based on eye images of multiple subjects, the 3D eyeball model parameters for each person were obtained, including the center of the 3D eyeball model and the average radius of the sphere.

[0029] Based on the pupil circle information constructed from multiple eye images, the intersection point of the normal vectors of multiple pupil circles is determined as the center of the 3D eyeball model. Since all intersection points cannot be at the same point due to numerical measurement errors, they can be determined by the least squares method.

[0030] Once the center of the 3D eyeball model is determined, the radius of all eye images is calculated based on the pupil circle and the center of the 3D eyeball model. The final 3D eyeball radius R is obtained by averaging the radii of all eye images.

[0031] R = mean(R) i =||p i -c||)

[0032] Where: R i Let be the radius obtained by the distance between the pupil circle and the center of the sphere in each eye image, c represents the center of the sphere in the 3D eye model, and pi is the pupil coordinate of the i-th eye image;

[0033] Based on the pupil center obtained from the eye image and the parameters of the 3D eye model, the pupil center projected onto the 3D eye model is obtained, and the line connecting the center of the 3D eye model and the pupil center projected from each eye image is determined as the three-dimensional line of sight vector.

[0034] In some embodiments, mapping the three-dimensional gaze vectors of the user's left and right eyes onto a virtual screen as the user's gaze points includes:

[0035] Based on the fitted 3D eye model and three-dimensional gaze vector, when the user gazes at a known point S on the screen, the conversion relationship between the actual gaze point coordinates and the estimated gaze point coordinates can be determined through simple calibration:

[0036]

[0037] Where M is the rotation matrix; To estimate the 3D line-of-sight vector; This is the actual 3D view vector;

[0038] The 3D eyeball center O = (a, b, c) and the estimated 3D gaze vector have been calculated. The coordinates of the intersection point of the two-dimensional plane equation Ax + By + Cz + D = 0;

[0039] The parametric equation of the line is:

[0040] x = mt + a

[0041] y = nt + b

[0042] z = pt + c

[0043] Substituting into the plane coordinate equation, we get:

[0044] Substituting t back into the parametric equation of the line, we can obtain the estimated coordinates P(x,y,z) of the intersection point between the line of sight and the two-dimensional screen.

[0045] Due to the differences between the left and right eyes, there will be discrepancies in the estimated two-dimensional plane coordinates between the two eyes. Therefore, the final estimated two-dimensional plane fixation point coordinates are obtained according to the following formula:

[0046]

[0047] The rotation matrix can be obtained from a known point S on the screen. i (i = 1, 2, 3) and the three-dimensional gaze vector at the screen-predicted gaze point P i Solve the following equation (i = 1, 2, 3) obtained through the above transformation relationship:

[0048]

[0049]

[0050]

[0051] In some embodiments, before determining the category information corresponding to the target being gazed at by the user based on the scene image being gazed at and a pre-trained DETR algorithm, the method further includes:

[0052] Detect each target in the scene images in the public dataset, and annotate the target bounding regions of each target in the scene images;

[0053] Each scene image is fed into DETR's Transformer-based end-to-end object detection algorithm for training. After feature extraction, decoding, and prediction stages, each region of the scene image is divided into target-containing and non-target-containing categories. The target bounding box region with the target-containing category is then matched with the category information describing the target.

[0054] In some embodiments, the category information corresponding to the target being gazed at by the user is determined based on the scene image being gazed at and a pre-trained DETR algorithm, including:

[0055] The scene image that the user is looking at is used as input to the DETR algorithm;

[0056] Based on the target selection area where the user's gaze point falls, determine the category information that matches the target selection area;

[0057] Controlling the AR / VR / MR device to perform the operation of displaying the category information corresponding to the target includes:

[0058] The AR / VR / MR device is controlled to display the category information corresponding to the target in the center of the target selection area where the gaze point falls.

[0059] In some embodiments, the preset conditions are that the gaze point falls within any preset target selection area, the gaze time is greater than a first threshold, and the fluctuation of the gaze point within the first threshold does not exceed a preset gaze point fluctuation range.

[0060] In some embodiments, after controlling the AR / VR / MR device to perform the operation of displaying the category information corresponding to the target, the method further includes:

[0061] Get the user's eye-closed duration;

[0062] If the duration of eye closure exceeds the second threshold, it is determined that the user is voluntarily closing their eyes, and the AR / VR / MR device is controlled to stop displaying the category information corresponding to the target.

[0063] Secondly, the present invention also provides an auxiliary understanding system based on gaze estimation, comprising:

[0064] An eye image acquisition module is used to acquire eye images of users wearing AR / VR / MR devices via the near-eye infrared camera of the AR / VR / MR device;

[0065] A gaze estimation module is used to process the eye image and determine the three-dimensional gaze vector;

[0066] The gaze point mapping module is used to map the three-dimensional gaze vectors of the user's left and right eyes onto the virtual screen as the user's gaze point;

[0067] An eye-tracking interaction module is used to acquire the gaze duration of the gaze point. When the gaze point and the gaze duration meet preset conditions, the module determines the category information corresponding to the target being gazed at by the user based on the scene image being gazed at and the pre-trained DETR algorithm, and controls the AR / VR / MR device to perform the operation of displaying the category information corresponding to the target. The category information includes the object shape, color, and the relative positions of the objects.

[0068] In this embodiment of the invention, AR / VR / MR devices can be controlled to display category information corresponding to the target. A series of operations, such as switching and returning to the scene image and generating text boxes, are achieved through eye-tracking interaction. By integrating gaze estimation, target detection, and eye-tracking interaction technologies, this system assists in understanding objects in scene images observed on mobile devices through eye-tracking interaction. It is used to describe objects within scene images on AR / VR / MR devices and can help infants, the elderly, and adults with intellectual disabilities due to illness to interact with computers through eye movements, assisting in understanding people or objects in scene images. When using this system, when the gaze point falls on a target object of interest to the user, not only is the name and confidence level of the detected object displayed, but further descriptions of the object of interest are also provided, such as the object's shape, color, and the relative positions of objects. Attached Figure Description

[0069] Figure 1 A flowchart illustrating the line-of-sight estimation-based auxiliary understanding method provided in an embodiment of the present invention;

[0070] Figure 2 Provided for embodiments of the present invention Figure 1 Flowchart of step S102;

[0071] Figure 3 This is a schematic diagram of the fitted eyeball model for a single eye image in an auxiliary understanding method based on gaze estimation according to the present invention.

[0072] Figure 4 A flowchart of the DETR algorithm in an auxiliary understanding method based on gaze estimation provided in an embodiment of the present invention;

[0073] Figure 5 A flowchart for generating and canceling visual text boxes in text aided by gaze estimation, provided as an embodiment of the present invention;

[0074] Figure 6 The diagram illustrates the expected effect of a line-of-sight estimation-based auxiliary understanding method as provided in an embodiment of the present invention.

[0075] Figure 7 The diagram shows the composition and connection of the line-of-sight estimation-based auxiliary understanding system provided in the embodiments of the present invention. Detailed Implementation

[0076] Exemplary embodiments will be described more fully below with reference to the accompanying drawings; however, these exemplary embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will enable those skilled in the art to fully understand the scope of this disclosure.

[0077] As used herein, the term “and / or” includes any and all combinations of one or more relevant enumeration rules.

[0078] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the said feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded.

[0079] The embodiments described herein can be described with reference to plan views and / or cross-sectional views using the ideal schematic diagrams of this disclosure. Therefore, the example illustrations can be modified according to manufacturing techniques and / or tolerances. Therefore, the embodiments are not limited to those shown in the drawings, but include modifications to configurations formed based on manufacturing processes. Therefore, the areas illustrated in the drawings are schematic in nature, and the shapes of the areas shown in the figures illustrate specific shapes of areas of an element, but are not intended to be limiting.

[0080] To describe objects within scene images on AR / VR / MR devices, and to assist infants, the elderly, and adults with intellectual disabilities due to illness in understanding people or objects in scene images, this disclosure provides a gaze-based assisted understanding method and system. The following detailed description, in conjunction with the accompanying drawings of the embodiments provided by this invention, will illustrate these methods.

[0081] Example 1

[0082] Firstly, such as Figure 1 As shown, this embodiment of the invention provides an auxiliary understanding method based on gaze estimation, including:

[0083] Step S101: Collect data from the near-eye infrared camera of the AR / VR / MR device on the user wearing the AR / VR / MR device.

[0084] Eye image;

[0085] Step S102: Process the eye image to determine the three-dimensional gaze vector;

[0086] Step S103: Map the three-dimensional gaze vectors of the user's left and right eyes onto the virtual screen as the user's gaze points;

[0087] Step S104: Obtain the gaze duration of the gaze point. When the gaze point and the gaze duration meet preset conditions, determine the category information corresponding to the target being gazed at by the user based on the scene image being gazed at and the pre-trained DETR algorithm, and control the AR / VR / MR device to perform the operation of displaying the category information corresponding to the target.

[0088] The category information includes the object's shape, color, and the relative positions of the objects.

[0089] It should be noted that the eye images are acquired using a near-eye infrared camera in an AR / VR / MR device, and the eye image acquisition module can be configured within the AR / VR / MR device. First, each location to be captured is visualized on a virtual screen within the visualization interface. Second, the user captures K eye images at each visualized location by gazing at it, simultaneously generating the ground truth coordinates of that location for subsequent calibration. Finally, the hardware parameters of the near-eye infrared camera, such as focal length, frame rate, and sensor size, need to be determined to facilitate the construction of a spatial coordinate system when fitting the eyeball model. Step S101 is implemented by the eye image acquisition module.

[0090] It should be noted that the range of K in this invention is 10-50.

[0091] In a preferred embodiment, the category information corresponding to the target is displayed via a text box. The visualized gaze point and scene image are input into the virtual screen of the AR / VR / MR device via a display interface module. A text box describing the text information is generated by comparing the visualized gaze point estimated by the gaze with the information after object detection.

[0092] First, there's the gaze point visualization: the 3D gaze vector of the gaze point is mapped to obtain the estimated 2D gaze point location, and a visualization point is drawn centered on this point in the mobile deployment application. Second, there's the scene image visualization: each time a user uses the app, a certain number of images can be randomly selected for text descriptions of targets in the scene images. Third, there's the text box generation visualization: after the gaze point and the target selection area are matched, eye-tracking behavior determines whether to generate a text box. The text box generation is implemented by inputting the marked object information into the text box according to the mobile deployment configuration. Finally, there's the button configuration: when it's necessary to change to the next or previous image, the scene image can be switched by looking at the button at the top of the virtual screen.

[0093] It is understandable that the display of category information corresponding to the target is not limited to a text box; for example, voice prompts may be used. This is not a limitation here.

[0094] In this embodiment of the invention, AR / VR / MR devices can be controlled to display category information corresponding to the target. A series of operations, such as switching and returning to the scene image and generating text boxes, are achieved through eye-tracking interaction. By integrating gaze estimation, target detection, and eye-tracking interaction technologies, this system assists in understanding objects in scene images observed on mobile devices through eye-tracking interaction. It can be used in AR / VR / MR devices to describe objects within scene images, helping infants, the elderly, and adults with intellectual disabilities to interact with computers through eye movements and understand people or objects in scene images. When using this system, when the gaze point falls on a target object of interest to the user, not only is the name and confidence level of the detected object displayed, but further descriptions of the object, such as its shape, color, and the relative positions of objects, are also provided.

[0095] In some embodiments, such as Figure 2 As shown, the step of processing the eye image to determine the three-dimensional gaze vector (i.e., step S102) includes:

[0096] The eye image is input into the RITnet network segmentation model to segment the eye image and obtain segmentation maps of the edge information of the four parts: background, sclera, iris, and pupil.

[0097] The obtained segmentation map is fitted with an edge detector and the least squares method to obtain the pupil ellipse and determine the parameter information of the pupil ellipse;

[0098] Determine the camera parameters, and then use these parameters to establish the camera coordinate system in three-dimensional space.

[0099] In the camera coordinate system, a 3D eye model is fitted based on the pupil ellipse of multiple eye images of the user. The fitted pupil ellipse is then triangulated to obtain a pupil circle tangent to the 3D eye model. The parameters of the 3D eye model are determined based on multiple eye images, the pupil circle information corresponding to each eye image, and the least squares method. The three-dimensional gaze vector is also determined.

[0100] In this embodiment of the invention, a three-dimensional cone with the camera as the vertex is constructed using triangulation geometry, with the camera as the focal point. The pupil ellipse segmented from the eye image is the intersection of this plane and the cone. Multiple acquired eye images are then fitted to create an eyeball model, obtaining the center coordinates and radius information. Finally, the gaze vector is visualized by calculating the pitch angle on each eye image, and the three-dimensional gaze vector is output.

[0101] The RITnet network segmentation model consists of five downsampling modules and four upsampling modules. Each downsampling module contains five convolutional layers and LeakyReLU activation functions, with pooling layers consisting of 2×2 convolutional kernels. Each upsampling module contains four convolutional layers and LeakyReLU activation functions. Each sampling channel has 32 channels to reduce parameters, and all convolutional layers share connections from previous convolutional layers.

[0102] In some embodiments, the obtained segmentation map is fitted to the pupil ellipse using an edge detector and least squares method, and the parameter information of the pupil ellipse is determined, including:

[0103] The segmentation map, consisting of the background, sclera, iris, and pupil, obtained through the RITnet network segmentation model, is used to extract the pupil boundary after thresholding using an edge detector. The boundary information is then fitted using the least squares method to obtain the parameter information of the pupil ellipse. The pupil ellipse is then drawn, and the mathematical expression of its general equation is determined as follows:

[0104] G(x, y) = Ax 2 +Bxy+Cy 2 +Dx+Ey+F

[0105]

[0106]

[0107] Where: the parameters of the general equation for an elliptical pupil are (A, B, C, D, E, F); the center of the pupil ellipse is (X... e Y e ).

[0108] The above steps achieve pupil ellipse fitting.

[0109] In some embodiments, in a camera coordinate system, a 3D eyeball model is fitted based on the pupil ellipse of multiple eye images of the user, and the fitted pupil ellipse is triangulated to obtain a pupil circle tangent to the 3D eyeball model, including:

[0110] Using triangulation geometry, a three-dimensional cone is constructed with the camera as the focal point and the camera as the vertex. The pupil ellipse segmented from the eye image is the intersection line between the eye image and the cone.

[0111] In a fully perspective projection model, the pupil ellipse is back-projected onto a 3D eyeball model, ensuring that the back-projected pupil circle is tangent to the 3D eyeball model. The pupil circle projected onto the surface of the 3D eyeball model is considered as a cross-section of a cone with the camera focus as its vertex, and the boundary of the pupil ellipse in the eye image is the intersection point of the two-dimensional plane intersecting the cone (e.g., ...). Figure 3 (As shown); Once the pupil circle is determined, the pupil circle information is obtained:

[0112] Pupile circle = (p, m, r)

[0113] Where: p represents the three-dimensional pupil coordinates; m represents the normal vector of the pupil; r represents the radius of the pupil circle;

[0114] The process of determining 3D eyeball model parameters and three-dimensional gaze vectors based on multiple eye images, pupil circle information corresponding to each eye image, and least squares method includes:

[0115] Based on eye images of multiple subjects, the 3D eyeball model parameters for each person were obtained, including the center of the 3D eyeball model and the average radius of the sphere.

[0116] Based on the pupil circle information constructed from multiple eye images, the intersection point of the normal vectors of multiple pupil circles is determined as the center of the 3D eyeball model. Since all intersection points cannot be at the same point due to numerical measurement errors, they can be determined by the least squares method.

[0117] Once the center of the 3D eyeball model is determined, the radius of all eye images is calculated based on the pupil circle and the center of the 3D eyeball model. The final 3D eyeball radius R is obtained by averaging the radii of all eye images.

[0118] R = mean(R) i =||p i -c||)

[0119] Where: R i Let be the radius obtained by the distance between the pupil circle and the center of the sphere in each eye image, c represents the center of the sphere in the 3D eye model, and pi is the pupil coordinate of the i-th eye image;

[0120] Based on the pupil center obtained from the eye image and the parameters of the 3D eye model, the pupil center projected onto the 3D eye model is obtained, and the line connecting the center of the 3D eye model and the pupil center projected from each eye image is determined as the three-dimensional line of sight vector.

[0121] The above steps enable the generation and visualization of a three-dimensional view vector.

[0122] In some embodiments, the three-dimensional gaze vectors of the user's left and right eyes are mapped onto the virtual screen as the user's gaze points (i.e., step S103), including:

[0123] Based on the fitted 3D eye model and three-dimensional gaze vector, when the user gazes at a known point S on the screen, the conversion relationship between the actual gaze point coordinates and the estimated gaze point coordinates can be determined through simple calibration:

[0124]

[0125] Where M is the rotation matrix; To estimate the 3D line-of-sight vector; This is the actual 3D view vector;

[0126] The 3D eyeball center O = (a, b, c) and the estimated 3D gaze vector have been calculated. The coordinates of the intersection point of the two-dimensional plane equation Ax + By + Cz + D = 0;

[0127] The parametric equation of the line is:

[0128] x = mt + a

[0129] y = nt + b

[0130] z = pt + c

[0131] Substituting into the plane coordinate equation, we get:

[0132] Substituting t back into the parametric equation of the line, we can obtain the estimated coordinates P(x,y,z) of the intersection point between the line of sight and the two-dimensional screen.

[0133] Due to the differences between the left and right eyes, there will be discrepancies in the estimated two-dimensional plane coordinates between the two eyes. Therefore, the final estimated two-dimensional plane fixation point coordinates are obtained according to the following formula:

[0134]

[0135] The rotation matrix can be obtained from a known point S on the screen. i (i = 1, 2, 3) and the three-dimensional gaze vector at the screen-predicted gaze point P i Solve the following equation (i = 1, 2, 3) obtained through the above transformation relationship:

[0136]

[0137]

[0138]

[0139] The above steps establish the gaze point mapping model. In step S103, the three-dimensional gaze vector generated by the gaze estimation module is mapped onto the virtual screen as an estimation point. The accuracy error of the gaze estimation algorithm can be obtained by comparing the estimated coordinate point with the real coordinate point.

[0140] In some embodiments, before determining the category information corresponding to the target being gazed at by the user based on the scene image being gazed at and a pre-trained DETR algorithm, the method further includes:

[0141] Detect each target in the scene images in the public dataset, and annotate the target bounding regions of each target in the scene images;

[0142] Each scene image is fed into DETR's Transformer-based end-to-end object detection algorithm for training. After feature extraction, decoding, and prediction stages, each region of the scene image is divided into target-containing and non-target-containing categories. The target bounding box region with the target-containing category is then matched with the category information describing the target.

[0143] The category information includes the specific object shape, color, and relative positional relationships between objects. The object name, the location of the target selection area, and other information will be stored as tag information in a text file.

[0144] For object detection tasks, understanding the relationships between objects is crucial for improving detection performance. Therefore, this system employs the DETR algorithm. While traditional object detection algorithms utilize the relationships between objects, they don't effectively leverage attention mechanisms. This is primarily because the number, category, and size of each image are not identical. DETR uses a Transformer to model the relationships between different objects, incorporating relational information into the feature values, thus achieving true end-to-end object detection.

[0145] like Figure 4 As shown, the DETR algorithm consists of four steps: feature extraction, decoding, decryption, and prediction.

[0146] 1. Feature extraction stage: The traditional CNN backbone is used to extract features from the image as input, and then positional encoding commonly used in NLP is added to generate a batch of serialized data;

[0147] 2. Decoding stage: After flattening the data, the model sends the serialized data to the Transformer decoder in the decoding stage, and uses the attention mechanism to extract features from the data.

[0148] 3. Decoding stage: A small number of position embeddings learned by the Transformer decoder (which we call object queries) are used as input. That is, N randomly initialized vectors are input into the Transformer decoder. Each object query focuses on a different position in the image. Finally, the decoder will output N vectors.

[0149] 4. Prediction Stage: After the decoding stage, each vector corresponds to a detected target. Finally, these N vectors are input into the FFN neural network, which can predict the target category if there is a relative positional relationship or if there is no category.

[0150] In this embodiment of the invention, a target detection module is used to detect and label target objects in scene images from a publicly available dataset.

[0151] In some embodiments, step S104, determining the category information corresponding to the target being gazed at by the user based on the scene image being gazed at and the pre-trained DETR algorithm, includes:

[0152] The scene image that the user is looking at is used as input to the DETR algorithm;

[0153] Based on the target selection area where the user's gaze point falls, determine the category information that matches the target selection area;

[0154] Controlling the AR / VR / MR device to perform the operation of displaying the category information corresponding to the target includes:

[0155] The AR / VR / MR device is controlled to display the category information corresponding to the target in the center of the target selection area where the gaze point falls.

[0156] When displaying the category information of the target being gazed at by the user in the form of a text box, the display interface module mainly includes gaze point visualization, text box generation, etc. Figure 5 Create a flowchart for generating and canceling text boxes.

[0157] 1. Foveation visualization:

[0158] After establishing the above gaze point model, a gaze point corresponding to a two-dimensional plane is visualized as a solid red dot that follows the gaze through a visualization interface; that is, after determining the gaze point position, a solid circle with a radius of 5mm is drawn with the gaze point coordinates as the center. This visualized gaze point can be used not only for initial calibration but also for tracking the gaze to facilitate the user's selection of the target.

[0159] 2. To aid in understanding the generation of text boxes:

[0160] When the gaze point falls within a target bounding box and exceeds a first threshold of gaze duration, the target bounding box of the object will be highlighted and centered as shown in the attached image. Figure 6The pop-up text box shown is an illustration of the expected effect of displaying the category information of the target being viewed by the user in the form of a text box. This invention inputs a scene image; when the user's gaze falls on an object in the image, text information describing the object pops up in the center of the object through eye-tracking, thus providing a textual description of the target to help the user understand the scene image.

[0161] In some embodiments, the preset conditions are that the gaze point falls within any preset target selection area, the gaze time is greater than a first threshold, and the fluctuation of the gaze point within the first threshold does not exceed a preset gaze point fluctuation range.

[0162] Step S104 is implemented through the eye-tracking interaction module. This module uses eye-tracking signals as operation signals to facilitate user operation on areas of interest. It mainly includes two aspects: the first type of eye-tracking signal is fixation, that is, when the fixation point stays within a certain range for a certain period of time (greater than a set first threshold), the module determines that the user is currently fixating on that point and will execute the corresponding operation.

[0163] Specifically, due to the unique physiological response of the human eye, it will involuntarily jump and blink when observing an object. Therefore, it is necessary to set up an eye-tracking interaction module to judge the eye-tracking behavior in the current and future events.

[0164] When the human eye looks at an object, it will focus its gaze on a point and the surrounding area for a long time.

[0165] Based on this characteristic, a fixation threshold is set to determine whether the human eye is fixating on a certain object. The fixation threshold, also known as the first threshold, is determined to be an eye movement behavior of fixation when the fixation point stays on a point or in its designated nearby area for a period of time greater than t = 1500 ms.

[0166] Defining the gaze point fluctuation range: During prolonged fixation, if the gaze point coordinates fluctuate within a predetermined range, this fluctuation range is measured by a circle. Fixation is defined as the fluctuation not exceeding the radius of the measuring circle within a first threshold. The fixation model is shown in the following formula:

[0167] Gaze = (X g ,Y g ,r,t)

[0168] Where: (Xg, Yg) are the initial gaze point position coordinates; r represents the radius of the circular measurement region; t represents the gaze time threshold.

[0169] In some embodiments, after controlling the AR / VR / MR device to perform the operation of displaying the category information corresponding to the target, the method further includes:

[0170] Obtain the user's eye-closed duration;

[0171] If the eye-closed duration is greater than a second threshold, determine that the user closes their eyes voluntarily, and control the AR / VR / MR device to perform an operation of stopping displaying the category information corresponding to the target.

[0172] Step S104 is implemented through the eye movement interaction module. This module uses the eye movement signal as an operation signal to facilitate the user to operate on the area of interest. It also includes that the second type of eye movement signal is closing the eyes, which is divided into voluntary eye closure and involuntary eye closure. The main judgment basis is the eye-closed duration. When the eye-closed duration is less than the set second threshold, it is determined as involuntary eye closure. When the eye-closed duration is greater than the set second threshold, it is voluntary eye closure. This module performs different operations according to different judgment results.

[0173] Due to the unique physiological reaction of the human eye, there will be involuntary behaviors such as blinking when observing an object. Therefore, an eye movement interaction module for eye movement behaviors needs to be set to judge the eye movement behaviors within a certain period of time currently and in the future.

[0174] Prolonged use of the eyes will cause a dry feeling. Therefore, it is inevitable to blink involuntarily when using this system. Therefore, it is necessary to reduce the frequent change of the fixation point caused by blinking. Involuntary blinking usually has a very short eye-closed time. Therefore, only need to set a time threshold Td = 500ms (the second threshold) to determine whether the time between the current frame and the eye-closed image frame exceeds this time threshold; if T < Td, it indicates involuntary blinking and no operation is performed; if T > Td, it indicates voluntary eye closure, so a return operation will be performed on it; the eye-closed time calculation formula is as follows:

[0175] T = (F k -F n ) / frame

[0176] Where, T represents the blinking time; Fk and Fn respectively represent the number of frames with the eyes open at time k and the number of frames with the eyes closed at time n; frame represents the frame rate.

[0177] In the embodiment of the present invention, the visualization of the text box is mainly achieved by placing an oval dialog box above the scene image through the visualization interface of this system, and its text content needs to match the annotation information of the currently detected object, so as to describe the object in the scene picture; if you want to cancel the current dialog box, you only need to close your eyes voluntarily anywhere in the current image, that is, the eye-closed time is greater than the second threshold, the text box will be cancelled and the highlighted target box selection area will also disappear, and this system will be reset to the initial state (as shown in the appendix Figure 5 、 6 ). The present invention designs a visualization interface for the text interaction system, and realizes a series of operations such as switching, returning, and generating text boxes for the scene picture through eye movement interaction.

[0178] The following example illustrates the line-of-sight estimation-based auxiliary understanding method provided by this invention:

[0179] S1: Users need to wear the AR headset correctly first. There is an option button to view the eye image on the headset's display interface. Based on the eye image transmitted back by the real-time camera, users can adjust their current wearing posture to ensure that the pupil, iris, etc. are clearly visible in the eye image.

[0180] S2: After wearing the AR headset, use the near-eye camera of the AR headset to collect images of the wearer's eyes. While obtaining a large number of eye images of the user, calibrate each left and right eye image to obtain the real two-dimensional plane gaze point position information and output the information as a text file.

[0181] S3: The eye images of the left and right eyes obtained above are fed into the RITnet segmentation model in parallel, that is, the left and right eyes are simultaneously and separately input into the segmentation network model.

[0182] S4: Edge information of the background, sclera, iris, and pupil is obtained through the RITnet segmentation model;

[0183] S5: After obtaining the above segmentation map, determine the threshold of the pupil boundary pixels. That is, if the pixels in the eye image are higher than the threshold, it is determined that they do not belong to the pupil region.

[0184] S6: After the above threshold determination, the approximate area of ​​the pupil can be obtained. Then, the pupil of the eye image after threshold determination is separated by the Canny edge detector and the rough boundary of the pupil is determined. Finally, the elliptical pupil is fitted by the least squares method according to the rough boundary information, the parameters of the pupil ellipse are obtained, and the pupil ellipse is drawn.

[0185] S7: Determine parameters such as camera focal length and sensor size. The camera's focal length and sensor size are used to determine the camera coordinate system in three-dimensional space.

[0186] S8: Fit a 3D eyeball model based on the pupil ellipse of multiple eye images of each user, and obtain a pupil circle tangent to the 3D eyeball model by triangulation of the fitted pupil ellipse.

[0187] S9: Based on the fitting method of a single eye image, the intersection of multiple pupil circle normal vectors is determined as the center of the 3D eyeball model. The average distance between the center of the sphere and the center of each pupil circle tangent to the eyeball model is the radius of the center of the sphere.

[0188] S10: After estimating the center of the 3D eyeball model, the vector between the center of the pupil circle projected onto the surface of the eyeball model from each eye image and the center of the eyeball model is the three-dimensional gaze estimation vector.

[0189] S11: Perform the above steps S8-S10 in parallel for both the left and right eyes;

[0190] S12: Map the three-dimensional gaze estimation vectors of the left and right eyes onto the 2D plane according to the mapping relationship to obtain the gaze point coordinates;

[0191] S13: After completing the above calibration, a visual gaze point that tracks the real-time gaze will appear on the virtual two-dimensional plane;

[0192] S14: Based on the image and text information of the target detection task in the publicly available COCO dataset, select targets and describe them in text, including the shape and color attributes of the target object and its relative relationship with surrounding objects.

[0193] S15: Select the scene image that the user is looking at as the input of the object detection algorithm, and mark the target bounding area of ​​each object in the scene image;

[0194] S16: Match the location coordinates of the target bounding area with the text information describing the target using a target detection algorithm;

[0195] S17: Based on the threshold for determining the fixation time and range, if the user fixates on a point for a long time within the first threshold or fluctuates within the fixation point band range, it is determined that the current fixation point has not changed, and fixation eye movement operation is performed.

[0196] S18: The gaze eye movement behavior includes, when the gaze time exceeds a set first threshold, if the gaze point is within the target selection area, text information describing the target being gazed at will pop up in the center of the target selection area; if the gaze point is not within the target selection area, no operation will be performed.

[0197] S19: Cancel text box operation. When a user needs to cancel the currently displayed text box, the user must close their eyes for more than the second threshold time. The system will determine that the user has voluntarily closed their eyes, cancel the current text box, and return the highlighted target selection area to its initial state.

[0198] Secondly, such as Figure 7 As shown, embodiments of the present invention also provide an auxiliary understanding system based on gaze estimation, comprising:

[0199] The eye image acquisition module 11 is used to acquire eye images of users wearing AR / VR / MR devices through the near-eye infrared camera of the AR / VR / MR device;

[0200] The gaze estimation module 12 is used to process the eye image to determine the three-dimensional gaze vector;

[0201] The gaze point mapping module 13 is used to map the three-dimensional gaze vectors of the user's left and right eyes onto the virtual screen as the user's gaze point;

[0202] The eye-tracking interaction module 14 is used to obtain the gaze time of the gaze point. When the gaze point and the gaze time meet the preset conditions, it determines the category information corresponding to the target being gazed at by the user based on the scene image being gazed at and the pre-trained DETR algorithm, and controls the AR / VR / MR device to perform the operation of displaying the category information corresponding to the target. The category information includes the object shape, color and the relative positional relationship between objects.

[0203] like Figure 7 As shown, it also includes a display interface module and an object detection module. The display interface module inputs the visualized gaze point and scene image into the virtual screen of the AR / VR / MR device. It compares the visualized gaze point with the object detection information to generate a text box describing the target being gazed at by the user, and then displays the category information of the target in the form of a text box. The object detection module detects and labels target objects in the scene images from the public dataset.

[0204] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of the present invention, and the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. An auxiliary understanding method based on gaze estimation, characterized in that, include: Images of the eyes of users wearing AR / VR / MR devices are captured using a near-eye infrared camera. The eye image is processed to determine the three-dimensional gaze vector; The three-dimensional gaze vectors of the user's left and right eyes are mapped onto the virtual screen as the user's gaze points; The gaze time of the gaze point is obtained. When the gaze point and the gaze time meet the preset conditions, the category information corresponding to the target gazed by the user is determined according to the scene image gazed at by the user and the pre-trained DETR based Transformer end-to-end target detection algorithm. The AR / VR / MR device is then controlled to perform the operation of displaying the category information corresponding to the target. The category information includes the object shape, color and the relative positional relationship between objects. The process of processing the eye image to determine the three-dimensional gaze vector includes: The eye image is input into the RITnet network segmentation model to segment the eye image and obtain segmentation maps of the edge information of the four parts: background, sclera, iris, and pupil. The obtained segmentation map is fitted with an edge detector and the least squares method to obtain the pupil ellipse and determine the parameter information of the pupil ellipse; Determine the camera parameters, and then use these parameters to establish the camera coordinate system in three-dimensional space. In the camera coordinate system, a 3D eyeball model is fitted based on the pupil ellipse of multiple eye images of the user. The fitted pupil ellipse is then triangulated to obtain a pupil circle tangent to the 3D eyeball model. The parameters of the 3D eyeball model are determined based on multiple eye images, the pupil circle information corresponding to each eye image, and the least squares method. The three-dimensional gaze vector is also determined. In the camera coordinate system, a 3D eyeball model is fitted based on the pupil ellipse from multiple eye images of the user. The fitted pupil ellipse is then triangulated to obtain a pupil circle tangent to the 3D eyeball model, including: Using triangulation geometry, a three-dimensional cone is constructed with the camera as the focal point and the camera as the vertex. The pupil ellipse segmented from the eye image is the intersection line between the eye image and the cone. In a fully perspective projection model, the pupil ellipse is back-projected onto a 3D eyeball model, ensuring that the back-projected pupil circle is tangent to the 3D eyeball model. The pupil circle projected onto the surface of the 3D eyeball model is considered as a cross-section of a cone with the camera focus as its vertex, and the boundary of the pupil ellipse in the eye image is the intersection point of the two-dimensional plane intersecting the cone. Once the pupil circle is determined, the pupil circle information is obtained: Where: p represents the three-dimensional pupil coordinates; m represents the normal vector of the pupil; r represents the radius of the pupil circle; The process of determining 3D eyeball model parameters and three-dimensional gaze vectors based on multiple eye images, pupil circle information corresponding to each eye image, and least squares method includes: Based on eye images of multiple subjects, the 3D eyeball model parameters for each person were obtained, including the center of the 3D eyeball model and the average radius of the sphere. Based on the pupil circle information constructed from multiple eye images, the intersection point of the normal vectors of multiple pupil circles is determined as the center of the 3D eyeball model. Since all intersection points cannot be at the same point due to numerical measurement errors, the least squares method is used to determine the center. Once the center of the 3D eyeball model is determined, the radius of all eye images is calculated based on the pupil circle and the center of the 3D eyeball model. The final 3D eyeball radius R is obtained by averaging the radii of all eye images. Where: R i Let be the radius obtained by the distance between the pupil circle and the center of the sphere in each eye image, c represents the center of the sphere in the 3D eye model, and pi is the pupil coordinate of the i-th eye image; Based on the pupil center obtained from the eye image and the parameters of the 3D eye model, the pupil center projected onto the 3D eye model is obtained, and the line connecting the center of the 3D eye model and the pupil center projected from each eye image is determined as the three-dimensional line of sight vector. Eye images are acquired using a near-eye infrared camera in an AR / VR / MR device. When acquiring eye images, the AR / VR / MR device is configured. First, each location to be captured is visualized on a virtual screen within a visualization interface. Second, the user captures K eye images at each visualized location by gazing at it, and simultaneously generates the ground truth coordinates of that location for subsequent calibration. Finally, the focal length, frame rate, and sensor size hardware parameters of the infrared near-eye camera are determined to construct a spatial coordinate system when fitting the eye model. Wherein, K ranges from 10 to 50; The category information corresponding to the target is displayed through a text box. Through the display interface module, the visualized gaze point and scene image are input into the virtual screen of the AR / VR / MR device. The visualized gaze point and the information after object target detection are compared to generate a text box with descriptive text information. First, there's the gaze point visualization: the 3D gaze vector of the gaze point is mapped to obtain the estimated 2D gaze point location, and a visualization point is drawn centered on this point in the mobile deployment application. Second, there's the scene image visualization: each time a user uses the app, a certain number of images can be randomly selected for text descriptions of targets in the scene images. Third, there's the text box generation visualization: after the gaze point and the target selection area are matched, eye-tracking behavior determines whether to generate a text box. The text box generation is implemented by inputting the marked object information into the text box according to the mobile deployment configuration. Finally, there's the button configuration: when it's necessary to change to the next or previous image, the user can switch scene images by looking at the button at the top of the virtual screen.

2. The auxiliary understanding method based on gaze estimation according to claim 1, characterized in that, The obtained segmentation map is fitted with an edge detector and least squares method to obtain the pupil ellipse, and the parameter information of the pupil ellipse is determined, including: The segmentation map, consisting of the background, sclera, iris, and pupil, obtained through the RITnet network segmentation model, is used to extract the pupil boundary after thresholding using an edge detector. The boundary information is then fitted using the least squares method to obtain the parameter information of the pupil ellipse. The pupil ellipse is then drawn, and the mathematical expression of its general equation is determined as follows: Where: the parameters of the general equation for an elliptical pupil are (A, B, C, D, E, F); the center of the pupil ellipse is (X). e Y e ).

3. The auxiliary understanding method based on gaze estimation according to claim 1, characterized in that, The three-dimensional gaze vectors of the user's left and right eyes are mapped onto the virtual screen as the user's gaze points, including: Based on the fitted 3D eye model and three-dimensional gaze vector, when the user gazes at a known point S on the screen, the conversion relationship between the actual gaze point coordinates and the estimated gaze point coordinates can be determined through simple calibration: Where M is the rotation matrix; To estimate the 3D line-of-sight vector; This is the actual 3D view vector; The 3D eyeball center O=(a,b,c) and the estimated 3D gaze vector have been calculated. In the two-dimensional plane equation The coordinates of the intersection point; The parametric equation of the line is: Substituting into the plane coordinate equation, we get: ; Substituting t back into the parametric equation of the line, we can obtain the estimated coordinates P(x,y,z) of the intersection point between the line of sight and the two-dimensional screen. Due to the differences between the left and right eyes, there will be discrepancies in the estimated two-dimensional plane coordinates between the two eyes. Therefore, the final estimated two-dimensional plane fixation point coordinates are obtained according to the following formula: The rotation matrix can be obtained from known points on the screen. (i=1, 2, 3) and the three-dimensional gaze vector at the screen to predict the gaze point Solve the following equation (i=1, 2, 3) obtained through the above transformation relationship: 。 4. The auxiliary understanding method based on gaze estimation according to claim 1, characterized in that, Before determining the category information of the target being gazed at based on the scene image and the pre-trained DETR algorithm, the process also includes: Detect each target in the scene images in the public dataset, and annotate the target bounding regions of each target in the scene images; Each scene image is fed into the DETR algorithm for training. After feature extraction, decoding, and prediction, each region of the scene image is divided into target-category and non-target-category categories. The target selection region with target category is then matched with the category information describing the target.

5. The auxiliary understanding method based on gaze estimation according to claim 4, characterized in that, Based on the scene image being viewed by the user and a pre-trained DETR algorithm, the category information corresponding to the target being viewed by the user is determined, including: The scene image that the user is looking at is used as input to the DETR algorithm; Based on the target selection area where the user's gaze point falls, determine the category information that matches the target selection area; Controlling the AR / VR / MR device to perform the operation of displaying the category information corresponding to the target includes: The AR / VR / MR device is controlled to display the category information corresponding to the target in the center of the target selection area where the gaze point falls.

6. The auxiliary understanding method based on gaze estimation according to claim 4, characterized in that, The preset conditions are that the gaze point falls within any preset target selection area, the gaze time is greater than a first threshold, and the fluctuation of the gaze point within the first threshold does not exceed a preset gaze point fluctuation range.

7. The auxiliary understanding method based on gaze estimation according to claim 1, characterized in that, After controlling the AR / VR / MR device to perform the operation of displaying the category information corresponding to the target, the method further includes: Get the user's eye-closed duration; If the duration of eye closure exceeds the second threshold, it is determined that the user is voluntarily closing their eyes, and the AR / VR / MR device is controlled to stop displaying the category information corresponding to the target.

8. An auxiliary understanding system based on the gaze estimation-based auxiliary understanding method according to any one of claims 1 to 7, characterized in that, include: An eye image acquisition module is used to acquire eye images of users wearing AR / VR / MR devices via the near-eye infrared camera of the AR / VR / MR device; A gaze estimation module is used to process the eye image and determine the three-dimensional gaze vector; The gaze point mapping module is used to map the three-dimensional gaze vectors of the user's left and right eyes onto the virtual screen as the user's gaze point; An eye-tracking interaction module is used to acquire the gaze duration of the gaze point. When the gaze point and the gaze duration meet preset conditions, the module determines the category information corresponding to the target being gazed at based on the scene image being gazed at by the user and the pre-trained DETR-based end-to-end target detection algorithm using Transformer. The module then controls the AR / VR / MR device to perform the operation of displaying the category information corresponding to the target. The category information includes the object shape, color, and the relative positions of the objects.

Images