Endoscope sharing control method based on human body signal and game theory

Abstract

The invention provides an endoscope sharing control method based on a human body signal and a game theory, and belongs to the field of medical instruments, and the endoscope sharing control method comprises the following steps: obtaining eyeball movement data; based on the eyeball movement data, determining eye movement control position data used for representing a manual control intention through gaze thermodynamic diagram analysis; based on the eyeball movement data, through quantitative analysis of sight, eye opening degree and pupil size, obtaining reference indexes used for representing intention integrating degree and manual control burden; machine control position data used for representing the expected moving distance of the endoscope are obtained by analyzing the position relation between a target object and the endoscope in an endoscope image, and the endoscope image is collected in real time through the endoscope; based on the machine control position data, the eye movement control position data and the reference index, determining shared control position data through a cooperative game strategy; movement of the endoscope is controlled based on the shared control position data.

Description

Technical Field

[0001] This invention relates to the field of medical devices, and more specifically, to an endoscope sharing control method based on human body signals and game theory. Background Technology

[0002] In minimally invasive surgery, endoscopic field of view adjustment directly affects the quality of lesion exposure and surgical safety, and is a crucial factor in surgical success. Current endoscopic control methods mainly include three categories: manual control, automatic control, and hybrid control. Manual control involves the physician directly manipulating the endoscope, resulting in high intent matching, but requires continuous participation, leading to high workload and low comfort. Automatic control typically predicts instrument or tissue position and adjusts the field of view based on visual recognition or deep learning methods, reducing the physician's burden, but it struggles to replicate clinical experience and is poorly adaptable to complex situations such as tissue deformation and bleeding obstruction. While existing hybrid control incorporates human signals such as eye movements, voice, or gestures, it remains a passive trigger control, essentially adding operational steps and failing to achieve true human-machine collaboration at the decision-making level.

[0003] Therefore, there has long been a contradiction in endoscopic control: manual control has a high degree of consistency with intent but low comfort, while automatic control has high comfort but lacks the ability to express intent. Summary of the Invention

[0004] In view of this, the present invention provides an endoscope sharing control method based on human body signals and game theory.

[0005] One aspect of the present invention provides an endoscope shared control method based on human signals and game theory, comprising: acquiring eye movement data; determining eye movement control position data for characterizing human control intention by means of gaze heatmap analysis based on the eye movement data; obtaining reference indicators for characterizing intention fit and human control burden by means of quantitative analysis of line of sight, eye opening degree and pupil size based on the eye movement data; obtaining machine control position data for characterizing the expected movement distance of the endoscope by analyzing the positional relationship between the target object and the endoscope in the endoscope image, wherein the endoscope image is acquired in real time by the endoscope; determining shared control position data by means of a cooperative game strategy based on the machine control position data, the eye movement control position data and the reference indicators; and controlling the movement of the endoscope based on the shared control position data.

[0006] According to an embodiment of the present invention, the above-mentioned method of obtaining reference indicators for characterizing intention fit and human control burden based on the above-mentioned eye movement data through quantitative analysis of gaze, eye opening degree, and pupil size includes: obtaining a first reference indicator for characterizing intention fit based on the above-mentioned eye movement data through quantitative analysis of the degree of gaze deviation and concentration degree; obtaining a second reference indicator for characterizing human control burden based on the above-mentioned eye movement data through quantitative analysis of eye opening degree and pupil size; and obtaining a reference indicator based on the first reference indicator and the second reference indicator.

[0007] According to an embodiment of the present invention, the above-mentioned method of obtaining a first reference index for characterizing intention fit based on the above-mentioned eye movement data by quantitative analysis of the degree of deviation and concentration of gaze includes: generating a gaze heatmap based on the above-mentioned eye movement data by kernel density estimation; performing statistical analysis on the above-mentioned gaze heatmap to determine the visual field deviation value; performing data analysis on the above-mentioned gaze heatmap based on the Gaussian kernel density value of each pixel of the above-mentioned gaze heatmap to determine the visual concentration; and performing a weighted calculation on the above-mentioned visual field deviation value and the above-mentioned visual concentration to obtain the first reference index.

[0008] According to an embodiment of the present invention, the above-mentioned statistical analysis of the gaze heatmap to determine the visual field deviation value includes: determining the weighting center of the gaze heatmap as a first coordinate point by performing a weighted calculation on the gaze heatmap; determining the geometric center of the gaze heatmap as a second coordinate point; and determining the visual field deviation value based on the difference between the first coordinate point and the second coordinate point.

[0009] According to an embodiment of the present invention, the above-mentioned data analysis of the gaze heatmap based on the Gaussian kernel density values ​​of each pixel in the gaze heatmap to determine visual concentration includes: normalizing the Gaussian kernel density values ​​of each pixel in the gaze heatmap to obtain the probability distribution values ​​of each corresponding pixel; selecting multiple target pixels from the gaze heatmap whose probability distribution values ​​are greater than the preset probability density values ​​based on preset probability density values; determining visual concentration efficiency based on the ratio of the number of the multiple target pixels to the total number of pixels in the gaze heatmap; determining spatial compactness based on the ratio of the convex hull area formed by the multiple target pixels to the total area of ​​the region corresponding to the gaze heatmap; and determining visual concentration based on the visual concentration efficiency and the spatial compactness.

[0010] According to an embodiment of the present invention, the above-mentioned method of obtaining a second reference index for characterizing the burden of manual control based on the eye movement data and through quantitative analysis of eye opening degree and pupil size includes: determining the operator's eye opening degree and pupil size based on the eye movement data through image processing and feature extraction; obtaining the duration of the endoscope in manual-dominated mode as the manual-dominated duration, wherein the manual-dominated mode is an endoscope control mode in which the weight of manual control of the endoscope is greater than the weight of machine control of the endoscope, and the weight of manual control and the weight of machine control are determined through a cooperative game strategy based on machine control position data, eye movement control position data, and the reference index; determining the eye opening degree score based on the eye opening degree through Gaussian decay function analysis; determining the pupil size score based on the pupil size through Gaussian penalty function analysis; determining the manual-dominated score based on the manual-dominated duration through ramp function analysis; and performing a weighted calculation on the eye opening degree score, the pupil size score, and the manual-dominated score to determine the second reference index.

[0011] According to an embodiment of the present invention, the determination of shared control position data based on the machine control position data, the eye-tracking control position data, and the reference index through a cooperative game strategy includes: obtaining a human-machine decision position gap value based on the machine control position data and the eye-tracking control position data; obtaining the initial machine control weight and the initial human control weight for the current endoscopic control cycle; obtaining the machine weight change and the human weight change based on a preset position gap threshold, the human-machine decision position gap value, and the reference index; adding the machine weight change to the initial machine control weight and the human weight change to the initial human control weight to obtain the machine control weight and the human control weight; and weighting the machine control position data and the eye-tracking control position data based on the machine control weight and the human control weight to determine the shared control position data.

[0012] According to an embodiment of the present invention, the above-mentioned method of obtaining the machine weight change and the human weight change based on the preset position difference threshold, the human-machine decision position difference value, and the reference index includes: when the human-machine decision position difference value is greater than the preset position difference threshold; when the first reference index is less than or equal to the preset first reference index, the machine weight change is determined to be the negative of the preset machine weight step size value, and the human weight change is determined to be the preset human weight step size value; when the first reference index is greater than the preset first reference index and the second reference index is greater than the preset second reference index, the machine weight change is determined to be the negative of the preset machine weight step size value, and the human weight change is determined to be the negative of the preset human weight step size value; when the first reference index is greater than the preset first reference index and the second reference index is less than or equal to the preset second reference index, the machine weight change is determined to be the preset machine weight step size value, and the human weight change is determined to be the preset human weight step size value.

[0013] According to an embodiment of the present invention, the above-mentioned method of obtaining the machine weight change and the human weight change based on the preset position difference threshold, the human-machine decision position difference value, and the reference index includes: when the human-machine decision position difference value is less than or equal to the preset position difference threshold; when the second reference index is less than or equal to the preset second reference index, the machine weight change is determined to be a preset machine weight step size value, and the human weight change is a negative value of the preset human weight step size value; when the second reference index is greater than the preset second reference index and the first reference index is less than or equal to the preset first reference index, the machine weight change is determined to be a preset machine weight step size value, and the human weight change is a preset human weight step size value; when the second reference index is greater than the preset second reference index and the first reference index is greater than the preset first reference index, the machine weight change is determined to be a negative value of the preset machine weight step size value, and the human weight change is a negative value of the preset human weight step size value.

[0014] According to an embodiment of the present invention, the above-mentioned determination of eye movement control position data for characterizing artificial control intention by means of gaze heatmap analysis based on the above-mentioned eye movement data includes: generating a gaze heatmap by kernel density estimation based on the above-mentioned eye movement data; determining the weighting center of the gaze heatmap as a first gaze point by weighted calculation of the gaze heatmap; determining the movement trajectory of the first gaze point based on the above-mentioned eye movement data; determining the displacement of the first gaze point based on the movement trajectory of the first gaze point; and performing data mapping on the displacement of the first gaze point based on a first preset coefficient to obtain eye control position data.

[0015] According to embodiments of the present invention, eye-tracking control position data for characterizing the intention of manual control, machine control position data for characterizing the expected movement distance of the endoscope, and reference indicators for characterizing the degree of intention fit and the burden of manual control can be determined. Based on the machine control position data, eye-tracking control position data, and reference indicators, shared control position data is determined through a cooperative game strategy. By controlling the movement of the endoscope through the shared control position data, a balance between the degree of intention fit and comfort during the endoscope control process is achieved. Under the premise of meeting the surgical operation intention, the operator's eye fatigue and operational burden can be significantly reduced, making the endoscope control process more natural, comfortable, and easy to operate stably for a long time. Attached Figure Description

[0016] The above and other objects, features and advantages of the present invention will become more apparent from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0017] Figure 1 A flowchart of an endoscope sharing control method based on human body signals and game theory according to an embodiment of the present invention is shown;

[0018] Figure 2 A flowchart illustrating an endoscope sharing control method based on human body signals and game theory according to another embodiment of the present invention is shown.

[0019] Figure 3 A schematic diagram of an endoscope system architecture according to an embodiment of the present invention is shown;

[0020] Figure 4 A flowchart of a cooperative game based on an embodiment of the present invention is shown. Detailed Implementation

[0021] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0022] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms "comprising," "including," etc., as used herein indicate the presence of the above-described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0023] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0024] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0025] Existing endoscope shared control methods based on manual and machine control are all hybrid control methods, which involve switching between different control modes at different times and the control system switching between multiple independent control strategies. However, this control mode is a passively triggered control, which essentially increases the operator's steps and does not achieve true human-machine collaboration at the decision-making level. Therefore, it is conceivable to adopt a shared control mode to control the endoscope, in which humans and machines apply control actions to the endoscope in parallel, with control rights dynamically allocated according to weights, and the inputs from both sides merged into the final endoscope control command.

[0026] Figure 1 A flowchart of an endoscope sharing control method based on human body signals and game theory according to an embodiment of the present invention is shown.

[0027] like Figure 1 As shown, the endoscope sharing control method based on human body signals and game theory includes operations S110-S160.

[0028] In operation S110, eye movement data is acquired.

[0029] In embodiments of the present invention, eye movement data can be acquired from the operator's eye movements using a Mixed Reality (MR) head-mounted display device. A mixed reality head-mounted display device is a wearable device that integrates virtual content with the real world in real time. It combines the features of augmented reality (AR) and virtual reality (VR), allowing the operator to manipulate virtual digital objects while interacting with the real environment. The core of the mixed reality head-mounted display device is the precise fusion of virtual digital information with the real physical scene through optical modules, while relying on multi-sensor perception, spatial positioning, and interaction technologies to achieve natural interaction between the operator and the mixed scene. The mixed reality head-mounted display device integrates dedicated perception sensors and optical acquisition modules, thereby enabling the acquisition of eye movement data, and using algorithms to complete the real-time capture, processing, and calibration of the corresponding data.

[0030] Eye movement data can include fixation point, gaze direction, blink frequency, and the spatial position of the pupil center. Eye movement data can be obtained by capturing eye features based on the principle of infrared light reflection. When an infrared light source illuminates the eye, a clear, tiny reflective point is formed on the cornea. The position of this reflective point is determined by the fixed position of the infrared light source and can serve as a fixed reference point. Simultaneously, an eye-tracking camera can clearly capture the position of the pupil center, which shifts relative to eye movement. By calculating the relative positional relationship between the pupil center and the corneal reflective point using algorithms, the eye rotation angle can be accurately deduced, thereby determining the operator's gaze direction and fixation point.

[0031] In embodiments of the present invention, head motion data can also be acquired from the operator's head movements using an MR head-mounted display device, and the endoscope's attitude control can be achieved based on the head motion data. The head motion data can include the head's three-dimensional position coordinates and three-dimensional attitude in space, including the head's yaw angle, pitch angle, and roll angle. A mature attitude calculation algorithm can be used to convert the head motion data into relative attitude data of the head in a preset spatial coordinate system, serving as a basic attitude reference for mapping to the endoscope. The calculation process includes initial attitude calibration and real-time attitude calculation. The initial attitude calibration process involves guiding the operator to adjust their head to a reference pose upon system startup. Using this as the zero point, an initial reference coordinate system for the head attitude is established, and the Euler angles, quaternions, or rotation matrix of the initial attitude are recorded. The real-time attitude calculation process involves using the initial attitude as a reference, fusing and calculating the inertial data of the inertial measurement unit through a sensor data fusion algorithm, and outputting the head's attitude parameters in the preset coordinate system in real time.

[0032] In operation S120, eye movement data is used to determine eye movement control position data to characterize the intention of artificial control through gaze heatmap analysis.

[0033] Gaze heatmaps are the core visualization product of eye-tracking technology. Through color gradients and brightness variations, they visually present the observer's gaze position, duration, and frequency on a target interface. Their core function is to quantify the distribution patterns of visual attention, making them an important analytical tool in fields such as human-computer interaction, user experience, industrial design, and psychology. By analyzing gaze heatmaps, we can determine the operator's visual focus, thereby representing their intention to control the system—where the operator focuses indicates where they wish to move the endoscope.

[0034] In embodiments of the present invention, the eye-tracking control position data of the endoscope can be the three-dimensional coordinate data of the endoscope in space. For example, when the eye-tracking control position data of the endoscope is (X0, Y0, Z0), it represents the desired control of the endoscope to move to the position corresponding to the three-dimensional coordinates (X0, Y0, Z0) in space.

[0035] In operation S130, based on eye movement data, reference indicators for characterizing intention fit and human control burden are obtained through quantitative analysis of gaze, eye opening degree and pupil size.

[0036] In embodiments of the present invention, the degree of intent fit refers to the matching degree between the operator's desired field of view and the actual displayed field of view, which can be obtained by quantitative analysis of the operator's gaze. For example, when the operator's gaze is located at the center of the virtual screen, it means that the operator's desired field of view and the actual displayed field of view are consistent, and the degree of intent fit is high; when the operator's gaze is located at a corner of the virtual screen, it means that the operator's desired field of view and the actual displayed field of view are inconsistent, and the degree of intent fit is low.

[0037] Manual control burden refers to the total expenditure of the operator's perception, decision-making, movement, cognition, and physiology during the process of controlling the endoscope. The higher the manual control burden, the more likely problems such as hand tremors, misoperation, and fatigue will occur. Manual control burden can be obtained by quantifying the degree of eye opening and pupil size. For example, the lower the degree of eye opening and the smaller the pupil, the greater the operator's manual control burden.

[0038] In operation S140, by analyzing the positional relationship between the target object and the endoscope in the endoscopic image, machine control position data is obtained to characterize the expected movement distance of the endoscope. The endoscopic image is acquired in real time by the endoscope.

[0039] During operation, the endoscope acquires images in real time, obtaining endoscopic images. The operator uses these endoscopic images to obtain the surgical field of view inside the body. In embodiments of the present invention, the endoscopic images can also be analyzed and processed using target detection algorithms to identify the position of target objects within the endoscopic images. For example, the endoscopic images can be input into open-source models such as YOLOX to obtain the position coordinates of the target objects in the endoscopic images. Based on the current position coordinates of the endoscope and the position coordinates of the target objects in the endoscopic images, and through a preset relative positional relationship, the machine control position data of the endoscope when the preset relative positional relationship is met can be analyzed and obtained. The aforementioned machine control position data is based on the endoscopic images being processed entirely by relevant algorithms; it is a position coordinate value of where the endoscope should move, predicted based on automatic vision processing. In embodiments of the present invention, the endoscopic images can also be obtained based on binocular imaging. Inputting the endoscopic images obtained based on binocular imaging into a target detection model can yield the three-dimensional position coordinates of the target objects in the endoscopic images.

[0040] In operation S150, based on machine control position data, eye-tracking control position data, and reference indicators, shared control position data is determined through a cooperative game strategy.

[0041] In each endoscope control cycle, reference indicators, machine control position data, and eye-tracking control position data can be used as inputs to a cooperative game to determine the human control weights and machine control weights. The game only involves the position control of the endoscope; the endoscope's posture control can be achieved based on head movement data. The game type of this invention can be a non-constant sum game, meaning the total payoff for both parties is not a fixed value. The game type of this invention can be pure, meaning the game's reward is independent of probability. The participants in the game of this invention are the operator and the position prediction module. The operator determines the eye-tracking control position data, while the position prediction module determines the machine control position data. The game reward is the reference indicator, and the game strategy is for the operator and the position prediction module to each increase (acquiring) or decrease (yielding) their control weights. The game strategy design of this invention satisfies Pareto optimality, that is, increasing the reward of one party without decreasing the reward of the other.

[0042] In operation S160, the movement of the endoscope is controlled based on shared control position data.

[0043] Through the embodiments of the present invention, eye-tracking control position data for characterizing human control intention, machine control position data for characterizing the expected movement distance of the endoscope, and reference indicators for characterizing intention fit and human control burden can be determined. Based on machine control position data, eye-tracking control position data, and reference indicators, shared control position data is determined through a cooperative game strategy. By controlling the movement of the endoscope through shared control position data, a balance between intention fit and comfort is achieved during the endoscope control process. Under the premise of meeting the surgical operation intention, the operator's eye fatigue and operational burden can be significantly reduced, making the endoscope control process more natural, comfortable, and easy to operate stably for a long time.

[0044] According to an embodiment of the present invention, eye movement control position data for characterizing artificial control intention is determined based on eye movement data and through gaze heatmap analysis, including: generating a gaze heatmap based on eye movement data through kernel density estimation; determining the weighting center of the gaze heatmap as a first gaze point by weighted calculation; determining the movement trajectory of the first gaze point based on eye movement data; determining the displacement of the first gaze point based on the movement trajectory of the first gaze point; and mapping the displacement of the first gaze point based on a first preset coefficient to obtain eye movement control position data.

[0045] The weighted center of a gaze heatmap is the core coordinate point obtained by weighting the effective gaze dataset from the gaze heatmap. It represents the "center of gravity" of the observer's visual attention. The weighted center accurately represents the core location of the highest concentration of attention within the entire heatmap, rather than simply the geometric center. The weighted center of a gaze heatmap is a key feature value for quantitative analysis and an important reference indicator for eye movement control and interface design optimization.

[0046] A gaze point is the specific location in a virtual or real-world scene coordinate system where the operator's gaze is directed. In embodiments of this invention, the directionality of the gaze can be characterized by two types of physical postures: one is the head reference gaze, which is the spatial direction the head is pointing towards, representing the expected visual direction corresponding to the head posture; the other is the eye reference gaze, which is the spatial direction pointed to by the rotation posture of the eyeball within the eye socket, representing the actual target direction of the eyeball's gaze. In embodiments of this invention, the weighted center of the gaze heatmap corresponding to the current eye state can be obtained through eye movement data and used as the first gaze point. The operator controls eye movement, and the first gaze point corresponding to the operator's eye state also moves accordingly. By processing the eye movement data, the movement trajectory of the first gaze point can be obtained, thereby determining the displacement of the first gaze point on a reference plane, where the reference plane can be the display plane of the virtual screen.

[0047] In embodiments of the present invention, a first preset coefficient can be used to map the displacement of the first gaze point on the reference plane to the displacement of the endoscope on the working plane, i.e., the eye movement control position data of the endoscope. The working plane can be a plane perpendicular to the optical axis of the endoscope lens. For example, if the first gaze point moves 2 cm to the left and 1 cm upward on the reference plane, the endoscope will move to the left by the product of the first preset coefficient and 2 cm, and upward by the product of the first preset coefficient and 1 cm. The first preset coefficient can be determined based on the pixel size of the virtual screen, the working space range of the endoscope, and the movement speed of the endoscope. For example, when the length of the reference plane corresponding to the pixel size of the virtual screen is 50 cm, and the length of the working space range of the endoscope is 2 cm, the first preset coefficient can be 0.04 to ensure that the movement range of the first gaze point on the reference plane corresponds to the working space range of the endoscope. Similar to the second preset coefficient, the first preset coefficient can also be adjusted and determined by the user.

[0048] According to an embodiment of the present invention, based on eye movement data, reference indicators for characterizing intention fit and human control burden are obtained through quantitative analysis of gaze deviation and eye opening degree, including: obtaining a first reference indicator for characterizing intention fit based on eye movement data through quantitative analysis of gaze deviation and concentration degree; obtaining a second reference indicator for characterizing human control burden based on eye movement data through quantitative analysis of eye opening degree and pupil size; and obtaining a reference indicator based on the first and second reference indicators.

[0049] The degree of gaze deviation refers to the distance difference between the operator's current gaze point and the preset target point. The preset target point can be the exact center of the virtual screen viewed by the operator. The greater the distance between the operator's current gaze point and the exact center of the virtual screen, the greater the degree of gaze deviation. The degree of gaze concentration refers to the spatial concentration of the operator's gaze point over a period of time, used to quantify the operator's visual attention concentration. For example, when the operator's gaze point remains within a small area of ​​the virtual screen, it indicates a high degree of gaze concentration.

[0050] According to an embodiment of the present invention, based on eye movement data, a first reference index for characterizing intent fit is obtained through quantitative analysis of the degree of deviation and concentration of gaze, including: generating a gaze heatmap based on eye movement data by kernel density estimation; performing statistical analysis on the gaze heatmap to determine the visual field deviation value; performing data analysis on the gaze heatmap based on the Gaussian kernel density value of each pixel to determine the visual concentration; and performing a weighted calculation on the visual field deviation value and the visual concentration to obtain the first reference index.

[0051] The field of view deviation S1 can be obtained based on the following formula:

[0052] (1);

[0053] in, This is the field of view deviation value. It is half the length of the diagonal of the virtual screen.

[0054] The first reference index can be obtained by averaging the visual field deviation S1 and visual concentration. In an embodiment of the present invention, the first reference index can also be obtained by weighted summation of the visual field deviation S1 and visual concentration, wherein the weights of the weighted summation can be preset based on the actual situation.

[0055] According to an embodiment of the present invention, statistical analysis of a gaze heatmap to determine a visual field deviation value includes: determining the weighted center of the gaze heatmap as a first coordinate point by performing a weighted calculation on the gaze heatmap; determining the geometric center of the gaze heatmap as a second coordinate point; and determining the visual field deviation value based on the difference between the first coordinate point and the second coordinate point.

[0056] The gaze heatmap is obtained based on the virtual screen viewed by the operator; therefore, there is a correspondence between the pixels in the gaze heatmap and the pixels in the virtual screen. Let the horizontal and vertical indices of each pixel in the gaze heatmap be i and j, respectively. Then the pixel coordinates corresponding to each gaze point in the gaze heatmap are: The Gaussian kernel density corresponding to the gaze point during the detection sampling time is This serves as its weight. Further, the weighted center position of all fixation points on the fixation heatmap can be calculated, and the distance between this position and the geometric center of the fixation heatmap is defined as the visual field deviation value Δoffset, which is calculated as follows:

[0057] (2);

[0058] The field of view deviation value Δoffset is used to describe the degree of consistency between the field of view provided by the virtual screen and the operator's visual focus position. The smaller the deviation, the higher the degree of intent matching.

[0059] According to an embodiment of the present invention, visual concentration is determined by performing data analysis on the gaze heatmap based on the Gaussian kernel density values ​​of each pixel in the gaze heatmap, including: normalizing the Gaussian kernel density values ​​of each pixel in the gaze heatmap to obtain the probability distribution values ​​of each corresponding pixel; selecting multiple target pixels from the gaze heatmap whose probability distribution values ​​are greater than the preset probability density values ​​based on preset probability density values; determining visual concentration efficiency based on the ratio of the number of multiple target pixels to the total number of pixels in the gaze heatmap; determining spatial compactness based on the ratio of the convex hull area formed by the multiple target pixels to the total area of ​​the region corresponding to the gaze heatmap; and determining visual concentration based on visual concentration efficiency and the aforementioned spatial compactness.

[0060] In embodiments of the present invention, visual field deviation values ​​alone are insufficient to reflect the stability of the operator's operational intent. When the operator's gaze points are scattered over a period of time, it indicates that the operator is in a target search state; when the operator's gaze points are concentrated in a small area over a period of time, it indicates that the operator's target is clear. Therefore, visual concentration S2 can be constructed. First, the Gaussian kernel density of each pixel in the gaze heatmap is normalized to a probability distribution:

[0061] (3);

[0062] The visual attention density concentration efficiency, commonly used in viewpoint heatmap analysis, can then be calculated. Compactness of the gaze area And by fusing them, we obtain:

[0063] (4);

[0064] The probability density is the ratio of the number of pixels whose probability density exceeds a preset probability density value to the total number of pixels. In an embodiment of the present invention, the preset probability density value can be 0.9. It is the ratio of the area of ​​the convex hull formed by pixels with a probability density exceeding a threshold to the total area of ​​the pixels. The closer the value is to 1, the higher the visual concentration and the more focused the visual attention.

[0065] According to an embodiment of the present invention, based on eye movement data, a second reference index for characterizing the burden of manual control is obtained through quantitative analysis of eye opening degree and pupil size. This includes: determining the operator's eye opening degree and pupil size based on eye movement data through image processing and feature extraction; obtaining the duration of the endoscope in a manually controlled mode as the manually controlled duration, wherein the manually controlled mode is an endoscope control mode where the weight of manual control is greater than the weight of machine control, and the weights of manual control and machine control are determined through a cooperative game strategy based on machine control position data, eye movement control position data, and the reference index; determining an eye opening degree score based on eye opening degree through Gaussian decay function analysis; determining a pupil size score based on pupil size through Gaussian penalty function analysis; determining a manually controlled duration score based on the manually controlled duration through ramp function analysis; and performing a weighted calculation of the eye opening degree score, pupil size score, and manually controlled duration score to determine the second reference index.

[0066] In embodiments of the present invention, eye movement data includes images of the operator's eyes. By processing these images and using feature extraction algorithms, the degree of eye opening and pupil size can be quantitatively calculated. Facial landmark detection can be performed on the eye images to locate the left and right eye contour regions. Through threshold segmentation, edge detection, or deep learning object detection models, the effective eye region can be separated from the background, excluding interfering areas such as the face, skin, and eyelashes. For the degree of eye opening, the coordinates of the upper and lower eyelid edges can be extracted within the eye region, and the maximum opening and closing distance of the eyelids in the vertical direction can be calculated. Simultaneously, using the horizontal width of the eye as a normalization benchmark, the ratio of the vertical opening and closing distance to the horizontal width is used to obtain the normalized degree of eye opening, achieving unified quantification for different faces and different distances. For pupil size, the eye image can be preprocessed with grayscale conversion, noise reduction, and contrast enhancement to highlight the grayscale differences between the pupil, iris, and sclera. The pupil contour is accurately extracted using edge detection, Hough transform, contour fitting, or a lightweight segmentation network to obtain the pupil's closed boundary. Based on the extracted pupil contour, its equivalent diameter, area, and minimum circumscribed circle radius are calculated. The equivalent diameter or normalized area is used as a quantitative indicator of pupil size, achieving a continuous and stable representation of pupil dilation and contraction. In embodiments of this invention, quantitative indicators such as eye opening degree and pupil size can be obtained based on a mixed reality head-mounted display device.

[0067] In the embodiments of the application, eye-tracking control position data is endoscopic motion control data reflecting manual control, obtained based on the operator's intention. Machine-controlled position data is endoscopic motion control data reflecting the effect of automatic machine control, obtained through automatic recognition and processing of endoscopic images. The final motion control result of the endoscope is obtained by weighting the eye-tracking control position data and the machine-controlled position data; that is, the eye-tracking control position data and the machine-controlled position data are weighted and summed using manual control weights and machine control weights to obtain shared control position data. The endoscope moves based on the shared control position data. In each control cycle of the endoscope, the manual control weight and the machine control weight are constantly changing based on external conditions; the manual control weight may be larger in some periods and the machine control weight may be larger in others. Therefore, the endoscope control mode where the manual control weight is greater than the machine control weight can be defined as the manual-dominated mode, and the cumulative duration of the endoscope in the manual-dominated mode can be defined as the manual-dominated duration.

[0068] The eye-opening score can be obtained based on the following formula:

[0069] (5);

[0070] in, The average degree of eye opening. The standard deviation of eye opening can be calibrated before the operator performs endoscopic control. To ensure that the denominator is non-zero, the offset can be set to a constant of a smaller order of magnitude than the other variable in the denominator. For the current moment The collected eye opening degree. In an embodiment of the present invention... It can be 10 -6 .

[0071] Pupil size rating can be obtained based on the following formula:

[0072] (6);

[0073] in, This is the average pupil size, which can be calibrated before the operator makes endoscopic control. To ensure that the denominator is non-zero, the offset can be set to a constant of a smaller order of magnitude than the other variable in the denominator. This is a scaling factor for the relative rate of change, adjusted according to the range of pupil size; For the current moment The size of the pupil collected. In an embodiment of the present invention, It can be 0.15. It can be 10 -6 .

[0074] Human-led ratings can be obtained based on the following formula:

[0075] (7);

[0076] in, To allow for the doctor's preferred level of control, calibration can be performed before the operator takes endoscopic control. For the current moment The duration of human intervention; This is a scaling factor that can be adjusted based on the duration range; To ensure that the variable value does not exceed Preprocessing functions for intervals. In embodiments of the present invention, It can be 0.25.

[0077] According to an embodiment of the present invention, based on machine control position data, eye-tracking control position data, and reference indicators, shared control position data is determined through a cooperative game strategy, including: obtaining a human-machine decision-making position gap value based on machine control position data and eye-tracking control position data; obtaining the initial machine control weight and initial human control weight for the current endoscopic control cycle; obtaining the machine weight change and human weight change based on a preset position gap threshold, the human-machine decision-making position gap value, and reference indicators; adding the machine weight change to the initial machine control weight and the human weight change to the initial human control weight to obtain the machine control weight and human control weight; and weighting the machine control position data and eye-tracking control position data based on the machine control weight and human control weight to determine the shared control position data.

[0078] The endoscope control cycle is related to the output frequency of the endoscope's control system. For example, when the output frequency of the control system is 50 Hz, the endoscope's control system executes a control cycle every 0.02 seconds. That is, every 0.02 seconds, the control system reads the initial machine control weight and initial manual control weight of the current endoscope control cycle, and obtains the machine control weight and manual control weight of the current endoscope control cycle based on the machine weight change and manual weight change obtained from reference indicators. Finally, the machine control position data and eye-tracking control position data are weighted based on the machine control weight and manual control weight to determine the shared control position data and control the endoscope movement. Simultaneously, the machine control weight and manual control weight of the current endoscope control cycle serve as the initial machine control weight and initial manual control weight for the next endoscope control cycle. Endoscope control is achieved through continuous updates of the machine control weight and manual control weight. In the first endoscope control cycle, both the initial machine control weight and the initial manual control weight are 0.5. In embodiments of the present invention, the machine control weight and manual control weight can be normalized so that the sum of the machine control weight and manual control weight is always 1.

[0079] According to an embodiment of the present invention, when the human-machine decision-making position gap value is greater than a preset position gap threshold, the change in machine weight and the change in human weight are obtained based on the preset position gap threshold, the human-machine decision-making position gap value, and reference indicators. This includes: when the first reference indicator is less than or equal to a preset first reference indicator, determining the change in machine weight as a negative value of a preset machine weight step size and the change in human weight as a preset human weight step size; when the first reference indicator is greater than the preset first reference indicator and the second reference indicator is greater than the preset second reference indicator, determining the change in machine weight as a negative value of the preset machine weight step size and the change in human weight as a negative value of the preset human weight step size; and when the first reference indicator is greater than the preset first reference indicator and the second reference indicator is less than or equal to the preset second reference indicator, determining the change in machine weight as a preset machine weight step size and the change in human weight as a preset human weight step size.

[0080] When the discrepancy between human and machine decision-making positions is too large, it indicates a significant deviation between eye-tracking control position data reflecting human control intent and machine control position data reflecting machine prediction results. In this case, a categorized discussion is needed based on the values ​​of the first reference indicator (intention fit) and the second reference indicator (comfort). When intention fit is low but comfort is high, it indicates that automatic control deviates from the operator's intention, requiring an increase in human control weight and a decrease in machine control weight. When intention fit is high but comfort is low, it indicates a heavy operator burden, but the human-machine decision-making gap is also significant, so both machine and human control weights should be increased simultaneously. When both intention fit and comfort are low, priority should be given to increasing intention fit, strengthening human control weight, and decreasing machine control weight. When both intention fit and comfort are high, both human and machine control weights can be decreased simultaneously to maintain stability, waiting for intention fit to decrease before adjusting the strategy. Both the preset human weight step size and the preset machine weight step size are positive numbers, which can be set in advance according to the operator's needs. The preset human weight step size can be three times the preset machine weight step size to highlight the importance of human intervention in decision-making.

[0081] According to an embodiment of the present invention, when the human-machine decision position gap value is less than or equal to a preset position gap threshold, the change in machine weight and the change in human weight are obtained based on the preset position gap threshold, the human-machine decision position gap value, and a reference index. This includes: when the second reference index is less than or equal to a preset second reference index, determining the change in machine weight as a preset machine weight step size and the change in human weight as a negative value of the preset human weight step size; when the second reference index is greater than the preset second reference index and the first reference index is less than or equal to a preset first reference index, determining the change in machine weight as a preset machine weight step size and the change in human weight as a preset human weight step size; and when the second reference index is greater than the preset second reference index and the first reference index is greater than the preset first reference index, determining the change in machine weight as a negative value of the preset machine weight step size and the change in human weight as a negative value of the preset human weight step size.

[0082] When the human-machine decision-making position gap is small, the deviation between the eye-tracking control position data (representing the human control intention) and the machine control position data (representing the machine prediction result) is small. In this case, a classification discussion is conducted based on the values ​​of the first and second reference indices. When the intention fit is low but comfort is high, comfort does not need to be improved; the focus should be on ensuring the long-term improvement of intention fit. Therefore, both human control and machine control weights are increased simultaneously. When intention fit is high but comfort is low, it indicates that automatic control is reliable but the operator's burden is heavy. In this case, machine control weights are increased, and human control weights are decreased. When both intention fit and comfort are low, the small human-machine decision-making position gap indicates that automatic control is trending towards matching the human intention, and the fit will passively improve. In this case, the focus should be on improving comfort; therefore, the strategy is to increase machine control weights and decrease human control weights. When both intention fit and comfort are high, both human control and machine control weights are decreased to maintain the stability of the control system.

[0083] In embodiments of the present invention, the endoscope can also be controlled in the depth direction based on eye movement data or head movement data. A second gaze point can be determined based on eye movement data or head movement data. The operator can control the movement trajectory of the second gaze point through eye or head movements, and determine the cumulative duration for which the second gaze point falls within a first preset area on the virtual screen, i.e., the duration for which the operator gazes at a certain area. In embodiments of the present invention, the cumulative duration refers to the length of time the second gaze point continuously falls within the first preset area. When the second gaze point leaves the range of the first preset area, the cumulative duration for which the second gaze point falls within the first preset area is zero. When the cumulative duration exceeds a preset time threshold, it indicates that the operator has gazed at the first preset area for a sufficient period of time, and further control commands can be issued to the endoscope to control its movement. The preset time threshold is a time constant predefined by the operator, such as 1 second, 2 seconds, 6 seconds, 10 seconds, etc. Depth-direction control of the endoscope can be achieved through depth position data, which refers to the amount of movement of the endoscope within the body in the depth direction. Depth direction refers to the direction in which the endoscope lens enters or exits the body. In embodiments of the present invention, a first gaze point can be used as a second gaze point; alternatively, head orientation data can be determined based on head motion data, and the intersection of the extended line of the line of sight corresponding to the head orientation and the reference plane can be determined based on the head orientation data, with this intersection point serving as the second gaze point. In embodiments of the present invention, when controlling the depth direction of the endoscope, motion control of the endoscope based on shared control position data is suspended.

[0084] In embodiments of the present invention, multiple safety constraint mechanisms can be set, including maximum displacement boundary, speed limit and direction lock failure mechanism. That is, when the amount of movement of the endoscope inside the human body reaches the preset safety threshold, the movement control of the endoscope is stopped. The operator needs to reconfirm before the movement control of the endoscope can be performed again, thereby avoiding the risk caused by the operator unconsciously maintaining the posture.

[0085] In embodiments of the present invention, when the target eye position corresponding to the eye movement control position data matches the initial eye position, the endoscope is determined to maintain its current position; when the target head position corresponding to the posture data matches the initial head position, the endoscope is determined to maintain its current posture. The initial eye position and initial head position are determined by position signals collected by a position sensor. The initial eye position and initial head position are absolute position data of the eyeball and head determined by the operator in real three-dimensional space. For example, the initial eye position and initial head position can be position data determined by the operator in a naturally relaxed state; when the operator turns their head to the right from the initial head position, the target head position does not match the initial head position, so the endoscope turns to the right accordingly; when the operator returns their head from the right-turn angle to the left to a naturally relaxed state, the target head position corresponding to the posture data matches the initial head position, so the endoscope does not turn to the left accordingly but maintains its current posture. Similarly, when the target eye position matches the initial eye position, the endoscope maintains its current position.

[0086] During endoscopic control, the operator may exhibit minute head and eye movements. However, these minute movements do not indicate a desire for precise control of the endoscope; rather, they are normal and unavoidable physiological activities. Therefore, threshold detection can be performed on posture data or eye movement control position data. Control commands are only sent to the endoscope when the posture data or eye movement control position data exceeds a preset threshold. This prevents excessively frequent endoscope posture adjustments, ensuring the stability of the surgical field and the safety of the procedure.

[0087] Figure 2 A flowchart illustrating an endoscope sharing control method based on human body signals and game theory according to another embodiment of the present invention is shown.

[0088] like Figure 2As shown, after the operator puts on the head-mounted display, they first activate the interactive control function to acquire eye movement data and head movement data. Based on the head movement data, posture data is obtained, and after posture threshold detection, incremental posture movement commands are generated. Eye movement data can be used to generate reference indicators and for cooperative game theory to obtain human control weights and machine control weights. Based on the eye movement data, eye movement control position data is obtained by analyzing the movement trajectory of the first gaze point; based on the acquired endoscopic images, machine control position data is obtained through a target detection algorithm; through human control weights, machine control weights, eye movement control position data, and eye movement control position data, shared control position data is finally obtained; position threshold detection is performed on the shared control position data to generate incremental position movement commands. Based on the eye movement data, depth position data is obtained by analyzing the cumulative duration of gaze at the first preset area; time threshold detection is performed on the depth position data to generate depth movement commands. Based on the incremental posture movement commands, incremental position movement commands, and depth movement commands, endoscopic movement commands are obtained to control the movement of the endoscope and acquire endoscopic images. The acquired endoscopic images are displayed on the head-mounted display device.

[0089] Figure 3 A schematic diagram of an endoscope system architecture according to an embodiment of the present invention is shown.

[0090] like Figure 3 As shown, the endoscope system architecture includes a motion control layer, an execution layer, an operator layer, and a master control layer. The operator views endoscopic images acquired by the endoscope through an MR headset interactive environment and performs head and eye movements. In a decoupled control space, the eye-tracking module and the cooperative game module obtain shared control position data based on eye movement data, and this shared control position data, along with depth position data, together form the position data. The head-tracking module obtains posture data based on head movement data. Based on the position and posture data, the motion controller controls the endoscope to perform corresponding movements. Simultaneously, based on stereoscopic vision streaming and data processing and communication, the MR headset interactive environment enables data interaction between the virtual screens of the left and right cameras and the optical camera sensors.

[0091] Figure 4 A flowchart of a cooperative game based on an embodiment of the present invention is shown.

[0092] like Figure 4As shown, after the cooperative game begins, the human-machine decision-making position gap value and reference indicators are first input. Then, the human-machine decision-making position gap value is compared with the preset gap threshold, the second reference indicator is compared with the preset second reference indicator, and the first reference indicator is compared with the preset first reference indicator to obtain the output decision. When the second reference indicator is greater than the preset second reference indicator, the corresponding comfort level is higher; when the first reference indicator is greater than the preset first reference indicator, the corresponding intention fit is higher. Human dominance represents an increase in human control weight; machine dominance represents an increase in machine control weight; human relinquishing power represents a decrease in human control weight; machine relinquishing power represents a decrease in machine control weight.

[0093] Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0094] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. An endoscope sharing control method based on human body signals and game theory, characterized in that, include: Acquire eye movement data; Based on the eye movement data, eye movement control position data for characterizing artificial control intentions are determined through gaze heatmap analysis. Based on the eye movement data, reference indicators for characterizing intention fit and human control burden are obtained through quantitative analysis of gaze, eye opening degree and pupil size. By analyzing the positional relationship between the target object and the endoscope in the endoscopic image, machine control position data is obtained to characterize the expected movement distance of the endoscope, wherein the endoscopic image is acquired in real time by the endoscope; Based on the machine control position data, the eye-tracking control position data, and the reference indicators, shared control position data is determined through a cooperative game strategy. The movement of the endoscope is controlled based on the shared control position data.

2. The method according to claim 1, characterized in that, Based on the eye movement data, through quantitative analysis of gaze, eye opening degree, and pupil size, reference indicators for characterizing intention fit and human control burden are obtained, including: Based on the eye movement data, a first reference index for characterizing the degree of intent matching is obtained through quantitative analysis of the degree of deviation and concentration of gaze. Based on the eye movement data, a second reference index for characterizing the burden of manual control is obtained through quantitative analysis of eye opening degree and pupil size. Based on the first reference indicator and the second reference indicator, a reference indicator is obtained.

3. The method according to claim 2, characterized in that, Based on the eye movement data, a first reference index for characterizing intention matching is obtained through quantitative analysis of the degree of deviation and concentration of gaze, including: Based on the eye movement data, a gaze heatmap is generated through kernel density estimation. Statistical analysis was performed on the gaze heatmap to determine the visual field deviation value; The gaze heatmap is analyzed based on the Gaussian kernel density values ​​of each pixel to determine visual concentration. The visual field deviation value and the visual concentration are weighted and calculated to obtain the first reference index.

4. The method according to claim 3, characterized in that, The statistical analysis of the gaze heatmap to determine the visual field deviation value includes: By performing a weighted calculation on the gaze heatmap, the weighted center of the gaze heatmap is determined as the first coordinate point; The geometric center of the gaze heatmap is determined as the second coordinate point; The field of view deviation value is determined based on the difference between the first coordinate point and the second coordinate point.

5. The method according to claim 3, characterized in that, The process of analyzing the gaze heatmap based on the Gaussian kernel density values ​​of each pixel to determine visual concentration includes: The Gaussian kernel density values ​​of each pixel in the gaze heatmap are normalized to obtain the probability distribution values ​​of each corresponding pixel. Based on a preset probability density value, multiple target pixels with probability distribution values ​​greater than the preset probability density value are selected from the gaze heatmap; Visual concentration efficiency is determined based on the ratio of the number of the multiple target pixels to the total number of pixels in the gaze heatmap. The spatial compactness is determined based on the ratio of the convex hull area formed by the multiple target pixels to the total area of ​​the region corresponding to the gaze heatmap. The visual concentration is determined based on the visual concentration efficiency and the spatial compactness.

6. The method according to claim 2, characterized in that, Based on the eye movement data, a second reference index for characterizing the burden of manual control is obtained through quantitative analysis of eye opening degree and pupil size, including: Based on the eye movement data, the operator's eye opening degree and pupil size are determined through image processing and feature extraction; The duration of endoscopy in human-dominated mode is obtained as the human-dominated duration. The human-dominated mode is an endoscopy control mode in which the human control weight of the endoscope is greater than the machine control weight of the endoscope. The human control weight and the machine control weight are determined by a cooperative game strategy based on machine control position data, eye-tracking control position data and reference indicators. Based on the eye opening degree, the eye opening degree score is determined by Gaussian decay function analysis; Based on the pupil size, a pupil size score is determined through Gaussian penalty function analysis; Based on the aforementioned duration of human-led instruction, a human-led score is determined through ramp function analysis. The eye opening score, pupil size score, and artificially induced score are weighted and calculated to determine the second reference index.

7. The method according to claim 2, characterized in that, The process of determining shared control position data based on the machine control position data, the eye-tracking control position data, and the reference indicators through a cooperative game strategy includes: Based on the machine control position data and the eye-tracking control position data, the human-machine decision position gap value is obtained; Obtain the initial machine control weights and initial manual control weights for the current endoscope control cycle; Based on the preset position gap threshold, the human-machine decision position gap value, and the reference index, the machine weight change and the human weight change are obtained. The change in machine weight is added to the initial machine control weight, and the change in manual weight is added to the initial manual control weight to obtain the machine control weight and the manual control weight. The machine control position data and the eye-tracking control position data are weighted based on the machine control weight and the human control weight to determine the shared control position data.

8. The method according to claim 7, characterized in that, The process of obtaining machine weight change and human weight change based on a preset position gap threshold, the human-machine decision position gap value, and the reference index includes: When the human-machine decision-making position difference value is greater than a preset position difference threshold: When the first reference index is less than or equal to the preset first reference index, the change in machine weight is determined to be the negative of the preset machine weight step size, and the change in manual weight is determined to be the preset manual weight step size. When the first reference indicator is greater than the preset first reference indicator and the second reference indicator is greater than the preset second reference indicator, the machine weight change is determined to be the negative of the preset machine weight step size, and the manual weight change is determined to be the negative of the preset manual weight step size. When the first reference indicator is greater than the preset first reference indicator and the second reference indicator is less than or equal to the preset second reference indicator, the change in machine weight is determined to be the preset machine weight step size value, and the change in manual weight is determined to be the preset manual weight step size value.

9. The method according to claim 7, characterized in that, The process of obtaining machine weight change and human weight change based on a preset position gap threshold, the human-machine decision position gap value, and the reference index includes: When the human-machine decision position difference value is less than or equal to a preset position difference threshold: When the second reference index is less than or equal to the preset second reference index, the change in machine weight is determined to be the preset machine weight step size value, and the change in manual weight is the negative value of the preset manual weight step size value. When the second reference indicator is greater than the preset second reference indicator and the first reference indicator is less than or equal to the preset first reference indicator, the change in machine weight is determined to be the preset machine weight step size value, and the change in manual weight is determined to be the preset manual weight step size value. When the second reference indicator is greater than the preset second reference indicator and the first reference indicator is greater than the preset first reference indicator, the change in machine weight is determined to be the negative of the preset machine weight step size, and the change in manual weight is determined to be the negative of the preset manual weight step size.

10. The method according to claim 1, characterized in that, The process of determining eye movement control position data to characterize artificial control intentions through gaze heatmap analysis based on the eye movement data includes: Based on the eye movement data, a gaze heatmap is generated through kernel density estimation. By performing a weighted calculation on the gaze heatmap, the weighting center of the gaze heatmap is determined as the first gaze point; The movement trajectory of the first gaze point is determined based on the eye movement data; Based on the movement trajectory of the first gaze point, determine the displacement of the first gaze point; Based on a first preset coefficient, the displacement of the first gaze point is mapped to obtain eye-tracking control position data.

Images