Interaction control method, device, electronic equipment and system based on eye movement tracking

Abstract

The embodiment of the invention provides a control method, device and system based on eye movement tracking and electronic equipment, and the method comprises the steps: obtaining the interaction intention of a user based on the visual behavior characteristics of the user, generating a control instruction based on the interaction intention of the user, and controlling the interaction with an interaction screen based on the control instruction. Physical contact with a screen or use of an operation screen of external equipment such as a mouse and a keyboard is not needed, so that the complexity of interactive operation is reduced, the convenience and smoothness of the interactive operation are improved, and the interactive experience is improved; due to the fact that participation of external equipment is not needed, limitation on hardware conditions is reduced, the application range of the interaction control method based on eye movement tracking is expanded, and hardware cost is reduced.

Description

Technical Field

[0001] The present application belongs to the field of human-computer interaction, human factors and ergonomics technology, and specifically relates to an interactive control method, device, electronic device and system based on eye tracking. Background Art

[0002] Eye tracking technology locates the pupil position of the eye to obtain the user's gaze point on the screen or device, and performs further operations based on the user's gaze point. At present, eye tracking technology is widely used in fields such as medical rehabilitation and human-computer interaction.

[0003] In the prior art, the position changes of the pupil are captured by eye movement acquisition devices to determine the user's gaze point on the screen or device. However, the user often needs to rely on external control devices (such as a handle, mouse or keyboard, etc.) to interact with the operation controls corresponding to the gaze point.

[0004] However, using the existing technology requires users to rely on external operating devices to complete the operation, and the interaction method is not very convenient. Summary of the invention

[0005] The embodiments of the present application provide a control method, device, system and electronic device based on eye tracking to improve the convenience of interaction.

[0006] In a first aspect, an embodiment of the present application provides an interactive control method based on eye tracking, comprising:

[0007] Acquire the user's interaction intention based on the user's visual behavior data, where the interaction intention is used to indicate the operation that the user expects to perform on the interactive screen;

[0008] Based on the user's interaction intention, generating a control instruction for the interactive screen;

[0009] The interactive screen is controlled based on the control instruction.

[0010] Optionally, the visual behavior data includes: eye movement data; the eye movement data includes: blink data and gaze data;

[0011] The acquiring the user's interaction intention based on the user's visual behavior data includes:

[0012] Based on the correspondence between the eye movement data and the interaction intention, obtaining the interaction intention corresponding to the eye movement data;

[0013] The generating a control instruction for the interactive screen based on the user's interactive intention includes:

[0014] determining an operation object based on the gaze data;

[0015] Based on the user's interaction intention and the operation object, a control instruction for the interaction screen is generated, wherein the control instruction is used to instruct to perform an operation corresponding to the interaction intention on the operation object.

[0016] Optionally, the acquiring the interaction intention corresponding to the eye movement data based on the correspondence between the eye movement data and the interaction intention includes:

[0017] Based on the first corresponding relationship between the blink data and the first interaction intention, obtaining the first interaction intention corresponding to the blink data;

[0018] and / or,

[0019] Based on a second correspondence between the gaze data and the second interaction intention, obtaining a second interaction intention corresponding to the gaze data;

[0020] The blink data includes: blink frequency, blink interval, blink number, blink duration and / or eye closure duration during blinking;

[0021] The interaction intention includes: single click, double click, scroll, long press, select, turn page, display definition, open page, reduce page and / or close page;

[0022] The operation objects include: buttons, scroll bars, links and / or text content.

[0023] Optionally, the gaze data includes: a gaze point;

[0024] The determining the operation object based on the gaze data comprises:

[0025] Determining coordinate information of the gaze point on the interactive screen;

[0026] Based on the coordinate information of the gaze point on the interactive screen, determining the position information of the gaze point on the display interface displayed on the interactive screen;

[0027] The operation object is determined based on the position information of the gaze point on the display interface displayed on the interactive screen and the display elements on the display interface.

[0028] Optionally, also include:

[0029] The interaction intention confirmation information is displayed, and the interaction intention confirmation information is used to request the user to confirm the correctness of the interaction intention recognition.

[0030] Optionally, the method further includes: visually displaying the gaze point on the display interface based on position information of the gaze point on the display interface displayed on the interactive screen; and / or,

[0031] The gaze data further includes: an eye movement trajectory and an eye movement direction; the method further includes: visually displaying the eye movement trajectory along the eye movement direction on the display interface in the form of animation.

[0032] Optionally, obtaining the user's interaction intention based on the user's visual behavior data includes:

[0033] Based on the interactive page displayed on the interactive screen, visual behavior data is obtained, wherein the interactive page includes: an interactive page for human-machine control, an interactive page for personnel ability assessment, and / or an interactive page for personnel ability training;

[0034] Formatting the visual behavior data to obtain formatted visual behavior data;

[0035] Based on the formatted visual behavior data, obtaining the user's interaction intention;

[0036] Wherein, obtaining visual behavior data includes:

[0037] Acquire eye image data; preprocess the eye image data to obtain preprocessed eye image data; recognize the preprocessed eye image data to acquire the visual behavior data;

[0038] The preprocessing of the eye image to obtain the preprocessed eye image data includes: performing denoising, image contrast enhancement, image brightness adjustment and data synchronization on the eye image to obtain the preprocessed eye image data:

[0039] Among them, the denoising process includes: sliding mean filtering, sliding median filtering and / or wavelet transform; the data synchronization process includes: aligning the timestamps of the eye image data and the content of the display interface displayed on the interactive screen based on the same time reference.

[0040] Optionally, also include:

[0041] Display a first related interface of a first evaluation mode and a second related interface of a second evaluation mode; wherein the first evaluation mode is used for a user to interact with the first related interface through visual behavior to input first evaluation information, and the second evaluation mode is used for a user to input second evaluation information by filling out a questionnaire;

[0042] Obtaining first evaluation information input by the user through interaction with the first related interface in the form of visual behavior, and second evaluation information input by filling out a questionnaire; obtaining a first evaluation result of the first evaluation mode based on the first evaluation information, and obtaining a second evaluation result of the second evaluation mode based on the second evaluation information;

[0043] Based on the first evaluation result and the second evaluation result, feedback information is generated and output.

[0044] In a second aspect, an embodiment of the present application provides an interactive control device based on eye tracking, comprising:

[0045] An intention recognition module, used to obtain the user's interaction intention based on the user's visual behavior data, where the interaction intention is used to indicate the operation that the user expects to perform on the interactive screen;

[0046] An instruction generation module, used to generate a control instruction for the interactive screen based on the user's interaction intention;

[0047] A control module is used to control the interactive screen based on the control instruction.

[0048] Optionally, the visual behavior data includes: eye movement data; the eye movement data includes: blink data and gaze data;

[0049] The intention recognition module is specifically used to obtain the interaction intention corresponding to the eye movement data based on the correspondence between the eye movement data and the interaction intention;

[0050] The instruction generation module is specifically used to determine the operation object based on the gaze data; based on the user's interaction intention and the operation object, generate a control instruction for the interactive screen, and the control instruction is used to instruct to perform the operation corresponding to the interaction intention on the operation object.

[0051] Optionally, the intention recognition module is specifically used to obtain a first interaction intention corresponding to the blink data based on a first correspondence between the blink data and the first interaction intention; and / or, based on a second correspondence between the gaze data and the second interaction intention, obtain a second interaction intention corresponding to the gaze data;

[0052] The blink data includes: blink frequency, blink interval, blink number, blink duration and / or eye closure duration during blinking;

[0053] The interaction intention includes: single click, double click, scroll, long press, select, turn page, display definition, open page, reduce page and / or close page;

[0054] The operation objects include: buttons, scroll bars, links and / or text content.

[0055] Optionally, the gaze data includes: a gaze point;

[0056] The instruction generation module is specifically used to determine the coordinate information of the gaze point on the interactive screen; based on the coordinate information of the gaze point on the interactive screen, determine the position information of the gaze point on the display interface displayed on the interactive screen; based on the position information of the gaze point on the display interface displayed on the interactive screen, and the display elements on the display interface, determine the operation object.

[0057] Optionally, the intention recognition module is also used to display the interaction intention confirmation information, and the interaction intention confirmation information is used to request the user to confirm the correctness of the interaction intention recognition.

[0058] Optionally, the control module is further used to control the position information of the gaze point on the display interface displayed on the interactive screen, and visually display the gaze point on the display interface; and / or,

[0059] The gaze data further includes: an eye movement trajectory and an eye movement direction; the method further includes: visually displaying the eye movement trajectory along the eye movement direction on the display interface in the form of animation.

[0060] Optionally, the intention recognition module is specifically used to obtain visual behavior data based on an interactive page displayed on the interactive screen, wherein the interactive page includes: an interactive page for human-machine control, an interactive page for personnel capability assessment, and / or an interactive page for personnel capability training; formatting the visual behavior data to obtain formatted visual behavior data; based on the formatted visual behavior data, obtaining the user's interaction intention;

[0061] The intention recognition module is specifically used to obtain eye image data; preprocess the eye image data to obtain preprocessed eye image data; recognize the preprocessed eye image data to obtain the visual behavior data; perform denoising, image contrast enhancement, image brightness adjustment and data synchronization on the eye image to obtain preprocessed eye image data:

[0062] Among them, the denoising process includes: sliding mean filtering, sliding median filtering and / or wavelet transform; the data synchronization process includes: aligning the timestamps of the eye image data and the content of the display interface displayed on the interactive screen based on the same time reference.

[0063] Optionally, it also includes: an evaluation module, which is used to display a first relevant interface of a first evaluation mode and a second relevant interface of a second evaluation mode; wherein, the first evaluation mode is used for users to interact with the first relevant interface through visual behavior to input first evaluation information, and the second evaluation mode is used for users to input second evaluation information by filling out a questionnaire; obtain the first evaluation information input by the user through interacting with the first relevant interface through visual behavior, and the second evaluation information input by filling out a questionnaire; obtain a first evaluation result of the first evaluation mode based on the first evaluation information, and obtain a second evaluation result of the second evaluation mode based on the second evaluation information; generate and output feedback information based on the first evaluation result and the second evaluation result.

[0064] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the interactive control method based on eye tracking as described in any one of the first aspects are implemented.

[0065] In a fourth aspect, an embodiment of the present application provides an interactive control system based on eye tracking, comprising: an interactive control device based on eye tracking and an eye movement data acquisition device as described in the second aspect; or, an electronic device and an eye movement data acquisition device as described in the third aspect.

[0066] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the interactive control method based on eye tracking as described in any one of the first aspects are implemented.

[0067] In a sixth aspect, an embodiment of the present application provides a computer program product, which, when executed by a processor, implements the steps of the interactive control method based on eye tracking as described in any one of the first aspects.

[0068] The control method, device, system and electronic device based on eye tracking provided in this embodiment obtain the user's interaction intention based on the user's visual behavior characteristics, generate control instructions based on the user's interaction intention, and control the interaction with the interactive screen based on the control instructions. There is no need to physically touch the screen or use external devices such as a mouse and keyboard to operate the screen, which reduces the complexity of the interactive operation, improves the convenience and fluency of the interactive operation, and improves the interactive experience; since no external device is required, the limitation on hardware conditions is reduced, the application scope of the interactive control method based on eye tracking is expanded, and the hardware cost is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A flowchart of an interactive control method based on eye tracking provided in this application;

[0070] Figure 2 A flowchart of another interactive control method based on eye tracking provided in this application;

[0071] Figure 3 A flowchart of another interactive control method based on eye tracking provided in this application;

[0072] Figure 4 A flowchart of another interactive control method based on eye tracking provided by the present application;

[0073] Figure 5 A flowchart of another interactive control method based on eye tracking provided by the present application;

[0074] Figure 6 A flowchart of another interactive control method based on eye tracking provided in an embodiment of the present application;

[0075] Figure 7 A flowchart of another interactive control method based on eye tracking provided in an embodiment of the present application;

[0076] Figure 8 A flowchart of another interactive control method based on eye tracking provided in an embodiment of the present application;

[0077] Fig. 9 A flowchart of another interactive control method based on eye tracking provided in an embodiment of the present application;

[0078] Fig.10 A structural schematic diagram of an interactive control device based on eye tracking is provided for an embodiment of the present application;

[0079] Fig.11 A schematic diagram of an interactive control system based on eye tracking is provided for an embodiment of the present application;

[0080] Fig.12 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0081] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0082] The terms "first", "second", etc. of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of one type, and the number of objects is not limited, for example, the first object can be one or more. In addition, "or" in the present application represents at least one of the connected objects. For example, "A or B" covers three schemes, namely, Scheme 1: including A but not including B; Scheme 2: including B but not including A; Scheme 3: including both A and B. The character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0083] The term "indication" in this application can be a direct indication (or explicit indication) or an indirect indication (or implicit indication). A direct indication can be understood as the sender explicitly informing the receiver of specific information, operations to be performed, or request results in the sent indication; an indirect indication can be understood as the receiver determining the corresponding information according to the indication sent by the sender, or making a judgment and determining the operation to be performed or the request result according to the judgment result.

[0084] The embodiment of the present application provides an interactive control method based on eye tracking, which obtains the user's interactive intention based on the user's visual behavior data, generates control instructions according to the user's interactive intention, and operates on the interactive screen based on the control instructions, thereby achieving the purpose of interaction without physical contact with the interface, improving the convenience of interaction of the interactive control method based on eye tracking, and improving the interactive experience. Since no external equipment is required, the limitation on hardware conditions is reduced, and the application scope of the interactive control method based on eye tracking is expanded.

[0085] The technical solution of the present application is described below with reference to several specific embodiments.

[0086] Figure 1 A flowchart of an interactive control method based on eye tracking provided in this application is shown in FIG. Figure 1 As shown, the method of this embodiment is as follows:

[0087] S100: Acquire the user's interaction intention based on the user's visual behavior characteristics.

[0088] The interaction intention is used to indicate the operation that the user expects to perform on the interactive screen.

[0089] The operations include but are not limited to at least one of the following: single click, double click, long press, selection, scrolling, page turning, displaying annotations, opening a page, zooming in a page, zooming out a page, and closing a page.

[0090] Visual behavior characteristics include: eye movement data.

[0091] Eye movement data includes but is not limited to blink data and gaze data.

[0092] The blink data includes at least one of the following: blink frequency, blink interval, number of blinks, duration of blinks, and duration of eye closure during blinking, etc.

[0093] Among them, blink frequency: refers to the number of blinks per unit time. Blink interval: refers to the time between two consecutive blinks, that is, the length of time from the end of one blink to the beginning of the next blink. Number of blinks: refers to the number of blinks within a preset time. Blink duration: refers to the duration of a continuous blink process. Continuous blinks can be defined as the time interval between two blinks being less than the preset time length. Eye closure duration during blinking: refers to the length of time the eyes are closed during a blink.

[0094] The gaze data includes at least one of the following: gaze point, gaze frequency, gaze duration, eye movement trajectory, and eye movement direction.

[0095] Among them, the fixation point refers to the specific location of the interactive screen where the user is currently looking, which can be represented by coordinates. The fixation duration refers to the length of time the user keeps looking at a fixation point. The eye movement trajectory refers to the path of the user's eyes moving on the interactive screen. The eye movement direction refers to the direction in which the user's eyes move.

[0096] Based on the user's visual behavior characteristics, a possible implementation method for obtaining the user's interaction intention is as follows:

[0097] A correspondence between the eye movement data and the interaction intention may be established, and based on the correspondence, the interaction intention corresponding to the eye movement data may be obtained.

[0098] Among them, a possible implementation method is: a first correspondence between blink data and a first interaction intention can be defined, and the user's interaction intention can be obtained based on the first correspondence; that is, when the blink data meets a first preset condition, the operation corresponding to the first interaction intention is triggered.

[0099] For example, the first preset condition may be a preset blinking time, for example, blinking continuously for 3-5 seconds, and the operation corresponding to the first interaction intention may be a single-click operation. When it is detected that the user blinks continuously for 3-5 seconds, it is considered that the user's interaction intention is to perform a single-click operation.

[0100] The first preset condition may also be a preset number of consecutive blinks, and the duration of eye closure during each blink meets a preset duration, for example, two consecutive blinks, and the duration of eye closure during each blink lasts for 3-5 seconds. The operation corresponding to the first interaction intention may also be a double-click operation. When it is detected that the user blinks twice in a row, and the duration of eye closure during each blink lasts for 3-5 seconds, it is considered that the user's interaction intention is to perform a double-click operation.

[0101] The first preset condition may also be that the eye closing duration during a blink is greater than a preset duration, for example, the eye closing duration during a blink lasts for 15 seconds, and the operation corresponding to the first interaction intention may also be closing the page. When it is detected that the user closes his eyes for 15 seconds during a blink, it is considered that the user's interaction intention is to execute the operation of closing the page.

[0102] Another possible implementation method is: a second correspondence between gaze data and the second interaction intention can be defined, and the user's interaction intention can be obtained based on the second correspondence, that is, when the gaze data meets the second preset condition, the operation corresponding to the second interaction intention is triggered.

[0103] For example, the second preset condition may be that the user moves his eyes in a first direction, for example, the user moves his eyes in the up and down directions, and the operation corresponding to the second interaction intention may be controlling the display interface to scroll in the vertical direction; when it is detected that the user moves his eyes in the up and down directions, it is considered that the user's interaction intention is to execute the scrolling of the display interface in the vertical direction.

[0104] The second preset condition may also be that the user moves his eyes in a second direction, for example, the user moves his eyes in the left-right direction, and the operation corresponding to the second interaction intention may be to control the display interface to turn the page horizontally, or to switch to the previous display interface or the next display interface. When it is detected that the user moves his eyes in the left-right direction, it is considered that the user's interaction intention is to execute horizontal page turning, or to switch to the previous display interface or the next display interface.

[0105] The second preset condition may also be that the fixation time is greater than a preset threshold. For example, in a reading scenario, if the user's fixation time on a certain word or a certain sentence is greater than 3-5 seconds, the operation corresponding to the second interaction intention may be to explain the content of the word or the sentence. When it is detected that the user's fixation time on a certain word or a certain sentence is greater than 3-5 seconds, it is considered that the user's interaction intention is to explain the word or the sentence.

[0106] If the user's operation fails to meet the first preset condition or the second preset condition, the user's eye movement behavior will continue to be monitored and the next valid instruction will be waited for. If two consecutive operations are invalid, the system will provide feedback to inform the user that the operation is invalid and prompt the user with the correct operation method, such as prompting the user through text or sound.

[0107] S200: Generate a control instruction for the interactive screen based on the user's interaction intention.

[0108] Among them, one possible implementation method is: based on the user's interaction intention, a control instruction for the interactive screen can be directly generated. For example, if the user's interaction intention is to close the page, a control instruction for closing the page can be directly generated. For another example, if the user's interaction intention is to turn the page, a control instruction for turning the page can be directly generated. For another example, if the user's interaction intention is to scroll the page, a control instruction for scrolling the page can be directly generated.

[0109] Another possible implementation is that the specific operation object needs to be determined first, specifically: the operation object is determined based on the gaze data; based on the user's interaction intention and the operation object, a control instruction for the interactive screen is generated, wherein the control instruction is used to instruct the operation object to perform the operation corresponding to the interaction intention. The operation object includes but is not limited to: a scroll bar, a button, a link, text content or other controls. The control instruction includes but is not limited to: scrolling, single-clicking, double-clicking, and long-pressing operations.

[0110] Optionally, the determination of the operation object can be achieved in the following ways: determining the coordinate information of the gaze point on the interactive screen, determining the position information of the gaze point on the display interface displayed on the interactive screen based on the coordinate information of the gaze point on the interactive screen, determining the operation object based on the position information of the gaze point on the display interface displayed on the interactive screen and the display elements on the display interface.

[0111] For example, if the user's current gaze point on the interactive screen is on button A, it can be determined that the operation object is button A. If the user continues to blink for 3-5 seconds, it can be determined that the interaction intention is a single-click operation. Then, when it is detected that the user's gaze point on the interactive screen is on button A and continues to blink for 3-5 seconds, the generated control instruction is to click button A.

[0112] For another example, if the user's current gaze point on the interactive screen is on the display area of ​​link B, it can be determined that the operation object is link B. If the user continues to blink for 3-5 seconds, it can be determined that the interaction intention is a single-click operation. Then, when it is detected that the user's gaze point on the interactive screen is on link B and continues to blink for 3-5 seconds, a control instruction is generated to click link B.

[0113] For another example, if the user's current gaze point on the interactive screen is on scroll bar C, it can be determined that the operation object is the scroll bar; if the user moves his eyes from bottom to top, it can be determined that the interaction intention is to scroll the display interface upward in the vertical direction. Then, when it is detected that the user's gaze point on the interactive screen is on scroll bar C and moves his eyes from bottom to top, a control instruction is generated to scroll the display interface upward.

[0114] For another example, if the user's current gaze point on the interactive screen is a certain word or a certain sentence, it can be determined that the operation object is a certain word or a certain sentence; if the user's gaze point is on the certain word or a certain sentence for more than 3-5 seconds, it can be determined that the interaction intention is to display the interpretation, then if it is detected that the user's gaze point is on a certain word or a certain sentence for more than 3-5 seconds, a control instruction is generated to display the interpretation corresponding to the content of the word or the sentence.

[0115] S300: Controlling the interactive screen based on the control instruction.

[0116] For example, if the control instruction is to click button A, the user's operation of clicking button A is simulated, and the button A is clicked in response. For another example, if the control instruction is to click link B, the user's operation of clicking link B is simulated, and the link B is clicked to enter the page corresponding to link B. For another example, if the control instruction is to scroll down the display interface, the display interface is scrolled down. For another example, if the control instruction is to display the interpretation corresponding to the content of the word or sentence, the corresponding interpretation is displayed next to the word or sentence.

[0117] In this embodiment, the user's interaction intention is obtained based on the user's visual behavior characteristics, and a control instruction is generated based on the user's interaction intention, and the interaction with the interactive screen is controlled based on the control instruction. There is no need to physically touch the screen or use an external device such as a mouse or keyboard to operate the screen, which reduces the complexity of the interactive operation, improves the convenience and fluency of the interactive operation, and improves the interactive experience; since no external device is required, the limitation on hardware conditions is reduced, the application scope of the interactive control method based on eye tracking is expanded, and the hardware cost is reduced.

[0118] Figure 2 A flowchart of another interactive control method based on eye tracking provided in this application, Figure 2 is Figure 1 On the basis of the embodiment shown, further, before S200, it also includes: S100':

[0119] S100': displaying the interaction intention confirmation information, wherein the interaction intention confirmation information is used to request the user to confirm the correctness of the interaction intention recognition.

[0120] For example, the interaction intention is a single-click operation, and the interaction intention confirmation information can be: whether to perform the single-click operation. The interaction intention confirmation information can be displayed next to the gaze point. The scenario is that the user's current gaze point on the interactive screen is in the display area of ​​link B, and the operation object can be determined to be link B. The user blinks continuously for 3-5 seconds, and it can be determined that the interaction intention is a single-click operation. After the interaction intention is determined, the interaction intention confirmation information is displayed. The interaction intention confirmation information is used to prompt whether to perform the single-click operation. If so, please close your eyes for 2 seconds to confirm. If not, please shift your gaze point or automatically ignore the operation after 2 seconds. Figure 3 It is only a schematic diagram of the interaction intention confirmation information. As long as it can play the role of confirming the interaction intention, the embodiment of the present application does not limit the specific display method.

[0121] Optionally, the interaction intention confirmation information is not displayed after each interaction intention is determined, but only displayed for interaction intentions determined in some special scenarios. For example, some software icons may be at the edge of the interactive screen, or the icon size is small. In this case, the interaction intention confirmation information can be displayed after obtaining the user's interaction intention, such as "Do you need to open a certain software?" In this way, the interaction intention confirmation information can be displayed in a targeted manner, reducing interference with user behavior and improving user experience.

[0122] In this embodiment, by displaying the interaction intention confirmation information, the interaction intention confirmation information is used to request the user to confirm the correctness of the interaction intention recognition, which can improve the accuracy of the user interaction intention recognition and thus improve the user experience.

[0123] Figure 3 A flowchart of another interactive control method based on eye tracking provided in this application is shown below. Figure 3 is Figure 1 or Figure 2 On the basis of the embodiment shown, further, before S100, the method further includes:

[0124] S001: Visually displaying the gaze point on the display interface based on the position information of the gaze point on the display interface.

[0125] Optionally, the visualization implementation methods include but are not limited to the following methods:

[0126] The current gaze point is marked with a preset color and a preset shape. For example, the current gaze point is marked with a red circle, and the position of the red circle is dynamically updated as the gaze point moves.

[0127] In this embodiment, by visually displaying the gaze point on the display interface based on the position information of the gaze point on the display interface, the user can more intuitively understand that the system has recognized the user's line of sight, further improving the user experience, and can be used as auxiliary prompt information for operation. For example, when the user wants to click a button, when the visualized gaze point falls on the button, the user can continue to perform the visual behavior corresponding to the click operation, such as blinking for 3-5 seconds, thereby triggering the accurate operation of clicking the button, improving the effectiveness of the interaction and improving the efficiency of the interaction.

[0128] Figure 4 A flowchart of another interactive control method based on eye tracking provided in this application is shown below. Figure 4 is Figure 1-Figure 3 On the basis of the embodiment shown, further, the gaze data further includes: eye movement trajectory and eye movement direction; before S100, it also includes:

[0129] S002: Visually displaying the eye movement trajectory along the eye movement direction on the display interface in the form of animation.

[0130] Optionally, the eye movement trajectory can also be visualized on the display interface along the eye movement direction in the form of animation. For example, in a reading application, when a user gazes at text on an interactive screen, the user's gaze point is displayed in real time in the form of a dynamic cursor below the text as the user's gaze point moves, and the animation smoothly transitions from one gaze point to another, simulating a reading action.

[0131] In this embodiment, the eye movement trajectory is visually displayed along the eye movement direction on the display interface in the form of animation, which further improves the visualization effect and provides users with a more intuitive and natural interaction method through real-time display.

[0132] Optionally, based on the above embodiment, the user's gaze frequency and gaze duration can be displayed through a scatter plot, thereby intuitively showing the user's visual attention distribution in different screen areas. Optionally, the user's gaze concentration area can also be displayed through a heat map to highlight the part that the user is most interested in.

[0133] Figure 5 A flowchart of another interactive control method based on eye tracking provided in this application is shown below. Figure 5 Based on the above embodiments, a possible implementation of S100 is described. Figure 6 As shown:

[0134] S101: Acquire visual behavior data.

[0135] Optionally, visual behavior data may be acquired based on an interactive page displayed on the interactive screen, wherein the interactive page includes: an interactive page for human-machine control, an interactive page for personnel capability assessment, and / or an interactive page for personnel capability training.

[0136] Visual behavior data may include but is not limited to at least one of the following:

[0137] The continuous coordinates of the pupil center, which are recorded over time, reflect the position information of the pupil at each moment.

[0138] The speed and acceleration of eye movements provide dynamic information about eye movements.

[0139] Recording of blink events, including the start and end time of the blink and the duration of eye closure.

[0140] S102: Formatting the visual behavior data to obtain formatted visual behavior data.

[0141] The formatting of the visual behavior data includes but is not limited to at least one of the following processes:

[0142] Coordinate conversion processing: Convert pupil coordinates from the eye tracking device’s coordinate system to a coordinate system that matches the interactive screen.

[0143] Gaze point extraction: Determine the exact location where the user looks at the interactive screen, usually by calculating the average pupil position during the gaze.

[0144] After the visual behavior data is formatted, the formatted visual behavior data is obtained, including:

[0145] A series of gaze points, each of which includes the coordinates on the interactive screen and the gaze duration.

[0146] The path of the scan, including the starting point, end point, duration and speed of the scan.

[0147] Detailed information on blink events, including blink frequency, blink interval, number of blinks, blink duration, and eye closure duration during blinking.

[0148] The specific formatting method can be determined according to the application, and the formatting is performed in a format that matches the application.

[0149] S103: Acquire the user's interaction intention based on the formatted visual behavior data.

[0150] This step can be referred to the detailed description of S100 and will not be described here in detail.

[0151] In this embodiment, visual behavior data is acquired and formatted to obtain formatted visual behavior data. Based on the formatted visual behavior data, the user's interaction intention is acquired. The data format is unified through formatting, which is beneficial to data analysis, improves the efficiency of data analysis, and further improves the efficiency of acquiring the user's interaction intention.

[0152] In the above embodiment, the computer needs to obtain eye movement data and screen content: specifically, the user's eye movement data and the current content on the screen. Eye movement data covers the eye's gaze point, gaze duration, blink frequency, blink interval, etc., which are acquired in real time through an eye tracking device. At the same time, the computer also needs to identify the content displayed on the screen, such as icons, text, buttons, etc. These elements are target objects that the user may need to interact with through eye movement. The process of acquiring eye movement data is as follows: Figure 6 As shown. In addition, after obtaining the eye movement data and screen content, it is necessary to perform preset condition judgment. For eye movement data, check whether the gaze point is within the effective operation area of ​​the screen, and verify whether the blinking frequency and interval are within the normal range, such as 100 to 500 milliseconds. For screen content, the system will determine whether there are target objects that can be operated by eye movement, and check whether the position and size of these objects are suitable for eye tracking operations.

[0153] Figure 6 A flowchart of another interactive control method based on eye tracking provided in an embodiment of the present application is provided. Figure 6 is Figure 5 Based on the illustrated embodiment, a possible implementation of S101 is further described as follows:

[0154] S1011: Obtain user's eye image data.

[0155] Optionally, the user's eye image data can be obtained through an eye tracking device, where the eye tracking device can be, for example, an eye tracker, a camera, an infrared sensor, etc. The eye tracking device can capture the user's eye movement data in real time, including: pupil position, eye movement trajectory, blinking time, etc.

[0156] S1012: Preprocess the eye image data to obtain preprocessed eye image data.

[0157] The eye image is preprocessed by at least one of the following methods to obtain preprocessed eye image data:

[0158] 1) Denoising.

[0159] A possible implementation method: the eye image data generated by the user's normal eye behavior can be removed by denoising, and the eye image data generated by the user's normal eye behavior can be removed by dimensions such as blinking frequency, blinking duration, blinking timing and blinking pattern. For example: Under normal circumstances, people blink between 15-20 times per minute, so the eye image data generated within this blinking frequency range can be removed; for another example: the normal blinking duration is usually between 100-500 milliseconds, so the eye image data generated within this blinking duration range can be removed; for another example: when a blink is detected, the gaze point is in an irrelevant position, then the eye image data corresponding to the blinking behavior is removed; for another example: the blinking time is too long or too short, for example, more than 5 seconds, less than 1 second, are both considered to be blinks unrelated to interactive control, and the eye image data corresponding to the blinking behavior is removed.

[0160] The embodiment of the present application also performs denoising processing in at least one of the following ways:

[0161] Sliding mean filter: Smooths the data by calculating the average of a series of consecutive data points. In eye tracking, this filter can effectively remove high-frequency noise caused by rapid eye movements or device noise, thereby stabilizing the eye movement signal. In this way, random fluctuations in the data can be reduced, making the tracking of pupil position smoother and more accurate.

[0162] Sliding median filtering: Sliding median filtering is a nonlinear denoising method that works by replacing each data point with the median value of its surrounding neighborhood. Compared with mean filtering, median filtering is more effective for removing impulse noise because it is not affected by extreme values. In eye movement data, this can effectively remove outliers caused by instantaneous noise interference or atypical eye movements.

[0163] Wavelet transform: Wavelet transform is a multi-scale analysis technique that allows a signal to be decomposed into components of different frequencies. In eye tracking, wavelet transform can be used to identify and remove high-frequency noise components in the data while retaining low-frequency eye movement signals. In this way, wavelet transform can remove noise from eye movement data while retaining the natural fluctuations of eye movements and important visual behavior characteristics.

[0164] 2) Enhance image contrast processing.

[0165] Adjust the contrast of the image to make specific features in the image (such as the pupil) more obvious, which helps with subsequent feature extraction.

[0166] 3) Adjust the image brightness.

[0167] Adjust the brightness of the image to ensure that the lighting conditions of the image are suitable for pupil detection and eye tracking algorithms.

[0168] 4) Data Synchronization

[0169] The eye image data and the content of the display interface displayed on the interactive screen are time-stamped and aligned based on the same time reference:

[0170] Specifically, a possible implementation is as follows: data synchronization is performed based on timestamps to ensure that the collected eye movement data (eye image data) and other data (such as the content of the display interface displayed on the interactive screen) are synchronized in time. The timestamps are based on the same time reference, such as the system clock or the global standard time.

[0171] Eye movement data needs to be synchronized with other aspects of user operation, such as changes in screen content or other actions performed by the user. Each eye movement data point is given a timestamp, which is a marker that records the exact time when the data was collected. The timestamp ensures that the eye movement data can be aligned with other time-based data streams (such as video recordings, user input, etc.). In this way, the system can accurately know the exact time when each eye movement event occurred, so as to analyze the correlation between eye movements and other events.

[0172] Timestamp alignment is an operation performed after data collection, which ensures that the timestamps recorded by the eye tracking device are based on the same time reference as the timestamps of the system recording data. For example, if the system clock or global standard time (such as Coordinated Universal Time, UTC) is used as the time reference, then all devices and systems must first be synchronized to this reference. During the data collection process, each data point will be timestamped, and then through post-processing, these timestamps will be adjusted to ensure that they are aligned on a unified time axis. Thus, potential time deviations caused by system clock drift or other time reference changes can be eliminated.

[0173] S1013: Recognize the preprocessed eye image data to obtain the visual behavior data.

[0174] Among them, one possible implementation is as follows:

[0175] The pupil location in the image is identified through a pupil detection algorithm. This involves image processing techniques such as edge detection, which can be used to find the edge of the pupil in the eye image. Thresholding techniques may also be used to determine the exact boundaries of the pupil, where pupils detected within 50 to 600 milliseconds are considered valid. Through these steps, the system is able to extract the outline and features of the pupil, providing key information for subsequent eye movement analysis.

[0176] After the pupil position is identified, eye movement tracking will be performed. It involves continuous detection and tracking of pupil position information to determine the movement trajectory of the user's eye on the interactive screen. Eye movement tracking can be used to obtain the user's gaze point and gaze duration.

[0177] Blink data is used as part of interactive control. Blink data is collected through an eye tracking device, which records the time and duration of the blink. Blink compensation time refers to the time the system takes to process data immediately after detecting a blink event. For example, if the blink compensation time is set to 1 frame, it means that the system can quickly identify blinks and process them in a short time, thereby reducing the impact of blinks on eye movement data analysis.

[0178] In this embodiment, by acquiring eye image data of the user, preprocessing the eye image data to obtain preprocessed eye image data, recognizing the preprocessed eye image data, and acquiring the visual behavior data, the accuracy of the acquired visual behavior data can be improved.

[0179] Figure 7 A flowchart of another interactive control method based on eye tracking provided in an embodiment of the present application is provided. Figure 7 is Figure 6 On the basis of the embodiment shown, further, before S1011, the method further includes: S1010:

[0180] S1010: Calibrate the eye tracking device.

[0181] One possible implementation is: the user stands in front of a configured eye tracking device, which may include an image acquisition device (e.g., a non-contact eye tracker). The user calibrates by looking at a calibration card or specific calibration points displayed on the screen. During the calibration process, the eye tracking device records the position and movement of the user's eyes to establish a relationship between the user's eye movement and the screen coordinates.

[0182] After calibration, the user can move their head freely within a specific operating range, for example, 45 to 85 cm. Within this range, the user's eye movements can be tracked and recorded in real time by the system at a preset sampling frequency (for example, 60 Hz), which means that the system can capture 60 eye movement data points per second.

[0183] Optionally, the time of the image acquisition device and the eye tracking device may also be calibrated to ensure time synchronization and eliminate potential time deviations.

[0184] In this embodiment, the accuracy of the collected visual behavior data is improved by calibrating the eye tracking device.

[0185] Figure 8 A flowchart of another interactive control method based on eye tracking provided in an embodiment of the present application is provided. The interactive control method based on eye tracking is applied to an eye tracking system. Figure 8 On the basis of the above embodiments, further comprising:

[0186] S400: Displaying a first related interface of a first evaluation mode and a second related interface of a second evaluation mode.

[0187] The first evaluation mode is used for the user to interact with the first related interface through visual behavior to input first evaluation information.

[0188] The first related interface of the first evaluation mode may be a browser web page, an operating application, or a small game, etc. The user may select a corresponding evaluation scenario.

[0189] The second evaluation mode is a mode in which the user inputs the second evaluation information by filling out a questionnaire, and the second related interface is a related interface of the questionnaire.

[0190] S500: Acquire first evaluation information input by the user through interaction with the first related interface in the form of visual behavior, and second evaluation information input by the user through filling out a questionnaire.

[0191] S600: Obtain a first evaluation result of the first evaluation mode based on the first evaluation information, and obtain a second evaluation result of the second evaluation mode based on the second evaluation information.

[0192] Among them, the possible implementation method of obtaining the first evaluation result is as follows: according to the interactive response result, the accuracy, response speed and error rate of the eye tracking system in recognizing the eye movement data are obtained. More specifically, it can be: the accuracy of eye tracking, the accuracy and response speed of blink recognition, the accuracy and error rate of gaze point recognition, etc. For example: when the user's gaze point is related to a specific element (such as an icon, link or button, etc.) on the display interface, the recognized blink is considered a valid blink. If an invalid blink is recognized as a valid blink, it is considered that the blink recognition is wrong. The number of correct blink recognitions is compared with the total number of blink recognitions to obtain the accuracy of blink recognition. The error rate is calculated by recording the number of times the system makes mistakes in recognizing eye movement behaviors or gaze points, and comparing it with the total eye movement events.

[0193] A possible implementation method for obtaining the second evaluation result is as follows: based on a questionnaire filled out by the user, the user's evaluation data on the eye tracking system may be obtained.

[0194] S700: Generate and output feedback information based on the first evaluation result and the second evaluation result.

[0195] The feedback information may include but is not limited to: functions that need to be improved, such as the effectiveness of blinking behavior recognition, etc.

[0196] In this embodiment, by displaying the first relevant interface of the first evaluation mode and the second relevant interface of the second evaluation mode, obtaining the first evaluation result of the first evaluation mode and the second evaluation result of the second evaluation mode, and generating and outputting feedback information, the eye tracking system can be improved in a targeted manner and the performance of the eye tracking system can be optimized.

[0197] The technical solution provided in the embodiments of the present application can be applied to the field of medical rehabilitation, for example, rehabilitation therapy auxiliary applications and visual rehabilitation training applications.

[0198] The technical solution of the embodiment of the present application is described below with a detailed example. Fig. 9 As shown:

[0199] S901: Displaying a medical training interactive page on the interactive screen to obtain the user's blink data and gaze data.

[0200] Optionally, an interactive control system based on eye tracking applied to the field of medical rehabilitation will display a medical training interactive page on the interactive screen. The medical training interactive page can display the corresponding medical training interactive page for specific medical scenarios. For example, for patients with stroke or other brain injuries, the system will display a medical training interactive page containing multiple task icons, such as: "Reading Training", "Visual Search" and "Memory Game", etc. There is a number next to each icon. The user selects the task they want to perform by looking at the number and blinking. For visually impaired patients, the system will display an interface containing visual stimulation elements, such as flashing dots or moving graphics, to guide users to track these visual stimulation elements, thereby improving visual perception ability.

[0201] The interactive control system based on eye tracking can obtain the user's visual behavior data, including gaze data and blink data.

[0202] S902: Acquire the user's interaction intention based on the blink data and the gaze data.

[0203] For example, based on the gaze data, it can be determined that the user's gaze point is on the number corresponding to "reading training", and the blinking data includes blinking actions, so it can be determined that the user's interaction intention is to perform the task of "reading training".

[0204] S903: Generate a control instruction for the interactive screen based on the user's interaction intention.

[0205] For example, if it is determined based on step S902 that the user's interaction intention is to execute the task of "reading training", a control instruction for executing "reading training" is generated.

[0206] S904: Control the interactive screen based on the control instruction.

[0207] For example, based on the control instruction of running "reading training", an article is displayed for the user to read.

[0208] In this embodiment, by displaying a medical training interactive page on an interactive screen, the user's blink data and gaze data are obtained, and based on the blink data and gaze data, the user's interaction intention is obtained, and based on the user's interaction intention, a control instruction for the interactive screen is generated, and the interactive screen is controlled based on the control instruction. Thus, the user can perform rehabilitation training. This allows medical staff to analyze the patient's behavior pattern and better understand the patient's intention, thereby providing targeted treatment plans.

[0209] Fig.10 A structural diagram of an interactive control device based on eye tracking is provided for an embodiment of the present application, such as Fig.10 The example shown includes: an intention recognition module 1001, an instruction generation module 1002 and a control module 1003, wherein:

[0210] The intention recognition module 1001 is used to obtain the user's interaction intention based on the user's visual behavior data, where the interaction intention is used to indicate the operation that the user expects to perform on the interactive screen;

[0211] An instruction generation module 1002 is used to generate a control instruction for the interactive screen based on the user's interaction intention;

[0212] The control module 1003 is used to control the interactive screen based on the control instruction.

[0213] Optionally, the visual behavior data includes: eye movement data; the eye movement data includes: blink data and gaze data;

[0214] The intention recognition module 1001 is specifically used to obtain the interaction intention corresponding to the eye movement data based on the correspondence between the eye movement data and the interaction intention;

[0215] The instruction generation module 1002 is specifically used to determine the operation object based on the gaze data; based on the user's interaction intention and the operation object, generate a control instruction for the interactive screen, and the control instruction is used to instruct the operation object to perform the operation corresponding to the interaction intention.

[0216] Optionally, the intention recognition module 1001 is specifically used to obtain a first interaction intention corresponding to the blink data based on a first correspondence between the blink data and the first interaction intention; and / or, based on a second correspondence between the gaze data and the second interaction intention, obtain a second interaction intention corresponding to the gaze data;

[0217] The blink data includes: blink frequency, blink interval, blink number, blink duration and / or eye closure duration during blinking;

[0218] The interaction intention includes: single click, double click, scroll, long press, select, turn page, display definition, open page, reduce page and / or close page;

[0219] The operation objects include: buttons, scroll bars, links and / or text content.

[0220] Optionally, the gaze data includes: a gaze point;

[0221] The instruction generation module 1002 is specifically used to determine the coordinate information of the gaze point on the interactive screen; based on the coordinate information of the gaze point on the interactive screen, determine the position information of the gaze point on the display interface displayed on the interactive screen; based on the position information of the gaze point on the display interface displayed on the interactive screen, and the display elements on the display interface, determine the operation object.

[0222] Optionally, the intention recognition module 1001 is also used to display the interaction intention confirmation information, and the interaction intention confirmation information is used to request the user to confirm the correctness of the interaction intention recognition.

[0223] Optionally, the control module 1003 is further used to control the position information of the gaze point on the display interface displayed on the interactive screen, and visually display the gaze point on the display interface; and / or,

[0224] The gaze data further includes: an eye movement trajectory and an eye movement direction; the method further includes: visually displaying the eye movement trajectory along the eye movement direction on the display interface in the form of animation.

[0225] Optionally, the intention recognition module 1001 is specifically used to obtain visual behavior data based on the interactive page displayed on the interactive screen, wherein the interactive page includes: an interactive page for human-machine control, an interactive page for personnel ability assessment, and / or an interactive page for personnel ability training; formatting the visual behavior data to obtain formatted visual behavior data; based on the formatted visual behavior data, obtaining the user's interaction intention;

[0226] The intention recognition module 1001 is specifically used to obtain eye image data; preprocess the eye image data to obtain preprocessed eye image data; recognize the preprocessed eye image data to obtain the visual behavior data; perform denoising, image contrast enhancement, image brightness adjustment and data synchronization on the eye image to obtain preprocessed eye image data:

[0227] Among them, the denoising process includes: sliding mean filtering, sliding median filtering and / or wavelet transform; the data synchronization process includes: aligning the timestamps of the eye image data and the content of the display interface displayed on the interactive screen based on the same time reference.

[0228] Optionally, it also includes: an evaluation module, which is used to display a first relevant interface of a first evaluation mode and a second relevant interface of a second evaluation mode; wherein, the first evaluation mode is used for users to interact with the first relevant interface through visual behavior to input first evaluation information, and the second evaluation mode is used for users to input second evaluation information by filling out a questionnaire; obtain the first evaluation information input by the user through interacting with the first relevant interface through visual behavior, and the second evaluation information input by filling out a questionnaire; obtain a first evaluation result of the first evaluation mode based on the first evaluation information, and obtain a second evaluation result of the second evaluation mode based on the second evaluation information; generate and output feedback information based on the first evaluation result and the second evaluation result.

[0229] The device of this embodiment corresponds to the technical solution that can be used to execute the above-mentioned method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0230] The embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the following is achieved: Figure 1-Figure 9 The steps of the interactive control method based on eye tracking.

[0231] Fig.11 A schematic diagram of an interactive control system based on eye tracking is provided for an embodiment of the present application, including: an interactive control device based on eye tracking and an eye movement data acquisition device 1101 as described above. The interactive control device based on eye tracking includes: a data processing module 1102, a data display module 1103 and a behavior control module 1104. Alternatively, an electronic device and an eye movement data acquisition device as described above.

[0232] The embodiment of the present application also provides a computer-readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the following is implemented: Figure 1-Figure 9 The steps of the interactive control method based on eye tracking.

[0233] The present application also provides a computer program product, which, when executed by a processor, implements the following Figure 1-Figure 9 The steps of the interactive control method based on eye tracking.

[0234] The following specific reference Fig.12 , which shows a schematic diagram of the structure of an electronic device 1200 suitable for implementing the embodiment of the present application. The electronic device 1200 in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. The electronic device shown is only an example and should not bring any limitation to the functions and occupancy range of the embodiments of the present application.

[0235] like Fig.12 As shown, the electronic device 1200 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1210, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1220 or a program loaded from a storage device 1280 to a random access memory (RAM) 1230. In RAM 1230, various programs and data required for the operation of the electronic device are also stored. The processing device 1210, ROM 1220, and RAM 1230 are connected to each other via a bus 1240. An input / output (I / O) interface 1250 is also connected to the bus 1240.

[0236] Typically, the following devices may be connected to the I / O interface 1250: input devices 1260 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 1270 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1280 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1290. The communication device 1290 may allow the electronic device 1200 to communicate with other devices wirelessly or by wire to exchange data. Although the electronic device 1200 having various devices is shown, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead.

[0237] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1290, or installed from the storage device 1280, or installed from the ROM 1220. When the computer program is executed by the processing device 1210, the above-mentioned functions defined in the interactive control method based on eye tracking in the embodiment of the present application are executed.

[0238] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0239] In some embodiments, the user end and the management end may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0240] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0241] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0242] Acquire the user's interaction intention based on the user's visual behavior data, where the interaction intention is used to indicate the operation that the user expects to perform on the interactive screen;

[0243] Based on the user's interaction intention, generating a control instruction for the interactive screen;

[0244] The interactive screen is controlled based on the control instruction.

[0245] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0246] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0247] The units involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of a unit does not, in some cases, constitute a limitation on the unit itself.

[0248] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), etc.

[0249] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0250] According to one or more embodiments of the present application, the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is used to execute any eye tracking-based interactive control method provided in the embodiments of the present application.

[0251] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

[0252] In addition, although each operation is described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or to be performed in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the application. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0253] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An interactive control method based on eye tracking, characterized in that: include: Acquire the user's interaction intention based on the user's visual behavior data, where the interaction intention is used to indicate the operation that the user expects to perform on the interactive screen; Based on the user's interaction intention, generating a control instruction for the interactive screen; The interactive screen is controlled based on the control instruction.

2. The method according to claim 1, characterized in that The visual behavior data includes: eye movement data; the eye movement data includes: blink data and gaze data; The acquiring the user's interaction intention based on the user's visual behavior data includes: Based on the correspondence between the eye movement data and the interaction intention, obtaining the interaction intention corresponding to the eye movement data; The generating a control instruction for the interactive screen based on the user's interactive intention includes: determining an operation object based on the gaze data; Based on the user's interaction intention and the operation object, a control instruction for the interaction screen is generated, wherein the control instruction is used to instruct to perform an operation corresponding to the interaction intention on the operation object.

3. The method according to claim 2, characterized in that The acquiring the interaction intention corresponding to the eye movement data based on the correspondence between the eye movement data and the interaction intention includes: Based on the first corresponding relationship between the blink data and the first interaction intention, obtaining the first interaction intention corresponding to the blink data; and / or, Based on a second correspondence between the gaze data and the second interaction intention, obtaining a second interaction intention corresponding to the gaze data; The blink data includes: blink frequency, blink interval, blink number, blink duration and / or eye closure duration during blinking; The interaction intention includes: single click, double click, scroll, long press, select, turn page, display definition, open page, reduce page and / or close page; The operation objects include: buttons, scroll bars, links and / or text content.

4. The method according to claim 2, characterized in that: The gaze data includes: a gaze point; The determining the operation object based on the gaze data comprises: Determining coordinate information of the gaze point on the interactive screen; Based on the coordinate information of the gaze point on the interactive screen, determining the position information of the gaze point on the display interface displayed on the interactive screen; The operation object is determined based on the position information of the gaze point on the display interface displayed on the interactive screen and the display elements on the display interface.

5. The method according to any one of claims 1 to 4, characterized in that: Also includes: The interaction intention confirmation information is displayed, and the interaction intention confirmation information is used to request the user to confirm the correctness of the interaction intention recognition.

6. The method according to claim 2, characterized in that: The method further comprises: visually displaying the gaze point on the display interface based on the position information of the gaze point on the display interface displayed on the interactive screen; and / or, The gaze data further includes: an eye movement trajectory and an eye movement direction; the method further includes: visually displaying the eye movement trajectory along the eye movement direction on the display interface in the form of animation.

7. The method according to any one of claims 1 to 4, characterized in that: The acquiring the user's interaction intention based on the user's visual behavior data includes: Based on the interactive page displayed on the interactive screen, visual behavior data is obtained, wherein the interactive page includes: an interactive page for human-machine control, an interactive page for personnel ability assessment, and / or an interactive page for personnel ability training; Formatting the visual behavior data to obtain formatted visual behavior data; Based on the formatted visual behavior data, obtaining the user's interaction intention; Wherein, obtaining visual behavior data includes: Acquire eye image data; preprocess the eye image data to obtain preprocessed eye image data; recognize the preprocessed eye image data to acquire the visual behavior data; The preprocessing of the eye image to obtain the preprocessed eye image data includes: performing denoising, image contrast enhancement, image brightness adjustment and data synchronization on the eye image to obtain the preprocessed eye image data: Among them, the denoising process includes: sliding mean filtering, sliding median filtering and / or wavelet transform; the data synchronization process includes: aligning the timestamps of the eye image data and the content of the display interface displayed on the interactive screen based on the same time reference.

8. The method according to any one of claims 1 to 4, characterized in that: Also includes: Display a first related interface of a first evaluation mode and a second related interface of a second evaluation mode; wherein the first evaluation mode is used for a user to interact with the first related interface through visual behavior to input first evaluation information, and the second evaluation mode is used for a user to input second evaluation information by filling out a questionnaire; Obtaining first evaluation information input by the user through interaction with the first related interface in the form of visual behavior, and second evaluation information input by filling out a questionnaire; obtaining a first evaluation result of the first evaluation mode based on the first evaluation information, and obtaining a second evaluation result of the second evaluation mode based on the second evaluation information; Based on the first evaluation result and the second evaluation result, feedback information is generated and output.

9. An interactive control device based on eye tracking, characterized in that: include: An intention recognition module, used to obtain the user's interaction intention based on the user's visual behavior data, where the interaction intention is used to indicate the operation that the user expects to perform on the interactive screen; An instruction generation module, used to generate a control instruction for the interactive screen based on the user's interaction intention; A control module is used to control the interactive screen based on the control instruction.

10. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the interactive control method based on eye tracking as described in any one of claims 1 to 8 are implemented.

11. An interactive control system based on eye tracking, characterized in that: include: The interactive control device based on eye tracking and the eye movement data acquisition device according to claim 9; Alternatively, the electronic device and eye movement data acquisition device as described in claim 10.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the interactive control method based on eye tracking as described in any one of claims 1 to 8 are implemented.

Images