Eye tracking control method and system for multi-screen display screen

Abstract

The present invention provides an eye tracking control method and system for a multi-screen display screen. A first host and a second host connected via a network are respectively connected to the display screen by signals, wherein the first host is used to control the display of images within a first range of the display screen, and the second host is used to control the display of images within a second range of the display screen; the relative position relationship between the second range and the first range is obtained; the line of sight position of the user visually viewing the display screen is detected; when the line of sight position is within the second range, the first host converts the line of sight position according to the relative position relationship, and provides the converted line of sight position information to the second host, and the second host performs eye tracking control according to the converted line of sight position information. The present invention allows multiple hosts connected to the same display screen to achieve cross-host control through a single eye tracking module, thereby achieving the purpose of shared screen eye movement control.

Description

Technical Field

[0001] The present invention relates to a multi-image display technology, and in particular to an eye tracking control method and system for a multi-image display screen. Background Art

[0002] Eye tracking (or gaze tracking) is a technology that measures the gaze point of a person's eyes and the degree of movement relative to the head. It generally uses some non-contact optical methods to measure eye movement. Usually the eyes reflect infrared light, and cameras or other specially designed optical sensors are used for measurement. The measurement information is then analyzed to extract eye rotation information from the changes in the reflected light spot.

[0003] Traditional eye tracking control can only be applied to a single host screen when using display screens. With the development of large and wide screens, multiple hosts are often used in multiple screens such as picture by picture (PBP) or picture in picture (PIP). Although the screens can be shared, eye tracking control cannot be used across hosts. Summary of the invention

[0004] In response to the above technical problems, the present invention provides an eye tracking control method and system for a multi-screen display screen, which allows multiple source hosts connected to the same display screen to achieve cross-host control through a single eye tracking module.

[0005] The present invention provides an eye tracking control method for a multi-screen display screen, comprising: step 1, connecting a first host and a second host connected via a network to the display screen respectively by signal, the first host and the second host sharing the display screen in a multi-screen display mode, wherein the first host is used to control the display of a picture within a first range of the display screen, and the second host is used to control the display of a picture within a second range of the display screen; step 2, obtaining the relative position relationship between the second range and the first range; step 3, detecting the line of sight position of the user visually viewing the display screen; step 4, when the line of sight position is within the second range, the first host converts the line of sight position according to the relative position relationship, and provides the converted line of sight position information to the second host, and the second host performs eye tracking control according to the converted line of sight position information.

[0006] Preferably, after step 3, the method further includes: when the sight line position is within the first range, the first host directly performs eye tracking control according to the detected sight line position.

[0007] Preferably, in step 1, the first range and the second range correspond to a multi-image display mode of picture-outside-picture or picture-in-picture.

[0008] Preferably, in step 1, the first host signal is connected to an eye tracking device.

[0009] Furthermore, step 2 also includes the following steps: the first host controls the display screen to display a first calibration picture for calibrating the eye tracking device within the first range, and the first host requests the second host through the network to control the display screen to display a second calibration picture for calibrating the eye tracking device within the second range; the eye tracking device obtains first calibration data corresponding to the first host based on the first calibration picture, and the eye tracking device obtains second calibration data corresponding to the second host based on the second calibration picture; the first host obtains the relative position relationship between the second range and the first range based on the first calibration data and the second calibration data.

[0010] The present invention also provides an eye tracking control system for a multi-screen display screen, comprising a display screen, a first host, a second host and an eye tracking device; the first host is connected to the second host via a network, the first host and the second host are respectively connected to the display screen by signal, and the eye tracking device is connected to the first host by signal; when the first host and the second host share the display screen in a multi-screen display mode, the first host is used to control the display of images within a first range of the display screen, and the second host is used to control the display of images within a second range of the display screen; the first host is used to obtain a relative position relationship between the second range and the first range, and the eye tracking device is used to detect the line of sight position of the user when viewing the display screen, and when the line of sight position is within the second range, the first host converts the line of sight position according to the relative position relationship, and provides the converted line of sight position information to the second host, and the second host performs eye tracking control according to the converted line of sight position information.

[0011] Preferably, when the sight line position is within the first range, the first host directly performs eye tracking control according to the sight line position detected by the eye tracking device.

[0012] Preferably, the first range and the second range correspond to a multi-picture display mode of picture-outside-picture or picture-in-picture.

[0013] Preferably, when the first host and the second host share the display screen in a multi-screen display mode, the first host controls the display screen to display a first calibration picture for calibrating the eye tracking device within the first range, and the first host requests the second host through the network to control the display screen to display a second calibration picture for calibrating the eye tracking device within the second range.

[0014] Furthermore, the eye tracking device obtains first calibration data corresponding to the first host based on the first calibration screen, the eye tracking device obtains second calibration data corresponding to the second host based on the second calibration screen, and the first host obtains the relative position relationship between the second range and the first range based on the first calibration data and the second calibration data.

[0015] The eye tracking control method and system of the multi-screen display screen of the present invention connects two hosts through a network, the first host is used to control the display of the picture within a first range of the display screen, and the second host is used to control the display of the picture within a second range of the display screen, and there is a relative position relationship between the second range and the first range; the line of sight position in the multi-screen display (such as PBP or PIP) mode is detected, and the main control host (that is, the host equipped with an eye tracking device) selectively converts the line of sight position according to the line of sight position and the relative position relationship between the second range and the first range, thereby achieving the purpose of eye movement control of the shared screen. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 The present invention is a flow chart of an eye tracking control method for a multi-image display screen.

[0018] Figure 2 The figure is a schematic diagram of an eye tracking control system for a multi-image display screen according to the present invention.

[0019] Figure 3 FIG. 4 is a schematic diagram showing the principle of an eye tracking control system according to an embodiment of the present invention.

[0020] Figure 4 FIG. 4 is a schematic diagram showing the principle of an eye tracking control system according to another embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to provide a further understanding of the purpose, structure, features, and functions of the present invention, the following detailed description is given in conjunction with the embodiments.

[0022] Please refer to Figure 1 and Figure 2 , Figure 1 is a flow chart of an eye tracking control method for a multi-image display screen of the present invention, Figure 2The figure is a schematic diagram of an eye tracking control system for a multi-image display screen according to the present invention.

[0023] An eye tracking control method for a multi-image display screen of the present invention comprises the following steps.

[0024] Step S1, respectively connect a first host and a second host connected via a network to a display screen, and the first host and the second host share the display screen in a multi-screen display mode, wherein the first host is used to control the display of images within a first range of the display screen, and the second host is used to control the display of images within a second range of the display screen. Preferably, in step S1, the first range and the second range correspond to a multi-screen display mode of picture-outside-picture (PBP) or picture-in-picture (PIP).

[0025] Step S2, obtaining the relative position relationship between the second range and the first range.

[0026] Step S3, detecting the user's sight line position when viewing the display screen;

[0027] Step S4, when the line of sight position is within the second range, the first host converts the line of sight position according to the relative position relationship, and provides the converted line of sight position information to the second host, and the second host performs eye tracking control according to the converted line of sight position information.

[0028] Preferably, step S5 is further included after step S3, when the sight line position is within the first range, the first host directly performs eye tracking control according to the detected sight line position.

[0029] Preferably, in step S1, the first host signal is connected to an eye tracking device. Through the eye tracking device, it can be determined whether the sight line position is within the second range or within the first range.

[0030] In step 2 of obtaining the relative positional relationship between the second range and the first range, the following steps are also included: the first host controls the display screen to display a first calibration picture for calibrating the eye tracking device within the first range, and the first host requests the second host to control the display screen to display a second calibration picture for calibrating the eye tracking device within the second range through the network; the eye tracking device obtains first calibration data corresponding to the first host based on the first calibration picture, and the eye tracking device obtains second calibration data corresponding to the second host based on the second calibration picture; the first host obtains the relative positional relationship between the second range and the first range based on the first calibration data and the second calibration data.

[0031] In practical applications, the host computer with the eye tracking module installed is the master terminal, and the controlled terminal is required to start calibration at the same time. The calibration process of the controlled terminal uses the eye tracking module on the master terminal. From the perspective of the master terminal, the calibration process of the controlled terminal can use the eye tracking module to obtain the mark position in the above-mentioned second calibration screen, and the controlled terminal only turns on the calibration screen. The user visually calibrates the mark position prompted in the calibration screen to obtain the visual position data corresponding to the display screen. After the calibration is completed, the position of the screen viewed by the user is used as the visual position. When the user visually views the screen range corresponding to the controlled terminal, the master terminal converts it into a coordinate position and provides it to the controlled terminal for control. The above coordinates can be horizontal and vertical coordinates with the pixel arrangement direction of the display screen as the axis direction, but the present invention is not limited to this.

[0032] In the above-mentioned coordinate conversion process, since the resolution of the eye tracking module (such as 800*600) may not match the resolution of the display screen (such as 1920*1080), the actual position coordinates can be further generated by interpolation, but the present invention is not limited to this.

[0033] Please continue to see Figure 2 The eye tracking control system 1 of a multi-screen display screen of the present invention comprises a display screen 10, a first host 11, a second host 12 and an eye tracking device 13; the first host 11 is connected to the second host 12 via a network, the first host 11 and the second host 12 are respectively connected to the display screen 10 by signal, and the eye tracking device 13 is connected to the first host 11 by signal. In this embodiment, the first host 11 can be regarded as a master terminal, and the second host 12 can be regarded as a controlled terminal.

[0034] When the first host 11 and the second host 12 share the display screen 10 in a multi-screen display mode, the first host 11 is used to control the display of images within a first range of the display screen 10 , and the second host 12 is used to control the display of images within a second range of the display screen 10 .

[0035] The first host 11 is used to obtain the relative position relationship between the second range and the first range, and the eye tracking device 13 is used to detect the line of sight position of the user visually viewing the display screen 10. When the line of sight position is within the second range, the first host 11 converts the line of sight position according to the relative position relationship and provides the converted line of sight position information to the second host 12. The second host 12 performs eye tracking control according to the converted line of sight position information.

[0036] Preferably, when the sight line position is within the first range, the first host 11 directly performs eye tracking control according to the sight line position detected by the eye tracking device 13 .

[0037] See also Figure 3 , Figure 3Schematic diagram of the principle of an eye tracking control system according to an embodiment of the present invention. In this embodiment, the first range 110 and the second range 120 correspond to a multi-screen display mode of picture-outside-picture. When the first host 11 and the second host 12 share the display screen 10 in a multi-screen display mode, the first host 11 controls the display screen 10 to display a first calibration screen 111 for calibrating the eye tracking device 13 in the first range 110, and the first host 11 requests the second host 12 to control the display screen 10 to display a second calibration screen 121 for calibrating the eye tracking device 13 in the second range 120 through the network. The eye tracking device 13 obtains the first calibration data corresponding to the first host 11 according to the first calibration screen 111, the eye tracking device 13 obtains the second calibration data corresponding to the second host 12 according to the second calibration screen 121, and the first host 11 obtains the relative position relationship between the second range 120 and the first range 110 according to the first calibration data and the second calibration data.

[0038] See also Figure 4 , Figure 4 Schematic diagram of the principle of an eye tracking control system according to another embodiment of the present invention. In this embodiment, the first range 110' and the second range 120' correspond to a multi-screen display mode of picture-in-picture. When the first host 11 and the second host 12 share the display screen 10 in a multi-screen display mode, the first host 11 controls the display screen 10 to display a first calibration screen 111' for calibrating the eye tracking device 13 within the first range 110', and the first host 11 requests the second host 12 to control the display screen 10 to display a second calibration screen 121' for calibrating the eye tracking device 13 within the second range 120' through the network. The eye tracking device 13 obtains the first calibration data corresponding to the first host 11 according to the first calibration screen 111', and the eye tracking device 13 obtains the second calibration data corresponding to the second host 12 according to the second calibration screen 121', and the first host 11 obtains the relative position relationship between the second range 120' and the first range 110' according to the first calibration data and the second calibration data.

[0039] The above process of each embodiment of the present invention may be included in the calibration process before the eye tracking is enabled. For example, the control end (such as the first host 11) uses the eye tracking device 13 to capture data, and the screen sequentially indicates the position where the user should look to perform calibration, thereby detecting the relative position relationship between the screen of the controlled end (such as the second host 12) and the screen of the control end (such as the first host 11).

[0040] by Figure 3 For example, when the eye tracking device 13 obtains the first calibration data corresponding to the first host 11 according to the first calibration screen 111, the first calibration screen 111 sequentially reminds the user of the positions to be focused on, such as Figure 3The four marked positions of the four corners of the edge of the first calibration image 111 (which can be defined as the first upper left, first upper right, first lower left and first lower right) are obtained and the position information of the four positions is used as the first calibration data. As mentioned above, the resolution of the eye tracking device 13 (such as 800*600) may not match the resolution of the display screen (such as 1920*1080), and the actual position coordinates can be further generated by, for example, interpolation.

[0041] Similarly, when the eye tracking device 13 obtains the second calibration data corresponding to the second host 12 according to the second calibration screen 121, the second calibration screen 121 sequentially reminds the user of the positions to be focused on, such as Figure 3 Four marking positions at the four edge corners (which can be defined as the second upper left, second upper right, second lower left and second lower right) of the second calibration frame 121 are obtained and position information of the four positions is obtained as the second calibration data.

[0042] The first host 11 determines the first range 110 according to the above-mentioned first calibration data, and determines the second range 120 according to the above-mentioned second calibration data. The first range 110 is expressed as a first coordinate range, and the second range 120 is expressed as a second coordinate range. The first host 11 can determine the relative position relationship between the second range 120 and the first range 110 according to the first coordinate range and the second coordinate range.

[0043] by Figure 4 For example, when the eye tracking device 13 obtains the first calibration data corresponding to the first host 11 according to the first calibration screen 111', the first calibration screen 111' sequentially reminds the user of the positions to be focused on, such as Figure 4 The four marked positions of the four corners of the edge of the first calibration image 111' (which can be defined as the first upper left, first upper right, first lower left and first lower right) are obtained and the position information of the four positions is used as the first calibration data. As mentioned above, the resolution of the eye tracking device 13 (such as 800*600) may not match the resolution of the display screen (such as 1920*1080), and the actual position coordinates can be further generated by, for example, interpolation.

[0044] Similarly, when the eye tracking device 13 obtains the second calibration data corresponding to the second host 12 according to the second calibration screen 121 ′, the second calibration screen 121 ′ sequentially reminds the user of the positions to be focused on, such as Figure 3 Four marking positions at the four edge corners (defined as the second upper left, second upper right, second lower left and second lower right) of the second calibration frame 121' are obtained and position information of the four positions is obtained as the second calibration data.

[0045] The first host 11 determines the first range 110' according to the first calibration data, and determines the second range 120' according to the second calibration data. The first range 110' is, for example, represented by a first coordinate range, and the second range 120' is represented by a second coordinate range. The first host 11 can determine the relative position relationship between the second range 120' and the first range 110' according to the first coordinate range and the second coordinate range.

[0046] It should be noted that in Figure 4 In the picture-in-picture mode shown, the display screen of the controlled end (such as the second host 12) is included in the display screen range of the main control end (such as the first host 11). At this time, as long as the detected line of sight position is within the second range 120' of the controlled end, it is necessary to use the main control end to perform coordinate conversion, and the controlled end performs eye tracking control based on the converted line of sight position information. If the detected line of sight position is within the first range 110' other than this, the coordinate system of the main control end is used normally for eye tracking control.

[0047] The present invention can adopt the known eye tracking calibration and control technology. Taking the common desktop eye tracker as an example, its working principle is roughly as follows: function fitting is performed according to the corresponding relationship between the position coordinates of the gaze point on the screen and the pupil-corneal reflection vector. The calibration process is the process of solving the unknown parameters in the fitting function. After obtaining these parameters, the pupil-corneal reflection vector can be mapped to the coordinates of the user's gaze point on the computer screen. Its specific image processing and function fitting technology are not the focus of the present invention, so they are not described here. The focus of the present invention is to provide an eye tracking control method for a shared screen, a single screen has multiple screen display ranges corresponding to multiple hosts, eye tracking correction is performed for each screen display range corresponding to each host, by detecting real-time eye movement parameters, judging the host corresponding to the screen area viewed by the eye, converting the eye movement parameters into corresponding coordinates, and transmitting the corresponding coordinates to the corresponding host.

[0048] The eye tracking control method and system of the multi-screen display screen of the present invention connects two hosts through a network, the first host is used to control the display of the picture within a first range of the display screen, and the second host is used to control the display of the picture within a second range of the display screen, and there is a relative position relationship between the second range and the first range; the line of sight position in the multi-screen display (such as PBP or PIP) mode is detected, and the main control host (that is, the host equipped with an eye tracking device) selectively converts the line of sight position according to the line of sight position and the relative position relationship between the second range and the first range, thereby achieving the purpose of eye movement control of the shared screen.

[0049] The present invention has been described by the above-mentioned relevant embodiments, but the above-mentioned embodiments are only examples for implementing the present invention. In addition, the technical features involved in the different embodiments of the present invention described above can be combined with each other as long as they do not conflict with each other. It must be pointed out that the disclosed embodiments do not limit the scope of the present invention. On the contrary, changes and modifications made without departing from the spirit and scope of the present invention are all within the scope of patent protection of the present invention.

Claims

1. An eye tracking control method for a multi-screen display screen, characterized in that include: Step 1, connecting a first host and a second host connected via a network to a display screen respectively by signals, and the first host and the second host share the display screen in a multi-screen display mode, wherein the first host is used to control the display of a picture within a first range of the display screen, and the second host is used to control the display of a picture within a second range of the display screen; Step 2, obtaining a relative position relationship between the second range and the first range; Step 3, detecting the user's sight line position when viewing the display screen; Step 4, when the sight line position is within the second range, the first host converts the sight line position according to the relative position relationship, and provides the converted sight line position information to the second host, and the second host performs eye tracking control according to the converted sight line position information; Wherein in step 1, the first host signal is connected to the eye tracking device; in step 2, the following steps are also included: the first host controls the display screen to display a first calibration picture for calibrating the eye tracking device within the first range, and the first host requests the second host to control the display screen to display a second calibration picture for calibrating the eye tracking device within the second range through the network; The eye tracking device obtains first calibration data corresponding to the first host according to the first calibration image, and the eye tracking device obtains second calibration data corresponding to the second host according to the second calibration image; The first host obtains a relative position relationship between the second range and the first range according to the first calibration data and the second calibration data.

2. The eye tracking control method according to claim 1, characterized in that: After step 3, the method further includes: when the sight line position is within the first range, the first host directly performs eye tracking control according to the detected sight line position.

3. The eye tracking control method according to claim 1, characterized in that: In step 1, the first range and the second range correspond to a multi-picture display mode of picture-outside-picture or picture-in-picture.

4. An eye tracking control system for a multi-screen display, characterized in that include: A display screen, a first host, a second host, and an eye tracking device; The first host is connected to the second host via a network, the first host and the second host are respectively connected to the display screen by signal, and the eye tracking device is connected to the first host by signal; When the first host and the second host share the display screen in a multi-screen display mode, the first host is used to control the display of images within a first range of the display screen, and the second host is used to control the display of images within a second range of the display screen; The first host is used to obtain the relative position relationship between the second range and the first range, and the eye tracking device is used to detect the sight line position of the user when viewing the display screen. When the sight line position is within the second range, the first host converts the sight line position according to the relative position relationship and provides the converted sight line position information to the second host. The second host performs eye tracking control according to the converted sight line position information. When the first host and the second host share the display screen in a multi-screen display mode, the first host controls the display screen to display a first calibration image for calibrating the eye tracking device within the first range, and the first host requests the second host through a network to control the display screen to display a second calibration image for calibrating the eye tracking device within the second range; The eye tracking device obtains first calibration data corresponding to the first host based on the first calibration screen, the eye tracking device obtains second calibration data corresponding to the second host based on the second calibration screen, and the first host obtains the relative position relationship between the second range and the first range based on the first calibration data and the second calibration data.

5. The eye tracking control system according to claim 4, characterized in that: When the sight line position is within the first range, the first host directly performs eye tracking control according to the sight line position detected by the eye tracking device.

6. The eye tracking control system according to claim 4, characterized in that: The first range and the second range correspond to a multi-picture display mode of picture-outside-picture or picture-in-picture.

Images