Human eye region of interest positioning system

Abstract

A system for locating a region of interest of a human eye comprises a camera device and a processing device, wherein the camera device is fixed on a display device and faces the front of a display screen of the display device, and the processing device is communicatively connected to the camera device. The processing device is used to execute a positioning step, and the positioning step is based on image recognition of the center point of an eyeball. Compared with the method of traditional telemetry equipment relying on eyeball recognition, the system can better characterize the direction of sight, and determines the region of interest on the screen by detecting the positional relationship of the center point of the eyeball relative to the center point of the eye socket, the distance between the human body and the screen, and the preset visual range, thereby improving the positioning accuracy and providing a good user experience. At the same time, the shortcomings of traditional glasses-type and helmet-type equipment are overcome.

Description

Technical Field

[0001] The present invention belongs to the technical field of human eye attention tracking, and in particular relates to a human eye region of interest positioning system. Background Art

[0002] With the digital transformation of domestic enterprises, various operational data visualization systems represented by enterprise management cockpits are being promoted on a large scale. The traditional method of specifying the area of ​​interest on the visualization system display screen through mouse and keyboard has become difficult to meet user needs. Perceiving the area of ​​interest by tracking the human eye can greatly improve the viewer's user experience.

[0003] Existing human eye tracking devices are generally divided into three product forms: glasses, helmets, and telemetry. Glasses-type eye tracking devices generally require users to wear special glasses with integrated image sensors and image recognition systems. Wearing such devices usually affects the human body's walking vision, and user acceptance is low. In addition, such wearable devices are usually made of materials with light volume and weight. Limited computing resources greatly reduce the accuracy of image recognition, and the user experience is poor. Helmet-type eye tracking devices generally require the subject to wear a special helmet. The helmet integrates eye movement and head movement recognition devices. The two compensate for each other and can more accurately identify the movement direction of the human eyeball in real time. Based on this, the visual navigation system can respond in real time to improve the intelligent experience of product users. However, this special helmet is heavy and inconvenient to use. It is generally only used in scenarios such as virtual reality such as AR / VR. Remote eye tracking devices usually include desktop high-precision eye trackers and portable remote eye trackers. The devices are usually integrated with a screen and can sense the real-time eye movement direction of the subject from a distance. The devices adopt a non-contact design and do not require additional wearable devices. However, these devices are only based on image recognition of the eyeballs, with low accuracy and poor user experience. Summary of the invention

[0004] Based on this, in order to solve the above technical problems, a system for locating regions of interest of human eyes is provided.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A human eye region of interest positioning system includes a camera device and a processing device, wherein the camera device is fixed on a display device and faces the front of a display screen of the display device, and the processing device is communicatively connected with the camera device, and the processing device is used to perform the following positioning steps:

[0007] S101, collecting an image in front of the display screen by using the camera device;

[0008] S102, identifying a human target from the image;

[0009] S103, further identifying an eye socket target and an eyeball target in the eye socket target from the identified human body target, and determining a positional relationship between a corresponding actual eyeball center point and an actual eye socket center point;

[0010] S104, determining the distance between the display screen and the actual human body in the z-axis direction by the camera device, and using the distance as the distance H from the center point of the actual eye socket to the display screen, wherein the z-axis is an axis in the front-to-back direction with the center of the retina as the origin, and the z-axis passes through the center point of the actual eye socket and the center point of the display screen;

[0011] S105, determining the angle between the actual eyeball center point and the z-axis according to the positional relationship between the actual eyeball center point and the actual eye socket center point;

[0012] S106, determining the position of the mapping point of the actual eyeball center on the display screen according to the angle and the distance H;

[0013] S107: Map and determine a region of interest centered on the mapping point on the display screen according to a preset visible range.

[0014] The present invention is based on image recognition of the center point of the eyeball. Compared with traditional telemetry equipment that relies on eyeball recognition, it can better characterize the direction of the line of sight. It determines the area of ​​interest on the screen by detecting the positional relationship of the center point of the eyeball relative to the center point of the eye socket, the distance between the human body and the screen, and the preset visible range, thereby improving the positioning accuracy and providing a good user experience. At the same time, it also overcomes the shortcomings of traditional glasses and helmet-type devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments:

[0016] Figure 1 It is a structural schematic diagram of the present invention;

[0017] Figure 2 It is a schematic diagram of the camera device of the present invention being fixed on a display device;

[0018] Figure 3 is a flow chart of the positioning steps of the present invention;

[0019] Figure 4 It is a schematic diagram of identifying the eye socket target and the eyeball target of the present invention;

[0020] Figure 5 A schematic diagram of determining the position of a mapping point of an actual eyeball center point on a display screen according to the present invention;

[0021] Figure 6A schematic diagram of determining the distance between the left and right boundaries of an area of ​​interest according to a preset horizontal viewing angle mapping of the present invention;

[0022] Figure 7 A schematic diagram of determining the upper and lower boundary distances of an area of ​​interest according to a preset vertical viewing angle mapping of the present invention;

[0023] Figure 8 It is a schematic diagram of the region of interest located on the display screen according to the present invention. DETAILED DESCRIPTION

[0024] like Figure 1 As shown, an embodiment of the present specification provides a system for locating a region of interest of a human eye, including a camera device 11 and a processing device 12 .

[0025] like Figure 2 As shown, the camera device 11 is fixed on the display device 20 of the visualization system and faces the front of the display screen 21 of the display device 20 , and the processing device 12 is communicatively connected with the camera device 11 .

[0026] In this embodiment, the camera device 11 includes a depth camera for sensing the horizontal distance between the display screen and the human body and an RGB camera for capturing images in front of the display screen. As an example, the camera device 11 can adopt an Intel Real Sense camera, which has a depth camera and a binocular RGB camera consisting of left and right infrared cameras and an infrared dot matrix projector.

[0027] like Figure 3 As shown, the processing device 12 is used to perform the following positioning steps:

[0028] S101 , capturing an image in front of a display screen through the camera device 11 .

[0029] S102. Identify human targets from images: Identify human targets from images through a pre-trained first target detection model. As an example, the first target detection model adopts an SSD network model.

[0030] The first target detection model can be trained by a large number of pictures representing the front of human targets, so that it can recognize human targets from the pictures input to the model. This method is a conventional technical method and is not limited in detail here.

[0031] S103, further identifying an eye socket target and an eyeball target in the eye socket target from the identified human body target, and determining the positional relationship between the corresponding actual eyeball center point and the actual eye socket center point:

[0032] a. Through the pre-trained second target detection model, the eye socket target and the eyeball target in the eye socket target are further identified from the identified human target. As an example, the second target detection model adopts the YOLOv5 network model.

[0033] The second target detection model can be trained with a large number of images with eye sockets and eyeballs, so that it can identify eye sockets and eyeball targets from the images input to the model. This method belongs to conventional technical methods. When the eye socket targets and eyeball targets are identified, the second target detection model will mark the corresponding detection box in the picture, such as Figure 4 As shown, in the image coordinate system, the two diagonal coordinates of the detection frame of the eye socket target are (x1, y1) and (x2, y2), and the two diagonal coordinates of the detection frame of the eyeball target are (x3, y3) and (x4, y4). This method is a conventional technical method and is not limited in detail here.

[0034] b. According to the two diagonal coordinates of the detection frame of the orbital target, determine the coordinates of the center point m of the orbital target. Figure 4 In the figure, the center point m is the origin, and the coordinates of the center point n of the eyeball target are determined according to the two diagonal coordinates of the detection frame of the eyeball target.

[0035] c. Determine the positional relationship between the center point of the eyeball target and the center point of the orbital target based on the center point coordinates of the orbital target and the center point coordinates of the eyeball target.

[0036] d. According to the positional relationship between the center point of the eyeball target and the center point of the eye socket target, determine the positional relationship between the actual center point of the eyeball and the actual center point of the eye socket.

[0037] It should be pointed out that the above-mentioned eyeball refers to the middle part of the front of the eyeball, which is composed of the cornea, iris and pupil. Compared with the eyeball, the eyeball can better represent the direction of sight. At the same time, the color of the eyeball is different from white, usually black, blue, etc. Therefore, it is easier to identify through image recognition, which improves the accuracy of recognition.

[0038] After the coordinates of the center point m and the center point n are determined, we know the distances between the center point m and the center point n in the x-axis direction and the y-axis direction respectively, that is, the positional relationship between the center point m and the center point n. Since the x-axis direction is actually equivalent to the left-right direction of the human body and the y-axis direction is actually equivalent to the up-down direction of the human body, determining the positional relationship between the center point m and the center point n is equivalent to determining the positional relationship between the actual center point of the eyeball and the actual center point of the eye socket.

[0039] S104, determine the distance between the display screen and the actual human body in the z-axis direction (i.e., the horizontal distance in the front-to-back direction) through the camera device 11, and use this distance as the distance H from the actual eye socket center point to the display screen, such as Figure 5 As shown, the z-axis is the axis in the front-to-back direction with the retinal center Q1 as the origin. The present invention is based on the user facing the display screen and only considers one eye. The actual eye socket center point Q2 of the single eye and the center point Q3 of the display screen are located on the same horizontal straight line, so the z-axis passes through the actual eye socket center point Q2 and the center point Q3 of the display screen.

[0040] S105, according to the positional relationship between the actual eyeball center point N and the actual eye socket center point Q2, determine the angle α between the actual eyeball center point and the z-axis, see Figure 5 .

[0041] like Figure 5 As shown, according to the positional relationship between the actual eyeball center point N and the actual eye socket center point Q2, the distance between N and Q2 can be determined. Since the distance between Q1 and Q2 is regarded as a known value, the angle α can be calculated.

[0042] S106: Determine the position of the mapping point M of the actual eyeball center point N on the display screen according to the angle α and the distance H. Figure 5 .

[0043] Since the distance between N and Q2 has been determined, and Q2 and Q3 are on the same horizontal line, the distance between M and Q3 can be calculated through the angle α and the distance H, and then the coordinates of M can be calculated through the coordinates of Q3.

[0044] S107, according to the preset visual range, the region of interest G centered at the mapping point M is mapped on the display screen 21, see Figure 8 .

[0045] The preset visible range is a preset visible range of the human eye. Based on the visible range, corresponding mapping can be performed according to the distance H, so as to determine the area of ​​interest centered on the mapping point M on the display screen. At the same time, the size of the area of ​​interest can be constrained by the preset visible range.

[0046] In this embodiment, the preset visual range is composed of a preset horizontal visual angle β and a preset vertical visual angle γ. Accordingly, step S107 further includes:

[0047] like Figure 6 As shown, according to the preset horizontal viewing angle β, the left and right boundary distances L1 of the region of interest are determined by mapping.

[0048] like Figure 7 As shown, according to the preset vertical viewing angle γ, the upper and lower boundary distances L2 of the region of interest are determined by mapping.

[0049] The position of the rectangular region of interest G is determined according to the position of the mapping point M, the left and right boundary distances L1, and the upper and lower boundary distances L2.

[0050] Usually, the human eye can clearly observe things within a range of 36 degrees horizontally and 20 degrees vertically. Therefore, in a preferred embodiment, the preset horizontal viewing angle β is set to 36 degrees, and the preset vertical viewing angle γ is set to 20 degrees. In this way, the size of the located area of ​​interest is more reasonable.

[0051] However, those skilled in the art should recognize that the above embodiments are only used to illustrate the present invention, and are not intended to limit the present invention. As long as they are within the spirit of the present invention, any changes or modifications to the above embodiments will fall within the scope of the claims of the present invention.

Claims

1. A human eye region of interest positioning system, characterized in that: The invention comprises a camera device and a processing device, wherein the camera device is fixed on the display device and faces the front of the display screen of the display device, and the processing device is connected to the camera device for communication, and the processing device is used to perform the following positioning steps: S101, collecting an image in front of the display screen by using the camera device; S102, identifying a human target from the image; S103, further identifying an eye socket target and an eyeball target in the eye socket target from the identified human body target, and determining a positional relationship between a corresponding actual eyeball center point and an actual eye socket center point; S104, determining the distance between the display screen and the actual human body in the z-axis direction by the camera device, and using the distance as the distance H from the center point of the actual eye socket to the display screen, wherein the z-axis is an axis in the front-to-back direction with the center of the retina as the origin, and the z-axis passes through the center point of the actual eye socket and the center point of the display screen; S105, determining the angle between the actual eyeball center point and the z-axis according to the positional relationship between the actual eyeball center point and the actual eye socket center point; S106, determining the position of the mapping point of the actual eyeball center on the display screen according to the angle and the distance H; S107: Map and determine a region of interest centered on the mapping point on the display screen according to a preset visible range.

2. A human eye region of interest positioning system according to claim 1, characterized in that: The camera device includes a depth camera for sensing the distance between the display screen and the actual human body in the z-axis direction and an RGB camera for collecting an image in front of the display screen.

3. A human eye region of interest positioning system according to claim 2, characterized in that: The camera device adopts an Intel realsense camera.

4. The human eye region of interest positioning system according to claim 1, characterized in that: The step S102 further comprises: A human target is identified from the image by using a pre-trained first target detection model.

5. A human eye region of interest positioning system according to claim 4, characterized in that: The first target detection model adopts the SSD network model.

6. The human eye region of interest positioning system according to claim 1, characterized in that: The step S103 further comprises: Using the pre-trained second target detection model, further identifying the eye socket target and the eyeball target in the eye socket target from the identified human target; According to the two diagonal coordinates of the detection frame of the orbital target, the coordinates of the center point of the orbital target are determined, and according to the two diagonal coordinates of the detection frame of the eyeball target, the coordinates of the center point of the eyeball target are determined; Determine the positional relationship between the center point of the eyeball target and the center point of the orbit target according to the center point coordinates of the orbit target and the center point coordinates of the eyeball target; According to the positional relationship between the center point of the eyeball target and the center point of the eye socket target, the positional relationship between the actual eyeball center point and the actual eye socket center point is determined.

7. A human eye region of interest positioning system according to claim 6, characterized in that: The second target detection model adopts the YOLOv5 network model.

8. The human eye region of interest positioning system according to claim 1, characterized in that: The preset visual range consists of a preset horizontal visual angle and a preset vertical visual angle.

9. A human eye region of interest positioning system according to claim 8, characterized in that: The step S107 further comprises: According to the preset horizontal viewing angle, mapping determines the left and right boundary distances of the region of interest; Mapping and determining the upper and lower boundary distances of the region of interest according to the preset vertical viewing angle; The position of the rectangular region of interest is determined according to the position of the mapping point, the left and right boundary distances, and the upper and lower boundary distances.

10. The human eye region of interest positioning system according to claim 9, characterized in that: The preset horizontal viewing angle is 36 degrees, and the preset vertical viewing angle is 20 degrees.

Images