Gaze tracking method, system, processing device and computer readable storage medium

Abstract

A gaze tracking method, system, processing device, and computer-readable storage medium are disclosed. The method includes: acquiring raw images acquired by multiple image acquisition devices; stitching the raw images into a stitched image and determining the responsible area of ​​each image acquisition device in the stitched image; performing preliminary face localization on the stitched image; determining one or more raw images corresponding to the preliminary face position based on the preliminarily localized face position and the responsible area of ​​each image acquisition device in the stitched image; performing secondary face localization on the determined one or more raw images to obtain face pose information and the positional relationship of the pupil relative to the eye; and calculating the gaze point coordinates of the eye based on the obtained face pose information and the positional relationship of the pupil relative to the eye. This disclosure provides a method suitable for a large field of view that can simultaneously perform gaze tracking on multiple people, and effectively reduces computational complexity while improving computational accuracy.

Description

Technical Field

[0001] This disclosure relates to, but is not limited to, the field of intelligent display technology, and in particular to an eye-tracking method, system, processing device, and computer-readable storage medium. Background Technology

[0002] With the continuous development of Virtual Reality (VR) / Augmented Reality (AR) technologies, people have increasingly higher requirements for interaction methods, and non-invasive eye-tracking systems have become a research hotspot in this field of interaction methods.

[0003] Currently, in the process of calculating user gaze, the following problems exist for multi-person gaze tracking on large-size displays:

[0004] I. Medium and large-sized displays have high requirements for the camera's field of view (FOV);

[0005] Second, with a fixed resolution, a higher camera field of view will affect the calculation accuracy.

[0006] Third, performing simultaneous eye tracking on multiple people will consume a significant amount of computing resources.

[0007] In addition, current spatial eye tracking systems on the market are usually only suitable for small-sized displays and are typically designed for single-person use. They require prior calibration and their accuracy drops significantly when the person's head moves or rotates. Summary of the Invention

[0008] This disclosure provides an eye-tracking method, system, processing device, and computer-readable storage medium that can improve product yield.

[0009] This disclosure provides a gaze tracking method, including acquiring original images captured by multiple image acquisition devices, stitching the acquired original images into a stitched image, and determining the responsible area of ​​each image acquisition device in the stitched image; performing preliminary face localization on the stitched image, and determining one or more original images corresponding to the preliminary face location based on the preliminarily localized face position and the responsible area of ​​each image acquisition device in the stitched image; performing secondary face localization on the determined one or more original images to obtain face pose information and the positional relationship of the pupil relative to the eye, and calculating the gaze point coordinates of the eye based on the obtained face pose information and the positional relationship of the pupil relative to the eye.

[0010] In an exemplary embodiment, stitching together multiple acquired original images into a single stitched image includes: acquiring the center coordinates, lateral field of view, longitudinal field of view, and resolution of each image acquisition device; determining the target area acquired by each image acquisition device based on the center coordinates, lateral field of view, and longitudinal field of view of each image acquisition device; determining non-overlapping areas within the target areas acquired by adjacent image acquisition devices based on the target areas acquired by each image acquisition device; using one of the adjacent image acquisition devices as the primary acquisition device and the other as the secondary acquisition device, converting the non-overlapping areas within the target areas acquired by the secondary acquisition device into non-overlapping areas in the images acquired by the secondary acquisition device based on the resolution of each image acquisition device; and stitching the non-overlapping areas in the images acquired by the secondary acquisition devices onto the images acquired by the primary acquisition devices to obtain a stitched image.

[0011] In an exemplary embodiment, the step of using one of the adjacent image acquisition devices as the main acquisition device and the other as the auxiliary acquisition device includes: when two adjacent image acquisition devices are distributed left and right, the image acquisition device on the left is the main acquisition device and the image acquisition device on the right is the auxiliary acquisition device; when two adjacent image acquisition devices are distributed vertically, the image acquisition device on the upper side is the main acquisition device and the image acquisition device on the lower side is the auxiliary acquisition device.

[0012] In an exemplary embodiment, after the step of stitching together multiple acquired original images into a stitched image and before the step of determining the responsible area of ​​each image acquisition device in the stitched image, the method further includes: performing noise reduction processing and downsampling processing on the stitched image.

[0013] In an exemplary embodiment, the downsampling process of the stitched image includes: dividing the stitched image into multiple sub-regions evenly according to the resolution of the target image; calculating the number of pixels contained in each sub-region and the center coordinates of each sub-region; calculating the comprehensive pixel value of each sub-region according to the distance from all pixels in each sub-region to the center of the sub-region; and assigning the calculated comprehensive pixel value of each sub-region to the stitched image to obtain the downsampled stitched image.

[0014] In an exemplary embodiment, the step of uniformly dividing the stitched image into multiple sub-regions according to the resolution of the target image includes: uniformly dividing the stitched image into a*b sub-regions, each sub-region having a width of (M / a) and a height of (N / b), where a, b, M, and N are all positive integers, M>a, N>b, a*b is the resolution of the target image, and M*N is the resolution of the stitched image.

[0015] In an exemplary embodiment, determining the responsible area of ​​each image acquisition device in the stitched image includes: obtaining the target area acquired by each image acquisition device; determining the dividing line of the target area acquired by adjacent image acquisition devices based on the distribution position of each image acquisition device; obtaining the acquisition range of each image acquisition device based on the dividing line of the target area acquired by adjacent image acquisition devices; and determining the responsible area of ​​each image acquisition device in the stitched image based on the acquisition range of each image acquisition device.

[0016] In an exemplary embodiment, the step of performing secondary face localization on one or more determined original images to obtain face pose information and the positional relationship of the pupil relative to the eye, and calculating the gaze point coordinates of the eye based on the obtained face pose information and the positional relationship of the pupil relative to the eye, includes: performing face detection on one or more determined original images to obtain a face region; performing feature point detection on the face region to obtain the center of the eye region; solving for the face pose in space to obtain the direction vector of the face, wherein the pose includes position coordinates and angle coordinates; performing pupil localization, calculating the movement difference between the pupil and the center of the eye region, and obtaining the direction vector of the pupil relative to the face based on the calculated movement difference, the direction vector of the face, and the diameter of the eyeball; converting the direction vector of the pupil relative to the face into the direction vector of the pupil in the world coordinate system; and calculating the gaze point coordinates of the pupil on the screen based on the direction vector of the pupil in the world coordinate system.

[0017] In an exemplary embodiment, the method further includes: averaging the gaze coordinates of the left and right eyes to obtain the final gaze coordinates.

[0018] This disclosure also provides a processing device, including: a processor and a memory storing a computer program executable on the processor, wherein the processor executes the program to implement the steps of the gaze tracking method as described above.

[0019] This disclosure also provides an eye-tracking system, including: multiple image acquisition devices, and the processing device as described above.

[0020] This disclosure also provides a computer-readable storage medium storing executable instructions that, when executed by a processor, can implement the eye-tracking method as described in any of the preceding embodiments.

[0021] The gaze tracking method, system, processing device, and computer-readable storage medium disclosed herein, through image stitching of original images acquired by multiple image acquisition devices, preliminary face localization of the stitched image, determination of the original image corresponding to the face position, and secondary localization of the original image to obtain face pose information and the positional relationship of the pupil relative to the eye, and calculating the gaze point coordinates of the eye based on the obtained face pose information and the positional relationship of the pupil relative to the eye, provides a method suitable for a large field of view and capable of simultaneous gaze tracking of multiple people. This solves the problem that current spatial gaze tracking devices on the market are usually only suitable for single people and have a small tracking field of view. Furthermore, by adding the influence of face pose to gaze tracking, the calculation accuracy when the face moves or deflects is improved, solving the problem of significant accuracy drop when the face undergoes pose changes. In addition, through two localizations, the calculation accuracy is improved while the computational complexity is effectively reduced.

[0022] Other features and advantages of this disclosure will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the disclosure. Other advantages of this disclosure may be realized and obtained by means of the methods described in the description and the accompanying drawings. Attached Figure Description

[0023] The accompanying drawings are used to provide an understanding of the technical solutions of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0024] Figure 1 This is a flowchart illustrating a gaze tracking method according to an embodiment of the present disclosure;

[0025] Figure 2 This is a schematic diagram showing one possible arrangement of multiple image acquisition devices according to an embodiment of the present disclosure;

[0026] Figure 3 for Figure 2 A schematic diagram showing the optical axis directions of the multiple image acquisition devices;

[0027] Figure 4 for Figure 2 The diagram shown illustrates the calculation principle of the system's lateral nearest-use distance.

[0028] Figure 5 for Figure 2 The diagram shows the calculation principle of the system's lateral field of view.

[0029] Figure 6 for Figure 2 The diagram shows the calculation principle of the longitudinal field of view of the system.

[0030] Figure 7 for Figure 2 A schematic diagram of the overlapping areas in the shooting areas of adjacent image acquisition devices;

[0031] Figure 8 for Figure 7 A schematic diagram of the vertex coordinates of the shooting areas of camera 1 and camera 2;

[0032] Figure 9 for Figure 7 A schematic diagram showing the stitching result of images captured by camera 1 and camera 2;

[0033] Figure 10 This is a schematic diagram illustrating the principle of re-segmenting a downsampled stitched image according to an embodiment of the present disclosure;

[0034] Figure 11 This is a schematic diagram illustrating the principle of solving the face orientation vector according to an embodiment of the present disclosure;

[0035] Figure 12 This is a schematic diagram illustrating the principle of facial feature point detection according to an embodiment of the present disclosure;

[0036] Figure 13 This is a schematic diagram illustrating the coordinate transformation principle of the pupil direction vector according to an embodiment of the present disclosure;

[0037] Figure 14 This is a schematic diagram of the structure of a processing device according to an embodiment of the present disclosure. Detailed Implementation

[0038] The specific embodiments of this disclosure will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate this disclosure, but are not intended to limit its scope. It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be arbitrarily combined with each other.

[0039] This disclosure provides a gaze tracking method. It involves acquiring images of the area in front of a screen using multiple image acquisition devices; then, using distributed adaptive face detection technology to obtain approximate information about the face area in front of the screen for initial face localization; next, based on the face's position information in the image, secondary localization is performed on its corresponding camera image to obtain precise face pose information; based on the obtained face pose information and the current pupil position relative to the eyes, the coordinates of the gaze point on the screen are calculated. This gaze tracking method is applicable to displays of any size and can simultaneously track one or multiple people, solving the problem that most current spatial gaze tracking devices are only suitable for single people and have a small tracking field of view. This disclosure, through distributed adaptive face detection, improves computational accuracy while effectively reducing computational complexity, solving the problem of high resource consumption in multi-camera face detection. Furthermore, this disclosure incorporates the influence of face pose into gaze calculation, improving computational accuracy when the face moves or deflects, solving the problem of significant accuracy degradation when the face's pose changes; and the method proposed in this disclosure does not require prior calibration, making it more convenient to use.

[0040] like Figure 1 As shown, this disclosure provides an eye-tracking method, including steps 10 to 30.

[0041] Step 10: Acquire raw images from multiple image acquisition devices, stitch the acquired raw images into a stitched image, and determine the area of ​​responsibility of each image acquisition device in the stitched image.

[0042] In one exemplary implementation, such as Figure 2 As shown, multiple image acquisition devices are spaced apart around the screen area that the human eye is looking at.

[0043] In one exemplary embodiment, the shape of the screen area is rectangular, circular, elliptical, or any other arbitrary shape.

[0044] In one exemplary embodiment, the image acquisition device is a camera or other image acquisition device of any type, used to acquire images of the area in front of the screen. For example, the camera may be an RGB camera.

[0045] In one exemplary embodiment, such as Figure 2 As shown, 12 cameras are used, each with a field of view (FOV) of 75°, capturing an area 60° horizontally and 45° vertically in front of it; each camera has a resolution of 640*480 and a frame rate of 30 frames per second (FPS). Figure 3 As shown, the optical axes of all cameras are parallel to the vertical line in the center of the screen.

[0046] In one exemplary embodiment, the screen area is set to W*H in width and height. The distribution parameters of the multiple cameras are as follows: the horizontal spacing between adjacent cameras is W / 3, and the vertical spacing between adjacent cameras is H / 3. The distance from the boundary of the shooting areas of the multiple cameras to the screen area is the nearest access distance L of this eye-tracking system. The calculation method for the nearest access distance L of this eye-tracking system is as follows:

[0047] For the nearest horizontal distance Lw or the nearest vertical distance Lh, the requirement is that the camera's shooting range exactly overlaps at this distance. Since the camera's horizontal shooting area can be regarded as a horizontal fan-shaped area, the shooting of two adjacent cameras will not overlap when the distance is less than a certain distance. Here, the nearest horizontal distance Lw or the nearest vertical distance Lh can be obtained by using trigonometric functions.

[0048] like Figure 4 As shown, given that the distance between two adjacent cameras is W / 3 and the lateral FOV of the cameras is 60°, the lateral closest usable distance Lw is:

[0049]

[0050] Similarly, the longitudinal nearest usage distance Lh is calculated using the following formula:

[0051]

[0052] The most recently used distance L is the maximum value between the horizontal most recently used distance Lw and the vertical most recently used distance Lh.

[0053] At the nearest distance L, the lateral and longitudinal shooting ranges of all cameras are calculated, and the lateral and longitudinal field of view αw and αh of this eye-tracking system can be obtained using trigonometric and inverse trigonometric functions. For example, still using... Figure 2 Taking the camera matrix shown as an example, as Figure 5 and Figure 6 As shown, the field of view angles of multiple cameras at the nearest distance L are calculated as follows:

[0054] The lateral field of view αw is:

[0055]

[0056] The longitudinal field of view αh is:

[0057]

[0058] Whether the nearest usable distance L is taken as the lateral nearest usable distance Lw or the longitudinal nearest usable distance Lh determines whether the lateral field of view αw or the longitudinal field of view αh is used as the reference. If the lateral field of view αw is used as the reference, it means that both the lateral and longitudinal nearest usable distances are Lw. In this case, when calculating the longitudinal field of view αh, the corresponding lateral nearest usable distance Lw needs to be substituted back into the calculation formula of the longitudinal field of view αh (that is, Lh in the calculation formula of the longitudinal field of view αh is replaced with Lw) to obtain the longitudinal field of view αh at this time.

[0059] Taking a 75-inch screen as an example, the closest usable distance L of this eye-tracking system is 473mm, and the corresponding horizontal field of view αw at the closest usable distance is 133.2°, and the vertical field of view αh is 105.8°.

[0060] In one exemplary embodiment, stitching together multiple acquired original images into a single stitched image includes the following steps:

[0061] Step 101: Obtain the center coordinates, horizontal field of view, vertical field of view, and resolution of each image acquisition device;

[0062] Step 102: Determine the vertex coordinates of the target area acquired by each image acquisition device based on the center coordinates, horizontal field of view, and vertical field of view of each image acquisition device;

[0063] Step 103: Based on the vertex coordinates of the target area acquired by each image acquisition device, determine the vertex coordinates of the non-overlapping areas in the target areas acquired by adjacent image acquisition devices.

[0064] Step 104: Using one of the adjacent image acquisition devices as the main acquisition device and the other as the auxiliary acquisition device, based on the resolution of each image acquisition device, convert the vertex coordinates of the non-overlapping areas in the target area acquired by the auxiliary acquisition device into the vertex coordinates of the non-overlapping areas in the image acquired by the auxiliary acquisition device, thereby obtaining the non-overlapping areas in the images acquired by each auxiliary acquisition device.

[0065] Step 105: Stitch the non-overlapping areas in the images acquired by each auxiliary acquisition device onto the images acquired by each main acquisition device to obtain a stitched image.

[0066] In one exemplary embodiment, such as Figure 7As shown, taking the center coordinates of the top left camera 1 as the origin, the center coordinates of camera 2 are (W / 3, 0), camera 3 is (2W / 3, 0), camera 4 is (W, 0), camera 5 is (0, H / 3), camera 6 is (W, H / 3), camera 7 is (0, 2H / 3), camera 8 is (W, 2H / 3), camera 9 is (0, H), camera 10 is (W / 3, H), camera 11 is (2W / 3, H), and camera 12 is (W, H).

[0067] In one exemplary embodiment, the shooting areas and shooting distances of cameras 1 and 2 are known. Assuming the resolution of each camera is m*n, the horizontal FOV is θw, and the vertical FOV is θh, the vertex coordinates of the shooting areas of cameras 1 and 2 can be obtained using trigonometric functions, such as... Figure 8 As shown, at a distance d (distance d > nearest-use distance L), the coordinates of the four vertices of the area captured by camera 1 are as follows: (-d*tan(θw / 2), -d*tan(θh / 2)), (d*tan(θw / 2), -d*tan(θh / 2)), (-d*tan(θw / 2), d*tan(θh / 2)), (d*tan(θw / 2), d*tan(θh / 2)); the coordinates of the four vertices of the area captured by camera 2 are as follows: The coordinates of the four vertices are as follows: (W / 3-d*tan(θw / 2), -d*tan(θh / 2)), (W / 3+d*tan(θw / 2), -d*tan(θh / 2)), (W / 3-d*tan(θw / 2), d*tan(θh / 2)), (W / 3+d*tan(θw / 2), d*tan(θh / 2)); similarly, the coordinates of the four vertices of the shooting area of ​​each camera at a distance d can be calculated.

[0068] In one exemplary embodiment, when two adjacent cameras are arranged horizontally, the left camera is designated as the primary camera and the right camera as the secondary camera; when two adjacent cameras are arranged vertically, the upper camera is designated as the primary camera and the lower camera as the secondary camera. For example, for camera 1 and camera 2, camera 1 is designated as the primary camera and camera 2 as the secondary camera; for camera 1 and camera 5, camera 1 is designated as the primary camera and camera 5 as the secondary camera, and the primary / secondary relationship of other adjacent cameras can be obtained similarly.

[0069] In one exemplary embodiment, the coordinates of the shooting areas of adjacent cameras are compared to obtain the vertex coordinates of the non-overlapping areas in the shooting area of ​​the auxiliary camera, for example, such as... Figure 8As shown, for cameras 1 and 2, at a distance d, the coordinates of the four vertices of the non-overlapping region in the shooting area of ​​camera 2 are (d*tan(θw / 2), -d*tan(θh / 2)), (W / 3+d*tan(θw / 2), -d*tan(θh / 2)), (d*tan(θw / 2), d*tan(θh / 2)), (W / 3+d*tan(θw / 2), d*tan(θh / 2)); similarly, the coordinates of the four vertices of the non-overlapping region of other adjacent cameras can be obtained.

[0070] In one exemplary embodiment, the shooting ratio of the non-overlapping area in the shooting area of ​​the auxiliary camera is calculated and converted into an image to obtain the non-overlapping area in the image acquired by each auxiliary acquisition device.

[0071] In one exemplary embodiment, such as Figure 8 As shown, utilizing the mapping relationship between the camera's shooting range and the resulting image, the non-overlapping areas in the auxiliary camera's shooting area are converted to non-overlapping areas in the image captured by the auxiliary camera using the following formula:

[0072] For the top-left vertex of the non-overlapping region, the coordinates P1(X1, Y1) mapped to it in the image are:

[0073]

[0074] Since the center coordinates of camera 1 and camera 2 are on the same horizontal line, Y1 = 0;

[0075] For the top right vertex of the non-overlapping region, the coordinates mapped to the image are P2 = (m, 0);

[0076] For the lower left vertex of the non-overlapping region, the coordinates P3(X3, Y3) mapped to the image are:

[0077]

[0078] Y3 = n;

[0079] For the bottom right vertex of the non-overlapping region, the coordinates mapped to it in the image are P4 = (m, n).

[0080] In one exemplary embodiment, such as Figure 9 As shown, the non-overlapping areas in the image mapped to camera 2 are extracted and stitched to the right side of the image captured by camera 1, thus stitching the images captured by camera 1 and camera 2 together. For other cameras, the same method is used for calculation, and finally, the image of the complete area in front of the screen is obtained.

[0081] This disclosure provides a spatial gaze tracking method suitable for large fields of view, with low resource consumption, high computational accuracy and speed, and capable of simultaneous gaze calculation for multiple users. The method uses 12 cameras evenly distributed around the screen, with all 12 cameras activated simultaneously to capture images of the area in front of the screen, forming a large field of view. The captured 12 images are then stitched together to obtain an image of the complete area in front of the screen. The stitching method of this embodiment has the advantages of low time consumption, low computational complexity, and low resource consumption.

[0082] In one exemplary embodiment, after stitching together multiple acquired original images into a single stitched image, the gaze tracking method further includes the following steps:

[0083] Step 106: Perform noise reduction and downsampling processing on the stitched image.

[0084] In one exemplary embodiment, Gaussian filtering is used to optimize the stitched image and remove noise from the image.

[0085] Since the final target image resolution required for the coarse face localization part is 640*480, and the resolution of the stitched image is greater than this resolution, it is necessary to downsample the stitched image.

[0086] In one exemplary embodiment, downsampling the stitched image includes the following steps:

[0087] Step 1061: Divide the stitched image into multiple sub-regions evenly according to the resolution of the target image;

[0088] Step 1062: Calculate the number of pixels contained in each sub-region and the center coordinates of each sub-region;

[0089] Step 1063: Calculate the combined pixel value of each sub-region based on the distance of all pixels in each sub-region to the center of the sub-region;

[0090] Step 1064: Assign the calculated combined pixel values ​​of each sub-region to the stitched image to obtain the downsampled stitched image.

[0091] In one exemplary embodiment, the stitched image is uniformly divided into multiple sub-regions according to the resolution of the target image, including:

[0092] The stitched image is evenly divided into a*b sub-regions, each with a width of (M / a) and a height of (N / b), where a, b, M, and N are all positive integers, M>a, N>b, a*b is the resolution of the target image, and M*N is the resolution of the stitched image.

[0093] In one exemplary embodiment, assuming the resolution of the stitched image is M*N, and the final required image resolution is known to be 640*480, the downsampling process of the stitched image includes the following steps:

[0094] Step 1061': Divide the stitched image evenly into 640*480 rectangular regions, with the width*height of each rectangular region being (M / 640)*(N / 480);

[0095] Step 1062': Calculate the pixels contained within each rectangular region and the center coordinates of each rectangular region sequentially. Calculate the distance from each pixel Pk (k = 1, 2, 3, ..., s) to the center (xc, yc) of its respective rectangular region using the following formula:

[0096]

[0097] Step 1063': Calculate the final pixel value P corresponding to each rectangular region according to the following formula:

[0098]

[0099] Step 1064': Assign the final pixel value P corresponding to each rectangular region to the final required image to obtain the downsampled image.

[0100] In one exemplary embodiment, determining the responsible area of ​​each image acquisition device in the stitched image includes the following steps:

[0101] Step 1065: Obtain the vertex coordinates of the target area acquired by each image acquisition device;

[0102] Step 1066: Based on the distribution location of each image acquisition device, determine the dividing line of the target area acquired by adjacent image acquisition devices;

[0103] Step 1067: Obtain the acquisition range of each image acquisition device based on the dividing lines of the target area acquired by adjacent image acquisition devices;

[0104] Step 1068: Determine the area of ​​responsibility of each image acquisition device in the stitched image based on the acquisition range of each image acquisition device.

[0105] In this embodiment, the obtained downsampled image is re-divided into camera image regions based on the relative positions of each camera. Since the parameters of each camera are the same, the only difference being the center coordinates, the area captured by each camera at the same shooting distance is also the same size. Therefore, as... Figure 10 As shown, the downsampled image is segmented using a method that divides image regions based on camera distribution density. The calculation method for the image region corresponding to each camera is as follows:

[0106] Based on the aforementioned method, the shooting range of each camera at distance d is obtained. Let the width * height of the total shooting range of these 12 RGB cameras at distance d be Wd * Hd. Given the coordinates of the four vertices of each camera's shooting range, for camera 1 and camera 2, the dividing line between them should be the midpoint between the centers of camera 1 and camera 2. Taking the center of camera 1 as the origin, and the coordinates of camera 2 as (W / 3, 0), then the dividing line between camera 1 and camera 2 should be (0 + W / 3) / 2 = W / 6. That is, the dividing line between the shooting ranges of camera 1 and camera 2 is:

[0107]

[0108] Similarly, for cameras 2 and 3, the dividing line of the shooting range is:

[0109]

[0110] Similarly, for cameras 3 and 4, the dividing line of the shooting range is:

[0111]

[0112] Similarly, for camera 1 and camera 5, the dividing line of the shooting range is:

[0113]

[0114] Similarly, for cameras 5 and 6, the dividing line of the shooting range is:

[0115] Similarly, the dividing lines of the shooting areas of every other two adjacent cameras can be obtained;

[0116] By inputting these dividing lines into the total shooting area, we can obtain the shooting range of each camera, as well as the proportional and positional relationship between these ranges and the total shooting area;

[0117] By substituting these proportional and positional relationships into the downsampled image, we can obtain the area of ​​responsibility for each camera in the image.

[0118] Step 20: Perform preliminary face localization on the stitched image. Based on the preliminary face location and the area of ​​responsibility of each image acquisition device in the stitched image, determine one or more original images corresponding to the preliminary face location.

[0119] In this step, a face detection algorithm can be used to initially locate faces in the downsampled image, and based on the regions where these faces are located, the corresponding original images captured by the camera (i.e., the images before stitching) can be determined.

[0120] Step 30: Perform secondary face localization on one or more original images to obtain face pose information and the positional relationship of the pupils relative to the eyes. Calculate the coordinates of the gaze point based on the obtained face pose information and the positional relationship of the pupils relative to the eyes.

[0121] In this embodiment of the disclosure, since the FOV of each camera is small, the computational accuracy is high when performing face detection and gaze tracking.

[0122] In this embodiment, step 30 mainly obtains the coordinates of the gaze point on the screen based on the spatial position coordinates and angular coordinates of the face, combined with the positional relationship between the pupil and the center of the eye.

[0123] In one exemplary embodiment, step 30 includes the following steps:

[0124] Step 301: Perform face detection in one or more determined original images to obtain face regions;

[0125] Step 302: Perform facial landmark detection in the face region to obtain the center of the eye region;

[0126] Step 303: Solve the pose of the face in space through n-point perspective (PnP) to obtain the orientation vector of the face, which includes position coordinates and angle coordinates;

[0127] Step 304: Perform pupil localization, calculate the movement difference between the pupil and the center of the human eye region, and obtain the direction vector of the pupil relative to the human face based on the calculated movement difference, the direction vector of the face and the diameter of the human eyeball;

[0128] Step 305: Convert the direction vector of the pupil relative to the face into the direction vector of the pupil in the world coordinate system;

[0129] Step 306: Calculate the coordinates of the pupil's gaze point on the screen based on the pupil's direction vector in the world coordinate system.

[0130] In this step, face detection is performed again in one or more original images to obtain the accurate region of the face in the current image; feature point detection is performed in the obtained face region to obtain 68 feature points of the face and the coordinates of the eye region; the pose of the face in space is solved by the PnP algorithm, including three position coordinates (horizontal coordinate, vertical coordinate, and distance coordinate) and three angle coordinates (pitch angle, yaw angle, and roll angle). The aforementioned distributed adaptive face detection algorithm reduces computational complexity and improves accuracy by employing a first coarse localization followed by a second precise localization for face pose calculation. Regarding the reduction in computational complexity, since face detection is not performed on every camera image but only on a subset of cameras, the number of face detection operations is reduced, consequently lowering computational complexity and resource consumption. For example, if three faces are simultaneously present on the screen, without the distributed adaptive face detection algorithm, 12 face detection operations would be required; with this method, only 4 operations are needed (one for the downsampled image and three for the subsequent three images). Furthermore, the distributed adaptive face detection algorithm also improves face detection accuracy. Due to the inherent distortion in camera lenses, the closer the face is to the camera's optical axis, the higher the pose calculation accuracy. The distributed adaptive face detection algorithm proposed in this disclosure can assign coarsely localized faces to corresponding cameras for secondary precise localization, resulting in more accurate face pose calculations under those cameras.

[0131] In one exemplary embodiment, the pose of the face in space is assumed to be (a, b, c, α, β, γ), where a represents the horizontal coordinate, b represents the vertical coordinate, c represents the distance coordinate, α represents the pitch angle, β represents the yaw angle, and γ represents the roll angle; the center of camera 1 in the upper left corner of the screen is taken as the origin of the world coordinate system, and the center coordinates of each camera have been given in the aforementioned image stitching section; as shown... Figure 10 As shown, if three faces are simultaneously present in front of the screen, assuming that after two precise positioning steps in step 301, face 1 is located in the shooting area of ​​camera 5, face 2 is located in the shooting area of ​​camera 9, and face 3 is located in the shooting area of ​​camera 8.

[0132] The center coordinates of camera 5 are (0, H / 3, 0), the center coordinates of camera 9 are (0, H, 0), and the center coordinates of camera 8 are (W, 2H / 3, 0). In step 303, the pose coordinates of face 1 calculated by camera 5 are (x1, y1, z1, α1, β1, γ1); the pose coordinates of face 2 calculated by camera 9 are (x2, y2, z2, α2, β2, γ2); and the pose coordinates of face 3 calculated by camera 8 are (x3, y3, z3, α3, β3, γ3).

[0133] like Figure 11As shown, the direction vectors of face 1, face 2, and face 3 are calculated respectively. Let α1 and β1 represent the pitch angle and yaw angle of the face, respectively. The unit length of the face direction vector is 1. Figure 11 As shown, using trigonometric relationships, we have: a1 = 1 * cos(α1) * sin(β1); similarly, we can obtain the other two coordinates b1 and c1. Therefore, the direction vector (a1, b1, c1) of face 1 can be calculated using the following formula:

[0134] a1 = cos(α1) * sin(β1);

[0135] b1 = sin(α1);

[0136] c1 = cos(α1) * cos(β1);

[0137] Similarly, the direction vectors of face 2 and face 3 can be calculated.

[0138] like Figure 12 As shown, in step 302, the coordinates of the human eye region can be obtained by detecting facial feature points, and then the coordinates of the center of the human eye region can be obtained. Taking face 1 as an example, it is assumed that the center coordinates of the left eye region are (XL, YL) and the center coordinates of the right eye region are (XR, YR).

[0139] In one exemplary embodiment, in step 304, the gradient vector field method can be used to detect the pupils and obtain the pupil coordinates (xL, yL) and (xR, yR) of the left and right eyes at the current time.

[0140] Calculate the coordinate differences (xL-XL, yL-YL) and (xR-XR, yR-YR) between the current pupil and the center of the human eye region. The ratio of this difference to the image is KLX = (xL-XL) / 640, KLY = (yL-YL) / 480, KRX = (xR-XR) / 640, KRX = (yR-YR) / 480. Then, the coordinate differences of the pupil's movement relative to the center of the human eye region in space are: (KLX*W5, KLY*H5), (KRX*W5, KRY*H5), where W5 and H5 represent the shooting range of camera 5 at a distance z1.

[0141] Taking the left pupil as an example, based on the calculated difference in pupil movement relative to the center of the human eye region in space, combined with the diameter of the human eyeball (the human eyeball diameter is taken as an empirical value of 24), the pupil direction vector (KLX*W5, KLY*H5, zL) can be obtained (this vector is the pupil vector in a coordinate system established with the center of the face). Figure 13As shown, this vector is relative to the direction of the face and needs to be converted into a vector (r1, s1, t1) in the world coordinate system. By calculating the difference in the origin coordinates between the coordinate system established at the center of the face and the world coordinate system, as well as the angle differences between the X, Y, and Z axes of the two coordinate systems, the conversion formula between the two coordinate systems can be obtained. The conversion formula is as follows:

[0142]

[0143]

[0144] Substituting the pupil direction vector obtained above into this conversion formula, we can obtain the pupil direction vector in world coordinates:

[0145]

[0146]

[0147]

[0148] Since the screen lies on a plane in the world coordinate system, the equation of this plane can be determined by the coordinates of the four vertices of the screen, thus yielding the plane equation of the screen. The pupil direction vector in the world coordinate system has already been calculated. Using the method of finding the intersection point of a line and a plane, the coordinates of the intersection point between the line of sight and the screen can be calculated. Therefore, the coordinates of the intersection point EL(Ex1, Ey1) between the pupil direction and the screen are:

[0149]

[0150] Where We is the horizontal distance between the center of the human eyeball and the center of the human face, and He is the vertical distance between the center of the human eyeball and the center of the human face. In this disclosure, both We and He can be taken as empirical values; the obtained EL(Ex1, Ey1) is the gaze point coordinate of the left pupil on the screen. Similarly, the gaze point coordinate ER of the right pupil on the screen can be obtained.

[0151] In one exemplary embodiment, step 30 further includes: averaging the fixation point coordinates of the left and right eyes to obtain the final fixation point coordinates.

[0152] In this embodiment, after obtaining the coordinates of the two pupils, due to calculation errors and other reasons, the two fixation points will not intersect at one point (theoretically, they should intersect at one point if there are no errors). Therefore, the average value of EL and ER is calculated and the average value is used as the final fixation point coordinates.

[0153] The gaze tracking method of this disclosure uses the current coordinates and rotation angle of the face, combined with the relative positional relationship between the pupil and the eyes, to calculate the coordinates of the gaze point on the screen. Since this method takes into account the three coordinate information and three angle information of the face, it also has high calculation accuracy when the face moves or rotates.

[0154] The gaze tracking method of this disclosure does not use a polynomial mapping model, but rather a method based on face pose and pupil model. This method can calculate the gaze as soon as it is started. In contrast, gaze tracking methods using a polynomial mapping model require calculating the parameters of the mapping model first, because the mapping model parameters are different for each person, and even for the same person at different times. Therefore, gaze tracking methods using a polynomial mapping model need to be calibrated after startup to calculate the mapping model parameters corresponding to that person at that moment. Only after calibration can the gaze be calculated. The gaze tracking method of this disclosure does not require the calculation of mapping model parameters. Face detection and pupil coordinate calculation are performed in real time, and no calibration is required before use, making it more convenient to use.

[0155] This disclosure also provides a processing device, which may include a processor and a memory storing a computer program executable on the processor, wherein the processor executes the computer program to implement the steps of the gaze tracking method as described in any of the preceding claims of this disclosure.

[0156] like Figure 14 As shown, in one example, the processing device 1400 may include a processor 1410, a memory 1420, a bus system 1430, and a transceiver 1440. The processor 1410, the memory 1420, and the transceiver 1440 are connected via the bus system 1430. The memory 1420 stores instructions, and the processor 1410 executes the instructions stored in the memory 1420 to control the transceiver 1440 to transmit signals. Specifically, under the control of the processor 1410, the transceiver 1440 can acquire raw images from an image acquisition device and, after detecting the coordinates of the human eye's gaze point, send notifications to other devices via the transceiver.

[0157] It should be understood that processor 1410 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0158] Memory 1420 may include read-only memory and random access memory, and provides instructions and data to processor 1410. A portion of memory 1420 may also include non-volatile random access memory. For example, memory 1420 may also store device type information.

[0159] In addition to a data bus, the bus system 1430 may also include a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 14 The general labeled all buses as Bus System 1430.

[0160] In implementation, the processing performed by the processing device can be accomplished through integrated logic circuits in the hardware of the processor 1410 or through software instructions. That is, the method steps of this embodiment can be executed by the hardware processor, or by a combination of hardware and software modules within the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other storage media. This storage medium is located in memory 1420. The processor 1410 reads information from memory 1420 and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, further details are omitted here.

[0161] This disclosure also provides an eye-tracking system, including multiple image acquisition devices and a processing device. The processing device may be the processing device 1400 described above. The image acquisition devices may be, for example, any device capable of capturing images.

[0162] This disclosure also provides a computer-readable storage medium storing executable instructions. When executed by a processor, these executable instructions can implement the gaze tracking method provided in any of the above embodiments of this disclosure. This gaze tracking method can be used to control the gaze tracking system provided in the above embodiments of this disclosure to perform gaze point detection. It can simultaneously track multiple people's gaze in a large field of view, has low computational resource consumption, and offers high tracking accuracy and speed. Furthermore, it requires no prior calibration and is easy to use. It has good application prospects for gaze tracking on medium to large-sized displays. The method of driving the gaze tracking system to perform gaze point detection by executing executable instructions is basically the same as the gaze tracking method provided in the above embodiments of this disclosure, and will not be described in detail here.

[0163] In the description of the embodiments of this disclosure, it should be understood that the terms "middle", "upper", "lower", "front", "rear", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings. They are only for the convenience of describing this disclosure and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this disclosure.

[0164] In the description of the embodiments of this disclosure, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the meaning of the above terms in this disclosure based on their understanding.

[0165] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0166] While the embodiments disclosed herein are as described above, the content is merely for the purpose of facilitating understanding of this disclosure and is not intended to limit this disclosure. Any person skilled in the art to which this disclosure pertains may make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed herein; however, the scope of protection of this disclosure shall still be determined by the scope defined in the appended claims.

Claims

1. A gaze tracking method, characterized in that, include: Acquire raw images from multiple image acquisition devices, stitch the acquired raw images into a stitched image, and determine the area of ​​responsibility of each image acquisition device in the stitched image; The stitched image is initially located for faces. Based on the initially located face position and the area of ​​responsibility of each image acquisition device in the stitched image, one or more original images corresponding to the initially located face position are determined. Secondary face localization is performed on one or more original images to obtain face pose information and the positional relationship between the pupil and the eye. Based on the obtained face pose information and the positional relationship between the pupil and the eye, the coordinates of the gaze point of the eye are calculated. The step of determining the responsible area of ​​each image acquisition device in the stitched image includes: obtaining the target area acquired by each image acquisition device; determining the dividing line of the target area acquired by adjacent image acquisition devices based on the distribution position of each image acquisition device; obtaining the acquisition range of each image acquisition device based on the dividing line of the target area acquired by adjacent image acquisition devices; and determining the responsible area of ​​each image acquisition device in the stitched image based on the acquisition range of each image acquisition device. Wherein, when the shooting parameters of multiple image acquisition devices are the same, the dividing line of the target area acquired by adjacent image acquisition devices is the perpendicular bisector of the line connecting the center positions of adjacent image acquisition devices. The process of stitching multiple acquired original images into a single stitched image includes: acquiring the center coordinates, horizontal field of view, vertical field of view, and resolution of each image acquisition device; determining the target area acquired by each image acquisition device based on its center coordinates, horizontal field of view, and vertical field of view; determining the non-overlapping areas within the target areas acquired by adjacent image acquisition devices based on their respective target areas; using one adjacent image acquisition device as the primary acquisition device and the other as the secondary acquisition device, converting the non-overlapping areas within the target areas acquired by the secondary acquisition device into non-overlapping areas within the images acquired by the secondary acquisition device, based on their respective resolutions; and stitching the non-overlapping areas from the images acquired by the secondary acquisition devices onto the images acquired by the primary acquisition device to obtain a stitched image.

2. The gaze tracking method according to claim 1, characterized in that, The method of using one of adjacent image acquisition devices as the main acquisition device and the other as the auxiliary acquisition device includes: When two adjacent image acquisition devices are distributed left and right, the image acquisition device on the left is the main acquisition device and the image acquisition device on the right is the auxiliary acquisition device. When two adjacent image acquisition devices are distributed vertically, the image acquisition device on the upper side is the main acquisition device, and the image acquisition device on the lower side is the auxiliary acquisition device.

3. The gaze tracking method according to claim 1, characterized in that, After the step of stitching together multiple acquired original images into a single stitched image and before the step of determining the responsible area of ​​each image acquisition device in the stitched image, the method further includes: The stitched images are subjected to noise reduction and downsampling.

4. The gaze tracking method according to claim 3, characterized in that, The downsampling process for the stitched image includes: Based on the resolution of the target image, the stitched image is evenly divided into multiple sub-regions; Calculate the number of pixels contained in each sub-region and the center coordinates of each sub-region; Calculate the combined pixel value of each sub-region based on the distance of all pixels in each sub-region to the center of the sub-region; The calculated combined pixel values ​​of each sub-region are assigned to the stitched image to obtain the downsampled stitched image.

5. The gaze tracking method according to claim 4, characterized in that, The step of uniformly dividing the stitched image into multiple sub-regions based on the resolution of the target image includes: The stitched image is evenly divided into a*b sub-regions, each sub-region having a width of (M / a) and a height of (N / b), where a, b, M, and N are all positive integers, M>a, N>b, a*b is the resolution of the target image, and M*N is the resolution of the stitched image.

6. The gaze tracking method according to claim 1, characterized in that, The process of performing secondary face localization on one or more determined original images to obtain face pose information and the positional relationship of the pupils relative to the eyes, and calculating the coordinates of the gaze point of the eyes based on the obtained face pose information and the positional relationship of the pupils relative to the eyes, includes: Face detection is performed on one or more original images to obtain face regions; Feature point detection is performed in the face region to obtain the center of the human eye region; Solve the pose of the face in space to obtain the orientation vector of the face, where the pose includes position coordinates and angle coordinates; Pupil localization is performed, and the movement difference between the pupil and the center of the human eye region is calculated. Based on the calculated movement difference, the direction vector of the face, and the diameter of the human eyeball, the direction vector of the pupil relative to the face is obtained. Convert the direction vector of the pupil relative to the face into the direction vector of the pupil in the world coordinate system; Calculate the coordinates of the pupil's gaze point on the screen based on the pupil's direction vector in the world coordinate system.

7. The gaze tracking method according to claim 6, characterized in that, The method further includes: The final fixation point coordinates are obtained by averaging the coordinates of the left and right eyes.

8. A processing apparatus, characterized in that, include: The processor and a memory storing a computer program executable on the processor, wherein the processor executes the computer program to implement the steps of the eye-tracking method as described in any one of claims 1-7.

9. A gaze tracking system, characterized in that, include: Multiple image acquisition devices, and the processing device as described in claim 8.

10. A computer-readable storage medium, characterized in that, The device stores computer-executable instructions for performing the method of any one of claims 1 to 7.

Images