Method and device for detecting sight direction, detection system and readable storage medium

Abstract

The present application discloses a method for detecting the direction of sight. The detection method includes: collecting infrared thermal imaging images; locating the target face area when the target face exists in the infrared thermal imaging image; determining the eye position in the target face area; focusing and shooting the eye position to obtain a visible light binocular image; using a sight detection model to process the visible light binocular image to obtain the sight point direction of the target face. The sight direction detection method of the present application can realize the detection of the sight direction of human eyes, and can also realize the sight direction detection of multiple people within a certain spatial range, thereby solving the sight direction monitoring needs of complex demand scenarios, such as student attention monitoring, anti-cheating monitoring and other scenarios. The present application also discloses a sight direction detection device, a sight direction detection system and a readable storage medium.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a line of sight direction detection method, device and detection system thereof and a readable storage medium. Background Art

[0002] Current cameras can only capture the entire surveillance image, but cannot track the line of sight of the human eye, that is, they cannot detect the line of sight of the human eye. Especially in complex surveillance scenarios with multiple people, such as student attention monitoring, anti-cheating monitoring, etc., it is impossible to detect the line of sight of multiple people. Summary of the invention

[0003] In view of this, the present invention aims to solve one of the problems in the related art at least to a certain extent. To this end, the purpose of the present application is to provide a line of sight direction detection method, device and detection system thereof and a readable storage medium.

[0004] The embodiment of the present application provides a method for detecting the direction of sight. The detection method includes: obtaining an infrared thermal imaging image captured by a thermal imaging camera; locating the target face area when there is a target face in the infrared thermal imaging image; determining the eye position in the target face area; controlling the target camera to focus on the eye position to obtain a visible light binocular image; and using a sight detection model to process the visible light binocular image to obtain the sight point direction of the target face.

[0005] In certain embodiments, locating the target face area includes: preliminarily determining the target detection area based on different color areas in the infrared thermal imaging image; distinguishing the face area and the non-face area in the target detection area based on color information and temperature information corresponding to the target detection area; and locating one or more of the target face areas according to the position of the face area in the infrared thermal imaging image.

[0006] In some embodiments, determining the eye position in the target facial area includes: performing image segmentation on one or more target faces in the infrared thermal imaging image according to facial position characteristics to obtain the eye positions in one or more target faces; and calculating the eye positions in the one or more target faces according to the eye position fitting algorithm.

[0007] In some embodiments, controlling the target camera to focus and shoot on the eye position to obtain the visible light binocular image includes: determining the target camera according to the eye position; controlling the target camera to scale the field of view to the field of view corresponding to the eye position; after the camera is scaled to a preset field of view, controlling the camera to automatically focus and shoot to obtain the visible light binocular image.

[0008] In certain embodiments, the use of a gaze detection model to process the visible light binocular image to obtain the gaze direction of the target face includes: inputting the visible light binocular image into the gaze detection model, outputting the facial angle of the target face and the gaze angle of the eyes in the target face; and calculating the gaze direction of the target face based on the facial angle of the target face, the gaze angle of the eyes in the target face, and the relative position of the target face and the camera.

[0009] In certain embodiments, the detection method includes: creating a database consisting of multiple eye images of multiple test subjects looking at multiple target locations in space from different facial angles; performing image preprocessing on the multiple eye images in the database; establishing a deep learning model based on a deep residual network and a deep convolutional neural network; and training the deep learning model using the preprocessed multiple eye images to obtain the gaze detection model.

[0010] In certain embodiments, the creation of a database consisting of multiple eye images of multiple test subjects looking at multiple target locations in space from different facial angles includes: recording the facial angles of the test subjects and the corresponding sight lines, and calculating the sight point angle of the test subjects based on the face; and using the sight point angle and the facial angle together as image tags of the eye images to obtain the database.

[0011] The present application also provides a device for detecting the direction of sight. The detection device includes: a thermal imaging module, a positioning module, a determination module, a visible light image acquisition module and a sight direction acquisition module. The thermal imaging module is used to collect infrared thermal imaging images; the positioning module is used to locate the target face area when there is a target face in the infrared thermal imaging image; the determination module is used to determine the eye position in the target face area; the visible light image acquisition module is used to focus and shoot the eye position to obtain a visible light binocular image; the sight direction acquisition module is used to process the visible light binocular image using a sight detection model to obtain the sight point direction of the target face.

[0012] The present application also provides a line of sight direction detection system, which includes a processor and a memory, wherein the memory is used to store a computer program, and the processor implements any one of the detection methods described in the above embodiments when executing the computer program.

[0013] The present application also provides a non-volatile computer-readable storage medium for a computer program. When the computer program is executed by one or more processors, the detection method described in any one of the above embodiments is implemented.

[0014] The line of sight direction detection method of the present application can realize the detection of the line of sight direction of the human eye, and can also realize the line of sight direction detection of multiple people in a certain spatial range, thereby solving the line of sight direction monitoring needs in complex demand scenarios, such as student attention monitoring, anti-cheating monitoring and other scenarios.

[0015] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0017] Figure 1 is a flow chart of a method for detecting a sight line direction in certain embodiments of the present application;

[0018] Figure 2 is a schematic structural diagram of a sight line direction detection device in certain embodiments of the present application;

[0019] Figure 3 is a schematic diagram of a scene of a method for detecting a sight direction in certain embodiments of the present application;

[0020] Figure 4 is a flow chart of a method for detecting a sight line direction in certain embodiments of the present application;

[0021] Figure 5 is a schematic structural diagram of a positioning module in a sight line direction detection device in certain embodiments of the present application;

[0022] Figure 6 is a schematic diagram of a scene of a method for detecting a sight direction in certain embodiments of the present application;

[0023] Figure 7 is a flow chart of a method for detecting a sight line direction in certain embodiments of the present application;

[0024] Figure 8is a schematic diagram of the structure of an eye position determination module in a sight line direction detection device in certain embodiments of the present application;

[0025] Fig. 9 is a flow chart of a method for detecting a sight line direction in certain embodiments of the present application;

[0026] Fig.10 is a schematic diagram of the structure of a visible light image acquisition module in a sight line direction detection device in certain embodiments of the present application;

[0027] Fig.11 is a schematic diagram of a scene of a method for detecting a sight direction in certain embodiments of the present application;

[0028] Fig.12 is a schematic diagram of a scene of a method for detecting a sight direction in certain embodiments of the present application;

[0029] Fig.13 is a flow chart of a method for detecting a sight line direction in certain embodiments of the present application;

[0030] Fig.14 is a structural schematic diagram of a sight direction acquisition module in a sight direction detection device in certain embodiments of the present application;

[0031] Fig.15 is a schematic diagram of a scene of a method for detecting a sight direction in certain embodiments of the present application;

[0032] Fig.16 is a schematic diagram of a scene of a method for detecting a sight direction in certain embodiments of the present application;

[0033] Fig.17 is a flow chart of a method for detecting a sight line direction in certain embodiments of the present application;

[0034] Fig.18 is a schematic structural diagram of a sight line direction detection device in certain embodiments of the present application;

[0035] Fig.19 is a flow chart of a method for detecting a sight line direction in certain embodiments of the present application;

[0036] Fig. 20 is a flow chart of a method for detecting a sight line direction in certain embodiments of the present application;

[0037] Fig.21 is a schematic diagram of the structure of a line of sight direction detection system in certain embodiments of the present application;

[0038] Fig. 22 It is a schematic diagram of the structure of a computer-readable storage medium of certain embodiments of the present application. DETAILED DESCRIPTION

[0039] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0040] In the description of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0041] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, or can communicate with each other; it can be a direct connection, or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0042] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or reference letters in different examples, and this repetition is for the purpose of simplification and clarity, which itself does not indicate the relationship between the various embodiments and / or settings discussed.

[0043] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0044] See also Figure 1 , the present application provides a method for detecting the direction of sight. The detection method includes:

[0045] 01: Get the infrared thermal imaging image collected by the thermal imaging camera;

[0046] 02: Locate the target face area when there is a target face in the infrared thermal imaging image;

[0047] 03: Determine the eye position in the target face area;

[0048] 04: Control the target camera to focus on the eye position to obtain visible light binocular images;

[0049] 05: Use the gaze detection model to process the visible light binocular image to obtain the gaze direction of the target face.

[0050] See also Figure 2 The present application also provides a sight direction detection device 10. The detection device 10 comprises: a thermal imaging image acquisition module 11, a positioning module 12, an eye position determination module 13, a visible light image acquisition module 14 and a sight direction acquisition module 15.

[0051] Step 01 can be implemented by the thermal imaging image acquisition module 11, step 02 can be implemented by the positioning module 12, step 03 can be implemented by the eye position determination module 13, step 04 can be implemented by the visible light image acquisition module 14, and step 05 can be implemented by the sight direction acquisition module 15. That is, the thermal imaging image acquisition module 11 is used to acquire the infrared thermal imaging image collected by the thermal imaging camera; the positioning module 12 is used to locate the target face area when the target face exists in the infrared thermal imaging image; the determination module 13 is used to determine the eye position in the target face area; the visible light image acquisition module 14 is used to control the target camera to focus on the eye position to obtain a visible light binocular image; the sight direction acquisition module 15 is used to process the visible light binocular image using the sight detection model to obtain the sight point direction of the target face.

[0052] It can be understood that the thermal imaging camera can be a part of the thermal imaging image acquisition device, which is a camera that displays and outputs infrared thermal images by receiving infrared rays emitted by an object. Any object with temperature will emit infrared rays, and the infrared thermal imager can receive infrared rays emitted by the object and display the temperature distribution on the surface of the measured object through a colored picture. Specifically, the thermal imaging image acquisition device can be an infrared thermal imager, which can specifically include a thermal infrared signal receiving module, an optical module and a digital signal conversion module. The thermal infrared signal receiving module is, for example, an infrared detector. The optical module includes an optical imaging objective lens. The digital signal conversion module includes a digital-to-analog converter. The infrared thermal imager can use the infrared detector and the optical imaging objective lens to receive the infrared radiation energy distribution graph of the target object to be measured, and reflect the infrared radiation energy distribution graph to the photosensitive element of the infrared detector. After the infrared radiation energy distribution diagram is reflected on the photosensitive element of the infrared detector, the infrared light of the infrared radiation of the target object to be measured needs to be converted into an electrical signal through a digital signal converter, and then the infrared thermal imaging image is output after non-uniformity correction, digital filtering, image enhancement and pseudo-color lookup table conversion.

[0053] Then, when the target face exists in the infrared thermal imaging image, the target face area is located. It can be understood that, since different colors in the infrared thermal imaging image represent different temperatures, when the target face exists in the infrared thermal imaging image, the color of the area where the target face is located is different from the colors of other areas in the infrared thermal imaging image. Therefore, the positioning module 12 can locate the target face area according to the color difference in the infrared thermal imaging image, and does not need to locate the target face area by calculating the face area and distance, etc., which saves the calculation process, improves the processing speed, and realizes the rapid positioning of the face. Among them, the target face area in the infrared thermal imaging image can be one or more.

[0054] Then, the eye position determination module 13 may determine the eye positions in one or more target face regions.

[0055] After determining the eye position in the target face area, the eye position determination module 13 can be automatically triggered to transmit the corresponding camera call command to the visible light image acquisition module 14 corresponding to the eye position, thereby controlling the target camera corresponding to the eye position to be enabled and automatically focus and shoot, thereby obtaining a visible light binocular image. The target camera can be a visible light camera, which can be used to zoom to the field of view corresponding to the position of the eye positioning. After the visible light camera field of view is zoomed to the corresponding range, it automatically shoots to obtain a visible light binocular image, that is, the visible light camera only shoots the eyes of the target face. Among them, the camera call command transmitted by the eye position determination module 13 determines to start the visible light camera at a certain position in the entire detection area, and there is no need to manually control the visible light camera to shoot the binocular image. After receiving the camera call command, the visible light camera automatically focuses and shoots to obtain the visible light binocular image, making the detection method of the present application more efficient and smarter.

[0056] It can be understood that the infrared thermal imaging image is an image formed by covering the entire detection area, and there can be multiple visible light cameras corresponding to different positions in the infrared thermal imaging image (such as Figure 3 shown).

[0057] When only one target face region is determined in the infrared thermal imaging image, and after the eye position in the target face region is determined, a visible light camera corresponding to the eye position in the infrared thermal imaging image can be triggered to be enabled, and the visible light camera can be automatically focused and photographed after being enabled, thereby obtaining a visible light binocular image corresponding to the eye position. Among them, the visible light camera is a high-speed camera that can quickly capture the instantaneous image of the target face.

[0058] When multiple target face areas are determined in the infrared thermal imaging image, and after multiple eye positions in the multiple target face areas are determined, multiple visible light cameras corresponding to the multiple eye positions in the infrared thermal imaging image can be triggered to be enabled. After the multiple visible light cameras are enabled, automatic focusing and shooting can be performed, thereby obtaining multiple visible light binocular images corresponding to the multiple eye positions.

[0059] Finally, the sight line detection model is used to process the visible light binocular image to obtain the sight line direction of the target face. That is, one or more visible light binocular images can be processed at the same time to obtain the sight line direction of one or more target faces, which can realize the sight line direction detection of multiple people within a certain spatial range. Among them, the sight line detection model is a model established based on the end-to-end algorithm of visible light binocular images.

[0060] The line of sight direction detection method of the present application can realize the detection of the line of sight direction of the human eye, and can also realize the line of sight direction detection of multiple people in a certain spatial range, thereby solving the line of sight direction monitoring needs in complex demand scenarios, such as student attention monitoring, anti-cheating monitoring and other scenarios.

[0061] See also Figure 4 In some embodiments, step 02 includes:

[0062] 021: Preliminarily determine the target detection area based on different color areas in the infrared thermal imaging image;

[0063] 022: distinguish the face area and the non-face area in the target detection area according to the color information and temperature information corresponding to the target detection area;

[0064] 023: One or more target face regions are located according to the positions of the face regions in the infrared thermal imaging image.

[0065] See also Figure 5 The positioning module 12 includes a target detection area determining unit 121, a distinguishing unit 122 and a positioning unit 123.

[0066] Step 021 can be implemented by the target detection area determination unit 121, step 022 can be implemented by the distinguishing unit 122, and step 023 can be implemented by the positioning unit 123. That is, the target detection area determination unit 121 is used to preliminarily determine the target detection area according to different color areas in the thermal imaging image; the distinguishing unit 122 is used to distinguish the face area and the non-face area in the target detection area according to the color information and temperature information corresponding to the target detection area; the positioning unit 123 is used to locate one or more target face areas according to the position of the face area in the thermal imaging image.

[0067] Specifically, see Figure 6 ,Since different colors in the infrared thermal imaging image represent different temperatures, the target detection area (ROI) is preliminarily determined based on the color area, which can be Figure 6 The shaded area shown in .

[0068] Then, the temperature information and color information of the target detection area can be input into the feature extraction model to perform clustering calculation, so as to distinguish the target detection areas at different positions as human face areas and non-human face areas in the infrared thermal imaging image. Figure 6 As shown, the target detection area at position 1 is a face area, while the target detection area at position 2 is a non-face area. The feature extraction model is a model that can recognize faces.

[0069] Next, one or more target face regions can be located according to the position of the face region in the infrared thermal imaging image. Figure 6 One or more target face regions are determined at position 1 in the infrared thermal imaging image. It can be understood that if position 1 of the infrared thermal imaging image has one target face, then correspondingly, one target face region can be determined; if position 1 of the infrared thermal imaging image has multiple target faces, then correspondingly, multiple target face regions can be determined.

[0070] The line of sight direction detection method of the present application can locate the target face area of ​​one or more persons within a certain spatial range according to the infrared thermal imaging image, and can realize rapid positioning of the target face within a certain spatial range, so as to facilitate the subsequent detection of the line of sight direction of the human eye.

[0071] See also Figure 7 In some embodiments, step 03 includes:

[0072] 031: Perform image segmentation on one or more target faces in the infrared thermal imaging image according to the face position characteristics to obtain eye positions in the one or more target faces;

[0073] 032: Calculate eye positions in one or more target faces according to an eye position fitting algorithm.

[0074] See also Figure 8 In some embodiments, the eye position determination module 13 includes a cutting unit 131 and a calculation unit 132 .

[0075] Step 031 can be implemented by the cutting unit 131, and step 032 can be implemented by the calculation unit 132. That is, the cutting unit 131 is used to perform image cutting on one or more target faces in the infrared thermal imaging image according to the face position characteristics to obtain the eye positions in the one or more target faces; the calculation unit 132 is used to calculate the eye positions in the one or more target faces according to the eye position fitting algorithm.

[0076] It can be understood that the target face area may include one target face or multiple target faces, and the eye position in the target face area can be determined. Correspondingly, one eye position of one target face can be determined, or multiple eye positions of multiple target faces can be determined.

[0077] Specifically, the face position characteristic may refer to the position characteristic that the eyes are necessarily located at both sides of the nose, below the eyebrows, and above the cheeks. The eye position fitting algorithm may include a Taubin fitting algorithm.

[0078] For example, when a target face region includes a target face, after image segmentation and fitting of the target face, the position of the eye of the target face in the infrared thermal imaging image can be determined as follows: Figure 6 Similarly, when a target face region includes multiple target faces, the eye positions of the multiple target faces in the infrared thermal imaging image can be determined.

[0079] When there are multiple target face regions and each target face region includes one or more target faces, after image cutting and fitting are performed on the multiple target faces in the multiple target face regions, the eye positions of the multiple target faces in the infrared thermal imaging image can also be determined respectively.

[0080] The detection method of the present application can perform image segmentation on one or more target faces in an infrared thermal imaging image to obtain the eye positions in one or more target faces, and can accurately locate and calculate the eye positions of one or more target faces based on the infrared thermal imaging image combined with the eye position fitting algorithm. Therefore, the present application can accurately determine the eye positions of one or more target faces in an infrared thermal imaging image.

[0081] See also Fig. 9 , step 04 includes:

[0082] 041: Determine the target camera based on the eye position;

[0083] 042: Control the target camera to scale the field of view to the field of view corresponding to the eye position;

[0084] 043: After the camera is zoomed to a preset field of view, the camera is controlled to automatically focus and shoot to obtain a visible light binocular image.

[0085] See also Fig.10 The visible light image acquisition module 14 includes a target camera determination unit 141 , a control unit 142 and a shooting unit 143 .

[0086] Step 041 can be implemented by the target camera determination unit 141, step 042 can be implemented by the control unit 142, and step 043 can be implemented by the shooting unit 143. That is, the target camera determination unit 141 is used to determine the target camera according to the eye position; the control unit 142 is used to control the target camera to scale the field of view to the field of view corresponding to the eye position; the shooting unit 143 is used to control the camera to perform automatic focusing shooting after the camera is scaled to the preset field of view, so as to obtain a visible light binocular image.

[0087] Specifically, the target camera is the high-speed camera mentioned above. It can be understood that the high-speed camera in the line of sight direction detection method of the present application has multiple high-speed cameras, and the multiple high-speed cameras are respectively set at corresponding positions in the infrared thermal imaging image (such as Figure 6 As shown), each high-speed camera corresponds to shooting the area corresponding to the field of view of the high-speed camera.

[0088] Therefore, after determining an eye position in the infrared thermal imaging image, a corresponding high-speed camera can be called to focus and shoot according to the eye position. Figure 6 If the area is covered by the field of view of the high-speed camera 1, the high-speed camera 1 is the target camera. The high-speed camera 1 can be called to scale the field of view to the field of view corresponding to the eye position, and then the high-speed camera 1 can be controlled to perform automatic focusing shooting to obtain a visible light binocular image.

[0089] In addition, after determining multiple eye positions in the infrared thermal imaging image, one or more corresponding high-speed cameras can be called for focusing and shooting according to the multiple eye positions. That is, the number of target cameras can be determined according to the number of determined eye positions and whether the positions are within the field of view of the same high-speed camera.

[0090] In an example, Fig.11 As shown, when there are three determined eye positions, and the distances between the three eye positions are close and are all within the coverage range of the field of view of the high-speed camera 2, the target camera determined at this time is the high-speed camera 2.

[0091] In another example, Fig.12 As shown, when there are also three determined eye positions and the distances between the three eye positions are relatively far, among which eye position 1 is within the coverage range of the field of view angle of high-speed camera 3, eye position 2 is within the coverage range of the field of view angle of high-speed camera 4, and eye position 3 is within the coverage range of the field of view angle of high-speed camera 5, then there are three determined target cameras at this time, namely high-speed camera 3, high-speed camera 4 and high-speed camera 5.

[0092] It should be noted that the field of view of each high-speed camera may be the same or different. The field of view of each high-speed camera may be a default value or may be set according to user needs.

[0093] See also Fig.13 In some embodiments, step 05 includes:

[0094] 051: Input the visible light binocular image into the gaze detection model, and output the facial angle of the target face and the gaze angle of the eyes in the target face;

[0095] 052: Calculate the direction of the target face's sight point according to the target face's facial angle, the sight point angle of the target face's eyes, and the relative position between the target face and the camera.

[0096] See also Fig.14 The sight direction acquisition module 15 includes an angle acquisition unit 151 and a direction calculation unit 152 .

[0097] Step 051 can be implemented by the angle acquisition unit 151, and step 052 can be implemented by the direction calculation unit 152. That is, the angle acquisition unit 151 is used to input the binocular image into the sight line detection model, and output the facial angle of the target face and the sight line angle of the eyes in the target face; the direction calculation unit 152 is used to calculate the sight line direction of the target face according to the facial angle of the target face, the sight line angle of the eyes in the target face, and the relative position of the target face and the camera.

[0098] Specifically, see Fig.15 The facial angle of the target face can be based on the angle of the target face when it is facing the high-speed camera, and the facial angle of the target face is 0 degrees at this time.

[0099] See also Fig.15 , the sight point angle can be based on the angle when the human eye is in the middle position of the eye frame. At this time, the sight point angle of the human eye in the target face is 0 degrees horizontally. In addition, when the facial angle of the target face remains unchanged and the position of the human eye in the eye frame changes, resulting in a different sight point angle, the relative distance between the position of the human eye in the eye frame and the middle position can be proportional to the sight point angle. For example, for every 1mm deviation of the position of the human eye in the eye frame from the middle position, the sight point angle deviates by 10 degrees. For example, if the position of the human eye in the eye frame deviates from the middle position by 2mm in the C direction, the corresponding sight point angle is deflected by 20 degrees in the C direction.

[0100] For more details, please also refer to Fig.15 and Fig.16 , Fig.16 relatively Fig.15 The visible light binocular image in the output is input into the gaze detection model, and the facial angle of the target face is output as follows: The target face can be deflected 30 degrees in the direction A, that is, the facial angle of the target face is A30 degrees. Fig.15 and Fig.16If the eye position in the image remains unchanged, the corresponding sight angle is deflected 30 degrees in the direction of B, and the sight angle at this time is recorded as B30 degrees. It should be noted that if the target face is rotated 30 degrees in the opposite direction of direction A, the corresponding facial angle can also be recorded as a negative angle of -A30 degrees, and the corresponding sight angle can also be recorded as a negative angle relative to direction B, which is -B30 degrees. The situation in which the target face is rotated in directions other than direction A is similar to that in direction A, and will not be repeated here.

[0101] Then, the direction of the target face's sight point is calculated based on the target face's facial angle, the angle of the target face's eye sight point, and the relative position of the target face and the camera, so that the real sight point in space can be calculated. Fig.16 As shown, when the relative position of the target face and the camera remains unchanged, that is, the relative distance between the target face and the camera remains unchanged, since the distance L1 from the target face to the ground can also be regarded as unchanged, and the facial angle and the sight point angle of the target face are both known, the final sight point position toward the ground can be calculated based on the distance L1 from the target face to the ground, the facial angle A30 degrees of the target face, and the sight point angle B30 degrees of the human eye in the target face. That is, the value of L2 between the projection of the target face on the ground and the sight point position can be calculated according to the Pythagorean theorem, thereby determining the sight point position.

[0102] The detection method of the present application can use the gaze detection model to quickly identify the facial angle of the target face and the gaze angle of the human eyes in the target face in the visible light binocular image, so as to finally obtain the gaze direction of the target face, and calculate the actual gaze point in space.

[0103] The following is a detailed description of the creation process of the gaze detection model. Fig.17 , detection methods include:

[0104] 001: Create a database consisting of multiple eye images of multiple test subjects looking at multiple target locations in space from different facial angles;

[0105] 002: Perform image preprocessing on multiple eye images in the database;

[0106] 003: Establish a deep learning model based on deep residual network and deep convolutional neural network;

[0107] 004: Use the preprocessed multiple eye images to train the deep learning model to obtain a gaze detection model.

[0108] Please combine Fig.18The sight line direction detection device 10 includes a database creation module 101, a preprocessing module 102, a learning model building module 103 and a sight line detection model generation module 104.

[0109] Step 001 can be implemented by the database creation module 101, step 002 can be implemented by the preprocessing module 102, step 003 can be implemented by the learning model building module 103, and step 004 can be implemented by the sight line detection model generation module 104. That is, the sight line detection model generation module 101 is used to create a database consisting of multiple eye images of multiple test subjects looking at multiple target positions in space from different facial angles; the preprocessing module 102 is used to perform image preprocessing on the multiple eye images in the database; the learning model building module 103 is used to establish a deep learning model based on a deep residual network and a deep convolutional neural network; the sight line detection model generation module 104 is used to train the deep learning model using the preprocessed multiple eye images to obtain a sight line detection model. Step 001, step 002, step 003 and step 004 can be performed after step 04 and before step 05 as follows Fig.17 As shown, it is also possible to perform the above four steps before step 01. The execution order of the above four steps is not limited here.

[0110] Specifically, first, create a database consisting of multiple eye images of multiple test subjects looking at multiple target positions in space from different facial angles. For example, invite multiple test subjects, and then control under different lighting conditions, let each test subject randomly turn his head to look at multiple target positions in space, so that multiple eye images of multiple test subjects at different facial angles can be obtained, thereby obtaining multiple eye images of multiple test subjects looking at multiple target positions in space from different facial angles. The more the number of multiple test subjects, the more realistic the eye fitting results of the deep learning model training, and the higher the credibility of the obtained gaze detection model.

[0111] Then, the multiple eye images are preprocessed. The image preprocessing process may include random rotation, Gaussian blur, normalization and other processing processes on the eye images to achieve the expansion of the eye image data set in the database.

[0112] Next, a deep learning model is established based on the deep residual network and the deep convolutional network. The deep learning model is a CNN model. Fig.19 As shown in the figure, the feature extraction part of the deep learning model is composed of several groups of deep residual network (ResNet) modules and deep convolutional neural network (Inception) modules arranged alternately. Among them, the loss function of the deep learning model is the sum of the regression L2 of the facial angle and the regression L1 of the sight angle.

[0113] Finally, the deep learning model is trained using the preprocessed multiple eye images to obtain a gaze detection model. Fig.19 As shown, the feature extraction part of the deep learning model is connected to a fully connected layer (Fullconnection), and the preprocessed multiple eye images are input into the deep learning model. After the multiple eye images pass through the feature extraction part, they are then sent to the fully connected layer to realize the fitting calculation of the direction vector, thereby obtaining a gaze detection model.

[0114] The present application trains a deep learning model through multiple eye images to obtain a gaze detection model, and directly processes the visible light binocular images according to the gaze detection model, which can greatly improve the speed at which current computers process visible light binocular images and improve the processing efficiency of visible light binocular images.

[0115] For details, see Fig. 20 , step 003 includes:

[0116] 0031: Record the test subject’s facial angle and the corresponding sight line position, and calculate the test subject’s sight line angle based on the face;

[0117] 0032: The gaze point angle and the facial angle are used together as image labels of eye images to obtain a database.

[0118] Please combine Fig.18 , step 0031 and step 0032 can be implemented by the database creation module 101. That is, the database creation module 101 is used to record the facial angle of the test subject and the corresponding sight line position, calculate the sight line angle of the test subject based on the face; and use the sight line angle and the facial angle as the image label of the eye image to obtain a database.

[0119] Specifically, the test subject's facial angle and the corresponding sight line position are recorded, and the test subject's sight line angle is calculated based on the face. For example, the test subject's facial angle is as follows: Fig.16As shown, it is deflected 30 degrees in the direction of A. At this time, the position of the subject's line of sight is the point b indicated by the dotted line. At this time, the angle of the subject's line of sight can be calculated based on the subject's face, that is, the subject's face can be regarded as a plane, and the subject's face can be used as a reference plane. Through a line of sight and the reference plane, the angle of the subject's line of sight can be calculated. In addition, the subject can randomly turn his head. For example, when the subject's head is facing left, right, up or down, the position of the line of sight will change when the position of the eye in the eye socket of the subject's eye remains unchanged. At this time, the line of sight angle when the face of the face is facing the camera position can be recorded as 0 degrees, and this can be used as a reference to calculate the line of sight angle of the human eye looking at other different positions in the space.

[0120] Then, the gaze point angle and the facial angle are used together as image labels of eye images to obtain a database, so that each eye image in the database has corresponding gaze point angle data and facial angle data, and the content of each eye image can be accurately identified according to the image label, thereby improving the retrieval efficiency and accuracy of multiple pre-processed eye images during the training process of the deep learning model, which is conducive to the rapid establishment of a gaze detection model.

[0121] See also Fig.21 The present application also provides a line of sight direction detection system 100. The line of sight direction detection system 100 includes a processor 110 and a memory 120, the memory 120 is used to store a computer program 121, and the processor 110 implements any one of the above-mentioned detection methods when executing the computer program 121.

[0122] The line of sight direction detection system 100 of the present application can realize the detection of the line of sight direction of the human eye, and can also realize the line of sight direction detection of multiple people in a certain spatial range, thereby solving the line of sight direction monitoring needs in complex demand scenarios, such as student attention monitoring, anti-cheating monitoring and other scenarios.

[0123] See also Fig. 22 The present application also provides a non-volatile computer-readable storage medium 200 of a computer program. When the computer program 210 is executed by one or more processors 220, the detection method of any one of the above embodiments is implemented.

[0124] The computer-readable storage medium 200 of the present application can realize the detection of the line of sight direction of the human eye, and can also realize the detection of the line of sight direction of multiple people in a certain spatial range, thereby solving the line of sight direction monitoring needs of complex demand scenarios, such as student attention monitoring, anti-cheating monitoring and other scenarios.

[0125] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for detecting a line of sight direction, characterized in that: The detection method comprises: Acquire infrared thermal imaging images collected by a thermal imaging camera; In the case where a target face exists in the infrared thermal imaging image, locating a target face area; Determining the eye position in the target face area; Controlling the target camera to focus and shoot the eye position to obtain a visible light binocular image; Processing the visible light binocular image using a gaze detection model to obtain the gaze direction of the target face; The controlling the target camera to focus and shoot the eye position to obtain a visible light binocular image comprises: Determining the target camera according to the eye position; Controlling the target camera to scale the field of view to the field of view corresponding to the eye position; After the camera is zoomed to a preset field of view, the camera is controlled to perform automatic focusing and shooting to obtain the visible light binocular image.

2. The detection method according to claim 1, characterized in that: The positioning target face area includes: Preliminarily determining a target detection area according to different color areas in the infrared thermal imaging image; Distinguishing a face area and a non-face area in the target detection area according to color information and temperature information corresponding to the target detection area; One or more target face regions are located correspondingly according to the positions of the face regions in the infrared thermal imaging image.

3. The detection method according to claim 1, characterized in that: Determining the eye position in the target face area includes: Performing image segmentation on one or more target faces in the infrared thermal imaging image according to the face position characteristics to obtain eye positions in one or more target faces; The eye positions in the one or more target faces are calculated according to the eye position fitting algorithm.

4. The detection method according to claim 1, characterized in that: The step of using the sight line detection model to process the visible light binocular image to obtain the sight line direction of the target face includes: Input the visible light binocular image into the sight line detection model, and output the facial angle of the target face and the sight line angle of the eyes in the target face; The sight line direction of the target face is calculated according to the facial angle of the target face, the sight line angle of the eyes in the target face, and the relative position of the target face and the camera.

5. The detection method according to claim 4, characterized in that: Detection methods include: Creating a database consisting of multiple eye images of multiple test subjects looking at multiple target locations in space from different facial angles; Performing image preprocessing on the plurality of eye images in the database; Establish deep learning models based on deep residual networks and deep convolutional neural networks; The deep learning model is trained using the preprocessed plurality of eye images to obtain the gaze detection model.

6. The detection method according to claim 5, characterized in that: The step of creating a database consisting of multiple eye images of multiple test subjects looking at multiple target positions in space from different facial angles includes: Record the test subject's facial angle and the corresponding sight line position, and calculate the test subject's sight line angle based on the face; The database is obtained by taking the sight point angle and the facial angle together as image labels of the eye image.

7. A device for detecting a line of sight direction, characterized in that: The detection device comprises: A thermal imaging module, which is used to obtain infrared thermal imaging images collected by a thermal imaging camera; A positioning module, the positioning module is used to locate the target face area when the target face exists in the infrared thermal imaging image; A determination module, the determination module is used to determine the eye position in the target face area; A visible light image acquisition module, the visible light image acquisition module is used to control the target camera to focus and shoot the eye position to obtain a visible light binocular image; A sight line direction acquisition module, the sight line direction acquisition module is used to process the visible light binocular image using a sight line detection model to obtain the sight line landing direction of the target face; The controlling the target camera to focus and shoot the eye position to obtain a visible light binocular image comprises: Determining the target camera according to the eye position; Controlling the target camera to scale the field of view to the field of view corresponding to the eye position; After the camera is zoomed to a preset field of view, the camera is controlled to perform automatic focusing and shooting to obtain the visible light binocular image.

8. A system for detecting the direction of sight, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store a computer program, and the processor implements the detection method according to any one of claims 1 to 6 when executing the computer program.

9. A non-volatile computer-readable storage medium for a computer program, characterized in that: When the computer program is executed by one or more processors, the detection method according to any one of claims 1 to 6 is implemented.

Images