Line-of-sight detection method, line-of-sight detection device, and control program product

By using ear and hair information in non-contact line of sight detection technology, combining single eye position, instead of binocular position relationship, high-precision detection of face orientation, the line of sight detection accuracy is solved in the case where only one eye can be detected in the prior art Low problems achieve higher line of sight detection accuracy.

CN119028006APending Publication Date: 2024-11-26SWALLOW INCUBATE CO LTD +1
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411122558.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2019-11-13
Filing Date
2020-02-06
Publication Date
2024-11-26

Smart Images

  • Figure CN119028006A_ABST
    Figure CN119028006A_ABST
Patent Text Reader

Abstract

The invention provides a line-of-sight detection method, a line-of-sight detection device, and a control program product. The line-of-sight detection method obtains image data of the face of a person; detecting a left eye region including a left eye of a person and a right eye region including a right eye of the person from the image data of the face; detecting at least one of an ear region including a person's ear and a hair region including a person's hair from the image data of the face when only one of the right eye region and the left eye region can be detected; detecting information indicating the orientation of the face on the basis of information indicating the position of one eye included in the one eye region and information indicating the position of at least one of the position of the ear included in the ear region and the position of the hair included in the hair region; on the basis of information indicating the position of one eye and information indicating the orientation of the face, information indicating the line of sight of a person is detected.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of the invention patent application with the application date of February 6, 2020, application number 202080078883.1 (international application number PCT / JP2020 / 004552), and invention name “Line of sight detection method, line of sight detection device and control program”. Technical Field

[0002] The present invention relates to a technology for detecting a person's sight line. Background Art

[0003] The line of sight detection technology is used for various purposes such as inferring objects of interest to a person, inferring the state of a person such as sleepiness, and user interfaces for inputting input to a device through line of sight. The line of sight detection technology is roughly divided into head-mounted type and non-contact type. The head-mounted line of sight detection technology is a technology that wears a device such as a head-mounted display on a person's head to detect the line of sight. The non-contact line of sight detection technology is a technology that can detect the line of sight without wearing the device on the human body. Since the non-contact line of sight detection technology does not require the device to be worn on the head like the head-mounted line of sight detection technology, it has the advantages of not causing trouble to people and not blocking people's field of vision. Therefore, the present invention focuses on the non-contact line of sight detection technology.

[0004] Here, non-contact gaze detection technology is roughly divided into pupil-corneal reflection method and method based on three-dimensional eyeball model. Pupil-corneal reflection method, for example, as described in Patent Document 1, is a method of irradiating light to a person's eye through a light-emitting element and detecting gaze based on the position of a reflected image projected onto the eye, the irradiation angle of the light, the amount of movement of the position of the reflected image, etc.

[0005] The method based on the three-dimensional eyeball model is a method of estimating the sight line direction based on the information related to the eyeball obtained by analyzing the facial image. For example, Patent Document 2 discloses a method of estimating the sight line direction based on the rotation angle of the eyeball and the orientation of the face obtained by analyzing the facial image. In addition, there is also a known method of estimating the sight line direction by a three-dimensional vector connecting the center position of the eyeball and the center position of the iris obtained by analyzing the facial image, or estimating the sight line direction by a normal line of a section relative to the center position of the pupil.

[0006] However, the non-contact sight line detection technology is difficult to detect the sight line direction with high accuracy when using a facial image representing one side of the face, in other words, when only one eye can be detected from the facial image. For this reason, for example, for an object set to the side relative to the direction of travel of a pedestrian, it is difficult to judge the degree of attention or the presence or absence of inattention based on the pedestrian's sight line information. The non-contact sight line detection technology disclosed in the above-mentioned patent documents 1 and 2 does not specifically consider the sight line detection method when only one eye can be detected.

[0007] Prior art literature

[0008] Patent Literature

[0009] Patent document 1: Japanese Patent Publication No. 61-172552

[0010] Patent Document 2: Japanese Patent Publication No. 2003-271932 Summary of the invention

[0011] The present invention has been made to solve the above-mentioned problem, and an object of the present invention is to further improve the line of sight detection accuracy when only one eye can be detected from a facial image.

[0012] The first embodiment of the present invention involves a line of sight detection method, which is a line of sight detection method of a line of sight detection device, acquiring image data of a person's face; detecting a left eye area including the person's left eye and a right eye area including the person's right eye from the image data of the face; when only the eye area of ​​the right eye area or the left eye area can be detected, detecting at least one of an ear area including the person's ear and a hair area including the person's hair from the image data of the face; based on information indicating the position of one eye included in the eye area of ​​the one party and information indicating the position of at least one of the position of the ear included in the ear area and the position of the hair included in the hair area, detecting information indicating the orientation of the face; based on the information indicating the position of the one eye and the information indicating the orientation of the face, detecting information indicating the line of sight of the person.

[0013] The second embodiment of the present invention involves a line of sight detection method, which is a line of sight detection method of a line of sight detection device, which obtains image data of a person's face; detects a left eye area including the person's left eye and a right eye area including the person's right eye from the image data of the face; when the right eye area and the left eye area are detected, detects information indicating the orientation of the face from the image data of the face, and detects information indicating the line of sight of the person through a first processing, the first processing utilizing information indicating the position of the right eye included in the right eye area and information indicating the position of the left eye included in the left eye area and information indicating the orientation of the face; when only the eye area of ​​the right eye area or the left eye area can be detected, based on information indicating the position of one eye included in the eye area of ​​the one eye and information indicating the position of a part of the face different from the eye, detects information indicating the orientation of the face, and through a second processing utilizing information indicating the position of one eye and information indicating the orientation of the face, detects information indicating the line of sight of the person. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram showing an example of the overall configuration of the image processing system according to the first embodiment of the present invention.

[0015] Figure 2 This is a block diagram showing an example of a detailed configuration of the image processing system according to the first embodiment.

[0016] Figure 3 : is a schematic diagram showing an example of a right eye area and a left eye area.

[0017] Figure 4 This is a block diagram showing an example of the detailed configuration of a monocular sight line detection unit.

[0018] Figure 5 is a schematic diagram showing an example of the ear region.

[0019] Figure 6 is a schematic diagram showing an example of a face region.

[0020] Figure 7 : is a schematic diagram showing an example of a binarized face region.

[0021] Figure 8 This is a schematic diagram showing another example of a binarized face region.

[0022] Fig. 9 This is a schematic diagram showing an example of the orientation of a person's face when viewed from above.

[0023] Fig.10 It is a schematic diagram showing the human perspective.

[0024] Fig.11 This is a diagram for explaining a method for detecting a gaze direction.

[0025] Fig.12 This is a flowchart showing an example of the operation of the image processing device according to the first embodiment.

[0026] Fig.13 This is a flowchart showing an example of the operation of the monocular sight line detection unit according to the first embodiment.

[0027] Fig.14 This is a block diagram showing an example of a detailed configuration of an image processing system according to the second embodiment. DETAILED DESCRIPTION

[0028] Basic knowledge of the present invention

[0029] The existing line of sight detection method based on the three-dimensional eyeball model analyzes the facial image to detect information related to the eyeballs of both eyes, and infers the line of sight direction based on the detected information. However, there are cases where a facial image containing only one eye is used for line of sight detection because the person being detected is not paying attention. In this case, it can only be assumed that the one eye that can be detected from the facial image and the one eye that cannot be detected from the facial image are facing approximately the same direction.

[0030] Thus, the conventional line of sight detection method of the three-dimensional eyeball model is difficult to detect the line of sight direction with high accuracy when only a facial image representing one side of the face can be detected. If the line of sight direction cannot be detected with high accuracy, for example, it is difficult to determine the degree of attention or the presence or absence of inattention based on the line of sight information of the pedestrian, for an object located to the side relative to the direction of travel of the pedestrian.

[0031] The inventors of the present invention have conducted detailed research on this issue and have come to the conclusion that, when only one eye can be detected from a facial image, the accuracy of gaze detection can be improved by utilizing information related to parts other than the eye, and therefore conceived the various embodiments shown below.

[0032] The first embodiment of the present invention involves a line of sight detection method, which is a line of sight detection method of a line of sight detection device, acquiring image data of a person's face; detecting a left eye area including the person's left eye and a right eye area including the person's right eye from the image data of the face; when only the eye area of ​​the right eye area or the left eye area can be detected, detecting at least one of an ear area including the person's ear and a hair area including the person's hair from the image data of the face; based on information indicating the position of one eye included in the eye area of ​​the one party and information indicating the position of at least one of the position of the ear included in the ear area and the position of the hair included in the hair area, detecting information indicating the orientation of the face; based on the information indicating the position of the one eye and the information indicating the orientation of the face, detecting information indicating the line of sight of the person.

[0033] According to this configuration, when only the right eye region or the left eye region can be detected from the facial image data, at least one of the ear region and the hair region is also detected from the facial image data. Furthermore, based on information indicating the position of one eye included in the eye region of the one and information indicating the position of the ear or the hair included in at least one of the detected ear region and the hair region, information indicating the orientation of the face is detected. Furthermore, based on information indicating the orientation of the detected face and information indicating the position of the one eye, information indicating the line of sight of the person is detected.

[0034] Thus, even when image data of a face including only the right eye or the left eye is obtained, the face orientation can be detected with high accuracy based on the positional relationship between the detected single eye and the ear or hair instead of the positional relationship between the two eyes. As a result, even when only one eye can be detected from a face image, the face orientation can be determined with high accuracy, thereby improving the detection accuracy of the line of sight.

[0035] In the above-mentioned embodiment, when only the eye region of the one party can be detected, the ear region can be detected from the image data of the face, and when the ear region can be detected, the information indicating the orientation of the face can be detected based on the information indicating the position of the eyes of the one party and the information indicating the position of the ears.

[0036] According to this configuration, when only one of the right eye region or the left eye region can be detected from the face image data, the ear region is also detected from the face image data, and information indicating the orientation of the face is detected based on information indicating the position of one eye included in the one eye region and information indicating the position of the ear included in the detected ear region. Thus, even when only one eye can be detected from the face image, this configuration can detect the orientation of the face with high accuracy based on the positional relationship between a single eye and an ear instead of the positional relationship between both eyes.

[0037] In the above-mentioned embodiment, when the ear region cannot be detected, the hair region may be detected from the image data of the face, and information indicating the orientation of the face may be detected based on information indicating the position of the eyes of the one party and information indicating the position of the hair.

[0038] According to this configuration, when only one of the right eye region or the left eye region can be detected from the face image data, the hair region is also detected from the face image data, and information indicating the orientation of the face is detected based on information indicating the position of one eye included in the one eye region and information indicating the position of the hair included in the detected hair region. Thus, even when only one eye can be detected from the face image, this configuration can detect the orientation of the face with high accuracy based on the positional relationship between the single eye and the hair instead of the positional relationship between the two eyes.

[0039] In the above-mentioned embodiment, it is also possible to detect a contour point representing a point on the contour line of the face that is located on the inner corner side of the eye of the one party compared to the position of the outer corner of the eye of the one party based on the image data of the face and the information representing the position of the eye of the one party, and detect information representing the orientation of the face based on a first distance from the position of the eye of the one party to the contour point and a second distance from the position of the eye of the one party to the position of the ear.

[0040] According to this configuration, information indicating the orientation of the face is detected based on a first distance from the position of the eye of the one to a contour point indicating a point on the contour line of the face that exists on the inner corner side of the eye of the one compared to the position of the outer corner of the eye of the one, and a second distance from the position of the eye of the one to the position of the ear. Thus, this configuration can detect the orientation of the face with high accuracy based on the width of the face from the detected single eye to the inner corner side indicated by the first distance and the width of the face from the detected single eye to the ear side, i.e., the outer corner side indicated by the second distance.

[0041] In the above-mentioned embodiment, a contour point representing a point on the contour line of the face that is located on the inner corner side of the one eye compared to the position of the outer corner of the one eye may be detected based on the image data of the face and the information representing the position of the one eye, and information representing the orientation of the face may be detected based on a first distance from the position of the one eye to the contour point and a third distance from the position of the one eye to a point within the hair region that is located on the outer corner side compared to the position of the inner corner of the one eye.

[0042] According to this configuration, information indicating the orientation of the face is detected based on a first distance from the position of the one eye to a contour point indicating a point on the contour line of the face that is located on the inner corner side of the one eye compared to the position of the outer corner of the one eye, and a third distance from the position of the one eye to a point in the hair region that is located on the outer corner side of the one eye compared to the position of the inner corner of the one eye. Thus, this configuration can detect the orientation of the face with high accuracy based on the width of the face from the detected single eye to the inner corner side indicated by the first distance and the width of the face from the detected single eye to the outer corner side indicated by the third distance.

[0043] In the above-mentioned embodiment, when the right eye area and the left eye area are detected, information indicating the orientation of the face may be detected from the image data of the face, and information indicating the line of sight of the person may be detected based on information indicating the position of the right eye contained in the right eye area and the position of the left eye contained in the left eye area and information indicating the orientation of the face.

[0044] According to this configuration, when the right eye region and the left eye region are detected from the image data of the face, information indicating the orientation of the face is detected from the image data of the face. Furthermore, based on the information indicating the orientation of the detected face and the information indicating the positions of the two eyes, information indicating the sight line of the person is detected. Thus, in this configuration, when the two eyes can be detected from the image of the face, the sight line can be detected based on the positions of the two eyes obtained from the image data of the face and the orientation of the face.

[0045] The second embodiment of the present invention involves a line of sight detection method, which is a line of sight detection method of a line of sight detection device, which obtains image data of a person's face; detects a left eye area including the person's left eye and a right eye area including the person's right eye from the image data of the face; when the right eye area and the left eye area are detected, detects information indicating the orientation of the face from the image data of the face, and detects information indicating the line of sight of the person through a first processing, the first processing utilizing information indicating the position of the right eye included in the right eye area and information indicating the position of the left eye included in the left eye area and information indicating the orientation of the face; when only the eye area of ​​the right eye area or the left eye area can be detected, based on information indicating the position of one eye included in the eye area of ​​the one eye and information indicating the position of a part of the face different from the eye, detects information indicating the orientation of the face, and through a second processing utilizing information indicating the position of one eye and information indicating the orientation of the face, detects information indicating the line of sight of the person.

[0046] According to this configuration, when the right eye region and the left eye region are detected from the facial image data, information indicating the orientation of the face is detected from the facial image data. Furthermore, information indicating the line of sight of a person is detected by the first processing using the information indicating the orientation of the detected face and the information indicating the positions of the two eyes. Thus, in this configuration, when the two eyes can be detected from the facial image, the line of sight can be detected based on the positions of the two eyes and the orientation of the face by performing the first processing.

[0047] On the other hand, in the case where only the right eye area or the left eye area can be detected from the facial image data, information indicating the orientation of the face is detected based on information indicating the position of the eye of the one included in the eye area of ​​the one and information indicating the position of a part of the face different from the eye. Furthermore, information indicating the line of sight of a person is detected by a second process using the information indicating the orientation of the detected face and the information indicating the position of the eye of the one. Thus, in this configuration, in the case where only one eye can be detected from the facial image, the orientation of the face can be detected based on the positional relationship between the part different from the eye and the detected single eye, and the line of sight can be detected based on the position of the detected single eye and the orientation of the detected face by performing the second process.

[0048] In this manner, this configuration switches the processing according to the number of eyes detected from the facial image, thereby enabling appropriate line of sight detection.

[0049] In the above-mentioned embodiment, the different parts may be ears or hair.

[0050] According to this configuration, when only one of the right eye region or the left eye region can be detected from the image data of the face, information indicating the orientation of the face is detected based on information indicating the position of one eye included in the eye region of the one and information indicating the position of the ear or the hair. Thus, even when only one eye can be detected from the face image, this configuration can detect the orientation of the face with high accuracy based on the positional relationship between the detected single eye and the ear or the hair.

[0051] In the above-mentioned embodiment, it is also possible that, when only the eye region of the one party can be detected, the ear region including the ear of the person is detected from the image data of the face, and when the ear region can be detected, the information indicating the orientation of the face is detected based on the information indicating the position of the eye of the one party and the information indicating the position of the ear included in the ear region.

[0052] According to this configuration, when only one of the right eye region or the left eye region can be detected from the face image data, the ear region is detected from the face image data, and information indicating the orientation of the face is detected based on information indicating the position of one eye included in the eye region of the one and information indicating the position of the ear included in the detected ear region. Thus, even when only one eye can be detected from the face image, this configuration can detect the orientation of the face with high accuracy based on the positional relationship between the single eye and the ear instead of the positional relationship between the two eyes.

[0053] In the above-mentioned embodiment, when the ear area cannot be detected, a hair area including the hair of the person may be detected from the image data of the face, and information indicating the orientation of the face may be detected based on information indicating the position of the eyes of the one party and information indicating the position of the hair included in the hair area.

[0054] According to this configuration, when only one of the right eye region or the left eye region can be detected from the face image data, the hair region is detected from the face image data, and information indicating the orientation of the face is detected based on information indicating the position of one eye included in the one eye region and information indicating the position of the hair included in the detected hair region. Thus, even when only one eye can be detected from the face image, this configuration can detect the orientation of the face with high accuracy based on the positional relationship between the single eye and the hair instead of the positional relationship between the two eyes.

[0055] In the above-mentioned embodiment, it is also possible to detect a contour point representing a point on the contour line of the face that is located on the inner corner side of the eye of the one party compared to the position of the outer corner of the eye of the one party based on the image data of the face and the information representing the position of the eye of the one party, and detect information representing the orientation of the face based on a first distance from the position of the eye of the one party to the contour point and a second distance from the position of the eye of the one party to the position of the ear.

[0056] According to this configuration, information indicating the orientation of the face is detected based on a first distance from the position of the eye of the one to a contour point indicating a point on the contour line of the face that exists on the inner corner side of the eye of the one compared to the position of the outer corner of the eye of the one, and a second distance from the position of the eye of the one to the position of the ear. Thus, this configuration can detect the orientation of the face with high accuracy based on the width of the face from the detected single eye to the inner corner side indicated by the first distance and the width of the face from the detected single eye to the ear side, i.e., the outer corner side indicated by the second distance.

[0057] In the above-mentioned embodiment, a contour point representing a point on the contour line of the face that is located on the inner corner side of the one eye compared to the position of the outer corner of the one eye may be detected based on the image data of the face and the information representing the position of the one eye, and information representing the orientation of the face may be detected based on a first distance from the position of the one eye to the contour point and a third distance from the position of the one eye to a point within the hair region that is located on the outer corner side compared to the position of the inner corner of the one eye.

[0058] According to this configuration, information indicating the orientation of the face is detected based on a first distance from the position of the one eye to a contour point indicating a point on the contour line of the face that is located on the inner corner side of the one eye compared to the position of the outer corner of the one eye, and a third distance from the position of the one eye to a point in the hair region that is located on the outer corner side of the one eye compared to the position of the inner corner of the one eye. Thus, this configuration can detect the orientation of the face with high accuracy based on the width of the face from the detected single eye to the inner corner side indicated by the first distance and the width of the face from the detected single eye to the outer corner side indicated by the third distance.

[0059] The present invention can also be implemented as a control program that allows a computer to execute the characteristic components included in the sight line detection method, or as a sight line detection device that operates according to the control program. Moreover, it is needless to say that such a program can also be circulated through a computer-readable non-temporary recording medium such as a CD-ROM or a communication network such as the Internet.

[0060] In addition, each embodiment described below is an embodiment of a specific example of the present invention. The numerical values, shapes, constituent elements, steps, and the order of steps shown in the following embodiments are only a specific example and are not used to limit the present invention. Moreover, among the constituent elements in the following embodiments, the constituent elements that are not recorded in the independent claims representing the highest concept are described as arbitrary constituent elements. Moreover, for all embodiments, their contents can be arbitrarily combined.

[0061] First embodiment

[0062] Figure 1 1 is a schematic diagram showing an example of the overall configuration of an image processing system 1 according to the first embodiment of the present invention. The image processing system 1 is a system that captures a person 400 and detects line of sight information indicating the line of sight of the person 400 from the image data of the person 400 obtained. Figure 1 In the example of , the image processing system 1 determines which target 301 the person 400 is looking at among the multiple targets 301 displayed on the display device 300. However, this is only an example, and the image processing system 1 can determine not only the target 301 displayed on the display screen of the display device 300, but also the target 301 that the person 400 is looking at in the actual space.

[0063] exist Figure 1 In the example of FIG. 1 , the image processing system 1 is applied to a digital signage system. Therefore, the object 301 displayed on the display device 300 is an image of a signage such as an advertisement.

[0064] The image processing system 1 includes an image processing device 100, a camera 200, and a display device 300. The image processing device 100 is connected to the camera 200 and the display device 300 via a predetermined communication path. The predetermined communication path is, for example, a wired communication path such as a wired LAN or a wireless communication path such as a wireless LAN and Bluetooth (registered trademark). The image processing device 100 is, for example, composed of a computer installed around the display device 300. However, this is just an example, and the image processing device 100 may also be composed of a cloud server. In this case, the image processing device 100 is connected to the camera 200 and the display device 300 via the Internet. The image processing device 100 detects the line of sight information of the person 400 based on the image data of the person 400 captured by the camera 200, and outputs it to the display device 300. In addition, the image processing device 100 may be incorporated into the camera 200 or the display device 300 as hardware. In addition, the camera 200 or the display device 300 may be provided with a processor, and the image processing device 100 may be incorporated into it as software.

[0065] The camera 200, for example, acquires image data of a person 400 located around the display device 300 by photographing the environment around the display device 300 at a predetermined frame rate. The camera 200 sequentially outputs the acquired image data to the image processing device 100 at a predetermined frame rate. The camera 200 may be a visible light camera or an infrared camera. Furthermore, the camera 200 may also be a visible light camera or an infrared camera functionally provided in an electronic device such as the display device 300 or a smartphone.

[0066] The display device 300 is constituted by a display device such as a liquid crystal panel or an organic EL panel. Figure 1 In the example of , the display device 300 is a sign display. Figure 1 In the example, the image processing system 1 is described as including the display device 300, but this is only an example, and other devices may be used instead of the display device 300. For example, when the image processing system 1 is used as a user interface for accepting input to the device through sight, the image processing system 1 may also use household appliances such as refrigerators, televisions, and washing machines instead of the display device 300. For example, when the image processing system 1 is mounted on a vehicle, a vehicle such as a car may be used instead of the display device 300. In this case, the camera 200 may also be a visible light camera or an infrared camera functionally possessed by a household appliance or a vehicle. In addition, a storage device such as a hard disk drive or a solid state drive may be used instead of the display device 300.

[0067] Figure 21 is a block diagram showing an example of a detailed configuration of the image processing system 1 according to the first embodiment. The image processing device 100 includes a processor 110. The processor 110 is a circuit such as a CPU or an FPGA. The processor 110 includes a sight line detection device 120 and an output unit 130.

[0068] The sight line detection device 120 includes an image acquisition unit 121, an eye detection unit 122, a binocular sight line detection unit 123, and a monocular sight line detection unit 124. In addition, each module provided by the processor 110 may be implemented by causing the processor 110 to execute a control program that enables the computer to function as an image processing device, or may be constituted by a dedicated circuit.

[0069] The image acquisition unit 121 acquires image data captured by the camera 200. Here, the acquired image data includes the face of a person 400 (an example of a person) around the display device 300. In addition, the image data acquired by the image acquisition unit 121 may be image data posted on a website, for example, or may be image data stored in an external storage device.

[0070] The eye detection unit 122 detects a left eye region including the left eye of the person 400 and a right eye region including the right eye of the person 400 from the image data acquired by the image acquisition unit 121. Hereinafter, the left eye region and the right eye region are collectively referred to as eye regions. Specifically, the eye detection unit 122 can detect the eye region using a classifier pre-created for detecting the eye region. The classifier used here is, for example, a Haar-like cascade classifier pre-created for detecting the eye region in an open-source image processing library.

[0071] The eye region is a rectangular region having a size equal to the size of the eye plus a predetermined margin. However, this is merely an example, and the shape of the eye region may be other than a rectangle, such as a triangle, a pentagon, a hexagon, or an octagon. Furthermore, the position at which the boundary of the eye region is set relative to the eye depends on the performance of the classifier.

[0072] Figure 3 is a schematic diagram showing an example of the right eye area 50R and the left eye area 50L. Figure 3 As shown in FIG. 1 , in this embodiment, the eye refers to a region surrounded by the upper eyelid boundary 53 and the lower eyelid boundary 54 and including colored parts such as the white eyeball and the black eyeball. Figure 3As shown, the pupil refers to a colored portion including the pupil 55 and the annular iris 56 surrounding the pupil 55. In addition, in this embodiment, for the sake of convenience of explanation, the right eye refers to the eye located on the right side when the person 400 is viewed from the front, and the left eye refers to the eye located on the left side when the person 400 is viewed from the front. Figure 3 This shows an example in which the eye detection unit 122 detects a right eye region 50R including the right eye and a left eye region 50L including the left eye. However, this is only an example, and the eye located on the right side as viewed from the person 400 may be the right eye, and the eye located on the left side as viewed from the person 400 may be the left eye. In addition, in this embodiment, the direction on the right side of the paper is referred to as the right direction, and the direction on the left side of the paper is referred to as the left direction.

[0073] The binocular sight line detection unit 123 detects information indicating the orientation of the face of the person 400 (hereinafter referred to as face orientation information) from the image data acquired by the image acquisition unit 121 when the right eye region 50R and the left eye region 50L are detected by the eye detection unit 122. Furthermore, the binocular sight line detection unit 123 detects information indicating the sight line of the person 400 (hereinafter referred to as sight line information) by a first process using information indicating the position of the right eye included in the right eye region 50R and information indicating the position of the left eye included in the left eye region 50L and the detected face orientation information.

[0074] Specifically, the detection of facial orientation information and the first processing performed by the binocular sight line detection unit 123 may be applied to a known sight line detection process for detecting sight lines using a three-dimensional eyeball model. The binocular sight line detection unit 123 may detect facial orientation information from the image data acquired by the image acquisition unit 121 according to the known sight line detection process applied to the detection of facial orientation information. The facial orientation information includes, for example, an angle indicating the frontal direction of the face relative to the optical axis of the camera 200. Furthermore, the binocular sight line detection unit 123 may acquire information indicating the position of the right eye and the position of the left eye from the right eye area 50R and the left eye area 50L detected by the eye detection unit 122 according to the known sight line detection process applied as the first processing. The information indicating the position of the right eye includes, for example, the pupil of the right eye, the inner corner of the right eye, the outer corner of the right eye, and the position of the center of gravity of the right eye. Similarly, the information indicating the position of the left eye includes, for example, the pupil of the left eye, the pupil of the left eye, the inner corner of the left eye, the outer corner of the left eye, and the position of the center of gravity of the left eye.

[0075] The sight line information may include a vector that three-dimensionally indicates the direction of the sight line of the person 400, or may include coordinate data of a gaze point on a predetermined object plane (eg, display device 300). The gaze point is, for example, a position where the object plane intersects the vector indicating the sight line.

[0076] The monocular sight line detection unit 124 detects the facial orientation information based on the information indicating the position of the eye included in the eye area 50 of the right eye area 50R or the left eye area 50L and the information indicating the position of the part of the face of the person 400 different from the eye, when the eye detection unit 122 can detect only the eye area 50 of the right eye area 50R or the left eye area 50L. Furthermore, the monocular sight line detection unit 124 detects the sight line information by the second processing using the information indicating the position of the eye of the person 400 and the detected facial orientation information. The part different from the eye includes, for example, ears and hair. The details of the monocular sight line detection unit 124 will be described later.

[0077] The output unit 130 outputs the sight line information detected by the binocular sight line detection unit 123 and the monocular sight line detection unit 124 to the display device 300. The output unit 130 may also obtain information of the target 301 displayed on the display device 300, and determine the target 301 (hereinafter referred to as the gaze target) that the person 400 is gazing at based on the obtained information and the coordinate data of the gaze point, and output the determination result to the display device 300.

[0078] Since the camera 200 is Figure 1 It has been explained in , and its description is omitted here.

[0079] The display device 300 displays, for example, a mark indicating the line-of-sight information output from the output unit 130. The display device 300 may display, for example, a mark indicating the object 301 output from the output unit 130 at which the person 400 is looking.

[0080] For example, it is assumed that the coordinate data of the gaze point is output as the line of sight information to the display device 300. In this case, the display device 300 performs a process of causing a marker indicating the line of sight position to be displayed in an overlapping manner on the image being displayed at a position corresponding to the coordinate data. For example, it is assumed that the determination result of the gaze target is output to the display device 300. In this case, the display device 300 may perform a process of causing a marker indicating the gaze target to be displayed in an overlapping manner on the screen being displayed.

[0081] In addition, when the image processing system 1 is composed of a household appliance instead of the display device 300, the household appliance accepts the input of the line of sight information by the person 400. Furthermore, when the image processing system 1 is composed of a storage device instead of the display device 300, the storage device stores the line of sight information. In this case, the storage device can store the line of sight information and the time stamp in correspondence with each other.

[0082] Next, the details of the monocular sight line detection unit 124 will be described. Figure 4 1 is a block diagram showing an example of a detailed configuration of the monocular sight line detection unit 124. Figure 4 As shown, the monocular gaze detection unit 124 includes an ear detection unit 41 (an example of a part detection unit), a hair detection unit 42 (an example of a part detection unit), a face orientation detection unit 43, and a unilateral face gaze detection unit 44 (an example of a gaze detection unit).

[0083] The ear detection unit 41 detects a right ear region including the right ear of the person 400 and a left ear region including the left ear of the person 400 from the image data acquired by the image acquisition unit 121. Specifically, the ear detection unit 41 can detect the right ear region using a right ear classifier pre-created to detect the region including the right ear of the person 400, and detect the left ear region using a left ear classifier pre-created to detect the region including the left ear of the person 400. The right ear classifier and the left ear classifier used here are, for example, Haar-like cascade classifiers pre-created in an open-source image processing library to detect the right ear region and the left ear region, respectively. Hereinafter, when the right ear region and the left ear region are collectively referred to, they are described as ear regions.

[0084] The ear region is a rectangular region having a size approximately equal to that of an ear. However, this is merely an example, and the shape of the ear region may be other than a rectangle, such as a triangle, a pentagon, a hexagon, or an octagon. Furthermore, the position at which the boundary of the ear region is set relative to the ear depends on the performance of the classifier.

[0085] Figure 5 is a schematic diagram showing an example of the ear region 30. Figure 5 As shown, in this embodiment, the ear refers to the area representing the auricle surrounded by the dotted line 31. Moreover, in this embodiment, for the sake of convenience of explanation, the right ear refers to the ear located on the right side when the person 400 is viewed from the front, and the left ear refers to the ear located on the left side when the person 400 is viewed from the front. Figure 5 This shows an example in which the ear detection unit 41 does not detect the right ear region 30R (not shown) with the right ear classifier and can detect only the left ear region 30L with the left ear classifier. However, this is only an example, and the ear located on the right side as viewed from the person 400 may be the right ear, and the ear located on the left side as viewed from the person 400 may be the left ear.

[0086] The hair detection unit 42 detects a hair region including the hair of the person 400 from the image data acquired by the image acquisition unit 121. The details of the process of detecting the hair region will be described below.

[0087] First, the hair detection unit 42 detects a face region including at least a part of the face of the person 400 from the image data acquired by the image acquisition unit 121. Specifically, the hair detection unit 42 may detect the face region using a classifier created in advance for detecting the face region. The classifier used here is, for example, a Haar cascade classifier created in advance for detecting the face region in an open source image processing library.

[0088] Figure 6 is a schematic diagram showing an example of the face region 60. Figure 6 As shown, the face region 60 is, for example, a rectangular region having a size that includes the entire face. However, this is only an example, and the shape of the face region 60 may be other than a rectangle, such as a triangle, a pentagon, a hexagon, or an octagon. Furthermore, the hair detection unit 42 may detect the face region 60, for example, by pattern matching.

[0089] Next, the hair detection unit 42 binarizes the detected face region 60 . Figure 7 : is a schematic diagram showing a binarized face region 60. Here, for example, a method called Otsu's binarization is used. In addition, a region with a brightness lower than a threshold is represented by white, and a region with a brightness higher than the threshold is represented by black.

[0090] Next, the hair detection unit 42 performs labeling processing to assign label numbers to white islands appearing in the binary face region 60. Figure 7 In the example of , the hair is regarded as an island 71, and the label number "1" is assigned to the island 71. Furthermore, the left eye is regarded as an island 72, and the label number "2" is assigned to the island 72. Furthermore, the left hole of the nose is regarded as an island 73, and the label number "3" is assigned to the island 73. Furthermore, the lips are regarded as an island 74, and the label number "4" is assigned to the island 74.

[0091] Then, the hair detection unit 42 detects the island with the largest area among the labeled islands as a hair region. Figure 7 In the example shown in FIG. 1 , island 71 is detected as a hair region.

[0092] The face direction detection unit 43 detects the face direction information based on the information indicating the position of one eye of the eye area 50 included in one of the right eye area 50R or the left eye area 50L detected by the eye detection unit 122, the information indicating the position of the ear included in the ear area 30 detected by the ear detection unit 41, or the information indicating the position of the hair included in the hair area detected by the hair detection unit 42. The face direction information includes information indicating whether the face of the person 400 is facing right or left and the degree to which the face of the person 400 is tilted in the left-right direction relative to the optical axis of the camera 200 (hereinafter referred to as the face direction degree). The details of the process of detecting the face direction information are described below.

[0093] The face orientation detection unit 43 first determines whether the face of the person 400 is facing right or left. Specifically, when the left ear region 30L is detected by the ear detection unit 41, the face orientation detection unit 43 determines that the person 400 is facing right, and determines that the face orientation is facing right. On the other hand, when the right ear region 30R is detected by the ear detection unit 41, the face orientation detection unit 43 determines that the person 400 is facing left, and determines that the face orientation is facing left. Figure 5 In the example of , since the ear detection unit 41 detects the left ear area 30L, it is determined that the direction of the face is toward the right.

[0094] In addition, there is a risk that the ear detection unit 41 may misdetect the right ear region 30R and the left ear region 30L. In consideration of this misdetection, the face orientation detection unit 43 may also be configured as described below. That is, after the ear detection unit 41 detects the ear region 30, the face orientation detection unit 43 may detect the face region 60 from the image data acquired by the image acquisition unit 121, similarly to the hair detection unit 42. Furthermore, the face orientation detection unit 43 may determine that the face is oriented to the right when the position of the eye region 50 detected by the eye detection unit 122 is located on the right side of the ear region 30 detected by the ear detection unit 41, and may determine that the face is oriented to the left when the position of the eye region 50 is located on the left side of the ear region 30.

[0095] Furthermore, when the position of the center of gravity of the island detected as the hair region by the hair detection unit 42 exists in the left half of the face region 60, the face direction detection unit 43 determines that the person 400 is facing rightward, and determines that the face direction is facing rightward. On the other hand, when the position of the center of gravity of the island detected as the hair region by the hair detection unit 42 exists in the right half of the face region 60, the face direction detection unit 43 determines that the person 400 is facing leftward, and determines that the face direction is facing leftward. Figure 7In the example of , since the position P71 of the center of gravity of the island 71 detected as the hair region by the hair detection unit 42 exists in the left half of the face region 60, it is determined that the direction of the face is rightward.

[0096] Next, the face orientation detection unit 43 detects contour points representing points on the contour line of the face of the person 400 that are located closer to the inner corner of the eye than the outer corner of the eye, based on the image data acquired by the image acquisition unit 121 and the information indicating the position of the eye included in the eye region 50 of one side detected by the eye detection unit 122. The following describes in detail the process of detecting contour points when the orientation of the face is determined to be toward the right. In addition, when the orientation of the face is determined to be toward the left, the process of detecting contour points is the same as when the orientation of the face is determined to be toward the right, so the description thereof is omitted.

[0097] First, the face direction detection unit 43 detects and binarizes the face area 60 from the image data acquired by the image acquisition unit 121, similarly to the hair detection unit 42. Next, the face direction detection unit 43 performs labeling processing to assign label numbers to white islands appearing in the binarized face area 60. Figure 8 FIG. 2 is a schematic diagram showing another example of the face region 60 that has been binarized.

[0098] exist Figure 8 In the example, the hair is regarded as an island 81, and the label number "1" is assigned to island 81. The left eye is regarded as an island 82, and the label number "2" is assigned to island 82. The left hole of the nose is regarded as an island 83, and the label number "3" is assigned to island 83. The lips are regarded as an island 84, and the label number "4" is assigned to island 84. The left eyebrow is regarded as an island 85, and the label number "5" is assigned to island 85. A part of the right eye is regarded as an island 86, and the label number "6" is assigned to island 86.

[0099] Next, the face direction detection unit 43 detects an island farthest from the inner corner of the eye included in the eye region 50 of one side detected by the eye detection unit 122, and the island is an island having an area less than a predetermined area among the labeled islands. The predetermined area is, for example, an area of ​​the same size as the eye region 50 of one side detected by the eye detection unit 122. Figure 8In the example of FIG. 1 , the face direction detection unit 43 determines that the face direction is toward the right. Therefore, the face direction detection unit 43 determines that the eye region 50 detected by the eye detection unit 122 is the left eye region 50L, and the island 82 included in the left eye region 50L represents the left eye. Next, the face direction detection unit 43 detects an island 86 farthest to the right from the inner corner of the left eye from the left eye region 50L, and the island 86 has an area less than a predetermined area.

[0100] Next, the face direction detection unit 43 detects the contour point assuming that the contour point exists on the detected island. Figure 8 In the example of , the face orientation detection unit 43 detects the end P81 on the inner corner side of the left eye, that is, the right end, of the detected island 86 as the contour point 61 .

[0101] Next, the face direction detection unit 43 detects the center of gravity and the position of the inner corner of the eye from the eye region 50 detected by the eye detection unit 122 as information indicating the position of the eye included in the eye region 50. Figure 8 In the example of , since the orientation of the face is determined to be rightward, the face orientation detection unit 43 determines that the eye region 50 detected by the eye detection unit 122 is the left eye region 50L, and the island 82 included in the left eye region 50L represents the left eye. Here, the face orientation detection unit 43 detects the position P82 of the center of gravity of the island 82 as the position 62 of the center of gravity of the left eye. Furthermore, the face orientation detection unit 43 detects the end P83 on the inner corner side of the left eye in the island 82, that is, on the right side, as the position 63 of the inner corner of the left eye.

[0102] Next, when the ear region 30 is detected by the ear detection unit 41, the face direction detection unit 43 calculates a first distance from the position of the inner corner of the eye included in the eye region 50 of one side detected by the eye detection unit 122 to the contour point 61, and calculates a second distance from the position of the inner corner of the eye to the position of the ear included in the ear region 30 detected by the ear detection unit 41. Then, the face direction detection unit 43 detects face direction information based on the first distance and the second distance.

[0103] exist Figure 8In the example of FIG. 5 , the face direction detection unit 43 calculates the horizontal distance L1 from the position 63 of the inner corner of the left eye included in the left eye region 50L to the contour point 61 as a first distance (hereinafter referred to as the first distance L1). The face direction detection unit 43 uses the inner corner side of the left eye in the left ear region 30L detected by the ear detection unit 41, that is, the right end as the position of the ear included in the ear region 30. Furthermore, the face direction detection unit 43 calculates the horizontal distance L2 from the position 63 of the inner corner of the left eye to the position of the ear as a second distance (hereinafter referred to as the second distance L2).

[0104] Fig. 9 400 is a schematic diagram showing an example of the degree of the face orientation of the person 400 when viewed from above. The greater the degree to which the face of the person 400 is tilted in the left-right direction when facing the camera 200, the greater the degree to which the face of the person 400 is tilted in the left-right direction, such as Fig. 9 As shown, the angle 93 (hereinafter referred to as the posture angle) representing the front direction 92 of the face of the person 400 relative to the optical axis 91 of the camera 200 becomes larger. Moreover, the larger the degree of the face orientation becomes, the shorter the first distance L1 from the position of the inner corner of the single eye photographed by the camera 200 to the contour point becomes, and the longer the second distance L2 from the position of the inner corner of the eye to the position of the ear becomes. Therefore, the function (hereinafter referred to as the first function) of the relationship between the posture angle 93 representing the degree of face orientation and the ratio of the second distance L2 to the first distance L1 (= L2 / L1) or the ratio of the first distance L1 to the second distance L2 (= L1 / L2) can be set in advance based on experimental values, etc.

[0105] Here, the face orientation detection unit 43 calculates the ratio of the second distance L2 to the first distance L1, or the ratio of the first distance L1 to the second distance L2. Moreover, the face orientation detection unit 43 obtains the posture angle 93 corresponding to the input ratio as the face orientation degree by inputting the calculated ratio into the first function. The face orientation detection unit 43 detects the face orientation degree obtained in this way as face orientation information. In addition, the method of obtaining the face orientation degree is not limited to this. For example, a function representing the relationship between the difference between the first distance L1 and the second distance L2 and the posture angle 93 can be pre-set, and the face orientation degree can be obtained using the function.

[0106] The ear region 30 cannot be detected from the image data of the face of the person 400 whose ears are covered by hair. Here, in this case, the face direction detection unit 43 detects the hair region from the image data of the face of the person 400 using the hair detection unit 42. Next, the face direction detection unit 43 calculates the first distance L1 and calculates the third distance, which is the distance from the position of the eye included in the eye region 50 of one side detected by the eye detection unit 122 to the point in the hair region located on the outer corner side relative to the position of the inner corner of the eye. Then, the face direction detection unit 43 detects the face direction information based on the first distance L1 and the third distance.

[0107] Below, using Figure 7 The process of detecting face orientation information from the hair area is described below. Figure 7 In the example of Figure 8 Similarly to the example, the island 72 included in the left eye region 50L is determined to represent the left eye. Furthermore, the face direction detection unit 43 detects the end P75 on the inner corner side of the left eye in the island 73 as the contour point 61, and the island 73 has an area less than a predetermined area and is farthest from the right side of the inner corner of the left eye from the left eye region 50L. Furthermore, the face direction detection unit 43 detects the position P72 of the center of gravity of the island 72 representing the left eye as the position 62 of the center of gravity of the left eye, and detects the end P73 on the inner corner side of the left eye in the island 72 as the position 63 of the inner corner of the left eye. Then, the face direction detection unit 43 calculates the first distance L1 which is the horizontal distance from the detected position 63 of the inner corner of the left eye to the detected contour point 61.

[0108] Furthermore, the face direction detection unit 43 detects the position P71 of the center of gravity of the hair region detected by the hair detection unit 42 as the point 64 in the hair region. The point 64 in the hair region is not limited thereto, and may be a point P74 that is farthest to the left from the position 63 of the inner corner of the left eye included in the left eye region 50L, which is the outer corner of the left eye. The face direction detection unit 43 calculates the horizontal distance L3 from the position 63 of the inner corner of the left eye to the point 64 in the hair region as a third distance (hereinafter referred to as the third distance L3).

[0109] Furthermore, the face direction detection unit 43 and Figure 8Similarly to the example of , the ratio of the third distance L3 to the first distance L1, or the ratio of the first distance L1 to the third distance L3 is calculated. Then, the face orientation detection unit 43 inputs the calculated ratio into a second function that is the same as the first function, and obtains the posture angle 93 corresponding to the input ratio as the face orientation degree. The second function is a function that represents the posture angle 93 ( Fig. 9 ) and the ratio of the third distance L3 to the first distance L1 or the ratio of the first distance L1 to the third distance L3 can be preset based on experimental values ​​or the like. The face orientation detection unit 43 detects the face orientation degree obtained in this way as face orientation information. In addition, the method for obtaining the face orientation degree is not limited to this. For example, a function representing the relationship between the difference between the first distance L1 and the third distance L3 and the posture angle 93 can be preset, and the face orientation degree can be obtained using the function.

[0110] The one-sided face gaze detection unit 44 performs a second process of detecting gaze information based on information indicating the position of the eye included in the one eye region 50 detected by the eye detection unit 122 and the face orientation information detected by the face orientation detection unit 43. The second process will be described in detail below.

[0111] Fig.10 Schematic diagram showing the viewing angle 94 of the person 400. Generally speaking, the viewing angle 94 of the eyes of the person 400 is limited to a range of 0 degrees to 60 degrees to the left and right relative to the front direction 92 of the face of the person 400. Here, in this embodiment, it is assumed that the viewing angle 94 is preset to 60 degrees.

[0112] In the second process, the one-sided facial gaze detection unit 44 first detects the position of the outer corner of the eye included in the one eye region 50 from the one eye region 50 detected by the eye detection unit 122 .

[0113] Fig.11 This is a diagram for explaining a method for detecting a gaze direction. Fig.11 Indicates magnification Figure 8 Schematic diagram of the left eye area 50L shown. Fig.11 In this example, the unilateral facial gaze detection unit 44 detects the left end P85 of the island 82 determined by the facial direction detection unit 43 to represent the left eye as the position 65 of the outer corner of the left eye.

[0114] The unilateral facial sight line detection unit 44 obtains the horizontal distance from the position of the center of gravity 62 of the eye included in the eye region 50 of one side detected by the eye detection unit 122 to the position 65 of the outer corner of the eye (hereinafter referred to as the center-outer-corner distance). Furthermore, the unilateral facial sight line detection unit 44 obtains the horizontal distance from the position of the center of gravity 62 of the eye included in the eye region 50 of the detected one side to the position 63 of the inner corner of the eye (hereinafter referred to as the center-inner-corner distance). Then, the unilateral facial sight line detection unit 44 calculates the angle representing the sight line direction relative to the frontal direction 92 of the face (hereinafter referred to as the sight line direction angle) based on the obtained center-outer-corner distance and center-inner-corner distance and the preset angle of view 94.

[0115] Specifically, it is generally believed that the closer the position 62 of the center of gravity of the eye included in the eye region 50 detected by the eye detection unit 122 is to the position 63 of the inner corner of the eye, the closer the line of sight of the person 400 is to the direction close to the front direction 92 of the face. On the other hand, it is also believed that the closer the position 62 of the center of gravity of the eye is to the position 65 of the outer corner of the eye, the closer the line of sight of the person 400 is to the direction deviating from the front direction 92 of the face, that is, to the direction close to the angular direction deviating from the above-mentioned view angle 94 to the left or right with respect to the front direction 92 of the face.

[0116] Here, in Fig.11 In the example of , the one-sided facial gaze detection unit 44 acquires the distance K1 between the outer corners of the left eye, which is the horizontal distance from the position 62 of the center of gravity of the left eye included in the left eye area 50L detected by the eye detection unit 122 to the position 65 of the outer corner of the left eye. Furthermore, the one-sided facial gaze detection unit 44 acquires the distance K2 between the inner corners of the left eye, which is the horizontal distance from the position 62 of the center of gravity of the left eye to the position 63 of the inner corner of the left eye. Then, the one-sided facial gaze detection unit 44 calculates the gaze direction angle β by the following formula using the acquired distances K1 between the outer corners of the left eye and the distances K2 between the inner corners of the left eye and the value α (=60°) of the preset angle of view 94.

[0117] β=α×(K2 / K1)

[0118] The unilateral face sight line detection unit 44 calculates a value obtained by adding the calculated sight line direction angle β to the face orientation detected by the face orientation detection unit 43 as an angle representing the direction of the sight line relative to the optical axis 91 of the camera 200. The unilateral face sight line detection unit 44 detects the calculated angle representing the direction of the sight line relative to the optical axis 91 of the camera 200 as sight line information.

[0119] Next, the operation of the image processing device 100 will be described. Fig.12 This is a flowchart showing an example of the operation of the image processing device 100 according to the first embodiment.

[0120] The image acquisition unit 121 acquires image data of the face of the person 400 from the camera 200 (step S1). Next, the eye detection unit 122 detects the right eye region 50R and the left eye region 50L from the image data by inputting the image data acquired in step S1 into a classifier for detecting the eye region 50 (step S2).

[0121] In step S2, if Figure 3 As shown in the example of , it is assumed that the right eye region 50R and the left eye region 50L are detected ("Yes" in step S3). In this case, the binocular sight line detection unit 123 detects sight line information (step S4) through the first processing using the information indicating the position of the right eye included in the right eye region 50R and the information indicating the position of the left eye included in the left eye region 50L, and the face orientation information detected from the image data acquired in step S1.

[0122] On the other hand, Figure 5 as well as Figure 6 As shown in the example of , in step S2, it is assumed that only one of the eye regions 50 of the right eye region 50R or the left eye region 50L can be detected ("No" in step S3). In this case, the monocular sight line detection unit 124 detects the facial orientation information based on the information indicating the position of the eye included in the eye region 50 of the one party and the information indicating the position of the part of the face of the person 400 different from the eye. Furthermore, the monocular sight line detection unit 124 detects the sight line information through the second processing (step S5), and the second processing uses the information indicating the position of the eye of the one party and the detected facial orientation information. The details of the operation of the monocular sight line detection unit 124 in step S5 will be described later.

[0123] After step S4 or step S5 , the output unit 130 outputs the line-of-sight information detected in step S4 or step S5 to the display device 300 (step S6 ).

[0124] Next, the operation of the monocular sight line detection unit 124 in step S5 will be described. Fig.13 This is a flowchart showing an example of the operation of the monocular sight line detection unit 124 according to the first embodiment.

[0125] When step S5 is started, the ear detection unit 41 detects the ear region 30 from the image data by inputting the image data acquired in step S1 into a classifier for detecting the right ear region and the left ear region (step S51). Figure 5 As shown, the left ear region 30L is detected from the image data acquired in step S1.

[0126] When the ear area 30 is detected in step S51 ("Yes" in step S52), the face orientation detection unit 43 detects the face orientation information based on the information indicating the position of the eye included in the eye area 50 of the party detected in step S2 and the information indicating the position of the ear included in the ear area 30 detected in step S51 (step S53).

[0127] For example, in step S51, assuming that Figure 5 In this case, in step S53, since the left ear region 30L was detected in step S51, the face direction detection unit 43 determines that the direction of the face is toward the right.

[0128] In this case, the face direction detection unit 43, as shown in FIG. Figure 8 As shown, based on the image data acquired in step S1 and the position 63 of the inner corner of the left eye included in the left eye region 50L detected in step S2, the first distance L1 which is the horizontal distance from the position 63 of the inner corner of the left eye to the contour point 61 is calculated. The face orientation detection unit 43 also uses the end of the inner corner of the left eye in the left ear region 30L detected in step S51 as the position of the ear included in the ear region 30, and calculates the second distance L2 which is the horizontal distance from the position 63 of the inner corner of the left eye to the position of the ear.

[0129] Furthermore, the face direction detection unit 43 calculates the ratio of the second distance L2 to the first distance L1 or the ratio of the first distance L1 to the second distance L2, and inputs the calculated ratio into the first function. Thus, the face direction detection unit 43 calculates the posture angle 93 ( Fig. 9 The face orientation detection unit 43 detects the face orientation degree acquired in this way as face orientation information.

[0130] On the other hand, when the ear region 30 is not detected in step S51 ("No" in step S52), the face direction detection unit 43 detects the hair region from the image data acquired in step S1 using the hair detection unit 42 (step S54). Figure 7As shown, in step S54, the face region 60 is detected by the hair detection unit 42 from the image data acquired in step S1, and the island 71 having the largest area in the region where the face region 60 is binarized is detected as a hair region.

[0131] Then, the face direction detection unit 43 detects face direction information based on the information indicating the position of the eye included in the one eye region 50 detected in step S2 and the information indicating the position of the hair included in the hair region detected in step S54 (step S55).

[0132] For example, in step S54, if Figure 7 As shown, it is assumed that the island 71 is detected as the hair region. In this case, in step S55, as described above, since the position P71 of the center of gravity of the island 71 detected as the hair region exists in the left half of the face region 60, the face direction detection unit 43 determines that the direction of the face is toward the right.

[0133] In this case, the face direction detection unit 43, as shown in FIG. Figure 7 As shown, based on the image data acquired in step S1 and the position 63 of the inner corner of the left eye included in the left eye region 50L detected in step S2, the first distance L1 which is the horizontal distance from the position 63 of the inner corner of the left eye to the contour point 61 is calculated. Furthermore, the face direction detection unit 43 calculates the third distance L3 which is the horizontal distance from the position 63 of the inner corner of the left eye to the point 64 in the hair region, using the position P71 of the center of gravity of the hair region detected in step S54 as the point 64 in the hair region.

[0134] Then, the face direction detection unit 43 calculates the ratio of the third distance L3 to the first distance L1 or the ratio of the first distance L1 to the third distance L3, and inputs the calculated ratio into the second function. Thus, the face direction detection unit 43 calculates the posture angle 93 ( Fig. 9 The face orientation detection unit 43 detects the face orientation degree acquired in this way as face orientation information.

[0135] After step S53 or step S55, the unilateral facial gaze detection unit 44 detects gaze information (step S56) by a second processing using information indicating the position of the eye included in the eye area 50 on one side detected in step S2 and the facial orientation information detected in step S53 or step S55.

[0136] For example, in step S56, the unilateral face sight line detection unit 44 may Fig.11As shown, the horizontal distance from the position 62 of the center of gravity of the left eye included in the left eye area 50L detected in step S2 to the outer corner position 65 of the left eye, that is, the distance between the center of gravity and the outer corner of the left eye, i.e., the distance between the center of gravity and the inner corner of the left eye, i.e., the distance between the center of gravity and the inner corner of the left eye, i.e., the distance between the center of gravity and the inner corner of the left eye, i.e., the distance between the center of gravity and the inner corner of the left eye, i.e., the distance between the center of gravity and the inner corner of the left eye, i.e., the distance between the center of gravity and the inner corner of the left eye, is obtained. Then, the single-sided face sight line detection unit 44 calculates the sight line direction angle β by using the above formula (β=α×(K2 / K1) using the obtained distance between the center of gravity and the outer corner of the left eye and the distance between the center of gravity and the inner corner of the left eye and the value α (=60°) of the preset angle of view 94. The single-sided face sight line detection unit 44 adds the calculated sight line direction angle β to the face orientation degree detected as the face orientation information in step S53 or step S55, and detects it as the sight line information.

[0137] As described above, according to the present embodiment, when only the eye region 50 of the right eye region 50R or the left eye region 50L can be detected from the image data of the face, at least one of the ear region 30 and the hair region is detected from the image data of the face. Then, based on the information indicating the position of the eye included in the eye region 50 of the one and the information indicating the position of the ear or the hair included in at least one of the detected ear region 30 and the hair region, the face direction information is detected. Then, based on the detected face direction information and the information indicating the position of the eye, the line of sight information is detected.

[0138] Thus, in this embodiment, even when only image data of a face including only the right eye or the left eye can be obtained, facial orientation information can be detected with high accuracy based on the positional relationship between the ear or hair and the detected single eye, instead of the positional relationship between the two eyes. As a result, this configuration can determine the orientation of the face with high accuracy even when only one eye can be detected from a facial image, thereby improving the detection accuracy of the line of sight.

[0139] In addition, in step S54, a case where the hair region cannot be detected may also be considered. In this case, the output unit 130 may output an error message indicating that the sight line information cannot be detected to the display device 300. Alternatively, step S4 ( Fig.12 ) for subsequent processing.

[0140] Second embodiment

[0141] The second embodiment is an embodiment of estimating the degree of interest of the person 400 . Fig.141 is a block diagram showing the detailed structure of the image processing system 1A according to the second embodiment. In this embodiment, the same reference numerals are given to the same components as those in the first embodiment, and their descriptions are omitted. Fig.14 In Figure 2 Modules with the same name but different functions are given an A symbol at the end.

[0142] The sight line detection device 120A further includes a feature point detection unit 140 and an interest level estimation unit 150 .

[0143] The feature point detection unit 140 and the hair detection unit 42 ( Figure 3 ) Similarly, using a classifier created in advance for detecting a facial region, a facial region 60 including at least a portion of the face of the person 400 is detected from the image data acquired by the image acquisition unit 121. Furthermore, the feature point detection unit 140 detects facial feature points from the detected facial region 60. The facial feature points refer to one or more points located at a feature position in each of a plurality of parts constituting the face, such as the outer corner of the eye, the inner corner of the eye, the contour of the face, the bridge of the nose, the corner of the mouth, and the eyebrows. Feature points are also called landmarks. The feature point detection unit 140 can detect facial feature points by executing a landmark detection process using a model file of a framework, such as machine learning.

[0144] The interest level estimation unit 150 estimates the interest level of the person 400 through the following processing. First, the interest level estimation unit 150 detects eyebrows and corners of the mouth from the face area 60 using the feature points of the face detected by the feature point detection unit 140. Here, the interest level estimation unit 150 can detect eyebrows and corners of the mouth by identifying feature points to which landmark point numbers corresponding to eyebrows and corners of the mouth are assigned, respectively, from among the feature points of the face detected by the feature point detection unit 140.

[0145] Next, the interest level estimation unit 150 estimates the interest level of the person 400 based on the sight line information detected by the binocular sight line detection unit 123 or the monocular sight line detection unit 124 and the detected position of the eyebrows and the position of the corners of the mouth, and outputs the estimated interest level of the person 400 to the display device 300. Specifically, the interest level estimation unit 150 obtains pattern data (pattern data) from a memory (not shown), for example, which predescribes the standard positions of the eyebrows and the corners of the mouth when the person shows various expressions such as joy, surprise, anger, sadness, and expressionlessness. The interest level estimation unit 150 compares the detected positions of the eyebrows and the corners of the mouth of the person 400 with the pattern data to estimate the expression of the person 400. The interest level estimation unit 150 determines what kind of expression the person 400 shows when the person 400's sight line is directed in which direction or the person 400's gaze point is located at which position, using the estimated expression of the person 400 and the sight line indicated by the sight line information. That is, the interest level estimation unit 150 determines the interest level of the person 400 by using the data corresponding to the line of sight information and the expression of the person 400. In addition, here, the interest level estimation unit 150 estimates the interest level based on the eyebrows and the corners of the mouth, but this is only an example, and the interest level may also be estimated based on either the eyebrows or the corners of the mouth.

[0146] As described above, according to this embodiment, since the interest level of person 400 is estimated using eyebrows and mouth corners in addition to line of sight information, the interest level can be estimated more accurately than estimating the interest level based on line of sight information alone.

[0147] Modifications

[0148] (1) When an infrared camera is used as the camera 200, the infrared camera can be constructed by using an infrared camera that uses infrared light of a specified second wavelength band whose spectral intensity of sunlight is attenuated compared to a specified first wavelength. The specified first wavelength is, for example, 850 nm. The specified second wavelength is, for example, 940 nm. The second wavelength band is, for example, a wavelength band of a specified width that does not include 850 nm and is based on 940 nm (for example, the center). As an infrared camera that captures near-infrared light, an infrared camera that uses infrared light of 850 nm is known. However, since the spectral intensity of sunlight is not sufficiently attenuated at 850 nm, there is a possibility that high-precision line of sight detection cannot be performed outdoors where the spectral intensity of sunlight is strong. Therefore, the present invention uses, for example, a camera that uses infrared light of the 940 nm band as an infrared camera. As a result, high-precision line of sight detection can be performed even outdoors where the spectral intensity of sunlight is strong. Here, the specified second wavelength is 940 nm, but this is only an example and a wavelength slightly deviated from 940 nm may also be used. In addition, the infrared camera using the infrared light of the second wavelength is, for example, a camera including a light projector for irradiating the infrared light of the second wavelength.

[0149] (2) In the above embodiment, the sight line information includes coordinate data indicating the gaze point, but the present invention is not limited to this. For example, the sight line information may also include coordinate data indicating an area of ​​a predetermined shape (e.g., a circle, a quadrilateral, etc.) of a predetermined size with the gaze point as a reference (e.g., the center), i.e., a gaze surface. In this way, the gaze object can be appropriately determined without depending on the distance between the person 400 and the gaze object or the size of the gaze object.

[0150] Industrial Applicability

[0151] The present invention is suitable for using line of sight information to predict a person's object of interest, predict a person's state, and use a user interface using line of sight, because it can detect line of sight information with high accuracy.

Claims

1. A sight line detection method, which is a sight line detection method of a sight line detection device, characterized in that The following steps are involved: Acquire image data of a person's face; detecting, from the image data of the face, a left eye region including the left eye of the person and a right eye region including the right eye of the person; When the right eye region and the left eye region are detected, information indicating the orientation of the face is detected from the image data of the face, and information indicating the line of sight of the person is detected by a first process using information indicating the position of the right eye included in the right eye region, information indicating the position of the left eye included in the left eye region, and information indicating the orientation of the face; In a case where only one of the right eye area or the left eye area is detected, information indicating the orientation of the face is detected based on information indicating the position of one eye included in the eye area of ​​the one eye and information indicating the position of a predetermined part of the face existing on the outer corner of the one eye compared to the position of the inner corner of the one eye, and information indicating the line of sight of the person is detected through a second processing using the information indicating the position of the one eye and the information indicating the orientation of the face. The second processing is a processing that performs the following steps: (1) detecting the specified part from the image data of the face; (2) detecting a contour point representing a point on the contour line of the face that is located on the inner corner side of the outer corner of the eye of the one eye relative to the position of the eye of the one eye based on the image data of the face; (3) detecting information representing the orientation of the face based on a first distance from the position of the eye of the one eye to the contour point and a second distance from the position of the eye of the one eye to the position of the specified part; (4) Detecting information indicating the line of sight of the person based on the information indicating the position of the eye of the person and the information indicating the orientation of the face.

2. The sight line detection method according to claim 1, characterized in that: The prescribed part is an ear or hair.

3. The sight line detection method according to claim 2, characterized in that: In the second process, detecting an ear region including the ear of the person from the image data of the face, When the ear region is detectable, the distance from the position of the eye of the one to the position of the ear included in the ear region is set as the second distance, and information indicating the orientation of the face is detected.

4. The sight line detection method according to claim 3, characterized in that: In the second process, If the ear region cannot be detected, detecting a hair region including the hair of the person from the image data of the face, The distance from the position of the one eye to the position of the hair included in the hair region is defined as the second distance, and information indicating the orientation of the face is detected.

5. A sight line detection device, characterized in that include: An image acquisition unit, which acquires image data of a person's face; an eye detection unit that detects a left eye region including the left eye of the person and a right eye region including the right eye of the person from the image data of the face; a binocular sight line detection unit that detects information indicating the orientation of the face from the image data of the face when the right eye region and the left eye region are detected, and detects information indicating the sight line of the person by a first process using information indicating the position of the right eye included in the right eye region, information indicating the position of the left eye included in the left eye region, and information indicating the orientation of the face; The monocular sight line detection unit detects information indicating the orientation of the face based on information indicating the position of one eye included in the one eye area and information indicating the position of a predetermined part of the face existing on the outer corner of the one eye compared to the position of the inner corner of the one eye, when only the right eye area or the left eye area is detected, and detects information indicating the sight line of the person through a second processing using the information indicating the position of the one eye and the information indicating the orientation of the face. The second processing is a processing that performs the following steps: (1) detecting the specified part from the image data of the face; (2) detecting a contour point representing a point on the contour line of the face that is located on the inner corner side of the outer corner of the eye of the one eye relative to the position of the eye of the one eye based on the image data of the face; (3) detecting information representing the orientation of the face based on a first distance from the position of the eye of the one eye to the contour point and a second distance from the position of the eye of the one eye to the position of the specified part; (4) Detecting information indicating the line of sight of the person based on the information indicating the position of the eye of the person and the information indicating the orientation of the face.

6. A control program product for a sight line detection device, characterized in that: The computer included in the sight line detection device is made to function as the following parts: An image acquisition unit, which acquires image data of a person's face; an eye detection unit that detects a left eye region including the left eye of the person and a right eye region including the right eye of the person from the image data of the face; a binocular sight line detection unit that detects information indicating the orientation of the face from the image data of the face when the right eye region and the left eye region are detected, and detects information indicating the sight line of the person by a first process using information indicating the position of the right eye included in the right eye region, information indicating the position of the left eye included in the left eye region, and information indicating the orientation of the face; The monocular sight line detection unit detects information indicating the orientation of the face based on information indicating the position of one eye included in the one eye area and information indicating the position of a predetermined part of the face existing on the outer corner of the one eye compared to the position of the inner corner of the one eye, when only the right eye area or the left eye area is detected, and detects information indicating the sight line of the person through a second processing using the information indicating the position of the one eye and the information indicating the orientation of the face. The second processing is a processing that performs the following steps: (1) detecting the specified part from the image data of the face; (2) detecting a contour point representing a point on the contour line of the face that is located on the inner corner side of the outer corner of the eye of the one eye relative to the position of the eye of the one eye based on the image data of the face; (3) detecting information representing the orientation of the face based on a first distance from the position of the eye of the one eye to the contour point and a second distance from the position of the eye of the one eye to the position of the specified part; (4) Detecting information indicating the line of sight of the person based on the information indicating the position of the eye of the person and the information indicating the orientation of the face.

Citation Information

Patent Citations

  • Sight line direction detector

    JP2003271932A

  • Visual axis direction detection device and visual line direction detection method

    CN101489467A

  • Television angle regulating method based on sight watching angle detection

    CN104699124A

  • Goggle type display device, eye gaze detection method, and eye gaze detection system

    CN109464236A

  • Visual line detection device and method, and program

    JP2010134489A