Sight line detection device, sight line detection method, and sight line detection program

Abstract

To appropriately measure a distance between a detection target and an imaging unit.SOLUTION: A sight line detection device acquires a basic image obtained by imaging a subject in a state of being projected with imaging light by an imaging unit, acquires a left eye position and a right eye position of the subject, a face direction of the subject, and an on-image interval on the basic image on the basis of the acquired basic image, acquires a distance measurement image obtained by imaging the subject in a state of being projected with pattern light by the imaging unit, detects a pattern image from the acquired distance measurement image, acquires a left eye distance and a right eye distance on the basis of the detected pattern image, calculates a distance using a previously acquired actual interval according to a distance acquisition status when the acquisition of at least one of the left eye distance and the right eye distance fails due to a failure in detection of the pattern image, and detects a sight line of the subject on the basis of the acquired left eye position, the right eye position, the face direction, and the on-image interval which are acquired, and the left eye distance and the right eye distance which are acquired or calculated.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a gaze detection device, a gaze detection method, and a gaze detection program. [Background technology]

[0002] A three-dimensional distance measurement is known in which a detection target such as a person is photographed while being irradiated with a pattern light, and the distance between the photographing position and the detection target is measured based on the image of the pattern light included in the photographing data. This type of three-dimensional distance measurement can be applied to, for example, a gaze detection device, in which the distance between the left and right eyes of a subject and a photographing unit that photographs the subject is measured.

[0003] On the other hand, when performing three-dimensional distance measurement as described above, if the detection target is irradiated with strong light such as direct sunlight, the pattern light irradiated on the detection target becomes invisible, making it difficult to measure the distance. Patent Document 1 describes a technology for photographing a vehicle driver and judging the driver's driving state based on the obtained image. Patent Document 1 also describes that when judging the driver's driving state, the illumination state of sunlight on the driver is taken into consideration. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-079328 A Summary of the Invention [Problem to be solved by the invention]

[0005] In an environment where the detection target may be irradiated with strong light such as direct sunlight, it is necessary to appropriately measure the distance between the detection target and the imaging unit.

[0006] The present disclosure has been made in consideration of the above, and aims to provide a gaze detection device, a gaze detection method, and a gaze detection program that are capable of appropriately measuring the distance between a detection object and a shooting unit. [Means for solving the problem]

[0007] The gaze detection device according to the present disclosure includes an imaging unit that images a face of a subject, an imaging light projection unit that projects imaging light onto the subject, a pattern light projection unit that projects pattern light onto the subject, and a control unit that acquires a basic image of the subject imaged by the imaging unit with the imaging light projected thereon and a distance measurement image of the subject imaged by the imaging unit with the pattern light projected thereon, and detects the gaze of the subject based on the acquired basic image and distance measurement image, and the control unit detects a left eye position and a right eye position which are the positions of the subject's left and right eyes, a face direction which is the direction of the subject's face, and an image spacing which is the distance between the subject's left and right eyes in the basic image, based on the basic image. a distance between the subject and the image capturing unit, detects a pattern image which is an image of the pattern light from the distance measuring image, and obtains left eye distance and right eye distance which are the distance between the photographing unit and the subject's left and right eyes based on the detected pattern image; if obtaining at least one of the left eye distance and the right eye distance fails due to inability to detect the pattern image, calculates the distance using an actual distance which is the actual distance between the subject's left and right eyes which is obtained in advance for the subject, depending on the distance acquisition status; and detects the subject's line of sight based on the acquired left eye position, right eye position, face direction, and distance on the image, and the acquired or calculated left eye distance and right eye distance.

[0008] The gaze detection method according to the present disclosure includes acquiring a basic image of a subject photographed by an imaging unit in a state where imaging light is projected, acquiring a left eye position and a right eye position which are the positions of the left and right eyes of the subject, a face direction which is the direction of the face of the subject, and an on-image distance which is the distance between the left and right eyes of the subject in the basic image based on the acquired basic image, acquiring a distance measurement image of the subject photographed by the imaging unit in a state where pattern light is projected, detecting a pattern image which is an image of the pattern light from the acquired distance measurement image, and detecting the pattern image based on the detected pattern image by the imaging unit. and acquiring left eye distance and right eye distance, which are the distances between the left and right eyes of the subject, if acquisition of at least one of the left eye distance and the right eye distance fails due to inability to detect the pattern image, calculating the distance using an actual distance, which is the distance between the actual left and right eyes of the subject, which has been acquired in advance for the subject, depending on the acquisition status of the distance, and detecting the line of sight of the subject based on the acquired left eye position, right eye position, face direction, and distance on the image, and the acquired or calculated left eye distance and right eye distance.

[0009] The gaze detection program according to the present disclosure includes a process of acquiring a basic image of a subject photographed by an imaging unit with imaging light projected thereon, and acquiring a left eye position and a right eye position, which are the positions of the subject's left and right eyes, a face direction, which is the direction of the subject's face, and an on-image distance, which is the distance between the subject's left and right eyes in the basic image, based on the acquired basic image; acquiring a distance measurement image of the subject photographed by the imaging unit with pattern light projected thereon, detecting a pattern image, which is an image of the pattern light, from the acquired distance measurement image, and comparing the imaging unit with the subject based on the detected pattern image. The computer is caused to execute a process of acquiring left eye distance and right eye distance, which are the distance between the left and right eyes of the subject; if acquisition of at least one of the left eye distance and the right eye distance fails due to inability to detect the pattern image, a process of calculating the distance using an actual distance, which is the actual distance between the left and right eyes of the subject, acquired in advance for the subject, depending on the acquisition status of the distance; and a process of detecting the subject's line of sight based on the acquired left eye position, right eye position, face direction, and distance on the image, and the acquired or calculated left eye distance and right eye distance.

[0010] The state detection device according to the present disclosure includes an imaging unit that images a detection target, an imaging light projection unit that projects imaging light onto the detection target, a pattern light projection unit that projects pattern light onto the detection target, and a control unit that acquires a basic image of the detection target in a state where the imaging light is projected by the imaging unit and a distance measurement image of the detection target in a state where the pattern light is projected by the imaging unit, and acquires a state of the detection target based on the acquired basic image and distance measurement image, and the control unit acquires a first position that is a position of a first part of the detection target, a second position that is a position of a second part of the detection target, an object direction that is an orientation of the detection target, and an on-image interval that is an interval between the first position and the second position of the detection target in the basic image based on the basic image, and acquires a state of the detection target based on the distance measurement image. a pattern image, which is an image of the pattern light, is detected from the image capturing unit, and a first distance, which is the distance between the image capturing unit and the first position of the detection target, and a second distance, which is the distance between the image capturing unit and the second position of the detection target, are acquired based on the detected pattern image; if acquisition of at least one of the first distance and the second distance fails due to inability to detect the pattern image, the distance is calculated using an actual distance, which is the distance between the actual first position and the second position of the detection target that is acquired in advance for the detection target, depending on the distance acquisition status; and a state of the first part and the second part of the detection target is detected based on the acquired first position, the second position, the target direction, and the distance on the image, and the acquired or calculated first distance and second distance. Effect of the Invention

[0011] According to the present disclosure, it is possible to provide a gaze detection device, a gaze detection method, and a gaze detection program that are capable of appropriately measuring the distance between a detection target and an imaging unit. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram showing an example of a gaze detection device according to the present embodiment. [Diagram 2]FIG. 2 is a functional block diagram showing an example of the gaze detection device according to the present embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of the actual interval acquisition operation. [Figure 4] FIG. 4 is a diagram illustrating an example of the gaze detection operation. [Diagram 5] FIG. 5 is a diagram illustrating an example of the gaze detection operation. [Figure 6] FIG. 6 is a flowchart showing an example of a gaze detection method according to this embodiment. [Figure 7] FIG. 7 is a flowchart showing an example of the gaze detection method according to the present embodiment. [Figure 8] FIG. 8 is a schematic diagram showing an example of a state detection device according to the present embodiment. [Figure 9] FIG. 9 is a functional block diagram showing an example of a state detection device according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, as an example of a state detection device according to the present disclosure, an embodiment of a gaze detection device, a gaze detection method, and a gaze detection program will be described with reference to the drawings. Note that the present invention is not limited to these embodiments. In addition, the components in the following embodiments include those that are replaceable and easy for a person skilled in the art, or those that are substantially the same.

[0014] Fig. 1 is a schematic diagram showing an example of a gaze detection device according to the present embodiment. Fig. 2 is a functional block diagram showing an example of a gaze detection device according to the present embodiment. The gaze detection device 100 shown in Figs. 1 and 2 can be used as a part of a driver monitoring system that detects the state of a driver who drives a vehicle M, for example. As shown in Figs. 1 and 2, the gaze detection device 100 includes an imaging unit 10, an imaging light projection unit 20, a pattern light projection unit 30, and a control unit 40.

[0015] The photographing unit 10 photographs a subject R and generates photographed data. The subject R is, for example, a driver who drives a vehicle M equipped with the gaze detection device 100. The photographing unit 10 outputs the generated photographed data to the control unit 40. The photographing unit 10 is composed of a photographing device such as a visible light camera, a far-infrared camera, or a near-infrared camera. The photographing unit 10 may be composed of, for example, a combination of a visible light camera, a far-infrared camera, and a near-infrared camera. Furthermore, a camera provided in a driver monitoring system may be used as the photographing unit 10.

[0016] The photographing light projection unit 20 projects photographing light onto the subject R. As the photographing light, for example, far-infrared light, near-infrared light, etc. can be used. The photographing light is irradiated onto the surface of the subject R with a uniform or nearly uniform intensity.

[0017] The pattern light projection unit 30 projects pattern light onto the subject R. Examples of the pattern light include a dot pattern formed by far-infrared light, near-infrared light, etc. The pattern light is irradiated onto the surface of the subject R so that a predetermined intensity distribution is formed.

[0018] The control unit 40 controls the gaze detection device 100. The control unit 40 includes an input / output unit 41, a processing unit 42, and a storage unit 43.

[0019] The input / output unit 41 inputs and outputs signals to and from various external devices. The input / output unit 41 is connected to the photographing unit 10, the photographed light projection unit 20, and the patterned light projection unit 30. A video signal from the photographing unit 10 is input to the input / output unit 41. The input / output unit 41 outputs a control signal for controlling the photographing unit 10, the photographed light projection unit 20, and the patterned light projection unit 30.

[0020] The processing unit 42 performs various types of information processing. The processing unit 42 includes a processor such as a Central Processing Unit (CPU) and memories such as a Read Only Memory (ROM) and a Random Access Memory (RAM).

[0021] The processing unit 42 includes an imaging control unit 44 , a projection control unit 45 , and a detection unit 46 .

[0022] The photographing control unit 44 controls photographing by the photographing unit 10. The photographing control unit 44 acquires a video signal from the photographing unit 10.

[0023] The projection control unit 45 controls the operations of the photographing light projection unit 20 and the pattern light projection unit 30 .

[0024] The detection unit 46 performs a gaze detection operation to detect the gaze of the subject R. The detection unit 46 also performs an actual distance acquisition operation to acquire an actual distance between the subject R's left and right eyes.

[0025] The detection unit 46 extracts and acquires a basic image IM1 and a distance measurement image IM2 included in the video signal from the imaging unit 10. The basic image IM1 is an image of the subject R photographed by the imaging unit 10 in a state where the imaging light projection unit 20 projects imaging light onto the subject R. The distance measurement image IM2 is an image of the subject R photographed by the imaging unit 10 in a state where the pattern light projection unit 30 projects pattern light onto the subject R.

[0026] Based on the basic image IM1, the detection unit 46 acquires the left eye position and right eye position, which are the positions of the subject's left and right eyes, the face direction, which is the direction in which the subject's face is facing, and the image spacing, which is the distance between the subject's left and right eyes in the basic image IM1.

[0027] The detection unit 46 detects a pattern image IM3 of the left and right eyes of the subject R from the distance measurement image IM2. The detection unit 46 acquires the left eye distance and the right eye distance, which are the distances between the shooting unit 10 and the left and right eyes of the subject R, based on the detected pattern image IM3.

[0028] When strong light such as direct sunlight is irradiated on the subject, the pattern light irradiated on the subject becomes invisible. In this case, the detection unit 46 cannot detect the pattern image IM3 from the distance measurement image IM2, and it may be difficult to calculate the left eye distance and the right eye distance based on the distance measurement image IM2. In this embodiment, when the detection unit 46 fails to calculate at least one of the left eye distance and the right eye distance due to the inability to detect the pattern image IM3, the detection unit 46 calculates the distance using the actual distance between the subject R's left and right eyes, which is obtained in advance, according to the calculation status of the distance.

[0029] For example, if the detection unit 46 fails to acquire both the left eye distance and the right eye distance due to being unable to detect the pattern image IM3, it calculates the left eye distance and the right eye distance based on the on-image distance and face direction calculated based on the basic image IM1 and the actual distance acquired in advance.

[0030] Also, for example, if the detection unit 46 fails to calculate one of the left eye distance and the right eye distance due to an inability to detect pattern image IM3, it calculates the other distance for which the calculation of the left eye distance and the right eye distance failed, based on the calculated face direction, the other distance for which the calculation of the left eye distance and the right eye distance was successful, and the actual distance obtained in advance.

[0031] The detection unit 46 calculates the line of sight of the subject based on the left eye position, right eye position, face direction, and on-image distance calculated from the basic image IM1, and the left eye distance and right eye distance acquired from the pattern image IM3 or calculated using the actual distance. The detection unit 46 can calculate, for example, a line of sight vector defined in a three-dimensional coordinate system as the line of sight of the subject.

[0032] The storage unit 43 stores information such as various programs and data. The storage unit 43 includes storage such as a hard disk drive (HDD) and a solid state drive (SSD). In the present embodiment, the storage unit 43 stores information on the actual distance between the left and right eyes of the subject R, which is previously acquired about the subject R. The information on the actual distance may be information acquired in an actual distance acquisition operation described later, or may be information acquired by a measurement method different from the actual distance acquisition operation.

[0033] The storage unit 43 performs a process of, for example, acquiring a basic image IM1 obtained by photographing the subject R with the photographing unit 10 in a state where photographing light is projected, and acquiring the left eye position and right eye position, which are the positions of the left and right eyes of the subject R, the face direction, which is the direction of the face of the subject R, and the on-image distance, which is the distance between the left and right eyes of the subject R in the basic image IM1, based on the acquired basic image IM1; acquiring a distance measurement image IM2 obtained by photographing the subject R with the photographing unit 10 in a state where pattern light is projected, detecting a pattern image IM3, which is an image of the pattern light, from the acquired distance measurement image IM2, and performing photographing based on the detected pattern image IM3. The system stores a gaze detection program that causes a computer to execute the following processes: a process for acquiring the left eye distance and right eye distance, which are the distances between the shadow area 10 and the left and right eyes of subject R; a process for calculating the distance using an actual distance, which is the actual distance between the left and right eyes of subject R, acquired in advance for subject R, depending on the distance acquisition status if acquisition of at least one of the left eye distance and right eye distance fails due to inability to detect pattern image IM3; and a process for detecting the gaze of subject R based on the acquired left eye position, right eye position, face direction, and distance on the image, and the acquired or calculated left eye distance and right eye distance.

[0034] In the control unit 40, the processor in the processing unit 42 reads out various programs and loads them in memory, thereby executing information processing corresponding to the functions of the above-mentioned units. Examples of the various programs include a program input to the input / output unit 41, a program stored in the storage unit 43, and a program recorded on an external recording medium. The control unit 40 functions as an information processing device (computer) that executes various information processes. Note that the various programs may be executed by an information processing device other than the control unit 40, or the various programs may be executed in cooperation with the control unit 40 and the other information processing device.

[0035] Next, an example of the operation of the gaze detection device 100 configured as described above will be described. The gaze detection device 100 according to this embodiment performs a gaze detection operation and an actual interval acquisition operation. First, the actual interval acquisition operation will be described. The actual interval acquisition operation is performed as an initial operation when the gaze detection device 100 is started up, etc.

[0036] In the actual interval acquisition operation, the photographing light projection unit 20 projects photographing light onto the subject R, and in this state, the photographing unit 10 photographs the subject R to generate a basic image IM1. The photographing unit 10 outputs the generated basic image IM1 to the control unit 40.

[0037] The detection unit 46 of the control unit 40 acquires the basic image IM1 from the photographing unit 10. Based on the acquired basic image IM1, the detection unit 46 acquires the left eye position and the right eye position, which are the positions of the left and right eyes of the subject, the face direction, which is the direction of the face of the subject R, and the image interval, which is the interval between the left and right eyes of the subject R in the basic image IM1. For example, the detection unit 46 can detect the face and the left and right eyes of the subject R by performing known image processing on the basic image IM1. The detection unit 46 can acquire the left eye position and the right eye position of the subject R as three-dimensional coordinates by the image processing. For example, the detection unit 46 can obtain the center positions of the pupils of the left eye and the right eye of the subject R as the left eye position and the right eye position. In addition, the detection unit 46 can acquire the face direction of the subject R as an angle with respect to the optical axis AX of the photographing unit 10. In addition, the detection unit 46 can acquire the interval between the left and right eyes of the subject R in the basic image IM1 as the image interval. The detection unit 46 can obtain the distance between the positions corresponding to the centers of the pupils of the left and right eyes in the basic image IM1 as the on-image distance.

[0038] Next, the pattern light projection unit 30 projects the pattern light onto the subject R, and in this state, the photographing unit 10 photographs the subject R to generate a distance measurement image IM2. The photographing unit 10 outputs the generated distance measurement image IM2 to the control unit 40.

[0039] In the control unit 40, the detection unit 46 acquires a distance measurement image IM2 from the photographing unit 10, and detects a pattern image IM3 from the acquired distance measurement image IM2. The detection unit 46 calculates the left eye distance and the right eye distance, which are the distances between the photographing unit 10 and the left and right eyes of the subject, based on the detected pattern image IM3. The detection unit 46 detects the distances between the photographing unit 10 and the left and right eyes of the subject R as the left eye distance and the right eye distance, respectively, by applying a known distance measurement method such as structured light to the distance measurement image IM2. Note that in the above distance measurement method, the detection unit 46 detects a pattern image IM3, which is an image of pattern light, from the distance measurement image IM2. The detection unit 46 can determine, for example, the distance between the photographing unit 10 and the left eye position of the subject R as the left eye distance, and the distance between the photographing unit 10 and the right eye position of the subject R as the right eye distance, based on the detected pattern image IM3.

[0040] The detection unit 46 determines the actual distance between the subject R's left and right eyes based on the left eye position and right eye position, which are the positions of the subject R's left and right eyes, obtained based on the basic image IM1, the face direction, which is the direction of the subject R's face, and the on-image distance, which is the distance between the subject R's left and right eyes in the basic image IM1, and the left eye distance and right eye distance obtained based on the pattern image IM3.

[0041] FIG. 3 is a diagram showing an example of an actual interval acquisition operation. As shown in FIG. 3, the distance from the photographing unit 10 to the face of the subject R is D1, and the face direction of the subject R is θ (the angle formed in the left-right direction with respect to the optical axis AX of the photographing unit 10). The distance D1 can be, for example, the distance between the photographing unit 10 and the position of the midpoint of a virtual line connecting the left eye position E1 and the right eye position E2 of the subject R. Also, the total width in the left-right direction of the basic image IM1 photographed by the photographing unit 10 is Wa, and the image interval of the subject R in the basic image IM1 is W1. Also, the focal length of the lens of the photographing unit 10 is f, the dimension in the left-right direction of the image sensor provided in the photographing unit 10 is w, and the horizontal image photographing range by the photographing unit 10 at the position where the subject R is present is L.

[0042] In this case, the distance between the left and right eyes of the subject R when viewed in the axial direction of the optical axis AX of the imaging unit 10 is e, and the actual distance between the left and right eyes of the subject R is d. The actual distance d is the distance between the left eye position E1 and the right eye position E2.

number

number

[0043] Where:

number

number

[0044] After performing the above-described actual distance acquisition operation, when the subject R drives a vehicle, etc., a gaze detection operation is performed. In the gaze detection operation, similar to the actual distance acquisition operation, the photographing light projection unit 20 projects photographing light onto the subject R, and in this state, the photographing unit 10 photographs the subject R to generate a basic image IM1. The photographing unit 10 outputs the generated basic image IM1 to the control unit 40.

[0045] In the control unit 40, the detection unit 46 acquires the basic image IM1 from the imaging unit 10, and acquires the left eye position and right eye position, which are the positions of the subject's left and right eyes, the face direction, which is the direction of the subject R's face, and the on-image distance, which is the distance between the left and right eyes of the subject R in the basic image IM1, based on the acquired basic image IM1. The detection unit 46 can determine, for example, the center positions of the pupils of the left and right eyes of the subject R as the left eye position and right eye position. The detection unit 46 can also determine the distance between positions corresponding to the centers of the pupils of the left and right eyes in the basic image IM1 as the on-image distance.

[0046] Next, the pattern light projection unit 30 projects the pattern light onto the subject R, and in this state, the photographing unit 10 photographs the subject R to generate a distance measurement image IM2. The photographing unit 10 outputs the generated distance measurement image IM2 to the control unit 40.

[0047] In the control unit 40, the detection unit 46 acquires a distance measurement image IM2 from the photographing unit 10, detects a pattern image IM3 of the left and right eyes of the subject R from the acquired distance measurement image IM2, and acquires the left eye distance and right eye distance, which are the distances between the photographing unit 10 and the left and right eyes of the subject R, based on the detected pattern image IM3.

[0048] When strong light such as direct sunlight is irradiated on the subject, the pattern light irradiated on the subject R becomes invisible. In this case, the detection unit 46 cannot detect the pattern image IM3 from the distance measurement image IM2, and it may become difficult to calculate the left eye distance and the right eye distance based on the distance measurement image IM2.

[0049] In this embodiment, when the detection unit 46 fails to calculate at least one of the left eye distance and the right eye distance due to an inability to detect the pattern image IM3, it calculates the distance using the actual distance d, which is the distance between the actual left and right eyes of the subject R, which is obtained in advance, depending on the calculation status of the distance.

[0050] When the detection unit 46 fails to acquire both the left eye distance and the right eye distance due to an inability to detect the pattern image IM3, it calculates the left eye distance and the right eye distance based on the on-image distance and face direction calculated based on the basic image IM1 and the actual distance acquired in advance.

[0051] FIG. 4 is a diagram showing a schematic example of an eye gaze detection operation. FIG. 4 shows an example of a case where both the left eye distance and the right eye distance have failed to be acquired. As shown in FIG. 4, the distance from the image capture unit 10 to the face of the subject R is D2, and the face direction of the subject R is θ (the angle formed in the left-right direction with respect to the optical axis AX of the image capture unit 10). In addition, the distance on the image of the subject R in the basic image IM1 captured by the image capture unit 10 is W2. In addition, the distance on the image when the subject R faces forward at the distance D2 is W3. Note that the image capture unit 10 has the same parameters as the image capture unit 10 when the actual distance d is acquired. In this case, the distance D2 from the image capture unit 10 to the face of the subject R is obtained by the following equation 5.

number

[0052] Furthermore, the left eye distance l and the right eye distance r, which are the distances between the imaging unit 10 and the left and right eyes of the subject R, are obtained by the following Equations 6 and 7.

number

number

[0053] In addition, when the detection unit 46 fails to calculate one of the left eye distance and the right eye distance due to an inability to detect the pattern image IM3, it calculates the other distance for which the calculation of the left eye distance and the right eye distance failed, based on the calculated face direction, the other distance for which the calculation of the left eye distance and the right eye distance was successful, and the actual distance obtained in advance.

[0054] Fig. 5 is a diagram showing a schematic example of a gaze detection operation. Fig. 5 shows an example of a case where acquisition of one of the left eye distance and the right eye distance fails. In the example shown in Fig. 5, if acquisition of the right eye distance r is successful and acquisition of the left eye distance l fails, the left eye distance l can be obtained by the following equation 8, assuming that the face direction of the subject R is θ. Note that the imaging unit 10 has the same parameters as the imaging unit 10 when the actual distance d is acquired.

number

[0055] 5, if acquisition of the right eye distance r fails and acquisition of the left eye distance l succeeds, then assuming that the face direction of the subject R is θ, the right eye distance r can be obtained by the following equation 9. Note that the imaging unit 10 has the same parameters as the imaging unit 10 when the actual distance d is acquired.

number

[0056] The detection unit 46 calculates the line of sight of the subject based on the left eye position E1, right eye position E2, face direction θ, and on-image distance W2 calculated from the basic image IM1, and the left eye distance l and right eye distance r acquired from the pattern image IM3 or calculated using the actual distance d. The detection unit 46 can calculate, for example, a line of sight vector defined in a three-dimensional coordinate system as the line of sight of the subject.

[0057] Fig. 6 and Fig. 7 are flowcharts showing an example of the gaze detection method according to this embodiment. Fig. 6 shows an example of an actual distance acquisition operation, and Fig. 7 shows a gaze detection operation. First, with reference to Fig. 6, the actual distance acquisition operation for calculating the actual distance between the left and right eyes of a subject will be described.

[0058] In the actual interval acquisition operation, the photographing light projection unit 20 projects photographing light onto the subject R, and in this state, the photographing unit 10 photographs the subject R to generate a basic image IM1. The photographing unit 10 outputs the generated basic image IM1 to the control unit 40.

[0059] In the control unit 40, the detection unit 46 acquires a basic image IM1 from the photographing unit 10, and acquires a left eye position E1 and a right eye position E2 which are the positions of the left and right eyes of the subject R, a face direction θ which is the direction of the face of the subject R, and an on-image distance W1 which is the distance between the left and right eyes of the subject R in the basic image IM1 based on the acquired basic image IM1 (step S101). Note that if the detection unit 46 fails to acquire each value, it can repeat step S101.

[0060] Next, the pattern light projection unit 30 projects the pattern light onto the subject R, and in this state, the photographing unit 10 photographs the subject R to generate a distance measurement image IM2. The photographing unit 10 outputs the generated distance measurement image IM2 to the control unit 40.

[0061] In the control unit 40, the detection unit 46 acquires a distance measurement image IM2 from the photographing unit 10, detects a pattern image IM3 of the left and right eyes of the subject R from the acquired distance measurement image IM2, and acquires the left eye distance l and the right eye distance r, which are the distances between the photographing unit 10 and the left and right eyes of the subject R, based on the detected pattern image IM3 (step S102). Note that if the detection unit 46 fails to acquire each value, it can return to step S101 and perform processing.

[0062] The detection unit 46 calculates an actual distance d between the left and right eyes of the subject R based on the left eye position E1 and right eye position E2, which are the positions of the left and right eyes of the subject R, the face direction θ, which is the face direction of the subject R, and the on-image distance W1, which is the distance between the left and right eyes of the subject R in the basic image IM1, acquired in step S101, and the left eye distance l and right eye distance r acquired in step S102 (step S103). The detection unit 46 stores the actual distance d calculated in step S103 in the storage unit 43 (step S104). As a result, the actual distance d of the subject R is stored in the storage unit 43.

[0063] Next, the gaze detection operation will be described with reference to Fig. 7. In the gaze detection operation, similarly to the actual interval acquisition operation, the photographing light projection unit 20 projects photographing light onto the subject R, and in this state, the photographing unit 10 photographs the subject R to generate a basic image IM1. The photographing unit 10 outputs the generated basic image IM1 to the control unit 40.

[0064] In the control unit 40, the detection unit 46 acquires the basic image IM1 from the photographing unit 10, and acquires the left eye position E1 and the right eye position E2 which are the positions of the left and right eyes of the subject R, the face direction θ which is the direction of the face of the subject R, and the on-image distance W2 which is the distance between the left and right eyes of the subject R in the basic image IM1 based on the acquired basic image IM1 (step S201). Note that the detection unit 46 can repeat step S201 if it fails to acquire each value.

[0065] Next, the pattern light projection unit 30 projects the pattern light onto the subject R, and in this state, the photographing unit 10 photographs the subject R to generate a distance measurement image IM2. The photographing unit 10 outputs the generated distance measurement image IM2 to the control unit 40.

[0066] In the control unit 40, the detection unit 46 acquires a distance measurement image IM2 from the photographing unit 10, detects a pattern image IM3 of the left and right eyes of the subject R from the acquired distance measurement image IM2, and acquires the left eye distance l and right eye distance r, which are the distances between the photographing unit 10 and the left and right eyes of the subject R, based on the detected pattern image IM3 (step S202).

[0067] The detection unit 46 determines whether or not acquisition of both the left eye distance l and the right eye distance r has been successful (step S203).

[0068] If the detection unit 46 determines that acquisition of both the left eye distance l and the right eye distance r was unsuccessful (No in step S203), it determines whether acquisition of both the left eye distance l and the right eye distance r failed (step S204).

[0069] If it is determined in step S204 that acquisition of both the left eye distance l and the right eye distance r has failed (Yes in step S204), the detection unit 46 calculates the left eye distance l and the right eye distance r based on the face direction θ and the on-image distance W2 of the subject R acquired in step S201 and the actual distance d, which is the distance between the actual left and right eyes of the subject R calculated in advance in steps S101 to S104 (step S205).

[0070] If it is determined in step S204 that acquisition of one of the left eye distance l and the right eye distance r has failed (No in step S204), the detection unit 46 calculates the other of the left eye distance l and the right eye distance r that has failed to be calculated based on the face direction θ acquired in step S201, the other of the left eye distance l and the right eye distance r that has been successfully acquired in step S202, and the actual distance d calculated in advance in steps S101 to S104 (step S206).

[0071] If it is determined that both the left eye distance l and the right eye distance r have been successfully acquired (Yes in step S203), if the left eye distance l and the right eye distance r have been calculated in step S205, or if one of the left eye distance l and the right eye distance r has been acquired in step S202 and the other of the left eye distance l and the right eye distance r has been calculated in step S206, the gaze of subject R is detected based on the acquired left eye position E1, right eye position E2, face direction θ, and spacing W2 on the image and the acquired or calculated left eye distance l and right eye distance r (step S207).

[0072] As described above, the gaze detection device 100 according to this embodiment includes an imaging unit 10 that images the face of the subject R, an imaging light projection unit 20 that projects imaging light onto the subject R, a pattern light projection unit 30 that projects pattern light onto the subject R, and a control unit 40 that acquires a basic image IM1 captured by the imaging unit 10 of the subject R with the imaging light projected thereon and a distance measurement image IM2 captured by the imaging unit 10 of the subject R with the pattern light projected thereon, and detects the gaze of the subject R based on the acquired basic image IM1 and distance measurement image IM2. Based on the basic image IM1, the control unit 40 determines the left eye position and right eye position, which are the positions of the subject R's left and right eyes, the facial direction, which is the orientation of the subject R's face, and the basic image IM1. The system acquires an on-image distance, which is the distance between the left and right eyes of subject R, detects a pattern image IM3 of the left and right eyes of subject R from the distance measurement image IM2, and acquires the left eye distance and right eye distance, which are the distance between the shooting unit 10 and the left and right eyes of subject R, based on the detected pattern image IM3.If acquisition of at least one of the left eye distance and right eye distance fails due to inability to detect pattern image IM3, the distance is calculated using an actual distance, which is the actual distance between the left and right eyes of subject R acquired in advance for subject R, depending on the distance acquisition status.The system detects the line of sight of subject R based on the acquired left eye position, right eye position, face direction, and on-image distance, and the acquired or calculated left eye distance and right eye distance.

[0073] In addition, the gaze detection method according to this embodiment includes acquiring a basic image IM1 obtained by photographing the subject R with the photographing unit 10 while the photographing light is projected, acquiring the left eye position and right eye position, which are the positions of the left and right eyes of the subject R, the face direction, which is the direction of the face of the subject R, and the on-image distance, which is the distance between the left and right eyes of the subject R in the basic image IM1, acquiring a distance measurement image IM2 obtained by photographing the subject R with the photographing unit 10 while the pattern light is projected, detecting a pattern image IM3 of the left and right eyes of the subject R from the acquired distance measurement image IM2, and detecting the detected pattern. The method includes acquiring the left eye distance and right eye distance, which are the distances between the photographing unit 10 and the left and right eyes of subject R, based on the turn image IM3, and if acquisition of at least one of the left eye distance and the right eye distance fails due to inability to detect the pattern image IM3, calculating the distance using the actual distance, which is the distance between the actual left and right eyes of subject R, acquired in advance for subject R, depending on the distance acquisition status, and detecting the line of sight of subject R based on the acquired left eye position, right eye position, face direction, and distance on the image, and the acquired or calculated left eye distance and right eye distance.

[0074] The gaze detection program according to this embodiment acquires a basic image IM1 obtained by photographing the subject R with the photographing unit 10 while the photographing light is projected, and acquires the left eye position and right eye position, which are the positions of the left and right eyes of the subject R, the face direction, which is the direction of the face of the subject R, and the on-image distance, which is the distance between the left and right eyes of the subject R in the basic image IM1, based on the acquired basic image IM1; acquires a distance measurement image IM2 obtained by photographing the subject R with the photographing unit 10 while the pattern light is projected, detects a pattern image IM3 of the left and right eyes of the subject R from the acquired distance measurement image IM2, and detects the detected pattern The computer is caused to execute a process of acquiring the left eye distance and right eye distance, which are the distances between the photographing unit 10 and the left and right eyes of subject R, based on image IM3; if acquisition of at least one of the left eye distance and right eye distance fails due to inability to detect pattern image IM3, a process of calculating the distance using an actual distance, which is the actual distance between the left and right eyes of subject R, acquired in advance for subject R, depending on the distance acquisition status; and a process of detecting the gaze of subject R based on the acquired left eye position, right eye position, face direction, and distance on the image, and the acquired or calculated left eye distance and right eye distance.

[0075] According to this configuration, if acquisition of at least one of the left eye distance and the right eye distance fails due to inability to detect pattern image IM3, the left eye distance and the right eye distance are calculated using the actual distance between the subject R's left and right eyes acquired in advance, so that the distance between the subject R's left and right eyes and the imaging unit 10 can be appropriately measured in an environment where strong light such as direct sunlight may be irradiated onto the subject R.

[0076] In the gaze detection device 100 according to this embodiment, when the control unit 40 fails to acquire both the left eye distance and the right eye distance due to an inability to detect the pattern image IM3, it calculates the left eye distance and the right eye distance based on the on-image distance and face direction acquired based on the basic image IM1 and the actual distance acquired in advance. With this configuration, it is possible to appropriately perform three-dimensional distance measurement when the control unit 40 fails to acquire both the left eye distance and the right eye distance due to an inability to detect the pattern image IM3.

[0077] In the gaze detection device 100 according to this embodiment, when acquisition of one of the left eye distance and the right eye distance fails due to inability to detect the pattern image IM3, the control unit 40 calculates the one of the left eye distance and the right eye distance that failed to be acquired based on the face direction acquired based on the basic image IM1, the other of the left eye distance and the right eye distance that was successfully acquired, and the actual distance acquired in advance. With this configuration, three-dimensional distance measurement can be appropriately performed when acquisition of one of the left eye distance and the right eye distance fails due to inability to detect the pattern image IM3.

[0078] Fig. 8 is a schematic diagram showing an example of a state detection device 200 according to this embodiment. Fig. 9 is a functional block diagram showing an example of a state detection device 200 according to this embodiment. The state detection device 200 shown in Figs. 8 and 9 detects the state of a detection object Q. As shown in Figs. 8 and 9, the state detection device 200 includes an imaging unit 110, an imaging light projection unit 120, a pattern light projection unit 130, and a control unit 140.

[0079] The photographing unit 110 photographs the detection target Q to generate photographed data. The detection target Q may be a person or another living thing such as an animal. The detection target Q may also be a non-living thing such as a robot or a structure. In the example shown in FIG. 8, the detection target Q is a robot arm. The detection target Q has a main body Qa, a rotating part Qb, and an arm Qc. The detection target Q changes the orientation of the arm Qc when the rotating part Qb rotates relative to the main body Qa, for example. The tip of the arm Qc branches into two. In this embodiment, the tip portions of the branched arm Qc are referred to as a first part Q1 and a second part Q2.

[0080] The photographing unit 110 outputs the generated photographing data to the control unit 140. The photographing unit 110 is configured with a photographing device such as a visible light camera, a far-infrared camera, a near-infrared camera, etc. The photographing unit 110 may be configured with a combination of a visible light camera, a far-infrared camera, and a near-infrared camera, for example.

[0081] The imaging light projection unit 120 projects imaging light onto the detection object Q. As the imaging light, for example, far-infrared light, near-infrared light, etc., can be used. The imaging light is irradiated onto the surface of the detection object Q with a uniform or nearly uniform intensity.

[0082] The pattern light projection unit 130 projects pattern light onto the detection target Q. An example of the pattern light is a dot pattern formed by far-infrared light, near-infrared light, or the like. The pattern light is irradiated onto the surface of the detection target Q so that a predetermined intensity distribution is formed.

[0083] The control unit 140 controls the state detection device 200. The control unit 140 includes an input / output unit 141, a processing unit 142, and a storage unit 143.

[0084] The input / output unit 141 inputs and outputs signals to and from various external devices. The input / output unit 141 is connected to the photographing unit 110, the photographed light projection unit 120, and the patterned light projection unit 130. A video signal from the photographing unit 110 is input to the input / output unit 141. The input / output unit 141 outputs a control signal for controlling the photographing unit 110, the photographed light projection unit 120, and the patterned light projection unit 130.

[0085] The processing unit 142 performs various types of information processing. The processing unit 142 includes a processor such as a CPU, and memories such as a ROM and a RAM.

[0086] The processing unit 142 includes an imaging control unit 144 , a projection control unit 145 , and a detection unit 146 .

[0087] The imaging control unit 144 controls imaging by the imaging unit 110. The imaging control unit 144 acquires a video signal from the imaging unit 110.

[0088] The projection control unit 145 controls the operations of the capture light projection unit 120 and the pattern light projection unit 130 .

[0089] The detection unit 146 performs a state detection operation to detect the state of the first portion Q1 and the second portion Q2 of the detection target Q. The detection unit 146 also performs an actual distance acquisition operation to acquire an actual distance s that is the actual distance between a first position P1 that is the position of the first portion Q1 of the detection target Q and a second position P2 that is the position of the second portion Q2. The actual distance acquisition operation can be explained in the same manner as described with reference to FIG. 3. The actual distance s may be acquired by a measurement method different from that used in the actual distance acquisition operation.

[0090] The detection unit 146 extracts and acquires a basic image IM4 and a distance measurement image IM5 included in the video signal from the imaging unit 110. The basic image IM4 is an image of the detection target Q captured by the imaging unit 110 in a state in which the imaging light projection unit 120 projects imaging light onto the detection target Q. The distance measurement image IM5 is an image of the detection target Q captured by the imaging unit 110 in a state in which the pattern light projection unit 130 projects pattern light onto the detection target Q.

[0091] The detection unit 146 acquires a first position P1, which is the position of the first portion Q1 of the detection target Q, a second position P2, which is the position of the second portion Q2, an object direction θs, which is the direction in which the detection target Q faces, and an on-image interval Wb, which is the interval between the first position P1 and the second position P2 in the basic image IM4 of the detection target Q, based on the basic image IM4. The detection unit 46 can obtain, for example, the center positions of the first portion Q1 and the second portion Q2 as the first position P1 and the second position P2. The detection unit 46 can also obtain, for example, a direction that is a reference for the orientation of the detection target Q, such as the front direction of the detection target Q, as the object direction θs. The detection unit 46 can also obtain, for example, the distance between a position corresponding to the first position P1 and a position corresponding to the second position P2 in the basic image IM4 as the on-image interval Wb.

[0092] The detection unit 146 detects a pattern image IM6 of a first portion Q1 and a second portion Q2 of the detection target Q from the distance measurement image IM5. The detection unit 146 acquires a first distance C1, which is the distance between the imaging unit 110 and a first position P1 of the detection target Q, and a second distance C2, which is the distance between the imaging unit 110 and a second position P2 of the detection target Q, based on the detected pattern image IM6.

[0093] When strong light such as direct sunlight is irradiated onto the detection target Q, the pattern light irradiated onto the detection target Q becomes invisible. In this case, the detection unit 146 cannot detect the pattern image IM6 from the distance measurement image IM5, and it may be difficult to calculate the first distance C1 and the second distance C2 based on the distance measurement image IM5. In this embodiment, when the detection unit 146 fails to calculate at least one of the first distance C1 and the second distance C2 due to the inability to detect the pattern image IM6, the detection unit 146 calculates the distance using the actual distance between the first position P1 and the second position P2 of the detection target Q, which is obtained in advance for the detection target Q, according to the calculation status of the distance.

[0094] For example, if the detection unit 146 fails to acquire both the first distance C1 and the second distance C2 due to the inability to detect the pattern image IM6, the detection unit 146 calculates the first distance C1 and the second distance C2 based on the on-image interval Wb and the target direction θs calculated based on the basic image IM4 and the actual interval acquired in advance. The calculation principle of the first distance C1 and the second distance C2 in this case can be explained in the same manner as in the case explained using FIG. 4.

[0095] Furthermore, for example, when the detection unit 146 fails to calculate one of the first distance C1 and the second distance C2 due to the inability to detect the pattern image IM6, the detection unit 146 calculates the other of the first distance C1 and the second distance C2 that has not been calculated successfully, based on the calculated target direction θs, the other of the first distance C1 and the second distance C2 that has not been calculated successfully, and the actual interval s acquired in advance. The calculation principle of the first distance C1 and the second distance C2 in this case can be explained in the same manner as in the case explained using FIG. 5.

[0096] The detection unit 146 detects the state of the first portion Q1 and the second portion Q2 of the detection target Q based on the first position P1, the second position P2, the target direction θs, and the on-image interval Wb calculated from the basic image IM4, and the first distance C1 and the second distance C2 acquired from the pattern image IM6 or calculated using the actual interval. In this embodiment, for example, the orientations of the first portion Q1 and the second portion Q2 (the orientation of the arm Qc) can be detected as the state of the first portion Q1 and the second portion Q2 of the detection target Q.

[0097] The storage unit 143 stores information such as various programs and data. The storage unit 143 includes storage such as an HDD and an SSD. In this embodiment, the storage unit 143 stores information on an actual distance s between an actual first position P1 and a second position P2 of the detection target Q, which is acquired in advance for the detection target Q.

[0098] The storage unit 143 performs a process of, for example, acquiring a basic image obtained by capturing an image of the detection target Q with the capturing unit 110 in a state where the capturing light is projected, and acquiring, based on the acquired basic image, a first position P1 which is the position of a first portion Q1 of the detection target Q, a second position P2 which is the position of a second portion Q2 of the detection target Q, an object direction θs which is the orientation of the detection target Q, and an on-image interval Wb which is the interval between the first position P1 and the second position P2 of the detection target Q in the basic image IM4; acquiring a distance measurement image IM5 obtained by capturing an image of the detection target Q with the capturing unit 110 in a state where the pattern light is projected, detecting a pattern image IM6 which is an image of the pattern light from the distance measurement image IM5, and detecting a distance between the capturing unit 110 and the first position P1 of the detection target Q based on the detected pattern image IM6. The state detection program stored in the memory causes a computer to execute a process of acquiring a first distance C1, which is the distance between the photographing unit 110 and the second position P2 of the detection target Q, and a process of detecting a state of the first part Q1 and the second part Q2 of the detection target Q based on the acquired first position P1, second position P2, target direction θs, and on-image interval Wb, and the acquired or calculated first distance C1 and second distance C2, if acquisition of at least one of the first distance C1 and the second distance C2 fails due to inability to detect the pattern image IM6, calculating the distance using an actual distance s, which is the distance between the actual first position P1 and second position P2 of the detection target Q, depending on the distance acquisition status.

[0099] In the control unit 140, a processor in the processing unit 142 reads out various programs and loads them in a memory, thereby executing information processing corresponding to the functions of the above-mentioned units. Examples of the various programs include a program input to the input / output unit 141, a program stored in the storage unit 143, and a program recorded on an external recording medium. The control unit 140 functions as an information processing device (computer) that executes various information processes. Note that the various programs may be executed by an information processing device other than the control unit 140, or the various programs may be executed in cooperation with the control unit 140 and the other information processing device.

[0100] As described above, the state detection device 200 according to this embodiment includes an imaging unit 110 that images the detection target Q, an imaging light projection unit 120 that projects imaging light onto the detection target Q, a pattern light projection unit 130 that projects pattern light onto the detection target Q, and a control unit 140 that acquires a basic image IM4 obtained by imaging the detection target Q with the imaging unit 110 in a state where the imaging light is projected, and a distance measurement image IM5 obtained by imaging the detection target Q with the imaging unit 110 in a state where the pattern light is projected, and acquires the state of the detection target Q based on the acquired basic image IM4 and distance measurement image IM5. The control unit 140 acquires, based on the basic image IM4, a first position P1 that is the position of the first portion Q1 of the detection target Q, a second position P2 that is the position of the second portion Q2 of the detection target Q, an object direction θs that is the orientation of the detection target Q, and an image distance θs that is the distance between the first position P1 and the second position P2 of the detection target Q in the basic image IM4. An on-image interval Wb is acquired, a pattern image IM6, which is an image of pattern light, is detected from the distance measurement image IM5, and a first distance C1, which is the distance between the photographing unit 110 and the first position P1 of the detection target Q, and a second distance C2, which is the distance between the photographing unit 110 and the second position P2 of the detection target Q, are acquired based on the detected pattern image IM6. If acquisition of at least one of the first distance C1 and the second distance C2 fails due to inability to detect the pattern image IM6, a distance is calculated using an actual distance s, which is the distance between the actual first position P1 and second position P2 of the detection target Q acquired in advance for the detection target Q, depending on the distance acquisition status. The state of the first part Q1 and the second part Q2 of the detection target Q is detected based on the acquired first position P1, second position P2, target direction θs, and on-image interval Wb, and the acquired or calculated first distance C1 and second distance C2.

[0101] Furthermore, the state detection method according to the present embodiment includes acquiring a basic image by capturing an image of the detection target Q with the image capturing unit 110 in a state where the image capturing light is projected, and acquiring, based on the acquired basic image, a first position P1 which is the position of a first portion Q1 of the detection target Q, a second position P2 which is the position of a second portion Q2 of the detection target Q, an object direction θs which is the orientation of the detection target Q, and an on-image interval Wb which is the interval between the first position P1 and the second position P2 of the detection target Q in the basic image IM4; acquiring a distance measurement image IM5 by capturing an image of the detection target Q with the image capturing unit 110 in a state where the pattern light is projected, detecting a pattern image IM6 which is an image of the pattern light from the distance measurement image IM5, and performing a detection operation with the image capturing unit 110 based on the detected pattern image IM6. The method includes acquiring a first distance C1 which is the distance to a first position P1 of the detection target Q, and a second distance C2 which is the distance between the photographing unit 110 and a second position P2 of the detection target Q, and if acquisition of at least one of the first distance C1 and the second distance C2 fails due to inability to detect the pattern image IM6, calculating the distance using an actual distance s which is the distance between the actual first position P1 and second position P2 of the detection target Q which have been acquired in advance for the detection target Q, depending on the distance acquisition status, and detecting the state of the first part Q1 and the second part Q2 of the detection target Q based on the acquired first position P1, second position P2, target direction θs, and image distance Wb, and the acquired or calculated first distance C1 and second distance C2.

[0102] The state detection program according to this embodiment includes a process of acquiring a basic image obtained by capturing an image of the detection target Q with the image capturing unit 110 in a state where the image capturing light is projected, and acquiring a first position P1 which is the position of a first portion Q1 of the detection target Q, a second position P2 which is the position of a second portion Q2 of the detection target Q, an object direction θs which is the orientation of the detection target Q, and an on-image interval Wb which is the interval between the first position P1 and the second position P2 of the detection target Q in the basic image IM4, and acquiring a distance measurement image IM5 obtained by capturing an image of the detection target Q with the image capturing unit 110 in a state where the pattern light is projected, detecting a pattern image IM6 which is an image of the pattern light from the distance measurement image IM5, and determining a relationship between the image capturing unit 110 and the detection target Q based on the detected pattern image IM6. The computer is caused to execute a process of acquiring a first distance C1, which is the distance between the photographing unit 110 and a first position P1 of Q, and a second distance C2, which is the distance between the photographing unit 110 and a second position P2 of the detection target Q, and if acquisition of at least one of the first distance C1 and the second distance C2 fails due to inability to detect the pattern image IM6, a process of calculating the distance using an actual distance s, which is the distance between the actual first position P1 and second position P2 of the detection target Q that has been acquired in advance for the detection target Q, depending on the distance acquisition status, and detecting the state of the first part Q1 and the second part Q2 of the detection target Q based on the acquired first position P1, second position P2, target direction θs, and image distance Wb, and the acquired or calculated first distance C1 and second distance C2.

[0103] According to this configuration, if acquisition of at least one of the first distance C1 and the second distance C2 fails due to inability to detect the pattern image IM6, the first distance C1 and the second distance C2 are calculated using the actual distance s, which is the distance between the actual first position P1 and second position P2 of the detection target Q acquired in advance, so that the distance between the first part Q1 and the second part Q2 of the detection target Q and the imaging unit 110 can be appropriately measured in an environment where strong light such as direct sunlight may be irradiated onto the detection target Q.

[0104] The technical scope of the present invention is not limited to the above-described embodiment, and appropriate modifications can be made without departing from the spirit of the present invention. [Explanation of symbols]

[0105] M...vehicle, Q...detection target, Qa...main body, Qb...rotating part, Qc...arm, Q1...first part, Q2...second part, R...target, IM1, IM4...basic image, IM2, IM5...distance measurement image, IM3, IM6...pattern image, AX...optical axis, IM3...pattern image, 10,110...imaging unit, 20,120...imaging light projection unit, 30,130...pattern light projection unit, 40,140...control unit, 41,141...input / output unit, 42,142...processing unit, 43,143...storage unit, 44,144...imaging control unit, 45,145...projection control unit, 46,146...detection unit, 100...gaze detection device, 200...status detection device

Claims

1. A photographing unit that photographs the face of the subject; an imaging light projection unit that projects imaging light onto the subject; A pattern light projection unit that projects a pattern light onto the subject; a control unit that acquires a basic image of the subject with the photographing light projected thereon by the photographing unit and a distance measurement image of the subject with the photographing unit and detects a line of sight of the subject based on the acquired basic image and distance measurement image; Equipped with The control unit is Based on the basic image, a left eye position and a right eye position, which are the positions of the left and right eyes of the subject, a face direction, which is the direction of the face of the subject, and an on-image distance, which is the distance between the left and right eyes of the subject in the basic image, are obtained; A pattern image that is an image of the pattern light is detected from the distance measurement image, and a left eye distance and a right eye distance that are distances between the photographing unit and the left and right eyes of the subject are obtained based on the detected pattern image; if acquisition of at least one of the left eye distance and the right eye distance fails due to an inability to detect the pattern image, calculate the distance using an actual distance, which is the distance between the actual left and right eyes of the subject, that has been acquired in advance for the subject, depending on the acquisition status of the distance; Detecting the line of sight of the subject based on the acquired left eye position, right eye position, face direction, and on-image interval, and the acquired or calculated left eye distance and right eye distance. Gaze detection device.

2. When acquisition of both the left eye distance and the right eye distance fails due to an inability to detect the pattern image, the control unit calculates the left eye distance and the right eye distance based on the on-image distance and the face direction acquired based on the basic image and the actual distance acquired in advance. The gaze detection device according to claim 1 .

3. When acquisition of one of the left eye distance and the right eye distance fails due to an inability to detect the pattern image, the control unit calculates the one of the left eye distance and the right eye distance that has not been acquired based on the face direction acquired based on the basic image, the other of the left eye distance and the right eye distance that has been successfully acquired, and the actual distance that has been acquired in advance. The gaze detection device according to claim 1 .

4. Acquiring a basic image by photographing a subject with a photographing unit projecting photographing light, and acquiring a left eye position and a right eye position, which are the positions of the left and right eyes of the subject, a face direction, which is the direction of the face of the subject, and an on-image distance, which is the distance between the left and right eyes of the subject in the basic image, based on the acquired basic image; acquiring a distance measurement image obtained by photographing the subject with the photographing unit while the pattern light is projected, detecting a pattern image which is an image of the pattern light from the acquired distance measurement image, and acquiring a left eye distance and a right eye distance which are distances between the photographing unit and the left and right eyes of the subject based on the detected pattern image; When acquisition of at least one of the left eye distance and the right eye distance fails due to an inability to detect the pattern image, calculating the distance using an actual distance, which is an actual distance between the left and right eyes of the subject, that has been acquired in advance for the subject, depending on an acquisition status of the distance; Detecting the line of sight of the subject based on the acquired left eye position, right eye position, face direction, and on-image interval, and the acquired or calculated left eye distance and right eye distance; A gaze detection method comprising:

5. A process of acquiring a basic image of a subject with imaging light projected thereon by an imaging unit, and acquiring a left eye position and a right eye position, which are the positions of the subject's left and right eyes, a face direction, which is the orientation of the subject's face, and an on-image distance, which is the distance between the subject's left and right eyes in the basic image, based on the acquired basic image; a process of acquiring a distance measurement image of the subject with the photographing unit while the pattern light is projected, detecting a pattern image which is an image of the pattern light from the acquired distance measurement image, and acquiring a left eye distance and a right eye distance which are distances between the photographing unit and the left and right eyes of the subject based on the detected pattern image; a process of calculating the distance using an actual distance, which is an actual distance between the left and right eyes of the subject, according to a state of acquisition of the distance when acquisition of at least one of the left eye distance and the right eye distance fails due to an inability to detect the pattern image; A process of detecting the line of sight of the subject based on the acquired left eye position, right eye position, face direction, and on-image interval, and the acquired or calculated left eye distance and right eye distance; A gaze detection program that causes a computer to execute the above.

Images