Recognition device, recognition method, and recognition program

Abstract

To provide a recognition device and the like capable of accurately recognizing an orientation of an object covered with a covering member that recognizes an orientation of a face with a mask as a covering material put thereon.SOLUTION: A method acquires an image of an object covered with a covering member, captured in a state being irradiated with light from irradiation means, detects a front longitudinal centerline 212 on a surface of the object based on luminance levels of pixels contained in the image, and recognizes an orientation of the object based on inclination of the centerline.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present application relates to the technical field of recognition devices and the like that recognize the direction an object is facing based on an image obtained by capturing the object.

Background Art

[0002] Conventionally, there are techniques for recognizing the orientation of a person's face from an image obtained by capturing the person's face. Generally, face orientation is recognized using features of each part of the face, so if a part of the face is covered with a covering material such as a mask, recognition accuracy decreases.

[0003] In view of this, the technique described in Patent Document 1 calculates the left-right orientation angle of the face from the relative position of the nose at the center of the mask with respect to the face width of the face region when a mask is present.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problem to be Solved by the Invention

[0005] However, in the technique described in Patent Document 1, although the central portion of the mask is taken as the position of the nose, the central portion of the mask does not necessarily match the actual position of the nose, and thus the left-right orientation angle of the face may not be calculated accurately in some cases.

[0006] In view of these circumstances, an object of the present invention is to provide a recognition device and the like that can accurately recognize the orientation of an object covered by a covering member.

Means for Solving the Problem

[0007] The invention described in claim 1 comprises: an acquisition means for acquiring an image of an object covered by a covering member while it is illuminated with light; a detection means for detecting a center line in the front vertical direction on the surface of the object based on the brightness values ​​of pixels included in the image; and a recognition means for recognizing the vertical orientation of the object based on the inclination of the center line detected by the detection means.

[0008] The invention described in claim 5 is a recognition method by a recognition device, comprising: an acquisition step of acquiring an image of an object covered by a covering member while it is illuminated with light; a detection step of detecting a center line in the front vertical direction on the surface of the object based on the brightness values ​​of pixels included in the image; and a recognition step of recognizing the vertical orientation of the object based on the inclination of the center line detected in the detection step.

[0009] The invention described in claim 6 enables the computer to function as an acquisition means for acquiring an image of an object covered by a covering member while illuminated by light; a detection means for detecting a front vertical center line on the surface of the object based on the brightness values ​​of pixels included in the image; and a recognition means for recognizing the vertical orientation of the object based on the inclination of the center line detected by the detection means. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram of recognition device 1. [Figure 2] This is an example of a color-coded image. [Figure 3] This is an example of a color-coded image. [Figure 4] This is an example of a block diagram of the face orientation recognition device D. [Figure 5] This is an example of a functional block diagram of the face orientation recognition device D. [Figure 6] (A), (B), and (C) are diagrams illustrating examples of methods for detecting facial and mask features. [Figure 7] This is an example of color-coded images taken from various facial angles. [Figure 8]Figs. (A) and (B) are diagrams showing an example of a specially shaped mask. [Figure 9] It is a flowchart showing an example of face orientation determination processing performed by the face orientation recognition device D. [Figure 10] It is a flowchart showing an example of mask wearing determination processing performed by the face orientation recognition device D. [Figure 11] It is a flowchart showing an example of yaw angle determination processing performed by the face orientation recognition device D. [Figure 12] It is a flowchart showing an example of pitch angle determination processing performed by the face orientation recognition device D. [Figure 13] Figs. (A) and (B) are diagrams showing an example of features at the left and right ends of the mask and features of the face center line (near the upper end of the mask). [Figure 14] Figs. (A) and (B) are diagrams showing an example of features at the left and right ends of the mask and features of the face center line (the portion overlapping the mask). [Figure 15] It is a diagram showing an example of an image captured in a state where the camera C and the irradiation unit L are arranged in different directions as viewed from the target. MODE FOR CARRYING OUT THE INVENTION

[0011] A mode for carrying out the present invention will be described with reference to FIG. 1.

[0012] As shown in FIG. 1, the recognition device 1 includes an acquisition unit 111A, a detection unit 111B, a recognition unit 111C, a storage control unit 111D, and a storage unit 111E. Note that the storage unit may be provided outside the recognition device 1.

[0013] The acquisition unit 111A acquires an image obtained by capturing a target covered with a covering member in a state where light is irradiated onto the target by an irradiation unit. The target covered with a covering member is, for example, a face wearing a mask. It is preferable that the irradiation unit irradiates light onto the target from a fixed direction.

[0014] The detection means 111B detects the center line in the front vertical direction on the surface of the target based on the luminance values of pixels included in the image acquired by the acquisition means 111A. When the target is a face (a general face), the center line in the front vertical direction on the surface of the target is a line passing through the face surface that runs between the eyes, through the center of the nose and the center of the mouth.

[0015] Here, the center line will be described with reference to FIG. 2. FIG. 2 is a color-coded image 200 obtained by color-coding, based on luminance values, an image captured of a face wearing a mask (an example of a "covering material") while being irradiated with light from the camera direction (for privacy protection of the model, the eye portions are processed to be painted black; the same applies to images of the model captured hereinafter). Specifically, portions with a luminance value less than 100 are represented by the color indicated by reference numeral 201, portions with a luminance value of 100 or more and less than 160 are represented by the color indicated by reference numeral 202, portions with a luminance value of 160 or more and less than 220 are represented by the color indicated by reference numeral 203, and portions with a luminance value of 220 or more are represented by the color indicated by reference numeral 204. When irradiated with light, the luminance value of the mask region is higher than that of skin. As shown in FIG. 2, the mask region has a high luminance value (a luminance value of 220 or more). A face wearing a mask is symmetrical with respect to the face center, and the face center tends to protrude slightly relative to other facial regions. Therefore, based on an image captured with light irradiated from the front direction of the camera, the detection means 111B can detect the left end 211 of the mask and the face (mask) center line 212 (an example of the "center line"). That is, the detection means 111B detects the edge on the left end side of the mask (the right side in the image) in the region with a luminance value of 220 or more as the left end 211 of the mask, and detects the edge on the face center side (the opposite side to the left end side of the mask) in the region with a luminance value of 220 or more as the face center line 212. However, when the horizontal orientation (yaw angle) exceeds a certain angle (about 50 degrees) relative to the front of the camera, the edge on the face center side (the opposite side to the left end side of the mask) in the region with a luminance value of 220 or more no longer coincides with the face center line 212. Therefore, in this case, it is preferable to detect the edge on the face center side in the region with a luminance value of 160 or more and less than 220 as the face center line 212B.

[0016] The recognition means 111C recognizes the orientation of the target based on the detection result of the detection means 111B.

[0017] As described above, according to the operation of the recognition device 1 according to the embodiment, the orientation of the object is recognized based on the detection result of the center line in the front vertical direction on the surface of the object based on the brightness value of the pixels included in the image. Therefore, even if the object is covered by a covering member, the orientation of the object can be accurately recognized.

[0018] Furthermore, the recognition means 111C may recognize the vertical orientation (pitch angle) of the object based on the inclination of the face centerline 212. Since the inventors' research has shown that there is a correlation between the vertical orientation of the face and the inclination of the face centerline 212, a threshold for the inclination can be set for each vertical orientation (pitch angle) of the face (for example, every 10 degrees), and the vertical angle of the face can be determined by comparing the inclination of the face centerline 212 with the threshold. This makes it possible to recognize the vertical orientation of the object.

[0019] Furthermore, the detection means 111B further detects the upper boundary 213, which is the upper boundary of the covering member on the object, when the recognition means 111C recognizes the vertical orientation of the object. The memory control means 111D stores the inclination of the upper boundary 213 detected by the detection means 111B in the memory means 111E, corresponding it to the vertical orientation of the object recognized by the recognition means 111B. When an image of an object whose vertical orientation has not been recognized is newly acquired, the recognition means 111C may recognize the vertical orientation of the object in the image as the vertical orientation of the object if the inclination of the upper boundary 213 of the covering member detected for that image is approximately the same as the inclination of the upper boundary 213 of the covering member. This determination is possible because, as a result of the inventor's research, it has been found that if the inclination of the upper boundary 213 of the covering member is approximately the same, the vertical orientation of the object is also approximately the same. As a result, once the vertical orientation of an object is recognized, subsequent images can be recognized by detecting the upper boundary 213 and comparing it with the upper boundary 213 stored in the storage means. In other words, the processing burden for recognizing the vertical orientation is reduced.

[0020] Furthermore, the recognition means 111C may recognize the lateral orientation (yaw angle) of the object based on the position of the intersection point between the upper boundary line 213 and the face center line 212 within the width of the object.

[0021] Here, an example of recognizing the lateral orientation of an object will be explained using Figure 3. The recognition means 111C recognizes the lateral orientation of an object based on three perpendicular lines 221-223. Perpendicular line 221 is a perpendicular line passing through the point at the left earlobe. Perpendicular line 222 is a perpendicular line passing through the intersection of the face centerline 212 and the upper boundary 213. Perpendicular line 223 is a perpendicular line that moves horizontally from the intersection of the face centerline 212 and the upper boundary 213 towards the back of the face and passes through the point at the right end of the face. The recognition means 111C recognizes the lateral orientation of an object based on any two of the following widths: the width WA (front face width WA) of perpendicular line 221 and perpendicular line 222, the width WB (back face width WB) of perpendicular line 222 and perpendicular line 223, and the width WC of perpendicular line 221 and perpendicular line 223. The inventor's research has shown a correlation between the lateral orientation of the face and the ratio of the front face width WA to the back face width WB. Therefore, by setting a threshold for the ratio of the front face width WA to the back face width WB for each lateral orientation (angle) of the face (for example, every 10 degrees), and comparing the ratio of the front face width WA to the back face width WB with the threshold, the lateral angle of the face can be determined. This makes it possible to recognize the lateral orientation of the object.

[0022] Furthermore, the detection means 111B further detects the left and right boundaries, which are at least one of the left and right boundaries of the covering member. When the detection means 111B detects the left and right boundaries, the recognition means 111C recognizes the lateral orientation of the object based on the left and right boundaries detected by the detection means 111B and the center line. The left and right boundaries refer to at least one of the boundaries, for example, the left boundary indicated by the edge of the left end 211 of the mask (see Figure 2), and the right boundary indicated by the edge of the right end of the mask. In other words, the left and right boundaries can be said to be a general term for the left and right boundaries. The recognition means 111C recognizes that the object is facing to the right based on the detection of the edge of the left end 211 of the mask (left boundary), and recognizes that the object is facing to the left based on the detection of the edge of the right end of the mask (right boundary). [Examples]

[0023] Next, we will describe specific examples corresponding to the embodiments described above.

[0024] Examples will be described using Figures 4-14. The examples described below are examples in which the present invention is applied to a face orientation recognition device D (hereinafter sometimes referred to as "recognition device D").

[0025] In this embodiment, the recognition device D is connected to an illumination unit L, which is a light source such as an LED, and a camera C. The recognition device D recognizes the orientation of the face F, which is covered by a mask M, based on an image taken while the face F is illuminated by light from the illumination unit L.

[0026] Specifically, recognition device D detects a face (or facial features) by processing the captured image and determines whether a mask is being worn. If it determines that a mask is being worn, it detects mask features (mask shadow area, mask edge, mask edge slope, mask strap edge slope, etc.) and facial features (face centerline, face width, face contour, etc.), and recognizes the orientation of the face from these extracted features.

[0027] More specifically, first, it is determined whether a face is visible in the image and the location (mouth area) for mask-wearing detection is determined. For example, the approximate location of the mouth is determined within the face frame detected by conventionally known face detection processes. Alternatively, the location for mask-wearing detection may be determined by detecting the features of each part of the face. For example, the location of the eyes may be detected, and the area below them may be determined as the approximate location of the mouth, and mask-wearing detection may be performed. Next, it is determined whether a mask is being worn. Generally, when a face F wearing a mask M is illuminated with light and imaged, a high brightness value is obtained in the mask-wearing area, so it is determined whether a mask is being worn based on the high brightness value in the area around the mouth. Alternatively, it may be determined whether a mask is being worn based on the external shape information of the mask. If it is determined that a mask is being worn, the features of the mask and the face are detected, and the orientation of the face is recognized from these extracted features. Furthermore, based on the feature detection results when detecting face features and mask features from the image, the intensity of the illumination light or camera parameters may be adjusted to more clearly detect shadows and brightness gradients of the mask and face. In this embodiment, we will describe the case where the camera C and the illumination unit L are pointed towards the target from approximately the same direction. The case where the camera C and the illumination unit L are pointed towards the target from different directions will be described in the modified examples below.

[0028] [1. Configuration of Recognition Device D] Next, the configuration of the recognition device D according to this embodiment will be described using Figure 4. As shown in Figure 4, the recognition device D is broadly composed of a control unit 311, a storage unit 312, a communication unit 313, a display unit 314, and an operation unit 315. The communication unit 313 is connected to the camera C and the illumination unit L.

[0029] The memory unit 312 is composed of, for example, a hard disk drive, and stores the OS (Operating System) and various programs, including a face orientation recognition program for recognizing the orientation of the face. The memory unit 312 also stores images captured by the camera and various data used in the face orientation recognition program.

[0030] The communication unit 313 controls the communication status with the illumination unit L and the camera C.

[0031] The display unit 314 is composed of, for example, a liquid crystal display and is configured to display images captured by the camera C.

[0032] The control unit 315 is composed of, for example, a keyboard, a mouse, etc., and receives operation instructions from the operator and outputs the content of those instructions as instruction signals to the control unit 311.

[0033] The control unit 311 includes a CPU (Central Processing Unit) and ROM (Read Only Memory). It is composed of RAM (Random Access Memory), etc. The CPU then reads and executes various programs, including the face orientation recognition program, stored in ROM and memory unit 312, thereby realizing various functions.

[0034] [2. Functions of Recognition Device D] Next, the functions of the recognition device D (control unit 311) according to this embodiment will be explained using Figure 5. Figure 5 is a functional block diagram of the recognition device D. The recognition device D is composed of an image analysis unit 351, an illumination light adjustment unit 357, and a camera adjustment unit 358. The functions of the image analysis unit 351, the illumination light adjustment unit 357, and the camera adjustment unit 358 are realized by the control unit 311 executing a face orientation recognition program.

[0035] The image analysis unit 351 includes a mouth region detection unit 352, a mask wearing determination unit 353, a mask feature detection unit 354, a face feature detection unit 355, and a face orientation recognition unit 356.

[0036] The mouth region detection unit 352 performs face detection, eye detection, nose detection, and mouth detection processing on an image captured by the camera C while the illumination unit L illuminates a face F covered by a mask M. If the face region is determined during the face detection process, it can be determined that the mouth region is located in the area below the face detection frame. If the eye region is determined during the eye detection process, it can be determined that the mouth region is located below the eyes. If the nose region is determined during the nose detection process, it can also be determined that the mouth region is located below the nose. If the mouth region is detected during the mouth detection process, it is determined that the region is the mouth region.

[0037] The mask wearing determination unit 353 determines whether a mask M is being worn in the mouth area detected by the mouth area detection unit 352. Specifically, the determination is made based on the brightness value. For example, as shown in Figure 6(A), the average brightness value of the right eye area 401 and the average brightness value of the mouth area 402 are calculated. Generally, when light is shone towards the face, the brightness value of the mask is greater than the brightness value of the skin. For example, by comparing the average brightness value of the right eye area 401 and the average brightness value of the mouth area 402, if they are approximately the same, it is determined that a mask is not being worn. If the average brightness value of the mouth area 402 is greater (greater than the threshold), it is determined that a mask is being worn.

[0038] The mask feature detection unit 354 detects features of the mask region based on the brightness values ​​of the mask region. The face feature detection unit 355 detects features of the face region based on the brightness values ​​of the face region. Specifically, the mask feature detection unit 354 and the face feature detection unit 355 (sometimes collectively referred to as the "feature detection unit") detect features using a color-coded image obtained by dividing the brightness values ​​of an image of a face into four parts using three thresholds, as shown in Figure 2. Specifically, the image is divided into (1) the mask region, (2) the skin (light) region, (3) the skin (dark) region, and (4) the background region.

[0039] Here, we will explain an example of creating a color-coded image. First, the feature detection unit determines a first threshold that indicates the boundary between (1) the mask region and (2) the skin (bright) region, based on the maximum value Ja of the skin region (for example, the left eye region 401) (considering that the pixel value may include spike noise, etc., here we treat the fifth largest value as the maximum value) and the average brightness value Jb of the mouth region 402 when the mask is worn. For example, if Ja is "219" and Jb is "246", the first threshold is the value of Ja plus a margin. For example, the first threshold is "220", which is Ja plus "1" (we added "1" to make it a nice round number, but other values ​​are also acceptable. The same applies to the second and third thresholds below). A region composed of pixels with a brightness value greater than the first threshold can be detected as the mask region.

[0040] Next, the feature detection unit determines a second threshold that indicates the boundary between (2) the light skin region and (3) the dark skin region, based on the average value Jc of the skin region (for example, the lower half of the left eye region 403 shown in Figure 6(B)) (the lower half of the left eye region 403 was used for the eyebrows and pupil area because their brightness values ​​are small, but the average value of other parts can also be used). For example, if Jc is "157", the second threshold is the value obtained by subtracting the margin from Jc. For example, "150", which is Jc minus "7", is used as the second threshold.

[0041] Next, the feature detection unit determines a third threshold that indicates the boundary between (3) the skin (dark) region and (4) the background region, based on the minimum value Jd of the skin region (for example, the left half region 404 of the left eye shown in Figure 6(C)). (Considering that the pixel value may include spike noise, etc., the fifth smallest value is treated as the minimum value here. Here, the left half region 404 of the left eye is used, but the minimum value of any other skin area may also be used.) For example, if Jd is "78", the third threshold is the value obtained by subtracting the margin from Jd. For example, "70", obtained by subtracting "8" from Jd, is used as the third threshold.

[0042] The feature detection unit divides each pixel constituting the image into (1) a mask region, (2) a skin (light) region, (3) a skin (dark) region, and (4) a background region based on the brightness value of the pixel and the first, second, and third thresholds, and creates a color-coded image by representing these regions with four colors. Then, based on the color-coded image, it detects features of the mask region and features of the face region.

[0043] Figure 7 shows an example of images taken by changing the vertical orientation (pitch angle) and horizontal orientation (yaw angle) of the face by 10 degrees each. Similar to the example in Figure 2, the image is divided into areas with a brightness value of less than 100, areas with a brightness value of 100 to less than 160, areas with a brightness value of 160 to less than 220, and areas with a brightness value of 220 or more, and each is represented by a different color (same as in Figure 2). When light is irradiated, the brightness value of the mask area becomes larger than that of the skin. The mask area has a large brightness value (brightness value of 220 or more). When the face orientation recognition unit 356 is facing left or right (yaw angle: 20 to 50 degrees), the left and right boundaries, which are the boundaries between the left and right edges of the mask and the skin, can be clearly detected, and the unit can determine which direction the face is facing. On the other hand, when the face is facing directly in front of the camera (yaw angle: 0 degrees), the boundaries between the left and right edges of the mask and the skin (left and right boundaries) cannot be detected, and therefore the unit can determine that the face is facing forward. To estimate the left-right face orientation angle, if the center of the face, the edges of the face (such as the contour and ears), and the left and right edges of the mask can be detected, the approximate face orientation angle can be calculated. Below, the method by which the face orientation recognition unit 356 recognizes the vertical orientation (pitch angle) and the horizontal orientation (yaw angle) of the face will be explained in detail.

[0044] (When the pitch angle is 0 degrees) When the face is facing directly in front of the camera (yaw angle: 0 degrees, pitch angle: 0 degrees), the brightness value of the mouth area is high (compared to the skin area), but no prominent mask edges are detected on the left and right edges of the mask. When the face is facing 10 degrees to the left (yaw angle: 10 degrees, pitch angle: 0 degrees), the brightness value of the mouth area is high, and because the face is tilted to the left relative to the camera, a prominent mask edge can be detected on the left edge of the mask. This is because the light is shining from the direction directly in front of the camera. When the face is facing 20 degrees to the left (yaw angle: 20 degrees), the brightness value of the area of ​​the mask facing the camera becomes more prominent compared to the 10-degree case, because the face is tilted more. The left edge of the mask can also be clearly detected. In addition, since a face wearing a mask is symmetrical with respect to the center of the face, and the center of the face tends to protrude slightly from other parts of the face, when light is shining from the direction directly in front of the camera, the center line of the face can be clearly detected in addition to the left edge of the mask. Similarly, when the face is turned 30 degrees to the left, 40 degrees to the left, and 50 degrees to the left, the left edge of the mask and the face centerline can be clearly detected. Also, if there are no objects near the face, the background brightness value will not increase even if light is shone on it, so it is possible to detect the face width WA (forward) and face width WB (backward) relative to the camera. When the face is tilted 10 degrees to the left, the direction to the left of the face can be determined by detecting the edge of the left edge of the mask, the face centerline, the face width WA (forward), and the face width WB (backward) relative to the camera, as described above. When the face is turned to the right, it is simply a symmetrical representation of when the face is turned to the left, so a detailed explanation is omitted.

[0045] (When the pitch angle is 10 degrees upwards) When the face is facing upwards at 10 degrees (yaw angle: 0 degrees, pitch angle: upwards), the brightness value of the mouth area is higher than that of the skin area, but no prominent mask edges are detected at the left and right edges of the mask. The upper edge of the mask has a steeper slope compared to when the camera is facing forward (yaw angle: 0 degrees, pitch angle: 0 degrees), but the slope of the upper edge of the mask changes due to deformation each time the mask is worn, so this alone is not enough to determine that the face is facing upwards. However, if the reference edge slope of the upper edge of the mask when the mask is worn is known, it is possible to determine the vertical face angle as well. When the face is facing left at 10 degrees (yaw angle: 10 degrees), the brightness value of the mouth area is higher, similar to when the pitch angle is 0 degrees, and a prominent edge can be detected at the left edge of the mask due to the tilt of the face to the left relative to the camera. Furthermore, even when the face is tilted to the left, the left edge of the mask, the center line of the face, the face width WA (front of the face relative to the camera), the face width WB (back of the face relative to the camera), etc., can be detected, just as when the pitch angle is 0 degrees, to determine how many degrees the face is tilted to the left.

[0046] The upward angle (pitch angle) of the face can be determined by the inclination of the face's centerline when the face is facing 30 degrees to the left, 40 degrees to the left, 50 degrees to the left, etc. Comparing the inclination of the face's centerline when the face is facing 30 degrees to the left, 40 degrees to the left, and 50 degrees to the left, when the upward angle (pitch angle) of the face is 0 degrees and when it is 10 degrees, it can be clearly determined that the face is facing upward when the upward angle (pitch angle) is 10 degrees compared to when it is 0 degrees. If the slope of the upper edge of the mask when the face is facing upward at a pitch angle of 10 degrees is recorded as a reference, then, as long as the mask does not deform, if the slope of the upper edge of the mask in a later captured image is the same (preferably with a certain width and approximately the same angle), it can be determined that the upward angle (pitch angle) of the face is 10 degrees. When the face is facing 10 degrees to the right, it is simply symmetrical to when the face is facing 10 degrees to the left, so a detailed explanation is omitted.

[0047] (When the pitch angle is 20 degrees upwards) When the face is facing upwards at 20 degrees (yaw angle: 0 degrees, pitch angle: upwards at 20 degrees), the brightness value of the mouth area is higher than that of the skin area, but no prominent mask edges are detected at the left and right edges of the mask. The upper edge of the mask has a steeper slope compared to when the camera is facing forward (yaw angle: 0 degrees, pitch angle: 0 degrees), but the slope of the upper edge of the mask changes due to deformation each time the mask is worn, so this alone is not enough to determine that the face is facing upwards. However, if the reference edge slope of the upper edge of the mask when the mask is worn is known, the vertical face angle can also be determined. When the face is facing left at 10 degrees (yaw angle: 10 degrees, pitch angle: upwards at 20 degrees), the brightness value of the mouth area is higher, similar to when the pitch angle is 0 degrees, and a prominent edge can be detected at the left edge of the mask due to the tilt of the face to the left relative to the camera. Furthermore, when the face is tilted to the left, similar to when the pitch angle is 0 degrees, the left edge of the mask, the center line of the face, the face width WA (front of the face relative to the camera), the face width WB (back of the face relative to the camera), etc. can be detected to determine how many degrees the face is tilted to the left.

[0048] The upward angle (pitch angle) of the face can be determined by the slope of the face's centerline when the face is facing 30 degrees to the left, 40 degrees to the left, and 50 degrees to the left, similar to when the face is facing 10 degrees upward. Comparing the cases when the face is facing 10 degrees upward and 20 degrees upward, the slope of the face's centerline at 30 degrees to the left, 40 degrees to the left, and 50 degrees to the left is steeper at 20 degrees upward, so it can be determined to be 20 degrees upward. Furthermore, if the slope of the upper edge of the mask when the face is facing 20 degrees upward is recorded as a reference, then, as long as the mask does not deform, if the slope of the upper edge of the mask in a later captured image is the same (preferably with a certain width and approximately the same angle), it can be determined that the upward angle (pitch angle) of the face is 20 degrees. When the face is facing 20 degrees to the right, it is simply symmetrical to when the face is facing 20 degrees to the left, so a detailed explanation is omitted.

[0049] (When the pitch angle is -10 degrees) When the face is tilted 10 degrees downward (yaw angle: 0 degrees, pitch angle: downward 10 degrees), the brightness value of the mouth area is larger than that of the skin area, but no prominent mask edges are detected on the left and right edges of the mask. The upper edge of the mask has a gentler slope compared to when the camera is facing forward (yaw angle: 0 degrees, pitch angle: 0 degrees), but the slope of the upper edge of the mask changes due to deformation each time the mask is worn, so this alone is not enough to determine that it is tilted downward. However, if the reference edge slope of the upper edge of the mask when the mask is worn is known, it is possible to determine the face angle in the vertical direction as well (the same as when it is tilted upward 10 degrees). When the face is tilted 10 degrees to the left (yaw angle: 10 degrees, pitch angle: downward 10 degrees), the brightness value of the mouth area is larger, similar to when the pitch angle is 0 degrees, and a prominent edge can be detected on the left edge of the mask due to the tilt of the face to the left relative to the camera. When the face is tilted to the left, it is possible to determine how many degrees the face is tilted to the left by detecting the left edge of the mask, the center line of the face, the face width WA (front of the face relative to the camera), the face width WB (back of the face relative to the camera), etc., just as when the pitch angle is 0 degrees.

[0050] The downward angle of the face can be determined by the slope of the face's centerline when the face is facing 30 degrees to the left, 40 degrees to the left, and 50 degrees to the left, similar to when the face is facing 10 degrees upward. Comparing the case when the pitch angle is 0 degrees and the case when the pitch angle is 10 degrees downward, the slope of the face's centerline when the face is facing 30 degrees to the left, 40 degrees to the left, and 50 degrees to the left can be determined to be greater when the pitch angle is 10 degrees downward, allowing it to be determined to be 10 degrees downward. If the slope of the upper edge of the mask when the face is facing 10 degrees downward is recorded as a reference, then, as long as the mask does not deform, if the slope of the upper edge of the mask in a later captured image is the same (preferably with a certain width and approximately the same angle), it can be determined that the downward angle of the face (pitch angle) is 10 degrees. When the face is facing 10 degrees to the right, it is simply symmetrical to when the face is facing 10 degrees to the left, so a detailed explanation is omitted.

[0051] (When the pitch angle is -20 degrees) When the face is tilted downwards by 20 degrees (yaw angle: 0 degrees, pitch angle: downwards by 20 degrees), the brightness value of the mouth area is larger than that of the skin area, but no prominent mask edges are detected on the left and right edges of the mask. The upper edge of the mask has a gentler slope compared to when the camera is facing forward (yaw angle: 0 degrees, pitch angle: 0 degrees), but the slope of the upper edge of the mask changes due to deformation each time the mask is worn, so this alone is not enough to determine that it is tilted downwards. However, if the reference edge slope of the upper edge of the mask when worn is known, the vertical face angle can also be determined (the same as when it is tilted upwards by 10 degrees). When the face is tilted to the left by 10 degrees (yaw angle: 10 degrees, pitch angle: downwards by 20 degrees), the brightness value of the mouth area is larger, similar to when the pitch angle is 0 degrees, and a prominent edge can be detected on the left edge of the mask due to the tilt of the face to the left relative to the camera. Furthermore, when the face is tilted to the left, it is possible to detect the leftward tilt of the face by detecting the left edge of the mask, the center line of the face, the face width WA (front of the face relative to the camera), the face width WB (back of the face relative to the camera), etc., just as when the pitch angle is 0 degrees.

[0052] The downward angle (pitch angle) of the face can be determined by the slope of the face's centerline when the face is facing 30 degrees to the left, 40 degrees to the left, and 50 degrees to the left, similar to when the face is facing 10 degrees down. Comparing the cases when the face is facing 10 degrees down and 20 degrees down, the slope of the face's centerline when the face is facing 30 degrees to the left, 40 degrees to the left, and 50 degrees to the left indicates that the face is tilted more downwards, and can be determined to be 20 degrees down. If the slope of the upper edge of the mask when the downward angle (pitch angle) of the face is 20 degrees is recorded as a reference, then, as long as the mask does not deform, if the slope of the upper edge of the mask in a later captured image is the same (preferably with a certain width and approximately the same angle), it can be determined that the downward angle (pitch angle) of the face is 20 degrees. When the face is facing 20 degrees down to the right, it is simply symmetrical to when the face is facing 20 degrees down to the left, so a detailed explanation is omitted.

[0053] Furthermore, when the face is turned 50 degrees to the left, the brightness value of the skin near the left edge of the mask may be high. However, the brightness value of the mask straps that go over the ears also increases when illuminated with light, and a prominent edge can be detected. Based on this information, it is possible to detect the left edge of the mask. The up and down orientation of the face can also be determined from this strap edge information.

[0054] Furthermore, since face width WA and WB may differ depending on the vertical orientation of the face (depending on how the perpendicular line 222 in Figure 2 is drawn), it is also possible to determine the vertical orientation of the face and then determine the orientation based on the face width according to the criteria for each vertical orientation.

[0055] Furthermore, when light is shone from the same direction as camera C, if the face is tilted downwards at a 20-degree angle, and the forehead is not obscured by hair, the brightness of the forehead area will increase. The fact that the brightness of the forehead area exceeds a predetermined threshold may be used to determine that the face is tilted downwards at a 20-degree angle.

[0056] Furthermore, once the upper edge gradient of the mask, which serves as the reference for the vertical direction of the face, is determined, the vertical pitch angle can be determined based on that value. For example, if the upper edge gradient of the mask is determined when the face is pointing downwards at 20 degrees, and then the face direction changes to 20 degrees upwards, the edge gradient for 20 degrees upwards can be calculated based on the reference edge gradient for 20 degrees downwards. By comparing this value with the edge gradient detected from the image, if the values ​​are close, it can be determined that the face direction has changed to 20 degrees upwards. The reference edge gradient can be calculated as long as there is no deformation or removal of the mask while wearing it. It changes if the distance between the face and the camera changes, but if that distance is approximately constant, once the reference is determined, the determination can be made based on that reference. Also, even when the distance between the face and the camera changes, the face direction can be determined by changing the reference edge gradient based on the depth information and size information of the face.

[0057] Furthermore, since there are various types of masks, it is possible to determine which type of mask is being worn and estimate the direction of the face based on the shape information specific to the mask being worn. According to the inventor's research, there are various types of masks currently on the market, but regarding the left (right) edge of the mask as described above, although there are characteristics specific to each mask, it has been found that if light is shone on the mask when the face is turned left or right, the mask's unique shape at the left and right edges can be detected for almost all masks. Also, the center line of the face (mask) when the face is turned left or right differs in shape depending on the type of mask, but almost all masks have a protruding shape at the center (because the face has a shape that protrudes towards the center of the face, and the mask is designed to fit that shape), so if light is shone from the camera side, the center of the mask, which is the center of the face, can be detected. In addition, regarding the upper edge gradient of the mask as described above, it varies depending on the type of mask, but once a reference is determined using the above process, the vertical angle of the face can be determined based on that reference. As an example of a unique shape, in the mask shown in Figure 8(A), the shape of the left and right edges of the mask is not a vertical edge line, but the shape of the left and right edges of the mask can still be detected. Also, in the case of a hemispherical cup-shaped mask as shown in Figure 8(B), the mask center and the edge slope of the upper edge of the mask can be detected. Furthermore, even a black mask can be imaged as white when photographed with infrared light, although there may be some differences in brightness values, so similarly, the mask can be detected and the features of the left, right, and upper edges of the mask can be detected.

[0058] Furthermore, while the mask may deform, the characteristics of the edges on the left and right ends of the mask, as described above, hardly change. Regarding the center line of the mask when the face is turned left or right, because the mask undergoes flexible shape changes, there may be cases where the center line of the mask cannot be detected due to deformation. In such cases, when the face is turned from 30 degrees to 50 degrees to the left, the center line of the face can be detected in the upper half of the face, rather than the mask area, to determine the up or down direction of the face, or the tilt of the mask straps worn over the ears can be detected to determine the up or down direction of the face. The slope of the upper edge of the mask may also change due to mask deformation, but once a reference slope is determined, it is possible to handle deformation of the slope of the upper edge of the mask by periodically calculating the reference each time, taking mask deformation into consideration.

[0059] When the face orientation recognition unit 356 recognizes the orientation of a face, it may extract edge gradients and the like based on brightness values. However, depending on the shooting conditions, changing camera parameters (such as exposure time) or illumination light intensity (either one or both) may significantly improve the extraction of features such as edge gradients. In such cases, the illumination light adjustment unit 357 adjusts the intensity of the light emitted from the illumination unit L, or the camera adjustment unit 358 adjusts the parameters of camera C, based on the brightness values ​​detected by the mouth region detection unit 352, the mask wearing determination unit 353, the mask feature detection unit 354, and the face feature detection unit 355.

[0060] [3. Example of operation of recognition device D] Next, an example of the face orientation determination process of the recognition device D will be explained using the flowchart in Figures 9-12.

[0061] First, the control unit 311 of the recognition device D acquires an image taken by the camera C with the face F covered by the mask M illuminated by the illumination unit L (step S11).

[0062] Next, the control unit 311 detects the mouth region of the face in the image acquired in step S11 (step S12).

[0063] Next, the control unit 311 performs a mask wearing determination process on the mouth area detected in step S12 (step S13).

[0064] Here, we will explain the mask-wearing detection process using Figure 10.

[0065] First, the control unit 311 calculates the average brightness value Ia of the skin area (for example, the right eye area 401 in Figure 6) (step S21).

[0066] Next, the control unit 311 calculates the average brightness value Ib of the mouth region (for example, the mouth region 402 in Figure 6) (step S22).

[0067] Next, the control unit 311 determines whether "Ib - Ia > Ic" is true (step S23). Here, Ic is a threshold (for example, Ic = 40). Ic can be set according to the observed brightness value, etc. If the control unit 311 determines that "Ib - Ia > Ic" is true (step S23: YES), it determines that a mask is being worn (step S24) and terminates the mask wearing determination process. On the other hand, if the control unit 311 determines that "Ib - Ia > Ic" is not true (step S23: NO), it determines that a mask is not being worn (step S25) and terminates the mask wearing determination process.

[0068] Returning to Figure 9, the control unit 311 determines whether or not a mask is being worn in the mask wearing determination process (step S14). If the control unit 311 determines that a mask is not being worn (step S14: NO), it terminates the face orientation determination process. On the other hand, if the control unit 311 determines that a mask is being worn (step S14: YES), it then proceeds to perform the yaw angle determination process (step S15).

[0069] Here, we will explain the yaw angle determination process using Figure 11.

[0070] First, the control unit 311 determines whether the left and right edges of the mask can be detected from the image acquired in the process of step S11 or step S18 described later (step S31). For a mask having special-shaped left and right edges as shown in Fig. 8(A), by first determining and recording a mask portion having a luminance value larger than the luminance value of skin, subsequent processes can be performed in the same manner.

[0071] When the control unit 311 determines that the left and right edges of the mask cannot be detected (step S31: NO), the control unit 311 determines the yaw angle as "0 degrees" (step S32), and ends the yaw angle determination process. On the other hand, when it is determined that the left and right edges of the mask can be detected (step S31: YES), the control unit 311 next determines whether a face center line (see face center line 212 in Fig. 2) can be detected (step S33).

[0072] When the control unit 311 determines that the face center line cannot be detected (step S33: NO), the control unit 311 determines the yaw angle as "10 degrees" (step S34), and ends the yaw angle determination process. On the other hand, when it is determined that the face center line can be detected (step S33: YES), the control unit 311 then calculates a near-side face width WA and a far-side face width WB (see Fig. 3) (step S35).

[0073] Next, the control unit 311 determines the yaw angle based on the near-side face width WA and the far-side face width WB calculated in the process of step S35 (step S36). In the present embodiment, the yaw angle is determined by comparing WB / WA with predetermined thresholds Y1, Y2, Y3 (Y1>Y2>Y3). The values of the thresholds Y1, Y2, Y3 may be changed based on values such as an observed value of the distance between the camera C and the face F.

[0074] Specifically, the control unit 311 determines the yaw angle as follows. (1) Y1 < WB / WA → Yaw angle: 20 degrees (2) Y2 < WB / WA < Y1 → Yaw angle: 30 degrees (3) Y3 < WB / WA < Y2 → Yaw angle: 40 degrees (4) WB / WA < Y3 → Yaw angle: 50 degrees

[0075] Next, the control unit 311 performs face centerline correction processing (step S37) and terminates the yaw angle determination processing. Face centerline correction processing is performed to correct the face centerline in cases where, when the yaw angle with respect to the illumination unit L becomes large, the brightness value of the center of the mask decreases, and the mask centerline detected by the brightness value conditions described above no longer coincides with the face centerline (for example, when the yaw angle is 50 degrees in Figure 7. If the yaw angle is 40 degrees or less, the mask centerline and the face centerline tend to coincide). In this embodiment, when the yaw angle is less than 50 degrees, the line corresponding to the edge formed by pixels color-coded with a brightness value of 220 or higher is used as the face centerline. On the other hand, when the yaw angle is 50 degrees or more, the line corresponding to the edge formed by pixels color-coded with a brightness value of 160 or higher and less than 220 is used as the face centerline.

[0076] Returning to Figure 9, the control unit 311 performs pitch angle determination processing (step S16).

[0077] Here, we will explain the pitch angle determination process using Figure 12.

[0078] First, the control unit 311 determines whether the tilt of the upper edge of the mask has been recorded for all pitch angles (down 20 degrees, down 10 degrees, 0 degrees, up 10 degrees, up 20 degrees) (step S51). If the control unit 311 determines that the tilt of the upper edge of the mask has not been recorded for all pitch angles (step S51: NO), it determines the pitch angle based on the features of the left and right edges of the mask and the features of the face centerline (near the upper edge of the mask) (step S53). Specifically, as shown in Figures 13(A) and (B), the features of the left and right edges of the mask and the features of the face centerline differ depending on the pitch angle. Therefore, the reference features of the left and right edges of the mask and the face centerline for each pitch angle are recorded in the storage unit 312 in advance, and the pitch angle corresponding to the most similar features of the left and right edges of the mask and the face centerline in the acquired image is determined to be the pitch angle of the face in the image. In this embodiment, the inclination of the upper mask straps is recorded as a characteristic of the left and right edges of the mask, and the inclination near the top edge of the mask is recorded as a characteristic of the face's centerline.

[0079] A specific description will be given below. • Inclination of the string on the mask when the pitch is 0 degrees (horizontal position), recorded in the storage unit 312: m0 • Inclination of the face center line when the pitch is 0 degrees (horizontal position), recorded in the storage unit 312: v0 • Inclination of the string on the mask when the pitch is 10 degrees upward, recorded in the storage unit 312: m1 • Inclination of the face center line when the pitch is 10 degrees upward, recorded in the storage unit 312: v1 • Inclination of the string on the mask when the pitch is 20 degrees upward, recorded in the storage unit 312: m2 • Inclination of the face center line when the pitch is 20 degrees upward, recorded in the storage unit 312: v2 • Inclination of the string on the mask when the pitch is 10 degrees downward, recorded in the storage unit 312: m3 • Inclination of the face center line when the pitch is 10 degrees downward, recorded in the storage unit 312: v3 • Inclination of the string on the mask when the pitch is 20 degrees downward, recorded in the storage unit 312: m4 • Inclination of the face center line when the pitch is 20 degrees downward, recorded in the storage unit 312: v4 • Inclination of the string on the mask in the currently acquired image: mA • Inclination of the face center line in the currently acquired image: vA When the inclination of the string on the mask (an example of "characteristics of left and right ends of the mask") and the inclination of the face center line are defined as described above, the control unit 331 determines the pitch angle as follows.

[0080] (1)[(v3+v0) / 2<vA<(v0+v1) / 2] or / and [(m3+m0) / 2<mA<(m0+m1) / 2] → Pitch angle: 0 degrees

[0081] (2)[(v0+v1) / 2<vA<(v1+v2) / 2] or / and [(m0+m1) / 2<mA<(m1+m2) / 2] → Pitch angle: 10 degrees upward

[0082] (3)[(v1+v2) / 2<vA] or / and [(m1+m2) / 2<mA] → Pitch angle: 20 degrees upward

[0083] (4)[(v3+v4) / 2<vA<(v0+v3) / 2] or / and [(m3+m4) / 2<mA<(m0+m3) / 2] → Pitch angle: 10 degrees downward

[0084] (5)[vA<(v3+v4) / 2] or / and [mA<(m3+m4) / 2] → Pitch angle: 20 degrees downward

[0085] Next, the control unit 311 detects the upper edge of the mask (upper boundary 213 in Fig. 2), associates the pitch angle determined in the processing of step S53 with the inclination of the upper edge of the mask, records the association in the storage unit 312 (step S54), and terminates the pitch angle determination processing.

[0086] On the other hand, if the control unit 311 determines in the processing of step S51 that the inclination of the upper edge of the mask has been recorded for all pitch angles (step S51: YES), the control unit 311 compares the recorded inclination of the upper edge of the mask with the inclination of the upper edge of the mask in the currently acquired image, determines the pitch angle (step S52), and terminates the pitch angle determination processing. Here, a case where the pitch angle is determined by comparing the recorded inclination of the upper edge of the mask with the inclination of the upper edge of the mask in the image will be specifically described.

[0087] · Inclination of the upper edge of the mask at 0 degree pitch (horizontal state) recorded in the storage unit 312: g0 · Inclination of the upper edge of the mask at 10 degrees upward pitch recorded in the storage unit 312: g1 · Inclination of the upper edge of the mask at 20 degrees upward pitch recorded in the storage unit 312: g2 · Inclination of the upper edge of the mask at 10 degrees downward pitch recorded in the storage unit 312: g3 · Inclination of the upper edge of the mask at 20 degrees downward pitch recorded in the storage unit 312: g4 · Inclination of the upper edge of the mask in the currently acquired image: gA When the inclination of the upper edge of the mask is defined as described above, the control unit 331 determines the pitch angle as follows.

[0088] (1)[(g3+g0) / 2 < gA < (g0+g1) / 2] → Pitch angle: 0 degrees

[0089] (2)[(g0+g1) / 2 < gA < (g1+g2) / 2] → Pitch angle: 10 degrees upward

[0090] (3)[(g1+G2) / 2 < gA] → Pitch angle: 20 degrees upward

[0091] (4)[(g3+g4) / 2 < gA < (g0+g3) / 2] → Pitch angle: 10 degrees downward

[0092] (5)[gA < (g3+g4) / 2] → Pitch angle: 20 degrees downward

[0093] Returning to FIG. 9, the control unit 311 determines whether or not to continue the face orientation determination process (step S17). At this time, when the control unit 311 determines that the face orientation determination process is not to be continued (step S17: NO), it ends the face orientation determination process. On the other hand, when the control unit 311 determines that the face orientation determination process is to be continued (step S17: YES), it acquires the next image (step S18) and proceeds to the process of step S15.

[0094] As described above, the control unit 311 (an example of "acquiring means", "detecting means", and "recognizing means") of the recognition apparatus D according to the present embodiment acquires an image captured in a state where a face (an example of "object") covered by a mask (an example of "covering member") is irradiated with light by an irradiation unit L (an example of "irradiating means"), detects the face center line 212 (an example of "the center line in the front vertical direction on the surface of the object") based on the luminance values of pixels included in the acquired image, and recognizes the orientation of the object based on the detection result.

[0095] Therefore, according to the recognition device D of this embodiment, since the orientation of the face is recognized based on the detection result of the face center line 212 based on the brightness value of the pixels included in the image, the orientation of the face can be accurately recognized even when the face is covered by a mask. Furthermore, the orientation of the face can be recognized without processing loads such as pattern matching. In addition, since there is no need to store pattern images for pattern matching, the storage unit 312 can be a small-capacity storage device.

[0096] Furthermore, the control unit 311 of the recognition device D according to this embodiment recognizes the vertical orientation (pitch angle) of the face based on the inclination of the face centerline 212.

[0097] Furthermore, the control unit 311 (an example of "detection means," "storage control means," and "recognition means") of the recognition device D according to this embodiment further detects the upper boundary 213, which is the upper boundary of the mask, when the vertical orientation of the face is recognized, and stores the inclination of the upper boundary 213 in the storage unit 312 (an example of "storage means") in correspondence with the vertical orientation of the recognized object. When a new image of a face whose vertical orientation has not been recognized is acquired, the control unit recognizes the vertical orientation of the face in the image as the vertical orientation of the face in the image, which is associated with an inclination that is approximately the same as the inclination of the upper boundary 213 of the mask detected for that image. As a result, once the vertical orientation of the face has been recognized, for subsequent images, the vertical orientation can be recognized by detecting the upper boundary 213 and comparing it with the upper boundary 213 stored in the storage unit 312. In other words, the processing burden for recognizing the vertical orientation is reduced.

[0098] Furthermore, the control unit 311 (an example of "recognition means") of the recognition device D according to this embodiment recognizes the lateral orientation (yaw angle) of the face based on the position of the intersection point between the upper boundary line 213 and the face center line 212 in the width of the face (i.e., based on the front face width WA and the back face width WB).

[0099] Furthermore, the recognition device D according to this embodiment can be utilized by incorporating it into in-vehicle cameras (for driver safety confirmation by detecting face orientation), surveillance cameras (for detecting face orientation when wearing a mask, detecting people's interests and actions, etc.), medical cameras (for detecting face orientation when wearing a mask, detecting the actions of doctors and patients, etc.).

[0100] [4. Variant] Next, we will describe some modifications of the above embodiment. Note that the modifications described below can be combined as appropriate.

[0101] [4.1. Variation 1] In the above embodiment, as shown in Figures 13(A) and (B), the tilt of the reference upper mask string (an example of "features of the left and right edges of the mask") and the tilt of the face centerline (near the upper edge of the mask) (an example of "features of the face centerline") are recorded in the storage unit 312 in correspondence with the pitch angle, and the pitch angle is determined by comparing the tilt of the upper mask string and the tilt of the face centerline in a newly acquired image. Alternatively, as shown in Figures 14(A) and (B), for example, the features (shape) of the left and right edges of the mask and the features (shape) of the face centerline (the part covering the mask) are recorded in the storage unit 312 in correspondence with the pitch angle, and the features (shape) of the left and right edges of the mask and the features (shape) of the face centerline in a newly acquired image are compared, the pitch angle associated with the one with the highest similarity is obtained, and this is determined to be the pitch angle of the face F in the image.

[0102] [4.2. Variation 2] In the above embodiment, the camera C and the illumination unit L were assumed to be facing the target from approximately the same direction. However, the present invention can also be applied when the camera C and the illumination unit L are facing the target from different directions. That is, even when the camera C and the illumination unit L are located in different directions relative to the target, if the position of the illumination unit L is determined and it is known in advance how light is irradiated onto the face when wearing a mask and how an image is captured, then, similar to the above embodiment, it is possible to determine the mouth area, determine whether a mask is being worn, and estimate the direction of the face orientation by detecting mask features and face features.

[0103] Figure 15 shows an example of an image taken with the camera C and illumination unit L positioned in different directions relative to the subject, and light illuminating the face from the side. When light is illuminating the face from the side, the center line 601 of the face can be detected when the face is facing near the front of the camera. When the face faces in the direction of the light, the contour of the face, the mask features 602 facing the direction of the light, and the frontal area of ​​the face can be detected. These can be detected, and similar determinations can be made at 10-degree intervals of yaw angle. However, when light is illuminating the face from the side, if the face is facing in the opposite direction to the light illumination, the mask features and frontal features of the face due to the light illumination cannot be detected. Therefore, if a wider range of face orientation angles is desired, it is desirable to position the camera C and illumination unit L in the same direction relative to the face, as in the above embodiment. Depending on the application and environment, it may also be possible to determine the face orientation direction by illuminating the face from the side and detecting the mask features 602, etc.

[0104] Thus, even when the camera C and the illumination unit L are positioned in different directions relative to the object, mask features and frontal facial features can be clearly detected. Therefore, even when the illumination unit L is positioned away from the camera C due to the installation conditions of the camera C and the illumination unit L, and light illumination is to be performed from the side of the face, or when the dominant light (e.g., sunlight) is illuminating from the side of the camera C, face orientation determination can be performed using the same process.

[0105] [4.3. Variation 3] In determining the mouth region of a face and the face orientation angle, the results of determinations made using previously captured images may be used for the above determinations. For example, once the mouth region has been estimated, the process of searching for the mouth region around the previously estimated region can be considered in the image taken at the next time point. Also, in determining the face orientation direction, for example, if the face orientation is determined to be 10 degrees up and 20 degrees left, then in the image taken at the next time point, since it is highly likely that the face orientation is around the previously estimated face orientation, the process of prioritizing the detection of features such as face orientation 20 degrees up and 20 degrees left, face orientation 10 degrees up and 10 degrees left, face orientation 10 degrees up and 30 degrees left, and face orientation 20 degrees left can be considered. In other words, assuming that the face orientation does not change significantly in images taken at similar times, when determining the face orientation anew, the determination process can be performed starting from a direction close to the most recently determined face orientation. As shown in this modified example, by processing data in a time series and making the determination process more effective, the processing burden for determining the mouth region and face orientation angle can be reduced.

[0106] [4.4. Modification 4] The inventors of this invention have found that, in many cases, the mask region appears to have a higher brightness value compared to the skin in infrared imaging images. However, depending on the mask material, for example, if a material that absorbs infrared light is used, the mask region may appear darker (lower brightness value) compared to the skin in infrared imaging images. In such cases, the mask may be detected by determining that the brightness value of the mouth region is smaller than that of the skin region. In this case, the process of detecting the mask edge after mask detection can be performed in the same manner as in the above-described embodiment. [Explanation of Symbols]

[0107] 1 recognition device 111A Acquisition method 111B Theoretical means 111C Recognition means 111D Memory control means 111E Storage means D Face orientation recognition device D 311 Control Unit 312 Storage section 313 Communications Department 314 Display section 315 Operation section

Claims

1. An acquisition means for acquiring an image of an object covered by a covering member while it is illuminated with light, A detection means for detecting the center line in the front vertical direction on the surface of the target based on the brightness value of the pixels included in the aforementioned image, A recognition means that recognizes the vertical orientation of the object based on the inclination of the center line detected by the detection means, A recognition device equipped with the following features.

2. A recognition device according to claim 1, The detection means further detects the upper boundary, which is the upper boundary of the covering member on the object, when the recognition means recognizes the vertical orientation of the object. The system further includes a storage control means that causes the detected inclination of the upper boundary to be stored in a storage means in correspondence with the vertical orientation of the object recognized by the recognition means, The recognition means is a recognition device that, when the acquisition means acquires a new image of an object whose vertical orientation has not been recognized, recognizes as the vertical orientation of the object in the image a vertical orientation that is associated with a tilt substantially the same as the tilt of the upper boundary of the covering member detected for the image.

3. A recognition device according to claim 1, The detection means further detects the upper boundary, which is the upper boundary of the covering member, The recognition means is a recognition device that recognizes the lateral orientation of the object based on the position of the intersection point between the upper boundary and the center line within the width of the object.

4. A recognition device according to any one of claims 1 to 3, The detection means further detects the left and right boundaries, which are at least one of the left and right boundaries of the covering member. The recognition means is a recognition device that, when the detection means detects the left and right boundaries, recognizes the lateral orientation of the object based on the left and right boundaries detected by the detection means and the center line.

5. A recognition method using a recognition device, An acquisition step in which an image is taken of an object covered by a covering member while it is illuminated with light, A detection step of detecting the center line in the front vertical direction on the surface of the target based on the brightness value of the pixels included in the image, A recognition step in which the vertical orientation of the object is recognized based on the inclination of the center line detected in the above detection step, A recognition method that includes this.

6. Computers, An acquisition means for acquiring an image of an object covered by a covering member while it is illuminated with light. A detection means for detecting the center line in the front vertical direction on the surface of the target based on the brightness value of the pixels included in the aforementioned image. A recognition means that recognizes the vertical orientation of the object based on the inclination of the center line detected by the detection means. A recognition program that functions as such.

Images