Recognition apparatus, recognition method, and recognition program

Abstract

To provide a recognition apparatus, a recognition method and the like capable of accurately recognizing the orientation of an object covered by a covering member.SOLUTION: A face orientation recognition apparatus 1 comprises: acquisition means for acquiring an image of an object covered with a covering member while the object is irradiated with light by illumination means; detection means for detecting a center line in a front vertical direction on a surface of the object based on a luminance value of a pixel included in the image; and recognition means for recognizing the orientation of the object based on the detected center line.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present application relates to the technical field of a recognition device that recognizes the direction in which an object is facing based on an image of the object. [Background technology]

[0002] Conventionally, there is a technology that recognizes the direction of a person's face from an image of the face. Generally, the direction of the face is recognized using the characteristics of each part of the face, so the recognition accuracy decreases when part of the face is covered by a covering material such as a mask.

[0003] Therefore, in the technology described in Patent Document 1, when a mask is present, the left / right angle of the face is calculated from the relative position of the nose in the center of the mask with respect to the face width in the face region. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 5359266 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology described in Patent Document 1, the center of the mask is set to the position of the nose, but the center of the mask may not necessarily be the actual position of the nose, and it may not be possible to accurately calculate the left / right angle of the face.

[0006] In view of the above circumstances, an object of the present invention is to provide a recognition device or the like that can accurately recognize the orientation of an object covered by a covering member. [Means for solving the problem]

[0007] The invention described in claim 1 comprises an acquisition means for acquiring an image of an object covered with a covering member while being irradiated with light by an illumination means, a detection means for detecting a center line in the front vertical direction on the surface of the object based on the luminance values ​​of pixels contained in the image, and a recognition means for recognizing the orientation of the object based on the detection result of the detection means.

[0008] The invention described in claim 6 is a recognition method using a recognition device, comprising: an acquisition step of acquiring an image of an object covered with a covering member while being irradiated with light by an illumination means; a detection step of detecting a center line in the front vertical direction on the surface of the object based on the luminance values ​​of pixels contained in the image; and a recognition step of recognizing the orientation of the object based on the detection result of the detection step.

[0009] The invention described in claim 7 causes a computer to function as an acquisition means for acquiring an image of an object covered with a covering member while being irradiated with light by an illumination means, a detection means for detecting a center line in the front vertical direction on the surface of the object based on the luminance values ​​of pixels contained in the image, and a recognition means for recognizing the orientation of the object based on the detection result of the detection means. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram of a recognition device 1. [Diagram 2] 1 is an example of a color-coded image. [Diagram 3] 1 is an example of a color-coded image. [Figure 4] FIG. 2 is an example of a block diagram of a face direction recognition device D. [Diagram 5] FIG. 2 is an example of a functional block diagram of a face direction recognition device D. [Figure 6] 13A, 13B, and 13C are diagrams illustrating an example of a method for detecting features of a face or a mask. [Figure 7] 13 is an example of color-coded images taken with the face facing different directions. [Figure 8]13A and 13B are diagrams showing an example of a mask having a special shape. [Figure 9] 13 is a flowchart showing an example of a face direction determination process performed by the face direction recognition device D. [Figure 10] 13 is a flowchart showing an example of a mask wearing determination process performed by a face direction recognition device D. [Figure 11] 13 is a flowchart showing an example of a yaw angle determination process performed by a face direction recognition device D. [Figure 12] 13 is a flowchart showing an example of a pitch angle determination process performed by a face direction recognition device D. [Figure 13] 13A and 13B are diagrams showing an example of features of the left and right edges of a mask and features of the face center line (near the upper edge of the mask). [Figure 14] 13A and 13B are diagrams showing an example of features of the left and right edges of a mask and the features of the center line of a face (part that covers the mask). [Figure 15] FIG. 13 is a diagram showing an example of an image captured when a camera C and an irradiation unit L are disposed in different directions from the target. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] An embodiment of the present invention will be described with reference to FIG.

[0012] 1, the recognition device 1 includes an acquisition unit 111A, a detection unit 111B, a recognition unit 111C, a memory control unit 111D, and a memory unit 111E. The memory unit may be provided outside the recognition device 1.

[0013] The acquisition means 111A acquires an image of an object covered with a covering member, photographed in a state where the object is irradiated with light by an irradiating means. An example of the object covered with a covering member is a face wearing a mask. It is preferable that the irradiating means irradiates the object with light from a certain direction.

[0014] The detection means 111B detects a center line in the vertical direction of the front of the surface of the object based on the luminance value of the pixel included in the image acquired by the acquisition means 111A. When the object is a face (a typical face), the center line in the vertical direction of the front of the surface of the object is a line that passes between the eyes, the center of the nose, and the center of the mouth and passes through the surface of the face.

[0015] Here, the center line will be described with reference to FIG. 2. FIG. 2 shows a color-coded image 200 obtained by color-coding an image (in order to protect the model's privacy, the eyes are painted black. The same applies to images of models) of a face wearing a mask (an example of a "covering material") with light irradiated from the camera direction, based on the luminance value. Specifically, a portion with a luminance value of less than 100 is represented by the color indicated by reference symbol 201, a portion with a luminance value of 100 or more and less than 160 is represented by the color indicated by reference symbol 202, a portion with a luminance value of 160 or more and less than 220 is represented by the color indicated by reference symbol 203, and a portion with a luminance value of 220 or more is represented by the color indicated by reference symbol 204. When light is irradiated, the luminance value of the masked region is larger than that of the skin. As shown in FIG. 2, the masked region has a large luminance value (luminance value of 220 or more). Since a face wearing a mask is symmetrical with respect to the center of the face, and the center of the face tends to protrude slightly from other parts of the face, the detection means 111B can detect the left end 211 of the mask and the face (mask) center line 212 (an example of a "center line") based on an image in which light is irradiated from the front direction of the camera. That is, the detection means 111B detects the edge on the left end side of the mask (the right side in the image) of the area with a brightness value of 220 or more as the left end 211 of the mask, and detects the edge on the face center side (the side opposite the left end side of the mask) of the area with a brightness value of 220 or more as the face center line 212. However, if the horizontal direction (yaw angle) exceeds a certain angle (about 50 degrees) with respect to the front of the camera, the edge on the face center side (the side opposite the left end side of the mask) of the area with a brightness value of 220 or more will not match the face center line 212, so in this case, it is preferable to detect the edge on the face center side of the area with a brightness value of 160 or more and less than 220 as the face center line 212B.

[0016] The recognition means 111C recognizes the orientation of the target based on the detection result of the detection means 111B.

[0017] As described above, according to the operation of the recognition device 1 of the embodiment, the orientation of an object is recognized based on the detection result of the center line in the front vertical direction on the surface of the object which is based on the luminance values ​​of pixels contained in the image, so that the orientation of the object can be accurately recognized even if the object is covered by a covering member.

[0018] Furthermore, the recognition means 111C may recognize the vertical orientation (pitch angle) of the target based on the inclination of the face center line 212. Since the inventor's research has revealed that there is a correlation between the vertical orientation of the face and the inclination of the face center line 212, it is possible to determine the vertical angle of the face by setting a threshold value for the inclination for each vertical orientation (pitch angle) of the face (for example, for each 10 degrees) and comparing the inclination of the face center line 212 with the threshold value. This makes it possible to recognize the vertical orientation of the target.

[0019] Furthermore, the detection means 111B further detects an upper boundary 213, which is the upper boundary of the covering member in the target when the recognition means 111C recognizes the vertical orientation of the target, and the memory control means 111D stores the inclination of the upper boundary 213 detected by the detection means 111B in the memory means 111E in association with the vertical orientation of the target recognized by the recognition means 111B, and when an image of an object whose vertical orientation is not recognized is newly acquired, the recognition means 111C may recognize the vertical orientation associated with approximately the same inclination as the inclination of the upper boundary 213 of the covering member detected in the image as the vertical orientation of the object depicted in the image. Such a determination is possible because it has been found as a result of the inventor's research that when the inclination of the upper boundary 213 of the covering member is approximately the same, the vertical orientation of the target is also approximately the same. As a result, once the vertical orientation of the object is recognized, the vertical orientation of subsequent images can be recognized by detecting the upper boundary 213 and comparing it with the upper boundary 213 stored in the storage means. In other words, the processing load for recognizing the vertical orientation is reduced.

[0020] Furthermore, the recognition means 111C may recognize the lateral orientation (yaw angle) of the target based on the position of the intersection between the upper boundary line 213 and the face center line 212 in the lateral width of the target.

[0021] Here, an example of recognizing the lateral direction of an object will be described with reference to FIG. 3. The recognition unit 111C recognizes the lateral direction of an object based on three perpendicular lines 221-223. The perpendicular line 221 is a perpendicular line passing through the point at the end of the left ear. The perpendicular line 222 is a perpendicular line passing through the intersection of the face center line 212 and the upper boundary 213. The perpendicular line 223 is a perpendicular line moving horizontally from the intersection of the face center line 212 and the upper boundary 213 toward the back side of the face and passing through a point reaching the right end of the face. The recognition unit 111C recognizes the lateral direction of an object based on any two widths of the width WA (front side face width WA) between the perpendicular lines 221 and 222, the width WB (rear side face width WB) between the perpendicular lines 222 and 223, and the width WC between the perpendicular lines 221 and 223. Since the inventor's research has revealed that there is a correlation between the lateral orientation of the face and the ratio of the front face width WA to the rear face width WB, the lateral angle of the face can be determined by setting a threshold value for the ratio of the front face width WA to the rear face width WB for each lateral orientation (angle) of the face (for example, every 10 degrees) and comparing the ratio of the front face width WA to the rear face width WB with the threshold value. This makes it possible to recognize the lateral orientation of the target.

[0022] Furthermore, the detection means 111B further detects a left / right boundary, which is at least one of the boundaries on the left and right sides of the covering member, and the recognition means 111C recognizes the lateral orientation of the object based on the left / right boundary and the center line detected by the detection means 111B when the detection means 111B detects the left / right boundary. The left / right boundary means, for example, at least one of the left boundary indicated by the edge of the left end 211 of the mask (see FIG. 2) and the right boundary indicated by the edge of the right end of the mask. In other words, the left / right boundary can be said to be a general term for the left boundary and the right boundary. The recognition means 111C recognizes that the object is facing right based on the detection of the edge of the left end 211 of the mask (left boundary), and recognizes that the object is facing left based on the detection of the edge of the right end of the mask (right boundary). EXAMPLES

[0023] Next, a specific example corresponding to the above-described embodiment will be described.

[0024] The following embodiment will be described with reference to Figures 4 to 14. Note that the following embodiment is an embodiment in which the present invention is applied to a face direction recognition device D (hereinafter, sometimes referred to as "recognition device D").

[0025] The recognition device D of this embodiment is connected to an illumination unit L, which is a light source such as an LED, and a camera C. The recognition device D recognizes the orientation of the face F based on an image captured in a state in which the face F covered by the mask M is illuminated by the illumination unit L.

[0026] Specifically, the recognition device D detects the face (or facial features) by performing signal processing on the captured image and determines whether the person is wearing a mask. If it determines that the person is wearing a mask, it detects the mask features (mask shadow area, mask edge, mask edge gradient, mask string edge gradient, etc.) and facial features (face centerline, face width, face outline, etc.) and recognizes the face orientation from these extracted features.

[0027] More specifically, first, it is determined whether a face is present in the image, and a position (mouth area) for determining whether a mask is worn is determined. For example, an approximate position of the mouth is determined in a face frame detected by a conventionally known face detection process. The features of each part of the face may be detected to determine the position for determining whether a mask is worn. For example, the position of the eyes may be detected, and the area below the eyes may be determined as the approximate position of the mouth, and the mask wearing determination may be performed. Next, it is determined whether a mask is worn. Generally, when a face F wearing a mask M is irradiated with light and an image is taken, a high luminance value is obtained in the mask wearing area, and therefore, it is determined whether a mask is worn based on the high luminance value of the area around the mouth. It is also possible to determine whether a mask is worn based on the mask shape information or the like. When it is determined that a mask is worn, the features of the mask and the facial features are detected, and the orientation of the face is recognized based on these extracted features. In addition, based on the feature detection results when the features of the face and the mask are detected from the image, the intensity of the irradiated light is adjusted, or the camera parameters are adjusted, so that the shadows and luminance gradients of the mask and the face are more prominently detected. In this embodiment, a case will be described in which the camera C and the irradiating unit L face the target from approximately the same direction. A case in which the camera C and the irradiating unit L face the target from different directions will be described in a modified example described later.

[0028] [1. Configuration of recognition device D] Next, the configuration of the recognition device D according to this embodiment will be described with reference to Fig. 4. As shown in Fig. 4, the recognition device D is mainly composed of a control unit 311, a storage unit 312, a communication unit 313, a display unit 314, and an operation unit 315. The communication unit 313 is connected to the camera C and the irradiation unit L.

[0029] The storage unit 312 is configured, for example, with a hard disk drive, and stores various programs including an OS (Operating System) and a face direction recognition program for recognizing the direction of a face. The storage unit 312 also stores images captured by a camera and various data used in the face direction recognition program.

[0030] The communication unit 313 controls the communication state with the irradiation unit L and the camera C.

[0031] The display unit 314 is configured, for example, with a liquid crystal display or the like, and is configured to display images captured by the camera C and the like.

[0032] The operation unit 315 is constituted by, for example, a keyboard, a mouse, etc., and is adapted to receive operation instructions from an operator and output the contents of the instructions to the control unit 311 as instruction signals.

[0033] The control unit 311 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), The memory unit 312 is configured with a ROM and a random access memory (RAM), etc. The CPU reads out and executes various programs including a face direction recognition program stored in the ROM or the memory unit 312 to realize various functions.

[0034] [2. Functions of Recognition Device D] Next, the functions of the recognition device D (control unit 311) according to this embodiment will be described with reference to Fig. 5. Fig. 5 is a diagram showing a functional block diagram of the recognition device D. The recognition device D is configured to include an image analysis unit 351, an irradiation light adjustment unit 357, and a camera adjustment unit 358. The functions of the image analysis unit 351, the irradiation light adjustment unit 357, and the camera adjustment unit 358 are realized by the control unit 311 executing a face direction recognition program.

[0035] The image analysis unit 351 includes a mouth area detection unit 352 , a mask wearing determination unit 353 , a mask feature detection unit 354 , a face feature detection unit 355 , and a face direction recognition unit 356 .

[0036] The mouth area detection unit 352 performs face detection processing, eye detection processing, nose detection processing, and mouth detection processing on an image captured by the camera C with light being irradiated by the illumination unit L, for example, of a face F covered by a mask M. If the face area can be determined by the face detection processing, it can be determined that the mouth area is located in the area below the face detection frame. If the eye area can be determined by the eye detection processing, it can be determined that the mouth area is located below the eyes. If the nose area can be determined by the nose detection processing, it can also be determined that the mouth area is located below the nose. If the mouth area is detected by the mouth detection processing, it is determined that that area is the mouth area.

[0037] The mask wearing determination unit 353 determines whether or not a mask M is worn for the mouth area detected by the mouth area detection unit 352. Specifically, the determination is made based on the luminance value. For example, as shown in FIG. 6(A), the average luminance value of the right eye area 401 and the average luminance value of the mouth area 402 are calculated. In general, when light is irradiated toward the face, the luminance value of the mask is greater than the luminance value of the skin. For example, by comparing the average luminance value of the right eye area 401 and the average luminance value of the mouth area 402, if the values ​​are substantially the same, it can be determined that a mask is not worn, and if the average luminance value of the mouth area 402 is greater (greater than a threshold value), it can be determined that a mask is worn.

[0038] The mask feature detection unit 354 detects the features of the mask region based on the luminance value of the mask region. The face feature detection unit 355 detects the features of the face region based on the luminance value of the face region. Specifically, the mask feature detection unit 354 and the face feature detection unit 355 (collectively referred to as "feature detection unit") detect the features using a color-coded image in which the luminance value of an image of a face is divided into four regions using three threshold values, as shown in Fig. 2. Specifically, the region is divided into (1) mask region, (2) skin (light) region, (3) skin (dark) region, and (4) background region.

[0039] Here, an example of creating a color-coded image will be described. First, the feature detection unit determines a first threshold value indicating the boundary between (1) the mask region and (2) the skin (light) region based on the maximum value Ja (considering that pixel values ​​may include spike noise, etc., the fifth largest value is treated as the maximum value here) of the skin region (for example, the left eye region 401) and the average luminance value Jb of the mouth region 402 when the mask is worn. For example, if Ja is "219" and Jb is "246", a value obtained by adding a margin to Ja is set as the first threshold value. For example, "220" obtained by adding "1" to Ja is set as the first threshold value ("1" is added to make it a round number, but other values ​​may be used. The same applies to the second and third threshold values ​​below). A region composed of pixels with a luminance value greater than the first threshold value can be detected as a mask region.

[0040] Next, the feature detection unit determines a second threshold value indicating the boundary between the (2) skin (light) region and the (3) skin (dark) region based on the average value Jc of the skin region (for example, the left eye lower half region 403 shown in FIG. 6(B)) (the eyebrows and pupils have low luminance values, so the left eye lower half region 403 is used, but the average value of other parts may also be used). For example, if Jc is "157", the value obtained by subtracting a margin from Jc is set as the second threshold value. For example, the second threshold value is set to "150", which is obtained by subtracting "7" from Jc.

[0041] Next, the feature detection unit determines a third threshold value indicating the boundary between (3) the skin (dark) region and (4) the background region based on the minimum value Jd of the skin region (e.g., the left eye left half region 404 shown in FIG. 6(C)) (considering that pixel values ​​may include spike noise, etc., the fifth smallest value is treated as the minimum value here. Here, the left eye left half region 404 is used, but the minimum value of another skin part may also be used). For example, if Jd is "78", the value obtained by subtracting a margin from Jd is set as the third threshold value. For example, "70", obtained by subtracting "8" from Jd, is set as the third threshold value.

[0042] The feature detection unit divides each pixel constituting the image into (1) a mask region, (2) a skin (light) region, (3) a skin (dark) region, and (4) a background region based on the luminance value of the pixel and a first, second, and third thresholds, and creates a color-coded image by expressing the regions in four colors. Then, based on the color-coded image, the features of the mask region and the face region are detected.

[0043] FIG. 7 is an example of an image captured by changing the vertical direction (pitch angle) and the horizontal direction (yaw angle) of the face by 10 degrees each. As in the example of FIG. 2, the image is divided into a portion with a luminance value of less than 100, a portion with a luminance value of 100 or more and less than 160, a portion with a luminance value of 160 or more and less than 220, and a portion with a luminance value of 220 or more, and each portion is represented by a different color (as in the case of FIG. 2). When light is irradiated, the luminance value of the masked region becomes larger than that of the skin. The masked region has a large luminance value (luminance value of 220 or more). When the face is facing the left or right direction (yaw angle: 20 degrees to 50 degrees), the face direction recognition unit 356 can clearly detect the left and right boundaries between the left and right ends of the mask and the skin, and can determine whether the face is facing left or right. On the other hand, when the face is facing the front of the camera (yaw angle: 0 degrees), the boundaries between the left and right ends of the mask and the skin (left and right boundaries) cannot be detected, and this makes it possible to determine that the face is facing forward. In order to estimate the horizontal face direction angle, if the center of the face and the edges of the face (contour, ears, etc.) as well as the left and right edges of the mask can be detected, the approximate face direction angle can be calculated. Below, a detailed description is given of a method in which the face direction recognition unit 356 recognizes the vertical direction of the face (pitch angle) and the horizontal direction of the face (yaw angle).

[0044] (Pitch angle: 0 degrees) When the face is facing the camera (yaw angle: 0 degrees, pitch angle: 0 degrees), the luminance value of the mouth area is large (compared to the skin area), but no significant mask edges are detected at the left and right ends of the mask. When the face is facing 10 degrees to the left (yaw angle: 10 degrees, pitch angle: 0 degrees), the luminance value of the mouth area is large, and by tilting the face to the left relative to the camera, a significant mask edge can be detected at the left end of the mask. This is because light is irradiated from the front of the camera. When the face is facing 20 degrees to the left (yaw angle: 20 degrees), the face is tilted more than when facing 10 degrees, so the luminance value of the area of ​​the mask facing the camera becomes significant. The left end of the mask can also be detected clearly. In addition, a face wearing a mask is symmetrical with respect to the center of the face, and the center of the face tends to protrude slightly compared to other parts of the face, so when light is irradiated from the front of the camera, in addition to the left end of the mask, the center line of the face can be clearly detected. Similarly, when the face direction is 30 degrees, 40 degrees, or 50 degrees to the left, the left edge of the mask and the center line of the face can be clearly detected. Furthermore, when there is no object near the face, the brightness value of the background does not increase even when light is irradiated, so it is possible to detect the face width WA in front of the camera and the face width WB behind the camera. When the face direction is tilted 10 degrees to the left, it is possible to determine how many degrees to the left the face direction is by detecting the above-mentioned edge of the left edge of the mask, the center line of the face, the face width WA in front of the camera, and the face width WB behind the camera. When the face direction is to the right, the left and right are simply symmetrical to when the face direction is to the left, so a detailed explanation will be omitted.

[0045] (Pitch angle: 10 degrees up) When the face is turned 10 degrees up (yaw angle: 0 degrees, pitch angle: up 10 degrees), the luminance value of the mouth area is larger than that of the skin area, but no significant mask edges are detected at the left and right ends of the mask. The upper edge of the mask has a steeper gradient than when the camera is facing forward (yaw angle: 0 degrees, pitch angle: 0 degrees), but the gradient of the upper edge of the mask changes due to deformation each time the mask is worn, so this alone does not allow us to determine that the face is facing upwards. However, if the reference edge gradient of the upper edge of the mask when worn is known, it is possible to determine the angle of the face in the up and down direction. When the face is turned 10 degrees left (yaw angle: 10 degrees), the luminance value of the mouth area is larger, just like when the pitch angle is 0 degrees, and a significant edge can be detected at the left end of the mask due to the face being tilted leftward relative to the camera. Furthermore, when the face is tilted to the left, just as when the pitch angle is 0 degrees, the left edge of the mask, the center line of the face, the face width WA in front of the camera, the face width WB behind the camera, etc. can be detected to determine how many degrees the face is tilted to the left.

[0046] The face's upward angle (pitch angle) can be determined by the inclination of the face center line when the face is oriented 30 degrees to the left, 40 degrees to the left, 50 degrees to the left, etc. When the face's upward angle (pitch angle) is 0 degrees and 10 degrees, and the inclination of the face center line when the face is oriented 30 degrees to the left, 40 degrees to the left, and 50 degrees to the left, it can be determined that the face is clearly facing upward when the face's upward angle (pitch angle) is 10 degrees, rather than when the face's upward angle (pitch angle) is 0 degrees. If the gradient of the mask's upper edge when the face's upward angle (pitch angle) is 10 degrees is recorded as a reference, it can be determined that the face's upward angle (pitch angle) is 10 degrees if the gradient of the mask's upper edge in a later captured image is the same angle (preferably with a certain width and approximately the same angle) as long as the mask is not deformed. When the face is oriented 10 degrees up to the right, it is simply symmetrical left and right to the face when the face is oriented 10 degrees up to the left, so a detailed description will be omitted.

[0047] (Pitch angle: 20 degrees up) When the face is turned up 20 degrees (yaw angle: 0 degrees, pitch angle: up 20 degrees), the luminance value of the mouth area is larger than that of the skin area, but no significant mask edges are detected at the left and right ends of the mask. The upper edge of the mask has a steeper gradient than when facing the camera (yaw angle: 0 degrees, pitch angle: 0 degrees), but the gradient of the upper edge of the mask changes due to deformation each time the mask is worn, so it is not possible to determine that the mask is facing upwards based on this alone. However, if the reference edge gradient of the upper edge of the mask when worn is known, the angle of the face in the up and down direction can also be determined. When the face is turned 10 degrees left (yaw angle: 10 degrees, pitch angle: up 20 degrees), the luminance value of the mouth area is larger, just like when the pitch angle is 0 degrees, and a significant edge can be detected at the left end of the mask due to the face being tilted leftward relative to the camera. Furthermore, when the face is tilted to the left, just as when the pitch angle is 0 degrees, the left edge of the mask, the center line of the face, the face width WA in front of the camera, the face width WB behind the camera, etc. are detected to determine how many degrees the face is tilted to the left.

[0048] The face's upward angle (pitch angle) can be determined by the inclination of the face center line when the face direction is 30 degrees left, 40 degrees left, and 50 degrees left, in the same way as when the face direction is 10 degrees up. Comparing the face direction 10 degrees up with the face direction 20 degrees up, the face center line inclination when the face direction is 30 degrees left, 40 degrees left, and 50 degrees left is more inclined when the face direction is 20 degrees up, so it can be determined to be 20 degrees up. Also, if the gradient of the mask's upper edge when this face's upward angle (pitch angle) is recorded as a reference, it can be determined that the face's upward angle (pitch angle) is 20 degrees if it is compared with the gradient of the mask's upper edge in an image captured later and has the same angle (preferably with a certain width and approximately the same angle) as long as the mask is not deformed. When the face direction is 20 degrees up to the right, it is simply symmetrical left and right to the face direction when the face direction is 20 degrees up to the left, so a detailed description will be omitted.

[0049] (Pitch angle: 10 degrees down) When the face is turned 10 degrees downward (yaw angle: 0 degrees, pitch angle: 10 degrees downward), the luminance value of the mouth area is larger than that of the skin area, but no significant mask edges are detected at the left and right ends of the mask. The upper edge of the mask has a gentler gradient than when facing the camera (yaw angle: 0 degrees, pitch angle: 0 degrees), but the gradient of the upper edge of the mask changes due to deformation each time the mask is worn, so it is not possible to determine that the mask is facing downward based on this alone. However, if the reference edge gradient of the upper edge of the mask when worn is known, it is possible to determine the face direction in the vertical direction (same as when the face is turned 10 degrees upward). When the face is turned 10 degrees left (yaw angle: 10 degrees, pitch angle: 10 degrees downward), the luminance value of the mouth area is larger, just like when the pitch angle is 0 degrees, and a significant edge can be detected at the left end of the mask due to the face being tilted leftward relative to the camera. When the face is tilted to the left, just as when the pitch angle is 0 degrees, it is possible to determine how many degrees the face is tilted to the left by detecting the left edge of the mask, the center line of the face, the face width WA in front of the camera, and the face width WB behind the camera, etc.

[0050] The downward angle of the face can be determined by the inclination of the face center line when the face is oriented 30 degrees to the left, 40 degrees to the left, and 50 degrees to the left, in the same way as when the face is oriented 10 degrees up. Comparing the face oriented at a pitch angle of 0 degrees with the face oriented at a pitch angle of 10 degrees down, it can be determined that the face is more inclined downward at a pitch angle of 10 degrees down, and it is possible to determine the face oriented at 10 degrees down. If the gradient of the upper edge of the mask when this downward angle (pitch angle) of the face is 10 degrees is recorded as a reference, it can be determined that the downward angle (pitch angle) of the face is 10 degrees if the gradient of the upper edge of the mask in an image captured later is the same angle (preferably with a certain width and approximately the same angle) as long as the mask is not deformed. When the face is oriented 10 degrees down to the right, the left and right are simply symmetrical to each other when the face is oriented 10 degrees down to the left, and therefore a detailed description is omitted.

[0051] (Pitch angle: 20 degrees down) When the face is turned down 20 degrees (yaw angle: 0 degrees, pitch angle: 20 degrees down), the luminance value of the mouth area is larger than that of the skin area, but no significant mask edges are detected at the left and right ends of the mask. The upper edge of the mask has a gentler gradient than when facing the camera (yaw angle: 0 degrees, pitch angle: 0 degrees), but the gradient of the upper edge of the mask changes due to deformation each time the mask is worn, so it is not possible to determine that the mask is facing down based on this alone. However, if the reference edge gradient of the upper edge of the mask when worn is known, the vertical angle of the face can also be determined (same as when the face is turned up 10 degrees). When the face is turned left 10 degrees (yaw angle: 10 degrees, pitch angle: 20 degrees down), the luminance value of the mouth area is larger, just like when the pitch angle is 0 degrees, and a significant edge can be detected at the left end of the mask due to the face being tilted leftward relative to the camera. Furthermore, when the face is tilted to the left, just as when the pitch angle is 0 degrees, it is possible to detect the left edge of the mask, the center line of the face, the face width WA in front of the camera, and the face width WB behind the camera, and determine how many degrees the face is tilted to the left.

[0052] The face downward angle (pitch angle) can be determined by the inclination of the face center line when the face direction is 30 degrees left, 40 degrees left, and 50 degrees left, similarly to when the face direction is 10 degrees down. Comparing when the face direction is 10 degrees down and when the face direction is 20 degrees down, the inclination of the face center line when the face direction is 30 degrees left, 40 degrees left, and 50 degrees left can be determined as the face being inclined more downward, and can be determined as 20 degrees down. If the gradient of the upper edge of the mask when this face downward angle (pitch angle) is 20 degrees is recorded as a reference, and the gradient of the upper edge of the mask in an image captured later is the same angle (preferably with a certain width and approximately the same angle) as long as the mask is not deformed, the face downward angle (pitch angle) can be determined to be 20 degrees. When the face direction is 20 degrees down to the right, it is simply symmetrical to when the face direction is 20 degrees down to the left, so a detailed description will be omitted.

[0053] When the face is turned 50 degrees to the left, the brightness value of the skin near the left edge of the mask may be high, but the strings of the mask that hang from the ears also have a high brightness value when illuminated with light, making it possible to detect a prominent edge, and it is therefore possible to detect the left edge of the mask based on this information. The up-down orientation of the face can also be determined from this string edge information.

[0054] In addition, since the facial widths WA and WB may differ depending on the upward or downward direction of the face (depending on how the perpendicular line 222 in FIG. 2 is taken), it is also possible to determine whether the face is facing upward or downward and to perform a direction determination based on the facial width based on the criteria for each upward or downward direction.

[0055] Furthermore, when light is irradiated from the same direction as camera C, if the face is facing 20 degrees downward, and the forehead is not covered by hair, the luminance of the forehead area will be large. It may be determined that the face is facing 20 degrees downward when the luminance of the forehead area becomes greater than a predetermined threshold value.

[0056] Furthermore, once the mask upper edge gradient, which is the reference for the vertical direction of the face direction, is determined, the pitch angle in the vertical direction can be determined based on the value. For example, when the face direction is 20 degrees downward, after the mask upper edge gradient is determined once, when the face direction becomes 20 degrees upward, the edge gradient when the face direction is 20 degrees upward can be calculated based on the edge gradient when the mask is 20 degrees downward as the reference, and if the calculated value is close to the edge gradient detected from the image, it can be determined that the face direction is 20 degrees upward. The reference edge gradient can be calculated if there is no deformation or removal of the mask when it is worn. Although it changes if the distance between the face and the camera changes, if the distance is approximately constant, once the reference is determined, the face direction can be determined based on the reference. Even if the distance between the face and the camera changes, the face direction can be determined by changing the reference edge gradient based on the depth information and size information of the face.

[0057] In addition, since there are various types of masks, it is possible to determine which type of mask is being worn and estimate the face direction based on the shape information specific to the mask being worn. According to the inventor's research, there are various types of masks currently on the market, and it has been found that, although there are characteristics for each mask with regard to the edge of the left end (right end) of the mask described above, if light is irradiated when the face is turned left or right, the mask-specific shape of the left and right ends of the mask can be detected for almost all masks. In addition, the center line of the face (mask) when the face is turned left or right also differs in shape depending on the type of mask, but almost all masks have a shape in which the center of the mask protrudes (in order to fit the shape of the face, which protrudes toward the center of the face), so if light is irradiated from the camera side, the mask center, which is the center of the face, can be detected. In addition, the above-mentioned edge gradient of the upper side of the mask varies depending on the type of mask, but once a criterion is determined by the above-mentioned processing, the angle of the face in the up-down direction can be determined based on that criterion. For example, in the case of the mask shown in Fig. 8(A) as an example of a unique shape, the shape of the left and right edges of the mask is not a vertical edge line, but the shape of the left and right edges of the mask can be detected. Also, there is a hemispherical cup-shaped mask as shown in Fig. 8(B), but the mask center and the mask top edge gradient can be detected. Furthermore, even with a black mask, when infrared photography is performed, it is possible to capture an image as white, although there is a slight difference in brightness value, so it is possible to similarly detect the mask and detect the features of the left and right edges and top edge of the mask.

[0058] In addition, although the mask may be deformed, the above-mentioned characteristics of the edges at the left and right ends of the mask are hardly changed. As for the center line of the mask when the face is turned left or right, the mask undergoes a flexible shape change, and there is a possibility that the center line of the mask cannot be detected due to the deformation. In such a case, when the face is turned 30 degrees to the left and 50 degrees to the left, the center line of the face can be detected in the upper half of the face, not in the mask area, to determine the up-down direction of the face, or the inclination of the strings of the mask hanging on the ears can be detected to determine the up-down direction of the face. The gradient of the upper edge of the mask may also change due to mask deformation, but once a reference gradient is determined, the deformation of the gradient of the upper edge of the mask can be handled by periodically calculating the reference each time, taking into account the mask deformation.

[0059] In recognizing the face direction, the face direction recognition unit 356 may extract edge gradients and the like based on luminance values, but depending on the shooting conditions, changing the camera parameters (exposure time, etc.) or the intensity of the irradiated light (either one or both) may be more effective in extracting features such as edge gradients. In such cases, the irradiated light adjustment unit 357 adjusts the intensity of the light irradiated from the irradiating unit L, and the camera adjustment unit 358 adjusts the parameters of the camera C, based on the luminance values ​​detected by the mouth area detection unit 352, the mask wearing determination unit 353, the mask feature detection unit 354, and the face feature detection unit 355.

[0060] [3. Example of operation of recognition device D] Next, an example of the face direction determination process of the recognition device D will be described with reference to the flowcharts of FIGS.

[0061] First, the control unit 311 of the recognition device D acquires an image of a face F covered by a mask M captured by the camera C in a state where the illumination unit L is irradiating the face with light (step S11).

[0062] Next, control unit 311 detects the mouth area of ​​the face appearing in the image acquired in the process of step S11 (step S12).

[0063] Next, the control unit 311 performs a mask wearing determination process for the mouth area detected in the process of step S12 (step S13).

[0064] Here, the mask wearing determination process will be described with reference to FIG.

[0065] First, the control unit 311 calculates the average luminance value Ia of the skin region (for example, the right eye region 401 in FIG. 6) (step S21).

[0066] Next, the control unit 311 calculates an average luminance value Ib of a mouth region (for example, the mouth region 402 in FIG. 6) (step S22).

[0067] Next, the control unit 311 determines whether or not "Ib-Ia>Ic" is true (step S23). Here, Ic is a threshold value (for example, Ic=40). Ic can be set according to the observed brightness value, etc. If the control unit 311 determines that "Ib-Ia>Ic" is true (step S23: YES), it determines that a mask is being worn (step S24) and ends the mask-wearing determination process. On the other hand, if the control unit 311 determines that "Ib-Ia>Ic" is not true (step S23: NO), it determines that a mask is not being worn (step S25) and ends the mask-wearing determination process.

[0068] Returning to Fig. 9, the control unit 311 determines whether or not a mask is worn in the mask wearing determination process (step S14). If the control unit 311 determines that a mask is not worn (not worn) (step S14: NO), the control unit 311 ends the face direction determination process. On the other hand, if the control unit 311 determines that a mask is worn (step S14: YES), the control unit 311 then performs a yaw angle determination process (step S15).

[0069] Here, the yaw angle determination process will be described with reference to FIG.

[0070] First, the control unit 311 determines whether it can detect the left and right ends of the mask from the image acquired in the process of step S11 or step S18 described later (step S31). Note that for a mask with a special shape at the left and right ends as shown in FIG. 8(A), by first determining and recording the mask portion with a large luminance value with respect to the luminance value of the skin, the same processing can be performed in subsequent processing.

[0071] When the control unit 311 determines that it cannot detect the left and right ends of the mask (step S31: NO), it determines that the yaw angle is "0 degrees" (step S32), and ends the yaw angle determination process. On the other hand, when it determines that it can detect the left and right ends of the mask (step S31: YES), next, it determines whether it can detect the face center line (see the face center line 212 in FIG. 2) (step S33).

[0072] When the control unit 311 determines that it cannot detect the face center line (step S33: NO), it determines that the yaw angle is "10 degrees" (step S34), and ends the yaw angle determination process. On the other hand, when it determines that it can detect the face center line (step S33: YES), next, it calculates the front face width WA and the rear face width WB (see FIG. 3) (step S35).

[0073] Next, the control unit 311 determines the yaw angle based on the front face width WA and the rear face width WB calculated in the process of step S35 (step S36). In this embodiment, the yaw angle is determined by comparing WB / WA with predetermined threshold values Y1, Y2, Y3 (Y1>Y2>Y3). The threshold values Y1, Y2, Y3 may be changed based on values such as the distance between the camera C and the face F observed.

[0074] Specifically, the control unit 311 determines the yaw angle as follows. (1) Y1 < WB / WA → Yaw angle: 20 degrees (2) Y2 < WB / WA < Y1 → Yaw angle: 30 degrees (3) Y3 < WB / WA < Y2 → Yaw angle: 40 degrees (4) WB / WA < Y3 → Yaw angle: 50 degrees

[0075] Next, the control unit 311 performs face centerline correction processing (step S37) and ends the yaw angle determination processing. When the yaw angle becomes large with respect to the irradiation unit L, the luminance value of the mask center decreases, and the mask centerline detected by the above-mentioned luminance value condition may not match the face centerline (for example, when the yaw angle is 50 degrees in FIG. 7. If the yaw angle is 40 degrees or less, the mask centerline and the face centerline tend to match). In this embodiment, when the yaw angle is less than 50 degrees, the line corresponding to the edge formed by the pixels colored with a luminance value of 220 or more is set as the face centerline. On the other hand, when the yaw angle is 50 degrees or more, the line corresponding to the edge formed by the pixels colored with a luminance value of 160 or more and less than 220 is set as the face centerline.

[0076] Returning to FIG. 9, the control unit 311 performs a pitch angle determination process (step S16).

[0077] Here, the pitch angle determination process will be described with reference to FIG.

[0078] First, the control unit 311 judges whether the inclination of the upper end of the mask has been recorded for all pitch angles (20 degrees down, 10 degrees down, 0 degree, 10 degrees up, 20 degrees up) (step S51). At this time, if the control unit 311 judges that the inclination of the upper end of the mask has not been recorded for all pitch angles (step S51: NO), it judges the pitch angle based on the features of the left and right ends of the mask and the features of the face center line (near the upper end of the mask) (step S53). Specifically, as shown in Figs. 13(A) and 13(B), the features of the left and right ends of the mask and the features of the face center line differ depending on the pitch angle, so that the features of the left and right ends of the mask and the face center line as references for each pitch angle are recorded in advance in the storage unit 312, and the features of the left and right ends of the mask and the face center line in the currently acquired image are compared, and the pitch angle associated with the most similar features of the left and right ends of the mask and the face center line is judged to be the pitch angle of the face in the image. In this embodiment, the inclination of the upper string of the mask is recorded as a feature of the left and right ends of the mask, and the inclination near the upper end of the mask is recorded as a feature of the face centerline.

[0079] The following will be specifically described. · The inclination of the upper mask string at a pitch of 0 degrees (horizontal): m0 recorded in the memory unit 312 · The inclination of the face center line at a pitch of 0 degrees (horizontal): v0 recorded in the memory unit 312 · The inclination of the upper mask string at a pitch of +10 degrees: m1 recorded in the memory unit 312 · The inclination of the face center line at a pitch of +10 degrees: v1 recorded in the memory unit 312 · The inclination of the upper mask string at a pitch of +20 degrees: m2 recorded in the memory unit 312 · The inclination of the face center line at a pitch of +20 degrees: v2 recorded in the memory unit 312 · The inclination of the upper mask string at a pitch of -10 degrees: m3 recorded in the memory unit 312 · The inclination of the face center line at a pitch of -10 degrees: v3 recorded in the memory unit 312 · The inclination of the upper mask string at a pitch of -20 degrees: m4 recorded in the memory unit 312 · The inclination of the face center line at a pitch of -20 degrees: v4 recorded in the memory unit 312 · The inclination of the upper mask string in the image acquired this time: mA · The inclination of the face center line in the image acquired this time: vA When the inclination of the upper mask string (an example of "features at the left and right ends of the mask") and the inclination of the face center line are defined as described above, the control unit 331 determines the pitch angle as follows.

[0080] (1) [(v3 + v0) / 2 < vA < (v0 + v1) / 2] or / and [(m3 + m0) / 2 < mA < (m0 + m1) / 2] → Pitch angle: 0 degrees

[0081] (2) [(v0 + v1) / 2 < vA < (v1 + v2) / 2] or / and [(m0 + m1) / 2 < mA < (m1 + m2) / 2] → Pitch angle: +10 degrees

[0082] (3) [(v1 + v2) / 2 < vA] or / and [(m1 + m2) / 2 < mA] → Pitch angle: 20 degrees upward

[0083] (4)[(v3 + v4) / 2 < vA < (v0 + v3) / 2] or / and [(m3 + m4) / 2 < mA < (m0 + m3) / 2] → Pitch angle: 10 degrees downward

[0084] (5)[vA < (v3 + v4) / 2] or / and [mA < (m3 + m4) / 2] → Pitch angle: 20 degrees downward

[0085] Next, the control unit 311 detects the upper end of the mask (the upper boundary 213 in FIG. 2), associates the pitch angle determined in the process of step S53 with the inclination of the upper end of the mask, and records it in the storage unit 312 (step S54), and ends the pitch angle determination process.

[0086] On the other hand, in the process of step S51, when the control unit 311 determines that the inclination of the upper end of the mask has been recorded for all pitch angles (step S51: YES), it compares the recorded inclination of the upper end of the mask with the inclination of the upper end of the mask in the image acquired this time, determines the pitch angle (step S52), and ends the pitch angle determination process. Here, the case of comparing the recorded inclination of the upper end of the mask with the inclination of the upper end of the mask in the image and determining the pitch angle will be specifically described.

[0087] · Inclination of the upper end of the mask at pitch 0 degrees (horizontal): g0 recorded in the storage unit 312 · Inclination of the upper end of the mask at pitch 10 degrees upward: g1 recorded in the storage unit 312 · Inclination of the upper end of the mask at pitch 20 degrees upward: g2 recorded in the storage unit 312 · Inclination of the upper end of the mask at pitch 10 degrees downward: g3 recorded in the storage unit 312 · Inclination of the upper end of the mask at pitch 20 degrees downward: g4 recorded in the storage unit 312 · Inclination of the upper end of the mask in the image acquired this time: gA When the inclination of the upper end of the mask is defined as described above, the control unit 331 determines the pitch angle as follows.

[0088] (1) [(g3 + g0) / 2 < gA < (g0 + g1) / 2] → Pitch angle: 0 degrees

[0089] (2) [(g0 + g1) / 2 < gA < (g1 + g2) / 2] → Pitch angle: 10 degrees upward

[0090] (3) [(g1 + G2) / 2 < gA] → Pitch angle: 20 degrees upward

[0091] (4) [(g3 + g4) / 2 < gA < (g0 + g3) / 2] → Pitch angle: 10 degrees downward

[0092] (5) [gA < (g3 + g4) / 2] → Pitch angle: 20 degrees downward

[0093] Returning to FIG. 9, the control unit 311 determines whether to continue the face orientation determination process (step S17). At this time, when the control unit 311 determines not to continue the face orientation determination process (step S17: NO), it ends the face orientation determination process. On the other hand, when the control unit 311 determines to continue the face orientation determination process (step S17: YES), it acquires the next image (step S18) and proceeds to the process of step S15.

[0094] As described above, the control unit 311 (an example of the "acquisition means", "detection means", and "recognition means") of the recognition device D according to the present embodiment acquires an image captured in a state where a face (an example of the "target") covered with a mask (an example of the "coating member") is irradiated with light by the irradiation unit L (an example of the "irradiation means"), and based on the luminance values of the pixels included in the acquired image, detects the face center line 212 (an example of the "center line in the vertical front direction on the surface of the target"), and recognizes the orientation of the target based on the detection result.

[0095] Therefore, according to the recognition device D of this embodiment, the face direction is recognized based on the detection result of the face center line 212 based on the luminance value of the pixel included in the image, so that the face direction can be accurately recognized even if the face is covered by a mask. Also, the face direction can be recognized without performing a process with a high processing load such as pattern matching. Furthermore, since there is no need to store a pattern image for pattern matching, the storage unit 312 can be a small-capacity storage device.

[0096] Moreover, the control unit 311 of the recognition device D according to this embodiment recognizes the vertical direction of the face (pitch angle) based on the inclination of the face center line 212.

[0097] Furthermore, the control unit 311 (an example of a "detection means", "storage control means", and "recognition means") of the recognition device D according to this embodiment further detects the upper boundary 213, which is the upper boundary of the mask when the vertical orientation of the face is recognized, and stores the inclination of the upper boundary 213 in the memory unit 312 (an example of a "storage means") in association with the vertical orientation of the recognized target. When a face image whose vertical orientation is not recognized is newly acquired, the vertical orientation associated with the inclination substantially the same as that of the upper boundary 213 of the mask detected for the image is recognized as the vertical orientation of the face appearing in the image. As a result, once the vertical orientation of the face is recognized, the vertical orientation of subsequent images can be recognized by detecting the upper boundary 213 and comparing it with the upper boundary 213 stored in the memory unit 312. That is, the processing load for recognizing the vertical orientation is reduced.

[0098] Furthermore, the control unit 311 (an example of a "recognition means") of the recognition device D of this embodiment recognizes the lateral orientation of the face (yaw angle) based on the position of the intersection between the upper boundary line 213 and the facial center line 212 in the horizontal width of the face (i.e., based on the front side facial width WA and the back side facial width WB).

[0099] The recognition device D of this embodiment can be utilized by being incorporated into an in-vehicle camera (checking the driver's safety by detecting the face direction), a surveillance camera (detecting the face direction when wearing a mask, detecting people's interests and behavior, etc.), a medical camera (detecting the face direction when wearing a mask, detecting the behavior of doctors and patients, etc.), etc.

[0100] [4. Modifications] Next, modified examples of the above embodiment will be described. Note that the modified examples described below can be combined as appropriate.

[0101] [4.1. Variation 1] In the above embodiment, as shown in Figures 13(A) and (B), the inclination of the reference upper string of the mask (an example of "feature of the left and right ends of the mask") and the inclination of the face center line (near the upper end of the mask) (an example of "feature of the face center line") are associated with the pitch angle and recorded in the memory unit 312, and the inclination of the upper string of the mask and the inclination of the face center line in a newly acquired image are compared to determine the pitch angle. However, instead of this, for example, as shown in Figures 14(A) and (B), the feature (shape) of the left and right ends of the mask and the feature (shape) of the face center line (part of the mask) may be directly associated with the pitch angle and recorded in the memory unit 312, and the feature (shape) of the left and right ends of the mask and the feature (shape) of the face center line in a newly acquired image may be compared, the pitch angle associated with the one with the highest similarity is obtained, and it may be determined as the pitch angle of the face F depicted in the image.

[0102] [4.2. Variation 2] In the above embodiment, the camera C and the irradiation unit L face the target from approximately the same direction, but the present invention can also be applied to the case where the camera C and the irradiation unit L face the target from different directions. In other words, even if the camera C and the irradiation unit L are located in different directions as seen from the target, if the position of the irradiation unit L is determined and it is known in advance how light is irradiated onto the face when wearing a mask and an image is captured, it is possible to estimate the face direction by determining the mouth area, determining whether the mask is worn, and detecting mask features and facial features, as in the above embodiment.

[0103] FIG. 15 is an example of an image captured with the camera C and the irradiation unit L arranged in different directions from the subject and irradiated with light from the side of the face. When light is irradiated from the side of the face, the center line 601 of the face can be detected when the face is oriented toward the front of the camera. When the face is oriented in the direction of the light irradiation, the facial contour, the mask features 602 facing the direction of the light, and the front area of ​​the face can be detected. These can be detected and judged in the same manner for every 10 degrees of yaw angle. However, when light is irradiated from the side of the face, if the face is oriented in the opposite direction to the light irradiation, the mask features and the features of the front of the face due to the light irradiation cannot be detected. Therefore, if it is desired to detect the face direction angle over a wider range, it is desirable to install the camera C and the irradiation unit L in the same direction as seen from the face, as in the above embodiment. Depending on the application and environment, the face direction may be judged by irradiating light from the side of the face and detecting the mask features 602, etc.

[0104] In this way, even if the camera C and the irradiation unit L are positioned in different directions relative to the subject, mask features and frontal facial features can be clearly detected. Therefore, even if the installation conditions of the camera C and the irradiation unit L dictate that the irradiation unit L should be positioned away from the camera C to irradiate light from the side of the face, or if dominant light (e.g., sunlight) is irradiated from the side of the camera C, face orientation can be determined using similar processing.

[0105] [4.3. Variation 3] In determining the mouth area of ​​the face or the face direction angle, the results of the determination made in the past captured images may be used for the above determination. For example, once the mouth area is estimated, a process of searching for the mouth area around the previously estimated area in an image captured at the next time is considered. In addition, in determining the face direction, for example, if the face direction is determined to be 10 degrees up and 20 degrees left, it is highly likely that the face direction in an image captured at the next time will be around the previously estimated face direction, so a process of preferentially detecting the features of a face direction of 20 degrees up and 20 degrees left, 10 degrees up and 10 degrees left, 10 degrees up and 30 degrees left, and 20 degrees left is considered to be performed. In other words, it is assumed that the face direction does not change significantly in images captured at a nearby time, and when a new face direction is determined, the determination process can be performed from a direction close to the most recently determined face direction. As in this modified example, by processing data in chronological order and effectively performing the determination process, the processing load for determining the mouth area and the face direction angle can be reduced.

[0106] [4.4. Variation 4] The inventors of the present application have found that in many masks, the mask area has a higher brightness value than the skin in the infrared captured image. However, depending on the material of the mask, for example, if a material that absorbs infrared rays is used, the mask area may be captured darker (with a lower brightness value) than the skin in the infrared captured image. In such a case, the mask may be detected by determining that the brightness value of the mouth area is smaller than that of the skin area. In this case, after the mask is detected, the process of detecting the mask edge can be performed in the same manner as in the above-mentioned embodiment. [Explanation of symbols]

[0107] 1 recognition device 111A Acquisition method 111B Detection means 111C Recognition means 111D Storage control means 111E Storage means D Face direction recognition deviceD 311 Control Unit 312 Storage section 313 Communications Department 314 Display section 315 Operation section

Claims

[Claim 1] An acquisition means for acquiring an image of an object covered with a covering member while the object is irradiated with light by an illumination means; a detection means for detecting a center line in a front vertical direction on a surface of the object based on a luminance value of a pixel included in the image; A recognition means for recognizing a direction of the object based on a detection result of the detection means; A recognition device comprising:

Images