Recognition device, recognition method, and recognition program

Abstract

To provide a recognition device and the like capable of accurately recognizing an orientation of an object covered with a covering member that recognizes an orientation of a face with a mask as a covering material put thereon.SOLUTION: A method acquires an image of an object covered with a covering member, captured in a state being irradiated with light from irradiation means, detects a front longitudinal centerline 212 on a surface of the object based on luminance levels of pixels contained in the image, and recognizes an orientation of the object based on inclination of the centerline.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present application relates to the technical field of a recognition device that recognizes the direction in which an object is facing based on an image of the object. [Background technology]

[0002] There is a technology that recognizes the orientation of a person's face from a captured image. Generally, facial orientation is recognized using the characteristics of each part of the face, so recognition accuracy decreases when part of the face is covered by a covering material such as a mask.

[0003] Therefore, the technology described in Patent Document 1 calculates the left-right angle of the face when a mask is present from the relative position of the nose in the center of the mask with respect to the face width in the face area. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 5359266 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology described in Patent Document 1, the center of the mask is set to the position of the nose, but the center of the mask may not necessarily be the actual position of the nose, and it may not be possible to accurately calculate the left-right angle of the face.

[0006] In view of the above circumstances, an object of the present invention is to provide a recognition device or the like that can accurately recognize the orientation of an object covered by a covering member. [Means for solving the problem]

[0007] The invention described in claim 1 comprises an acquisition means for acquiring an image of an object covered with a covering member while being irradiated with light, a detection means for detecting a center line in the front vertical direction on the surface of the object based on the brightness values ​​of pixels included in the image, and a recognition means for recognizing the vertical orientation of the object based on the inclination of the center line detected by the detection means.

[0008] The invention described in claim 5 is a recognition method using a recognition device, and includes an acquisition step of acquiring an image of an object covered with a covering member while being irradiated with light, a detection step of detecting a center line in the front vertical direction on the surface of the object based on the brightness values ​​of pixels included in the image, and a recognition step of recognizing the vertical orientation of the object based on the inclination of the center line detected in the detection step.

[0009] The invention described in claim 6 causes a computer to function as an acquisition means for acquiring an image of an object covered with a covering member while being irradiated with light, a detection means for detecting a center line in the front vertical direction on the surface of the object based on the brightness values ​​of pixels included in the image, and a recognition means for recognizing the vertical orientation of the object based on the inclination of the center line detected by the detection means. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram of a recognition device 1. [Figure 2] 10 is an example of a color-coded image. [Figure 3] 10 is an example of a color-coded image. [Figure 4] FIG. 2 is an example of a block diagram of a face direction recognition device D. [Figure 5] FIG. 2 is an example of a functional block diagram of a face direction recognition device D. [Figure 6] 10A, 10B, and 10C are diagrams illustrating an example of a method for detecting features of a face or a mask. [Figure 7] 10A and 10B are examples of color-coded images taken with the face facing in various directions. [Figure 8]10A and 10B are diagrams showing an example of a mask with a special shape. [Figure 9] 10 is a flowchart showing an example of a face direction determination process performed by the face direction recognition device D. [Figure 10] 10 is a flowchart showing an example of a mask wearing determination process performed by a face direction recognition device D. [Figure 11] 10 is a flowchart showing an example of a yaw angle determination process performed by a face direction recognition device D. [Figure 12] 10 is a flowchart showing an example of a pitch angle determination process performed by a face direction recognition device D. [Figure 13] 10A and 10B are diagrams showing an example of features at the left and right edges of a mask and features of the center line of a face (near the top edge of the mask). [Figure 14] 10A and 10B are diagrams showing an example of features of the left and right edges of a mask and the center line of a face (part that overlaps with the mask). [Figure 15] FIG. 10 is a diagram showing an example of an image captured when a camera C and an irradiation unit L are placed in different directions as viewed from the target. DETAILED DESCRIPTION OF THE INVENTION

[0011] An embodiment of the present invention will be described with reference to FIG.

[0012] 1, the recognition device 1 includes an acquisition unit 111A, a detection unit 111B, a recognition unit 111C, a memory control unit 111D, and a memory unit 111E. The memory unit may be provided outside the recognition device 1.

[0013] The acquisition means 111A acquires an image of an object covered with a covering member, photographed while being irradiated with light by an irradiation means. An example of an object covered with a covering member is a face wearing a mask. It is preferable that the irradiation means irradiates the object with light from a certain direction.

[0014] The detection means 111B detects a center line in the vertical direction of the front of the surface of the object based on the luminance values ​​of pixels included in the image acquired by the acquisition means 111A. If the object is a face (a typical face), the center line in the vertical direction of the front of the surface of the object is a line that passes through the surface of the face between the eyes, the center of the nose, and the center of the mouth.

[0015] Here, the center line will be explained using FIG. 2. FIG. 2 shows a color-coded image 200 obtained by color-coding an image (the eyes have been painted black to protect the model's privacy; the same applies to images of models hereinafter) of a face wearing a mask (an example of a "covering material") with light irradiated from the camera, based on brightness values. Specifically, portions with a brightness value of less than 100 are represented by the color indicated by reference numeral 201, portions with a brightness value of 100 or more but less than 160 are represented by the color indicated by reference numeral 202, portions with a brightness value of 160 or more but less than 220 are represented by the color indicated by reference numeral 203, and portions with a brightness value of 220 or more are represented by the color indicated by reference numeral 204. When light is irradiated, the brightness value of the masked region is higher than that of the skin. As shown in FIG. 2, the masked region has a high brightness value (brightness value of 220 or more). A face wearing a mask is symmetrical with respect to the center of the face, and the center of the face tends to protrude slightly compared to other facial features. Therefore, based on an image illuminated with light from the front of the camera, the detection unit 111B can detect the left edge 211 of the mask and a face (mask) center line 212 (an example of a "center line"). That is, the detection unit 111B detects the edge on the left edge side of the mask (the right side in the image) of the area with a brightness value of 220 or more as the left edge 211 of the mask, and detects the edge on the center side of the face (the side opposite the left edge of the mask) of the area with a brightness value of 220 or more as the face center line 212. However, if the horizontal orientation (yaw angle) exceeds a certain angle (approximately 50 degrees) with respect to the front of the camera, the edge on the center side of the face (the side opposite the left edge of the mask) of the area with a brightness value of 220 or more will no longer coincide with the face center line 212. In this case, it is preferable to detect the edge on the center side of the face of the area with a brightness value of 160 or more but less than 220 as the face center line 212B.

[0016] The recognition means 111C recognizes the orientation of the target based on the detection result of the detection means 111B.

[0017] As described above, according to the operation of the recognition device 1 of the embodiment, the orientation of the object is recognized based on the detection result of the center line in the front vertical direction on the surface of the object, which is based on the brightness values ​​of the pixels included in the image, so that the orientation of the object can be accurately recognized even if the object is covered with a covering member.

[0018] Furthermore, the recognition unit 111C may recognize the vertical orientation (pitch angle) of the target based on the inclination of the face center line 212. As a result of research by the inventors, it has been found that there is a correlation between the vertical orientation of the face and the inclination of the face center line 212, so by setting a threshold value for the inclination for each vertical orientation (pitch angle) of the face (for example, every 10 degrees) and comparing the inclination of the face center line 212 with the threshold value, the vertical angle of the face can be determined. This makes it possible to recognize the vertical orientation of the target.

[0019] Furthermore, detection means 111B further detects upper boundary 213, which is the upper boundary of the covering member of the object when recognition means 111C recognizes the vertical orientation of the object, and memory control means 111D stores the tilt of upper boundary 213 detected by detection means 111B in memory means 111E in association with the vertical orientation of the object recognized by recognition means 111B, and when a new image of an object whose vertical orientation is not recognized is acquired, recognition means 111C may recognize the vertical orientation associated with approximately the same tilt as the tilt of upper boundary 213 of the covering member detected in the image as the vertical orientation of the object appearing in the image. Such a determination is possible because the inventor's research has shown that when the tilt of the upper boundary 213 of the covering member is approximately the same, the vertical orientation of the object is also approximately the same. As a result, once the vertical orientation of the object is recognized, the vertical orientation of subsequent images can be recognized by detecting the upper boundary 213 and comparing it with the upper boundary 213 stored in the storage means. In other words, the processing load for recognizing the vertical orientation is reduced.

[0020] Furthermore, the recognition means 111C may recognize the lateral orientation (yaw angle) of the target based on the position of the intersection between the upper boundary line 213 and the face center line 212 in the lateral width of the target.

[0021] An example of recognizing the lateral orientation of an object will now be described with reference to FIG. 3. The recognition unit 111C recognizes the lateral orientation of an object based on three perpendicular lines 221-223. The perpendicular line 221 is a perpendicular line passing through the point at the end of the left ear. The perpendicular line 222 is a perpendicular line passing through the intersection of the facial center line 212 and the upper boundary 213. The perpendicular line 223 is a perpendicular line that moves horizontally from the intersection of the facial center line 212 and the upper boundary 213 toward the back of the face and passes through a point at the right end of the face. The recognition unit 111C recognizes the lateral orientation of an object based on any two widths: the width WA (front-side face width WA) between the perpendicular lines 221 and 222, the width WB (rear-side face width WB) between the perpendicular lines 222 and 223, and the width WC between the perpendicular lines 221 and 223. The inventor's research has revealed that there is a correlation between the lateral orientation of the face and the ratio of the front face width WA to the back face width WB. Therefore, by setting a threshold value for the ratio of the front face width WA to the back face width WB for each lateral orientation (angle) of the face (for example, every 10 degrees), and comparing the ratio of the front face width WA to the back face width WB with the threshold value, the lateral angle of the face can be determined. This makes it possible to recognize the lateral orientation of the target.

[0022] Furthermore, the detection means 111B further detects a left-right boundary, which is a boundary between at least one of the left and right sides of the covering member. When the detection means 111B detects the left-right boundary, the recognition means 111C recognizes the lateral orientation of the object based on the left-right boundary detected by the detection means 111B and the center line. Note that the left-right boundary refers to, for example, at least one of the left boundary indicated by the edge of the left edge 211 of the mask (see FIG. 2) and the right boundary indicated by the edge of the right edge of the mask. In other words, the left-right boundary can also be said to be a general term for the left boundary and the right boundary. The recognition means 111C recognizes that the object is facing right based on the detection of the edge of the left edge 211 of the mask (left boundary), and recognizes that the object is facing left based on the detection of the edge of the right edge of the mask (right boundary). [Example]

[0023] Next, a specific example corresponding to the above-described embodiment will be described.

[0024] The following embodiments will be described with reference to Figures 4 to 14. The embodiments described below are embodiments in which the present invention is applied to a face direction recognition device D (hereinafter, sometimes referred to as "recognition device D").

[0025] The recognition device D of this embodiment is connected to an illumination unit L, which is a light source such as an LED, and a camera C. The recognition device D recognizes the orientation of the face F based on an image captured while the face F, which is covered with a mask M, is illuminated by light from the illumination unit L.

[0026] Specifically, the recognition device D performs signal processing on the captured image to detect the face (or facial features) and determine whether the person is wearing a mask. If it determines that the person is wearing a mask, it detects mask features (mask shadow area, mask edge, mask edge gradient, mask string edge gradient, etc.) and facial features (face center line, face width, face outline, etc.), and recognizes the face orientation from these extracted features.

[0027] More specifically, first, it is determined whether a face is present in the image, and a position (mouth area) for determining whether a mask is being worn is determined. For example, the approximate position of the mouth is determined within a face frame detected by a conventionally known face detection process. The position for determining whether a mask is being worn may also be determined by detecting features of each part of the face. For example, the position of the eyes may be detected, and the area below the eyes may be determined as the approximate position of the mouth, and then mask wear determination may be performed. Next, it is determined whether a mask is being worn. Generally, when a face F wearing a mask M is illuminated with light and an image is taken, a high luminance value is obtained in the mask-wearing area, and therefore, whether a mask is being worn is determined based on the high luminance value of the area around the mouth. Whether a mask is being worn may also be determined based on information such as the mask's outline. If it is determined that a mask is being worn, features of the mask and facial features are detected, and the orientation of the face is recognized from these extracted features. Furthermore, based on the feature detection results obtained when facial features and mask features are detected from the image, the intensity of the illuminating light or camera parameters may be adjusted to more clearly detect shadows, luminance gradients, etc., of the mask and face. In this embodiment, a case where the camera C and the irradiating unit L face the target from approximately the same direction will be described. A case where the camera C and the irradiating unit L face the target from different directions will be described in a modified example described later.

[0028] [1. Configuration of Recognition Device D] Next, the configuration of the recognition device D according to this embodiment will be described with reference to Fig. 4. As shown in Fig. 4, the recognition device D is broadly configured to include a control unit 311, a storage unit 312, a communication unit 313, a display unit 314, and an operation unit 315. The communication unit 313 is connected to the camera C and the irradiation unit L.

[0029] The storage unit 312 is configured with, for example, a hard disk drive, and stores various programs including an OS (Operating System) and a face direction recognition program for recognizing the direction of a face. The storage unit 312 also stores images captured by a camera and various data used in the face direction recognition program.

[0030] The communication unit 313 controls the communication state with the irradiation unit L and the camera C.

[0031] The display unit 314 is configured by, for example, a liquid crystal display or the like, and is configured to display images captured by the camera C and the like.

[0032] The operation unit 315 is configured with, for example, a keyboard, a mouse, etc., and is configured to receive operation instructions from an operator and output the contents of the instructions to the control unit 311 as instruction signals.

[0033] The control unit 311 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), , RAM (Random Access Memory), etc. The CPU reads and executes various programs, including a face direction recognition program, stored in the ROM or storage unit 312, to realize various functions.

[0034] [2. Functions of Recognition Device D] Next, the functions of the recognition device D (control unit 311) according to this embodiment will be described with reference to Fig. 5. Fig. 5 is a functional block diagram of the recognition device D. The recognition device D is configured to include an image analysis unit 351, an irradiation light adjustment unit 357, and a camera adjustment unit 358. The functions of the image analysis unit 351, the irradiation light adjustment unit 357, and the camera adjustment unit 358 are realized by the control unit 311 executing a face direction recognition program.

[0035] The image analysis unit 351 includes a mouth area detection unit 352 , a mask wearing determination unit 353 , a mask feature detection unit 354 , a face feature detection unit 355 , and a face direction recognition unit 356 .

[0036] The mouth area detection unit 352 performs face detection processing, eye detection processing, nose detection processing, and mouth detection processing on an image of, for example, a face F covered by a mask M, captured by a camera C with light being irradiated by an irradiation unit L. If the face area can be determined by the face detection processing, it can be determined that the mouth area is located in the area below the face detection frame. If the eye area can be determined by the eye detection processing, it can be determined that the mouth area is located below the eyes. If the nose area can be determined by the nose detection processing, it can also be determined that the mouth area is located below the nose. If the mouth area is detected by the mouth detection processing, it is determined that that area is the mouth area.

[0037] The mask wearing determination unit 353 determines whether a mask M is being worn for the mouth area detected by the mouth area detection unit 352. Specifically, the determination is made based on brightness values. For example, as shown in FIG. 6(A), an average brightness value of the right eye area 401 and an average brightness value of the mouth area 402 are calculated. Generally, when light is shone onto a face, the brightness value of the mask is greater than the brightness value of the skin. For example, by comparing the average brightness value of the right eye area 401 and the average brightness value of the mouth area 402, if the values ​​are approximately the same, it can be determined that a mask is not being worn, and if the average brightness value of the mouth area 402 is greater (exceeding a threshold value), it can be determined that a mask is being worn.

[0038] The mask feature detection unit 354 detects the features of the mask region based on the luminance value of the mask region. The face feature detection unit 355 detects the features of the face region based on the luminance value of the face region. Specifically, the mask feature detection unit 354 and the face feature detection unit 355 (sometimes collectively referred to as the "feature detection unit") detect features using a color-coded image in which the luminance values ​​in an image of a face are divided into four regions using three thresholds, as shown in FIG. 2. Specifically, the region is divided into (1) a mask region, (2) a skin (light) region, (3) a skin (dark) region, and (4) a background region.

[0039] Here, an example of creating a color-coded image will be described. First, the feature detection unit determines a first threshold value indicating the boundary between (1) the mask region and (2) the skin (light) region based on the maximum value Ja (considering that pixel values ​​may include spike noise, etc., the fifth largest value is treated as the maximum value here) of the skin region (e.g., left eye region 401) and the average luminance value Jb of the mouth region 402 when the mask is worn. For example, if Ja is "219" and Jb is "246," the value obtained by adding a margin to Ja is set as the first threshold value. For example, "220," obtained by adding "1" to Ja, is set as the first threshold value (although "1" is added to make the value round, other values ​​may also be used. The same applies to the second and third threshold values ​​below). A region composed of pixels with luminance values ​​greater than the first threshold value can be detected as a mask region.

[0040] Next, the feature detection unit determines a second threshold value indicating the boundary between the (2) skin (light) region and the (3) skin (dark) region based on the average value Jc of the skin region (for example, the lower half region of the left eye 403 shown in FIG. 6(B)). (The eyebrows and pupils of the eye have low brightness values, so the lower half region of the left eye 403 is used, but the average value of other parts may also be used.) For example, if Jc is "157," the value obtained by subtracting a margin from Jc is set as the second threshold value. For example, "150," obtained by subtracting "7" from Jc, is set as the second threshold value.

[0041] Next, the feature detection unit determines a third threshold value indicating the boundary between the (3) skin (dark) region and the (4) background region based on the minimum value Jd of the skin region (for example, the left eye left half region 404 shown in FIG. 6(C)). (Considering that pixel values ​​may include spike noise, for example, the fifth smallest value is treated as the minimum value. Here, the left eye left half region 404 is used, but the minimum value of another skin region may also be used.) For example, if Jd is "78," the value obtained by subtracting a margin from Jd is set as the third threshold value. For example, "70," obtained by subtracting "8" from Jd, is set as the third threshold value.

[0042] The feature detection unit divides each pixel constituting the image into (1) a mask region, (2) a skin (light) region, (3) a skin (dark) region, and (4) a background region based on the luminance value of the pixel and the first, second, and third thresholds, and creates a color-coded image by representing these regions in four colors.Then, based on the color-coded image, the features of the mask region and the face region are detected.

[0043] FIG. 7 shows an example of an image captured by changing the vertical orientation (pitch angle) and horizontal orientation (yaw angle) of the face in 10-degree increments. As with the example in FIG. 2, the image is divided into areas with a brightness value of less than 100, areas with a brightness value of 100 to 160, areas with a brightness value of 160 to 220, and areas with a brightness value of 220 or greater, and each area is represented by a different color (as in FIG. 2). When illuminated with light, the brightness value of the mask area is higher than that of the skin. The mask area has a high brightness value (brightness value of 220 or greater). When the face is facing left or right (yaw angle: 20 to 50 degrees), the face orientation recognition unit 356 can clearly detect the left and right boundaries between the left and right edges of the mask and the skin, and can determine whether the face is facing left or right. On the other hand, when the face is facing directly ahead of the camera (yaw angle: 0 degree), the boundaries between the left and right edges of the mask and the skin (left and right boundaries) cannot be detected, and this allows the face to be determined to be facing forward. In order to estimate the horizontal face direction angle, if the center of the face, the edges of the face (contour, ears, etc.), and the left and right edges of the mask can be detected, the approximate face direction angle can be calculated. Below, a method by which the face direction recognition unit 356 recognizes the vertical face direction (pitch angle) and the horizontal face direction (yaw angle) will be described in detail.

[0044] (Pitch angle: 0 degrees) When the face is facing the camera (yaw angle: 0 degrees, pitch angle: 0 degrees), the brightness value of the mouth area is high (compared to the skin area), but no noticeable mask edges are detected on the left or right edges of the mask. When the face is turned 10 degrees to the left (yaw angle: 10 degrees, pitch angle: 0 degrees), the brightness value of the mouth area is high, and because the face is tilted leftward relative to the camera, a noticeable mask edge can be detected on the left edge of the mask. This is because the light is irradiated from the front of the camera. When the face is turned 20 degrees to the left (yaw angle: 20 degrees), the face is tilted more than when turned 10 degrees, so the brightness value of the area of ​​the mask facing the camera is more noticeable. The left edge of the mask can also be clearly detected. In addition, a face wearing a mask is symmetrical with respect to the center of the face, and the center of the face tends to protrude slightly compared to other facial features. Therefore, when light is irradiated from the front of the camera, the center line of the face as well as the left edge of the mask can be clearly detected. Similarly, when the face is turned 30 degrees, 40 degrees, or 50 degrees left, the left edge of the mask and the center line of the face can be clearly detected. Furthermore, if there is no object near the face, the brightness value of the background does not increase even when light is shone on it, making it possible to detect the face width WA in front of the camera and the face width WB behind the camera. When the face is tilted 10 degrees to the left, the degree to the left of the face can be determined by detecting the edge of the left edge of the mask, the center line of the face, the face width WA in front of the camera, and the face width WB behind the camera, as described above. When the face is turned to the right, the left-right direction is simply mirrored to when the face is turned to the left, so a detailed explanation is omitted.

[0045] (Pitch angle: 10 degrees upward) When the face is turned 10 degrees up (yaw angle: 0 degrees, pitch angle: up 10 degrees), the brightness value of the mouth area is higher than that of the skin area, but no noticeable mask edges are detected on the left or right edges of the mask. The upper edge of the mask has a steeper slope than when facing the camera directly (yaw angle: 0 degrees, pitch angle: 0 degrees). However, since the upper edge slope of the mask changes depending on the deformation of the mask when worn, it is not possible to determine whether the face is facing up based on this alone. However, if the reference edge slope of the upper edge of the mask when worn is known, it is possible to determine the angle of the face in the up or down direction. When the face is turned 10 degrees left (yaw angle: 10 degrees), the brightness value of the mouth area is higher, just as when the pitch angle is 0 degrees. Furthermore, because the face is tilted leftward relative to the camera, a noticeable edge can be detected on the left edge of the mask. Furthermore, when the face is tilted to the left, just as when the pitch angle is 0 degrees, the edge of the left end of the mask, the center line of the face, the face width WA in front of the camera, the face width WB behind the camera, etc. can be detected to determine how many degrees the face is tilted to the left.

[0046] The face's upward tilt angle (pitch angle) can be determined by the tilt of the face's center line when the face is facing 30 degrees left, 40 degrees left, 50 degrees left, etc. Comparing the tilt of the face's center line when the face is facing 30 degrees left, 40 degrees left, and 50 degrees left when the face is facing upward at a tilt angle (pitch angle) of 0 degrees and 10 degrees, it is clearer that the face is facing upward when the face is facing upward at a tilt angle (pitch angle) of 10 degrees than when the face is facing upward at a tilt angle (pitch angle) of 0 degrees. By recording the gradient of the upper edge of the mask when the face is facing upward at a tilt angle (pitch angle) of 10 degrees as a reference, it is possible to determine that the face's upward tilt angle (pitch angle) is 10 degrees if the gradient of the upper edge of the mask in a subsequently captured image is the same angle (preferably with a certain width and approximately the same angle), provided the mask is not deformed. A face facing 10 degrees up to the right is simply symmetrical to a face facing 10 degrees up to the left, and therefore a detailed description thereof will be omitted.

[0047] (Pitch angle: 20 degrees upward) When the face is turned up 20 degrees (yaw angle: 0 degrees, pitch angle: up 20 degrees), the brightness value of the mouth area is higher than that of the skin area, but no noticeable mask edges are detected on the left or right edges of the mask. The upper edge of the mask has a steeper slope than when facing the camera directly (yaw angle: 0 degrees, pitch angle: 0 degrees). However, since the upper edge slope of the mask changes depending on the deformation of the mask when worn, it is not possible to determine whether the face is facing up based on this alone. However, if the reference edge slope of the upper edge of the mask when worn is known, it is possible to determine the vertical face orientation angle. When the face is turned 10 degrees left (yaw angle: 10 degrees, pitch angle: up 20 degrees), the brightness value of the mouth area is higher, just as when the pitch angle is 0 degrees. Furthermore, because the face is tilted leftward relative to the camera, a noticeable edge can be detected on the left edge of the mask. Furthermore, when the face is tilted to the left, just as when the pitch angle is 0 degrees, the edge of the left end of the mask, the center line of the face, the face width WA in front of the camera, the face width WB behind the camera, etc. can be detected to determine how many degrees the face is tilted to the left.

[0048] The face's upward tilt angle (pitch angle) can be determined by the tilt of the face centerline when the face is facing 30 degrees, 40 degrees, and 50 degrees left, just as when the face is facing 10 degrees up. Comparing the face facing 10 degrees up with the face facing 20 degrees up, the tilt of the face centerline when the face is facing 30 degrees, 40 degrees, and 50 degrees left is greater when the face is facing 20 degrees up, so the face can be determined to be facing 20 degrees up. Furthermore, if the gradient of the mask's upper edge when the face is facing 20 degrees up (pitch angle) is recorded as a reference, the face's upward tilt angle (pitch angle) can be determined to be 20 degrees if the gradient of the mask's upper edge in a subsequently captured image is the same angle (preferably with a certain width and approximately the same angle) as long as the mask is not deformed. A face facing 20 degrees up to the right is simply symmetrical to a face facing 20 degrees up to the left, and therefore a detailed description thereof will be omitted.

[0049] (Pitch angle: 10 degrees downward) When the face is turned down 10 degrees (yaw angle: 0 degrees, pitch angle: down 10 degrees), the brightness value of the mouth area is higher than that of the skin area, but no noticeable mask edges are detected on the left or right edges of the mask. The upper edge of the mask has a gentler gradient than when facing the camera directly (yaw angle: 0 degrees, pitch angle: 0 degrees). However, the upper edge gradient of the mask changes depending on the deformation of the mask when worn, so it is not possible to determine whether the mask is facing down based on this alone. However, if the reference edge gradient of the upper edge of the mask when worn is known, it is possible to determine the vertical face orientation angle (same as when facing up 10 degrees). When the face is turned 10 degrees left (yaw angle: 10 degrees, pitch angle: down 10 degrees), the brightness value of the mouth area is higher, just as when the pitch angle is 0 degrees. Furthermore, because the face is tilted leftward relative to the camera, a noticeable edge can be detected on the left edge of the mask. When the face is tilted to the left, just as when the pitch angle is 0 degrees, it is possible to determine the angle at which the face is tilted to the left by detecting the left edge of the mask, the center line of the face, the face width WA in front of the camera, and the face width WB behind the camera.

[0050] The downward angle of the face can be determined by the inclination of the face center line when the face is facing 30 degrees, 40 degrees, and 50 degrees left, just as when the face is facing 10 degrees up. Comparing a pitch angle of 0 degrees with a pitch angle of 10 degrees down, it can be determined that the face is more downwardly tilted at a pitch angle of 10 degrees down, making it possible to determine the face to be tilted 10 degrees down. If the gradient of the upper edge of the mask when this downward angle (pitch angle) of the face is 10 degrees is recorded as a reference, and the same angle (preferably with a certain width and approximately the same angle) is obtained by comparing it with the gradient of the upper edge of the mask in a subsequently captured image, it can be determined that the downward angle (pitch angle) of the face is 10 degrees, provided that the mask is not deformed. A face facing 10 degrees down to the right is simply symmetrical to a face facing 10 degrees down to the left, and therefore a detailed description thereof will be omitted.

[0051] (Pitch angle: 20 degrees downward) When the face is turned down 20 degrees (yaw angle: 0 degrees, pitch angle: down 20 degrees), the brightness value of the mouth area is higher than that of the skin area, but no noticeable mask edges are detected on the left or right edges of the mask. The upper edge of the mask has a gentler gradient than when facing the camera directly (yaw angle: 0 degrees, pitch angle: 0 degrees). However, the upper edge gradient of the mask changes depending on the deformation of the mask each time it is worn, so it is not possible to determine whether it is facing down based on this alone. However, if the reference edge gradient of the upper edge of the mask when worn is known, it is possible to determine the vertical face orientation angle (same as when facing up 10 degrees). When the face is turned 10 degrees left (yaw angle: 10 degrees, pitch angle: down 20 degrees), the brightness value of the mouth area is higher, just as when the pitch angle is 0 degrees. Furthermore, tilting the face to the left relative to the camera allows a noticeable edge to be detected on the left edge of the mask. Furthermore, when the face is tilted to the left, just as when the pitch angle is 0 degrees, the left edge of the mask, the center line of the face, the face width WA in front of the camera, the face width WB behind the camera, etc. are detected, and it is possible to determine how many degrees the face is tilted to the left.

[0052] The downward angle (pitch angle) of the face can be determined by the inclination of the face center line when the face is oriented 30 degrees, 40 degrees, and 50 degrees left, just as when the face is oriented 10 degrees downward. Comparing the face oriented 10 degrees downward with the face oriented 20 degrees downward, the inclination of the face center line when the face is oriented 30 degrees, 40 degrees, and 50 degrees left indicates that the face is tilted more downward, and can be determined to be 20 degrees downward. If the gradient of the upper edge of the mask when this downward angle (pitch angle) of the face is 20 degrees is recorded as a reference, and the same angle (preferably with a certain width and approximately the same angle) is obtained by comparing it with the gradient of the upper edge of the mask in a subsequently captured image, it can be determined that the downward angle (pitch angle) of the face is 20 degrees, provided that the mask is not deformed. When the face is oriented 20 degrees downward to the right, it is simply symmetrical left and right to the face oriented 20 degrees downward, and therefore a detailed description thereof will be omitted.

[0053] When the face is turned 50 degrees to the left, the brightness value of the skin near the left edge of the mask may be high, but the brightness value of the mask strings that hang from the ears also increases when illuminated with light, allowing for the detection of a distinct edge, so it is possible to detect the left edge of the mask based on this information.The up-down orientation of the face can also be determined from this string edge information.

[0054] Furthermore, since the facial widths WA and WB may differ depending on the up and down direction of the face (depending on how the perpendicular line 222 in Figure 2 is taken), it is also possible to determine whether the face is up or down and then determine the direction based on the facial width based on the criteria for each up and down direction.

[0055] Furthermore, when light is irradiated from the same direction as camera C, if the face is tilted downward by 20 degrees, the brightness of the forehead area will be high if the forehead is not covered by hair. It may be determined that the face is tilted downward by 20 degrees when the brightness of the forehead area is greater than a predetermined threshold.

[0056] Furthermore, once the mask upper edge gradient, which serves as the reference for the vertical direction of the face orientation, is determined, the vertical pitch angle can be determined based on that value. For example, when the face orientation changes from 20 degrees downward to 20 degrees upward after the mask upper edge gradient is determined, the edge gradient when the face orientation changes from 20 degrees downward to 20 degrees upward can be calculated based on the reference edge gradient when the face orientation is 20 degrees downward. If the calculated value is close to the edge gradient detected from the image, it can be determined that the face orientation has changed to 20 degrees upward. The reference edge gradient can be calculated as long as the mask is not deformed or removed while being worn. Although the reference edge gradient changes depending on the distance between the face and the camera, if the distance is approximately constant, once the reference is determined, the face orientation can be determined based on that reference. Even when the distance between the face and the camera changes, the reference edge gradient can be changed based on facial depth information and facial size information to determine the face orientation.

[0057] Furthermore, since there are various types of masks, it is possible to determine the type of mask being worn and estimate the face direction based on the shape information specific to the mask being worn. According to the inventor's research, there are various types of masks currently on the market. Regarding the edges of the left and right edges of the mask, although each mask has its own characteristics, it has been found that by irradiating light onto the mask when the face is turned left or right, it is possible to detect the unique shapes of the left and right edges of almost all masks. Furthermore, although the shape of the center line of the face (mask) when the face is turned left or right also varies depending on the type of mask, almost all masks have a protruding shape at the center (to fit the shape of the face, which protrudes toward the center). Therefore, by irradiating light from the camera, it is possible to detect the center of the mask, which is the center of the face. Furthermore, although the edge gradient of the upper side of the mask described above varies depending on the type of mask, once a standard is determined using the above-described process, the vertical angle of the face can be determined based on that standard. For example, in the mask shown in Figure 8(A), the left and right edges of the mask are not vertical edge lines, but the shapes of the left and right edges can be detected. There is also a hemispherical cup-shaped mask as shown in Figure 8(B), but the mask center and the edge gradient at the top of the mask can be detected. Furthermore, even a black mask can be captured as white when photographed with infrared light, although there may be some differences in brightness values. Therefore, it is possible to similarly detect the mask and detect the features of the left and right edges and top of the mask.

[0058] Furthermore, although masks may deform, the edge characteristics of the left and right edges of the mask described above hardly change. Regarding the center line of the mask when the face is turned left or right, because the mask flexibly changes shape, there is a possibility that the center line cannot be detected due to deformation. In such cases, when the face is turned 30 degrees to 50 degrees left, the center line of the face can be detected in the upper half of the face rather than the mask area to determine the up-down direction of the face, or the tilt of the mask strings hanging from the ears can be detected to determine the up-down direction of the face. The gradient of the upper edge of the mask may also change due to mask deformation, but once a reference gradient is determined, it is possible to deal with deformation of the gradient of the upper edge of the mask by periodically calculating the reference each time, taking mask deformation into account.

[0059] When recognizing the direction of the face, the face direction recognition unit 356 may extract edge gradients and the like based on brightness values, but depending on the shooting conditions, changing the camera parameters (exposure time, etc.) or the intensity of the irradiated light (either one or both) may be more effective in extracting features such as edge gradients. In such cases, the irradiated light adjustment unit 357 adjusts the intensity of the light irradiated from the irradiation unit L, and the camera adjustment unit 358 adjusts the parameters of the camera C, based on the brightness values ​​detected by the mouth area detection unit 352, mask wearing determination unit 353, mask feature detection unit 354, and face feature detection unit 355.

[0060] [3. Example of operation of recognition device D] Next, an example of the face direction determination process of the recognition device D will be described with reference to the flowcharts of FIGS.

[0061] First, the control unit 311 of the recognition device D acquires an image of a face F covered by a mask M, which is captured by the camera C while the illumination unit L is irradiating the face F with light (step S11).

[0062] Next, the control unit 311 detects the mouth area of ​​the face appearing in the image acquired in the process of step S11 (step S12).

[0063] Next, the control unit 311 performs a mask wearing determination process for the mouth area detected in the process of step S12 (step S13).

[0064] Here, the mask wearing determination process will be described with reference to FIG.

[0065] First, the control unit 311 calculates the average luminance value Ia of the skin region (for example, the right eye region 401 in FIG. 6) (step S21).

[0066] Next, the control unit 311 calculates the average brightness value Ib of the mouth region (for example, the mouth region 402 in FIG. 6) (step S22).

[0067] Next, the control unit 311 determines whether or not "Ib - Ia > Ic" is true (step S23). Here, Ic is a threshold value (for example, Ic = 40). Ic can be set according to the observed brightness value, etc. If the control unit 311 determines that "Ib - Ia > Ic" is true (step S23: YES), it determines that a mask is being worn (step S24) and ends the mask wearing determination process. On the other hand, if the control unit 311 determines that "Ib - Ia > Ic" is not true (step S23: NO), it determines that a mask is not being worn (step S25) and ends the mask wearing determination process.

[0068] 9, the control unit 311 determines whether or not a mask is being worn in the mask wearing determination process (step S14). If the control unit 311 determines that a mask is not being worn (not being worn) (step S14: NO), the control unit 311 ends the face direction determination process. On the other hand, if the control unit 311 determines that a mask is being worn (step S14: YES), the control unit 311 then performs yaw angle determination process (step S15).

[0069] Here, the yaw angle determination process will be described with reference to FIG.

[0070] First, the control unit 311 determines whether it can detect the left and right ends of the mask from the image obtained in the process of step S11 or step S18 described later (step S31). For a mask with a special shape at the left and right ends as shown in FIG. 8(A), by first determining and recording the mask part with a large luminance value with respect to the luminance value of the skin, the same processing can be performed in subsequent processing.

[0071] When the control unit 311 determines that it cannot detect the left and right ends of the mask (step S31: NO), it determines that the yaw angle is "0 degrees" (step S32), and ends the yaw angle determination process. On the other hand, when it determines that it can detect the left and right ends of the mask (step S31: YES), then it determines whether it can detect the face center line (refer to the face center line 212 in FIG. 2) (step S33).

[0072] When the control unit 311 determines that it cannot detect the face center line (step S33: NO), it determines that the yaw angle is "10 degrees" (step S34), and ends the yaw angle determination process. On the other hand, when it determines that it can detect the face center line (step S33: YES), then it calculates the front face width WA and the back face width WB (refer to FIG. 3) (step S35).

[0073] Next, the control unit 311 determines the yaw angle based on the front face width WA and the back face width WB calculated in the process of step S35 (step S36). In this embodiment, the yaw angle is determined by comparing WB / WA with predetermined threshold values Y1, Y2, Y3 (Y1>Y2>Y3). The threshold values Y1, Y2, Y3 may be changed based on values such as the distance between the camera C and the face F observed.

[0074] Specifically, the control unit 311 determines the yaw angle as follows. (1) Y1 < WB / WA → Yaw angle: 20 degrees (2) Y2 < WB / WA < Y1 → Yaw angle: 30 degrees (3) Y3 < WB / WA < Y2 → Yaw angle: 40 degrees (4) WB / WA < Y3 → Yaw angle: 50 degrees

[0075] Next, the control unit 311 performs a face centerline correction process (step S37) and terminates the yaw angle determination process. The face centerline correction process is performed to correct the face centerline in cases where the luminance value of the mask center decreases as the yaw angle increases relative to the irradiated area L, causing the mask centerline detected under the above-described luminance value conditions to no longer match the face centerline (for example, when the yaw angle is 50 degrees in FIG. 7 . If the yaw angle is 40 degrees or less, the mask centerline and the face centerline tend to match). In this embodiment, when the yaw angle is less than 50 degrees, the face centerline is determined to be the line corresponding to the edge formed by pixels color-coded with a luminance value of 220 or greater. On the other hand, when the yaw angle is 50 degrees or greater, the face centerline is determined to be the line corresponding to the edge formed by pixels color-coded with a luminance value of 160 or greater but less than 220.

[0076] Returning to FIG. 9, the control unit 311 performs a pitch angle determination process (step S16).

[0077] Here, the pitch angle determination process will be described with reference to FIG.

[0078] First, the control unit 311 determines whether the tilt of the upper end of the mask has been recorded for all pitch angles (down 20 degrees, down 10 degrees, 0 degree, up 10 degrees, up 20 degrees) (step S51). At this time, if the control unit 311 determines that the tilt of the upper end of the mask has not been recorded for all pitch angles (step S51: NO), the control unit 311 determines the pitch angle based on the features of the left and right ends of the mask and the features of the face center line (near the upper end of the mask) (step S53). Specifically, as shown in FIGS. 13A and 13B, the features of the left and right ends of the mask and the features of the face center line differ depending on the pitch angle. Therefore, the memory unit 312 stores the features of the left and right ends of the mask and the face center line as references for each pitch angle in advance, and compares the features of the left and right ends of the mask with the face center line in the currently acquired image to determine the pitch angle associated with the most similar feature of the left and right ends of the mask and the face center line as the pitch angle of the face appearing in the image. In this embodiment, the inclination of the upper strings of the mask is recorded as a feature of the left and right ends of the mask, and the inclination near the upper end of the mask is recorded as a feature of the face center line.

[0079] The following will be described in detail. · The inclination of the upper strap on the mask at a pitch of 0 degrees (horizontal): m0, recorded in the memory unit 312 · The inclination of the center line of the face at a pitch of 0 degrees (horizontal): v0, recorded in the memory unit 312 · The inclination of the upper strap on the mask at a pitch of 10 degrees above: m1, recorded in the memory unit 312 · The inclination of the center line of the face at a pitch of 10 degrees above: v1, recorded in the memory unit 312 · The inclination of the upper strap on the mask at a pitch of 20 degrees above: m2, recorded in the memory unit 312 · The inclination of the center line of the face at a pitch of 20 degrees above: v2, recorded in the memory unit 312 · The inclination of the upper strap on the mask at a pitch of 10 degrees below: m3, recorded in the memory unit 312 · The inclination of the center line of the face at a pitch of 10 degrees below: v3, recorded in the memory unit 312 · The inclination of the upper strap on the mask at a pitch of 20 degrees below: m4, recorded in the memory unit 312 · The inclination of the center line of the face at a pitch of 20 degrees below: v4, recorded in the memory unit 312 · The inclination of the upper strap on the mask of the image acquired this time: mA · The inclination of the center line of the face of the image acquired this time: vA When the inclination of the upper strap on the mask (an example of "features at the left and right ends of the mask") and the inclination of the center line of the face are defined as described above, the control unit 331 determines the pitch angle as follows.

[0080] <00003​​​​​​​​​​​​​[(m1 + m2) / 2 < mA] → Pitch angle: 20 degrees upward

[0083] (4)[(v3 + v4) / 2 < vA < (v0 + v3) / 2] or / and [(m3 + m4) / 2 < mA < (m0 + m3) / 2] → Pitch angle: 10 degrees downward

[0084] (5)[vA < (v3 + v4) / 2] or / and [mA < (m3 + m4) / 2] → Pitch angle: 20 degrees downward

[0085] Next, the control unit 311 detects the upper end of the mask (the upper boundary 213 in FIG. 2), associates the pitch angle determined in the process of step S53 with the inclination of the upper end of the mask, and records it in the storage unit 312 (step S54), and ends the pitch angle determination process.

[0086] On the other hand, if the control unit 311 determines in the process of step S51 that the inclination of the upper end of the mask has been recorded for all pitch angles (step S51: YES), it compares the recorded inclination of the upper end of the mask with the inclination of the upper end of the mask in the image acquired this time, determines the pitch angle (step S52), and ends the pitch angle determination process. Here, the case of comparing the recorded inclination of the upper end of the mask with the inclination of the upper end of the mask in the image and determining the pitch angle will be specifically described.

[0087] · Inclination of the upper end of the mask at pitch 0 degrees (horizontal): g0 recorded in the storage unit 312 · Inclination of the upper end of the mask at pitch 10 degrees upward: g1 recorded in the storage unit 312 · Inclination of the upper end of the mask at pitch 20 degrees upward: g2 recorded in the storage unit 312 · Inclination of the upper end of the mask at pitch 10 degrees downward: g3 recorded in the storage unit 312 · Inclination of the upper end of the mask at pitch 20 degrees downward: g4 recorded in the storage unit 312 · Inclination of the upper end of the mask in the image acquired this time: gA When the inclination of the upper end of the mask is defined as described above, the control unit 331 determines the pitch angle as follows.

[0088] (1) [(g3 + g0) / 2 < gA < (g0 + g1) / 2] → Pitch angle: 0 degrees

[0089] (2) [(g0 + g1) / 2 < gA < (g1 + g2) / 2] → Pitch angle: up 10 degrees

[0090] (3) [(g1 + G2) / 2 < gA] → Pitch angle: up 20 degrees

[0091] (4) [(g3 + g4) / 2 < gA < (g0 + g3) / 2] → Pitch angle: down 10 degrees

[0092] (5) [gA < (g3 + g4) / 2] → Pitch angle: down 20 degrees

[0093] Returning to FIG. 9, the control unit 311 determines whether to continue the face orientation determination process (step S17). At this time, when the control unit 311 determines not to continue the face orientation determination process (step S17: NO), it ends the face orientation determination process. On the other hand, when the control unit 311 determines to continue the face orientation determination process (step S17: YES), it acquires the next image (step S18) and proceeds to the process of step S15.

[0094] As described above, the control unit 311 (an example of the "acquisition means", "detection means", and "recognition means") of the recognition device D according to the present embodiment acquires an image taken in a state where a face (an example of the "object") covered by a mask (an example of the "coating member") is irradiated with light by the irradiation unit L (an example of the "irradiation means"), and based on the luminance values of the pixels included in the acquired image, detects the face center line 212 (an example of the "center line in the front vertical direction on the surface of the object"), and recognizes the orientation of the object based on the detection result.

[0095] Therefore, according to the recognition device D of this embodiment, the facial orientation is recognized based on the detection result of the facial center line 212 based on the luminance values ​​of pixels included in the image, so that the facial orientation can be accurately recognized even when the face is covered by a mask. Also, the facial orientation can be recognized without performing processing with a high processing load such as pattern matching. Furthermore, since there is no need to store pattern images for pattern matching, the memory unit 312 can be a small-capacity memory device.

[0096] Furthermore, the control unit 311 of the recognition device D according to this embodiment recognizes the vertical orientation (pitch angle) of the face based on the inclination of the face center line 212.

[0097] Furthermore, the control unit 311 (an example of a "detection unit," "storage control unit," or "recognition unit") of the recognition device D according to this embodiment further detects an upper boundary 213, which is the upper boundary of the mask when the vertical orientation of the face is recognized, and stores the inclination of the upper boundary 213 in the memory unit 312 (an example of a "storage unit") in association with the vertical orientation of the recognized subject. When a new image of a face whose vertical orientation is not recognized is acquired, the control unit 311 recognizes the vertical orientation associated with approximately the same inclination as the inclination of the upper boundary 213 of the mask detected for that image as the vertical orientation of the face appearing in that image. As a result, once the vertical orientation of the face is recognized once, the control unit 311 can recognize the vertical orientation of subsequent images by detecting the upper boundary 213 and comparing it with the upper boundary 213 stored in the memory unit 312. In other words, the processing load for recognizing the vertical orientation is reduced.

[0098] Furthermore, the control unit 311 (an example of a "recognition means") of the recognition device D according to this embodiment recognizes the lateral orientation (yaw angle) of the face based on the position of the intersection between the upper boundary line 213 and the face center line 212 in the horizontal width of the face (i.e., based on the front face width WA and the back face width WB).

[0099] The recognition device D according to this embodiment can be utilized by being incorporated into an in-vehicle camera (checking the driver's safety by detecting the direction of the face), a surveillance camera (detecting the direction of the face when wearing a mask, detecting people's interests and behavior, etc.), a medical camera (detecting the direction of the face when wearing a mask, detecting the behavior of doctors and patients, etc.), etc.

[0100] [4. Modifications] Next, modifications of the above embodiment will be described. The modifications described below can be combined as appropriate.

[0101] [4.1. Variation 1] In the above embodiment, as shown in FIGS. 13(A) and 13(B), the inclination of the reference upper string of the mask (an example of a "feature of the left and right edges of the mask") and the inclination of the face center line (near the upper edge of the mask) (an example of a "feature of the face center line") are associated with the pitch angle and recorded in the memory unit 312, and the pitch angle is determined by comparing the inclination of the upper string of the mask and the inclination of the face center line in a newly acquired image. However, instead of this, for example, as shown in FIGS. 14(A) and 14(B), the feature (shape) of the left and right edges of the mask and the feature (shape) of the face center line (portion of the mask) may be directly associated with the pitch angle and recorded in the memory unit 312, and the feature (shape) of the left and right edges of the mask and the feature (shape) of the face center line in a newly acquired image may be compared, and the pitch angle associated with the one with the highest similarity may be obtained and determined to be the pitch angle of the face F appearing in the image.

[0102] [4.2. Variation 2] In the above embodiment, the camera C and the irradiation unit L are assumed to face the target from approximately the same direction, but the present invention can also be applied to cases where the camera C and the irradiation unit L face the target from different directions. In other words, even if the camera C and the irradiation unit L are located in different directions as seen from the target, if the position of the irradiation unit L is determined and it is known in advance how light is irradiated onto the face when wearing a mask and how an image is captured, it is possible to estimate the face direction by determining the mouth area, determining whether the mask is being worn, and detecting mask features and facial features, as in the above embodiment.

[0103] FIG. 15 shows an example of an image captured with the camera C and the illuminating unit L positioned in different directions relative to the subject and illuminating the face from the side. When illuminating the face from the side, the center line 601 of the face can be detected when the face is facing toward the front of the camera. When the face faces in the direction of the light, the facial contour, mask features 602 facing the light, and the front face area can be detected. These can be detected and similarly determined for every 10 degrees of yaw angle. However, when illuminating the face from the side, if the face faces in the opposite direction to the light illumination, the mask features and front face features due to the light illumination cannot be detected. Therefore, if it is desired to detect a wider range of face orientation angles, it is desirable to install the camera C and the illuminating unit L in the same direction relative to the face, as in the above example. Depending on the application and environment, the face direction may be determined by illuminating the face from the side and detecting mask features 602, etc.

[0104] In this way, even if the camera C and the irradiation unit L are positioned in different directions from the subject, mask features and frontal facial features can be clearly detected. Therefore, even if the installation conditions of the camera C and the irradiation unit L require the irradiation unit L to be positioned away from the camera C so that light is irradiated from the side of the face, or even if dominant light (e.g., sunlight) is irradiated from the side of the camera C, the face orientation can be determined using similar processing.

[0105] [4.3. Variation 3] When determining the mouth area or the face direction angle, the results of determinations made on previously captured images may be used for the determination. For example, once the mouth area is estimated, a process may be considered in which the mouth area is searched for around the previously estimated area in an image captured at a subsequent time. Furthermore, when determining the face direction, for example, if the face direction is determined to be 10 degrees up and 20 degrees left, an image captured at a subsequent time is likely to be around the previously estimated face direction. Therefore, a process may be considered in which the features of a face direction of 20 degrees up and 20 degrees left, 10 degrees up and 10 degrees left, 10 degrees up and 30 degrees left, and 20 degrees left are preferentially detected. In other words, assuming that the face direction does not change significantly between images captured at similar times, when determining a new face direction, the determination process may be performed from a direction close to the most recently determined face direction. By processing data in chronological order and effectively performing the determination process as in this modified example, the processing load for determining the mouth area or face direction angle can be reduced.

[0106] [4.4. Variation 4] The inventors of the present application have found that for many masks, the mask area has a higher brightness value than the skin in an infrared image. However, depending on the material of the mask, for example, if a material that absorbs infrared rays is used, the mask area may appear darker (lower brightness value) than the skin in the infrared image. In such cases, the mask may be detected by determining that the brightness value of the mouth area is lower than that of the skin area. In this case, after the mask is detected, the process of detecting the mask edge can be performed in the same manner as in the above-described embodiment. [Explanation of symbols]

[0107] 1 recognition device 111A Acquisition method 111B Detection means 111C Recognition means 111D Storage control means 111E Storage means D Facial orientation recognition device D 311 Control Unit 312 Storage section 313 Communications Department 314 Display section 315 Operation section

Claims

1. an acquisition means for acquiring an image of an object covered with a covering member while being irradiated with light; a detection means for detecting a center line in a front vertical direction on the surface of the object based on the brightness values ​​of pixels included in the image; recognition means for recognizing the vertical orientation of the object based on the inclination of the center line detected by the detection means; A recognition device comprising:

2. 2. The recognition device according to claim 1, the detection means further detects an upper boundary that is an upper boundary of the covering member on the object when the recognition means recognizes the vertical orientation of the object; a storage control unit that stores the detected inclination of the upper boundary in a storage unit in association with the vertical orientation of the object recognized by the recognition unit; When the acquisition means acquires a new image containing an object whose vertical orientation has not been recognized, the recognition means recognizes the vertical orientation associated with approximately the same inclination as the inclination of the upper boundary of the covering member detected in the image as the vertical orientation of the object in the image.

3. 2. The recognition device according to claim 1, The detection means further detects an upper boundary that is an upper boundary of the covering member, The recognition means is a recognition device that recognizes the lateral orientation of the object based on the position of the intersection between the upper boundary and the center line in the lateral width of the object.

4. 4. The recognition device according to claim 1, wherein: the detection means further detects a left or right boundary of the covering member, the left or right boundary being a boundary of at least one of the left and right sides of the covering member; The recognition device is configured such that, when the detection means detects the left and right boundaries, the recognition means recognizes the lateral orientation of the object based on the left and right boundaries detected by the detection means and the center line.

5. A recognition method using a recognition device, an acquisition step of acquiring an image of the object covered by the covering member while being irradiated with light; a detection step of detecting a center line in a front vertical direction on the surface of the object based on the brightness values ​​of pixels included in the image; a recognition step of recognizing a vertical orientation of the object based on the inclination of the center line detected in the detection step; Recognition methods including.

6. Computer, an acquisition means for acquiring an image of an object covered by a covering member while being irradiated with light; a detection means for detecting a center line in a front vertical direction on the surface of the object based on the brightness values ​​of pixels included in the image; recognition means for recognizing the vertical orientation of the object based on the inclination of the center line detected by the detection means; A recognition program that acts as a

Images