Facial image learning device, facial image recognition device, and their programs

Abstract

To provide a face image recognition device which can easily perform learning and can precisely recognize a person who is wearing an attachment.SOLUTION: A face image recognition device 3 includes: a conversion unit 30 that converts to-be-processed regions of a registration face image 9B and a recognition face image 9C to a state having few image features; a registration unit 31 that obtains a feature vector of the registration face image 9B by inputting the registration face image 9B into a deep learning model and registers the obtained feature vector of the registration face image 9B for each person; and a recognition unit 32 that obtains a feature vector of the recognition face image 9C by inputting the recognition face image 9C into the deep learning model and recognizes a person on the basis of a comparison result of the obtained feature vector of the recognition face image 9C with the feature vector of each person registered by the registration unit 31.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a facial image learning device, a facial image recognition device, and a program therefor. [Background technology]

[0002] In the current COVID-19 pandemic, it is not uncommon for people to have parts of their faces obscured by accessories such as masks and sunglasses. Conventional image recognition devices experience reduced recognition accuracy when parts of the face are obscured by accessories in input or registered facial images.

[0003] To address this issue, several conventional technologies have been proposed (for example, Patent Documents 1 and 2 and Non-Patent Document 1). The technology described in Patent Document 1 determines whether or not an attachment is blocking the face in a facial image. If an attachment is present, the technology described in Patent Document 1 acquires height information about the attachment and creates a three-dimensional facial model based on the height information. The technology described in Patent Document 1 then reconstructs the facial image of the blocked portion from this three-dimensional facial model and applies facial image recognition processing.

[0004] The technology described in Patent Document 2 divides a facial image into multiple small regions, compares them with an image showing a standard face, and determines regions associated with accessories such as sunglasses, a mask, etc. The technology described in Patent Document 2 then performs facial image recognition based on occluded regions excluding the areas of accessories, thereby suppressing a decrease in accuracy.

[0005] The technology described in Non-Patent Document 1 generates a database of masked face images from an existing large-scale face image database by combining a mask image with an unmasked face image in accordance with the orientation of the face. The technology described in Non-Patent Document 1 makes it easy to build a large-scale database of masked face images, and can use this database to accurately recognize face images when wearing a mask. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2018-163481 [Patent Document 2] Japanese Patent Application Publication No. 2020-38731 [Non-patent literature]

[0007] [Non-Patent Document 1] Aqeel Anwar, Arijit Raychowdhury, “Masked Face Recognition for Secure Authentication,” arXiv:2008.22204v1, 2020 Summary of the Invention [Problem to be solved by the invention]

[0008] However, the technology described in Patent Document 1 is based on the premise that facial images of the same person without any clothing are acquired during face recognition. However, since it is difficult to acquire facial images without any clothing when wearing a mask on a daily basis during the COVID-19 pandemic, recognition accuracy is reduced.

[0009] The technology described in Patent Document 2 is premised on maintaining the accuracy of determining occluded areas, and there is a possibility that poor extraction of occluded areas will lead to a decrease in recognition accuracy. Furthermore, the technology described in Patent Document 2 uses feature amounts obtained by learning face images without occluded areas, and these feature amounts are not optimal for face areas excluding occluded areas, which may result in a decrease in recognition accuracy.

[0010] The technology described in Non-Patent Document 1 is based on the premise of using a composite image in which a mask image is combined with a face image, and the recognition accuracy may be affected by the accuracy of the composite image and the type of mask image to be combined. Furthermore, the technology described in Non-Patent Document 1 has the problem that, because a mask image is combined, it requires a lot of effort for learning.

[0011] Therefore, an object of the present invention is to provide a facial image learning device, a facial image recognition device, and programs therefor that are easy to learn and can accurately recognize a person wearing an attachment. [Means for solving the problem]

[0012] In order to solve the above problem, the facial image learning device of the present invention is a facial image learning device that uses training facial images in which the face is not occluded by clothing to generate a deep learning model for recognizing a person from recognition facial images in which part of the face is occluded by clothing, and is configured to include a conversion unit and a learning unit.

[0013] According to this configuration, the conversion unit pre-defines a processing target area where the presence of an attachment is expected, and converts the processing target area of ​​the learning face image into a state with fewer image features. The learning unit generates a deep learning model by performing deep learning on the learning face images converted by the conversion unit.

[0014] According to this configuration, the conversion unit pre-defines a processing target area where the presence of an attachment is expected, and converts the processing target area of ​​the learning face image into a state with fewer image features. The learning unit generates a deep learning model by performing deep learning on the learning face images converted by the conversion unit. The conversion unit replaces pixels included in the processing target area of ​​the learning face image with the same pixel value or random pixel values.

[0015] In addition, in order to solve the above problem, the facial image recognition device of the present invention is a facial image recognition device that uses a deep learning model generated by the facial image learning device to recognize a person from a recognition facial image in which part of the face is occluded by an attachment, and is configured to include a conversion unit, a registration unit, and a recognition unit.

[0016] According to this configuration, the conversion unit pre-sets a processing target area where the presence of an attached object is expected, and converts the processing target areas of the registration face image and the recognition face image in which the face is not obscured by the attached object into a state with fewer image features. The registration unit inputs the registration face image converted by the conversion unit into a deep learning model to obtain a feature vector of the registration face image, and registers the obtained feature vector of the registration face image for each person. The recognition unit acquires a feature vector of the recognition face image by inputting the recognition face image converted by the conversion unit into a deep learning model, and recognizes a person based on the comparison result between the feature vector of the acquired recognition face image and the feature vector of each person registered in the registration unit.

[0017] According to this configuration, the conversion unit pre-sets a processing target area where the presence of an attached object is expected, and converts the processing target areas of the registration face image and the recognition face image in which the face is not obscured by the attached object into a state with fewer image features. The registration unit inputs the registration face image converted by the conversion unit into a deep learning model to obtain a feature vector of the registration face image, and registers the obtained feature vector of the registration face image for each person. The recognition unit acquires a feature vector of the recognition face image by inputting the recognition face image converted by the conversion unit into a deep learning model, and recognizes a person based on the comparison result between the feature vector of the acquired recognition face image and the feature vector of each person registered in the registration unit. The conversion unit replaces pixels included in the processing target area of ​​the face image for registration and the face image for recognition with the same pixel value or with random pixel values.

[0018] The present invention can also be realized by a program for causing a computer to function as the face image learning device or face image recognition device described above. [Effects of the Invention]

[0019] According to the present invention, learning is easy and a person wearing an attachment can be recognized with high accuracy. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a block diagram showing a configuration of a face recognition device according to an embodiment. [Figure 2]In an embodiment, (a) is an explanatory diagram illustrating an example of a face image for learning, (b) is an explanatory diagram illustrating an example of a face image for registration, and (c) is an explanatory diagram illustrating an example of a face image for recognition. [Figure 3] 1A to 1C are explanatory diagrams illustrating the rotation of a face image for learning in an embodiment. [Figure 4] 1A to 1C are explanatory diagrams illustrating normalization of face images for learning in an embodiment. [Figure 5] 10A and 10B are explanatory diagrams illustrating interpolation of learning face images in an embodiment. [Figure 6] 10A and 10B are explanatory diagrams illustrating replacement of a processing target region with the same pixel value in an embodiment. [Figure 7] FIG. 1A is an explanatory diagram illustrating replacement of a processing target region with random pixel values, and FIG. 1B is an explanatory diagram illustrating deletion of a processing target region, in an embodiment. [Figure 8] 1A to 1D are explanatory diagrams illustrating slides of face images for learning in an embodiment. [Figure 9] 10A and 10B are explanatory diagrams illustrating expansion and contraction of a learning face image in an embodiment. [Figure 10] FIG. 10 is an explanatory diagram illustrating setting of a small processing target region in the embodiment. [Figure 11] In the embodiment, (a) and (b) are explanatory diagrams illustrating an example of a deep learning model. [Figure 12] FIG. 2 is an explanatory diagram illustrating learning, registration, and recognition for each pattern in the embodiment. [Figure 13] 4 is a flowchart illustrating an operation of the face recognition device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0021] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, each embodiment described below is intended to embody the technical concept of the present invention, and unless otherwise specified, the present invention is not limited to the following. Furthermore, the same means will be given the same reference numerals, and their description may be omitted.

[0022] [Facial Recognition Device Overview] Hereinafter, an overview of a face recognition device 1 according to an embodiment will be described with reference to FIG. As shown in Fig. 1, face recognition device 1 recognizes people and includes a face image learning device 2, a face image recognition device 3, and a storage unit 4. Furthermore, face recognition device 1 receives three types of face images as input: a face image for learning 9A, a face image for registration 9B, and a face image for recognition 9C.

[0023] The training face image 9A is a face image to be used for generating a deep learning model, which will be described later. For example, the training face image 9A may be acquired from a training face image database (not shown) that is composed of face images of various people. As shown in FIG. 2(a), the training face image 9A is a face image in which the face is not occluded by an attachment 90 such as a mask or sunglasses.

[0024] The registration face image 9B is a face image for registering a person to be recognized in the face image recognition device 3. This registration face image 9B is made up of a plurality of face images collected for each person to be identified. As shown in FIG. 2(b), the registration face image 9B is a face image in which the face is not covered by an attachment 90.

[0025] The face image for recognition 9C is a face image of a person to be recognized. As shown in Fig. 2(c), the face image for recognition 9C is a face image in which the face is covered by an attachment 90. In this embodiment, it is assumed that a part of the lower region of the face is covered by a mask serving as the attachment 90. It is assumed that the face area and the positions of both eyes of the learning face image 9A, the registration face image 9B, and the recognition face image 9C can be detected by a known face area detection process.

[0026] First, in the face recognition device 1, the face image learning device 2 generates a deep learning model using a learning face image 9A. Next, in the face recognition device 1, the face image recognition device 3 uses a registration face image 9B to register a person to be recognized by linking the person to identification information. Then, in the face recognition device 1, the face image recognition device 3 recognizes the person using a recognition face image 9C.

[0027] [Configuration of the facial image learning device] The configuration of the face image learning device 2 will be specifically described with reference to FIG. The facial image learning device 2 uses a training facial image 9A in which the face is not occluded by an attachment 90 to generate a deep learning model for recognizing a person from a recognition facial image 9C in which part of the face is occluded by an attachment 90. As shown in FIG. 1 , the facial image learning device 2 includes a conversion unit 20 and a learning unit 21.

[0028] The conversion unit 20 preliminarily sets a processing target area where the attachment 90 is expected to be present, and converts the processing target area of ​​the training face image 9A into a state with fewer image features. Then, the conversion unit 20 outputs the training face image 9A with the converted processing target area to the learning unit 21. Here, in order to generate a highly accurate deep learning model, the conversion unit 20 may convert the processing target area of ​​the training face image 9A after performing preprocessing as described below on the training face image 9A.

[0029] <Preprocessing of face images for training> First, the pre-processing of the learning face image 9A by the conversion unit 20 will be described. 3(a) to detect the positions of both eyes from the learning face image 9 A. For example, the method described in Reference 1 can be used as the face area detection process. Reference 1: OpenCV, [online], [searched September 21, 2021], Internet<URL:https: / / opencv.org / >

[0030] The conversion unit 20 may perform rotation correction on the learning face image 9A based on the detected positions of both eyes. Specifically, as shown in FIG. 3(b), the conversion unit 20 performs rotation correction on the learning face image 9A based on the detected positions of the right eye E R and left eye position E L A line F passing through LR is horizontal line F H The learning face image 9A is rotated so that it coincides with the face image 9A shown in Fig. 3(c) .

[0031] The conversion unit 20 may also normalize the face image 9A for learning after the rotation correction. Specifically, as shown in FIG. 4(a), the conversion unit 20 normalizes the face image 9A for learning after the rotation correction. R and left eye position E L and length F E The training face image 9A is expanded or contracted so that the length F is a certain ratio to the width W of the training face image 9A. E 4(b), the conversion unit 20 is positioned at the midpoint E C may be aligned with the center of the training face image 9A, and the training face image 9A may be transformed into a square (aspect ratio 1:1). Fig. 4(c) shows the training face image 9A after normalization.

[0032] As shown in FIG. 5(a), in an original image 9D before the learning face image 9A is cut out, a facial region may be located at the edge, resulting in missing pixel regions required for the learning face image 9A. Note that in FIG. 5(a), for ease of explanation, the missing pixel regions in the original image 9D are indicated by dashed lines. In this case, the conversion unit 20 may interpolate the missing pixel regions with a fixed pixel value. For example, as shown in FIG. 5(b), the conversion unit 20 interpolates the missing pixel regions in the learning face image 9A with a pixel value of 0 (black). The conversion unit 20 does not necessarily have to perform these pre-processing steps on the learning face image 9A, but may perform them as needed.

[0033] <Conversion of processing area> Next, the conversion of the processing target region by the conversion unit 20 will be described. The conversion unit 20 sets in advance a processing target area in the training face image 9A where the presence of an attachment 90 is expected. Here, the user of the face recognition device 1 manually sets the processing target area by operating a mouse or keyboard (not shown). As shown in FIG. 6(a), if the attachment 90 is expected to be a mask, a portion of the lower area of ​​the face that is obscured by the mask is set as the processing target area 91. If the attachment 90 is expected to be sunglasses, a portion of the upper area of ​​the face that is obscured by the sunglasses (not shown) may be set as the processing target area 91.

[0034] The conversion unit 20 converts the processing target region 91 of the training face image 9A into a state with fewer image features. Specifically, the conversion unit 20 replaces pixels included in the processing target region 91 of the training face image 9A with the same pixel value or random pixel values, or deletes the processing target region 91 of the training face image 9A. In this embodiment, the conversion unit 20 replaces pixels included in the processing target region 91 of the training face image 9A with pixel values ​​of 0 (black), as shown in FIG. 6(b). In this way, the processing target region 91 of the training face image 9A is converted into a state with fewer image features, so that facial features can be more effectively learned in regions other than the processing target region 91 in deep learning, which will be described later.

[0035] The conversion unit 20 may replace the pixel value of each pixel included in the processing target area 91 of the training face image 9A with a random pixel value calculated using pseudo-random numbers. Fig. 7(a) illustrates the training face image 9A in which each pixel of the processing target area 91 has been replaced with a random pixel value. Furthermore, when the conversion unit 20 deletes the processing target region 91 of the learning face image 9A, the learning face image 9A becomes a horizontally long image in which only the upper region of the face remains, as shown in FIG. 7(b).

[0036] Here, taking into consideration fluctuations in the position at which the attachment 90 is worn and inaccuracies in the face area and eye positions due to the face area detection process, the face images 9A may be converted into a plurality of patterns of training face images 9A so that different face areas fall outside the processing target area 91. In other words, the conversion unit 20 slides the training face images 9A by different amounts in the vertical direction, and converts the processing target areas 91 of the plurality of patterns of training face images 9A with different slide amounts into a state with fewer image features.

[0037] For example, the conversion unit 20 does not slide the learning face image 9A in Fig. 8(a), but replaces the processing target region 91 of the learning face image 9A1 with a pixel value of 0 (black) as shown in Fig. 8(b). Also, the conversion unit 20 slides the positions of both eyes of the learning face image 9A in Fig. 8(a) upward in the vertical direction, and replaces the pixel value of the learning face image 9A1 with a pixel value of 0 (black) as shown in Fig. 8(c). K 8(a) to replace the pixel value of the processing target region 91 with a pixel value of 0 (black) (where K is an integer that satisfies K≧1). In addition, the conversion unit 20 further slides the positions of both eyes of the learning face image 9A in FIG. 8(a) upward in the vertical direction, and replaces the learning face image 9A in FIG. 8(d) with a pixel value of 0 (black) (where K is an integer that satisfies K≧1). L The processing target area 91 is replaced with a pixel value of 0 (black) (where L is an integer satisfying L>K). Hereinafter, the training face image after the processing target area is converted may be referred to as a "converted training face image." In other words, one training face image 9A is converted into L patterns of converted training face images 9A1 to 9A2 with different slide amounts. L These facial images 9A1 to 9A L are images in which the processing target area 91 is arranged at the same position and the face area other than the processing target area 91 is different. The slide amount and the number of patterns L of the learning face image 9A can be set arbitrarily.

[0038] In this way, the conversion unit 20 uses the normalized training face image 9A as a base and slides the positions of both eyes of the base training face image 9A upward in the vertical direction to generate new converted training face images 9A1 to 9A. L and generate new training face images 9A1 to 9A LTherefore, the facial image learning device 2 converts the processing target region 91 of the facial images 9A1 to 9A2 for transformation learning, which contain many facial regions that represent the likeness of the person. L This allows deep learning to be performed, further improving recognition accuracy.

[0039] As shown in FIG. 9(a), when cutting out the training face image 9A, the pixel area of ​​the original image 9D may be insufficient, resulting in the training face image 9A being horizontally elongated. In FIG. 9(a), the insufficient pixel area in the original image 9D is indicated by a dashed line. In this case, the conversion unit 20 scales the processing target area 91 so that the converted training face image 9A has a predetermined aspect ratio, and then converts the processing target area 91 of the training face image 9A scaled to the predetermined aspect ratio into a state with fewer image features. For example, as shown in FIG. 9(b), the conversion unit 20 scales the training face image 9A so that the converted training face image 9A becomes a square (aspect ratio 1:1).

[0040] The processing target area 91 can be set to any shape and size. For example, as shown in Fig. 10, the conversion unit 20 may set the processing target area 91 to a smaller size than that shown in Fig. 6. In this case, the conversion unit 20 can also slide the learning face image 9A in the vertical direction by different amounts, as in Fig. 8.

[0041] Returning to FIG. 1, the description of the face image learning device 2 will continue. The learning unit 21 generates a deep learning model by performing deep learning on the learning face image 9A converted by the conversion unit 20. Here, the learning unit 21 can use a general deep learning model such as a convolutional neural network.

[0042] In this embodiment, as shown in FIG. 11(a), the learning unit 21 uses ResNeXt, a type of convolutional neural network, as the deep learning model 210. For example, the deep learning model 210 includes an input layer 211, an intermediate layer (fully connected layer) 212, and an output layer 213. For example, the output layer 213 has 12883 dimensions, and the fully connected layer 212A immediately preceding the output layer 213 has 2048 dimensions. In this case, the deep learning model 210 can use the feature map of the 2048-dimensional intermediate layer 212 as a feature vector. Furthermore, as shown in FIG. 11(b), the deep learning model 210 can reduce computational costs by using a small number of dimensions. Therefore, a fully connected layer 214 having fewer dimensions than the fully connected layer 212A may be inserted between the output layer 213 and the preceding fully connected layer 212A. For example, the fully connected layer 214 has 256 dimensions, and the feature map of the fully connected layer 214 is used as a feature vector.

[0043] Here, the learning unit 21 generates a deep learning model 210 for each pattern of the training face image 9A in which the person depicted is known in advance. That is, the learning unit 21 generates a deep learning model 210 for each pattern of the training face images 9A1 to 9A2 in which the person depicted is known in advance. L From each of the above, L deep learning models 210 are generated. Then, the learning unit 21 writes the generated deep learning models 210 into the storage unit 4.

[0044] <Actions and Effects> In this way, facial image learning device 2 converts processing target area 91, where the presence of attachment 90 such as a mask is expected, into a state with few image features, making learning easy and eliminating the need to prepare facial images of people wearing attachment 90 in order to generate deep learning model 210. Use of this deep learning model 210 reduces the difference in image features depending on whether attachment 90 is present or not, allowing people wearing attachment 90 to be recognized with high accuracy.

[0045] [Configuration of face image recognition device] Returning to FIG. 1, the configuration of the face image recognition device 3 will be specifically described. The facial image recognition device 3 uses the deep learning model 210 generated by the facial image learning device 2 to recognize a person from a recognition facial image 9C in which part of the face is occluded by an attachment 90. As shown in FIG. 1 , the facial image recognition device 3 includes a conversion unit 30, a registration unit 31, and a recognition unit 32.

[0046] The conversion unit 30 preliminarily sets a processing target area 91 where the attachment 90 is expected to be present, and converts the processing target area 91 of the registration face image 9B and the recognition face image 9C, where the face is not occluded by the attachment 90, into a state with fewer image features. Specifically, the conversion unit 30 replaces pixels included in the processing target area 91 of the registration face image 9B and the recognition face image 9C with the same pixel value or random pixel values, or deletes the processing target area 91 of the registration face image 9B and the recognition face image 9C. In this way, the processing target area 91 of the registration face image 9B and the recognition face image 9C is converted into a state with fewer image features.

[0047] Here, the conversion unit 30 may slide the registration face image 9B and the recognition face image 9C by different amounts in the vertical direction, and convert the processing target area 91 of multiple patterns of registration face images 9B and recognition face images 9C with different slide amounts into a state with fewer image features. For example, if there are P people, P registration face images 9B are input to the conversion unit 30 (where P is an integer that satisfies P≧1). In this case, similar to FIG. 8, the conversion unit 30 converts each of the P registration face images 9B into M patterns of registration face images 9B with different sliding amounts (where M is an integer that satisfies M>K). Also, for example, the conversion unit 30 converts one recognition-use face image 9C into N patterns of recognition-use face images 9C with different slide amounts (where N is an integer satisfying N>K), as in FIG.

[0048] 9, the conversion unit 30 may expand or contract the processing target area 91 so that the converted registration face image 9B and recognition face image 9C have a predetermined aspect ratio, and convert the processing target area 91 of the registration face image 9B and recognition face image 9C expanded or contracted to the predetermined aspect ratio into a state with fewer image features. Hereinafter, the registration face image after the conversion of the processing target area may be referred to as the "converted registration face image," and the recognition face image after the conversion of the processing target area may be referred to as the "converted recognition face image."

[0049] In this embodiment, when a person is registered, the conversion unit 30 receives a registration face image 9B as input, and converts the processing target area 91 of this registration face image 9B and outputs the converted registration face image 9B to the registration unit 31. When a person is recognized, the conversion unit 30 receives a recognition face image 9C as input, and converts the processing target area 91 of this recognition face image 9C and outputs the converted recognition face image 9C to the recognition unit 32. As described above, the conversion unit 30 performs the same processing as the conversion unit 20 except that it handles the registration face images 9B and the recognition face images 9C instead of the learning face images 9A, and therefore further explanation will be omitted.

[0050] The registration unit 31 acquires the feature vector of the registration face image 9B by inputting the registration face image 9B converted by the conversion unit 30 into the deep learning model 210, and registers the acquired feature vector of the registration face image 9B for each person.

[0051] In this embodiment, the registration unit 31 inputs a registration face image 9B of a certain person into the deep learning model 210 of the storage unit 4. Then, the feature vector of the registration face image 9B is output from the deep learning model 210 of the storage unit 4, and the registration unit 31 writes the feature vector of the registration face image 9B into the storage unit 4 in association with the identification information of each person. Here, the identification information refers to information (e.g., name or ID) that can uniquely identify each person. For example, a user of the face recognition device 1 manually inputs the identification information by operating a mouse or keyboard (not shown).

[0052] Here, the registration unit 31 may input the registration facial image converted by the conversion unit 30 into the deep learning model 210 generated for each pattern, thereby obtaining a feature vector of the registration facial image 9B for each person and pattern, and registering the obtained feature vector of the registration facial image 9B for each person and pattern (details will be described later).

[0053] The recognition unit 32 acquires the feature vector of the recognition face image 9C by inputting the recognition face image 9C converted by the conversion unit 30 into the deep learning model 210, and recognizes a person based on the comparison result between the feature vector of the acquired recognition face image 9C and the feature vector of each person registered by the registration unit 31.

[0054] In this embodiment, the recognition unit 32 inputs the recognition-use facial image 9C to the deep learning model 210 in the storage unit 4. Then, the recognition unit 32 outputs a feature vector of the recognition-use facial image 9C from the deep learning model 210 in the storage unit 4. The recognition unit 32 then compares the feature vector of the acquired recognition-use facial image 9C with the feature vector of the registration-use facial image 9B of each person stored in the storage unit 4, and recognizes which person's face is included in the recognition-use facial image 9C. For example, the recognition unit 32 can perform the comparison using a common method, such as calculating the dot product of the feature vector of the recognition-use facial image 9C with the feature vector of the registration-use facial image 9B of each person. In this case, the recognition unit 32 outputs, as the recognition result, identification information associated with the feature vector with the largest dot product.

[0055] Here, the recognition unit 32 obtains the feature vector of the recognition face image 9C for each pattern by inputting the recognition face image 9C converted by the conversion unit 30 into the deep learning model generated for each pattern, calculates the inner product of the combination of the feature vectors of the registration face image 9B and the recognition face image 9C, and may determine the person with the largest calculated inner product as the recognition result (details will be described later).

[0056] The storage unit 4 is a general storage device such as a memory, HDD (Hard Disk Drive), or SSD (Solid State Drive) that stores the deep learning model 210 and the feature vectors of registered people.

[0057] <Pattern-by-pattern learning, registration and recognition> Referring to FIG. 12, learning, registration and recognition for each pattern will be described. As shown in FIG. 12, the conversion unit 20 converts one training face image 9A into L patterns of converted training face images 9A1, . . . , 9A K ,…,9A L Then, the learning unit 21 generates a deep learning model 210 of the L pattern. The conversion unit 30 converts one registration face image 9B for each person into M patterns of conversion registration face images 9B1, . . . , 9B K ,…,9B M When there are P people, the registration unit 31 converts P×M patterns of conversion registration face images 9B1, ..., 9B K ,…,9B M and registers P×L×M feature vectors obtained from the deep learning model 210.

[0058] Furthermore, the conversion unit 30 converts one face image for recognition 9C into N patterns of converted face images for recognition 9C1, . . . , 9C K ,…,9C N In this case, the recognition unit 32 converts the L-pattern deep learning model 210 into N-pattern converted recognition face images 9C1, ..., 9C K ,…,9C N is input, and the feature vectors of the L×N facial images for recognition 9C are obtained from the deep learning model 210. Then, the recognition unit 32 calculates the dot product of the feature vectors of the P×L×M facial images for registration 9B and the feature vectors of the L×N facial images for recognition 9C. As shown in FIG. 12, there are P×L×M×L×N possible combinations of feature vectors. Then, the recognition unit 32 calculates the dot product of the feature vectors of these combinations, and outputs the person for whom the calculated dot product is largest as the recognition result.

[0059] The registration unit 31 may narrow down the P×L×M combinations and register the feature vectors, taking into consideration the balance between calculation cost and accuracy. The recognition unit 32 may narrow down the P×L×M×L×N combinations and calculate the dot products of the feature vectors, taking into consideration the balance between calculation cost and accuracy.

[0060] <Actions and Effects> In this way, the facial image recognition device 3 converts the processing target area 91, where the presence of an attachment 90 such as a mask is expected, into a state with fewer image features, and uses the deep learning model 210, so that the difference in image features depending on whether or not an attachment is present is reduced, and a person wearing the attachment 90 can be recognized with high accuracy.

[0061] [Facial recognition processing] The processing of the face recognition device 1 will be described with reference to FIG. 13, in step S1, the conversion unit 20 preliminarily sets a processing target area 91 and converts the processing target area 91 of the learning face image 9A to a state with fewer image features. Also, the conversion unit 30 preliminarily sets a processing target area 91 and converts the processing target areas 91 of the registration face image 9B and the recognition face image 9C to a state with fewer image features.

[0062] In step S2, the learning unit 21 performs deep learning on the training face image 9A converted in step S1 to generate a deep learning model 210. For example, the learning unit 21 uses ResNeXt, which is one of convolutional neural networks, as the deep learning model 210.

[0063] In step S3, the registration unit 31 obtains the feature vector of the registration face image 9B by inputting the registration face image 9B converted in step S1 into the deep learning model 210, and registers the obtained feature vector of the registration face image 9B for each person.

[0064] In step S4, the recognition unit 32 acquires a feature vector of the recognition face image 9C by inputting the recognition face image 9C converted by the conversion unit 30 into the deep learning model 210. Then, the recognition unit 32 recognizes a person based on a comparison result between the feature vector of the acquired recognition face image 9C and the feature vector of each person registered by the registration unit 31.

[0065] Although the embodiments have been described in detail above, the present invention is not limited to the above-described embodiments, and includes design modifications and the like within the scope of the present invention. In the above embodiment, the aspect ratio of the learning face image, the registration face image, and the recognition face image is 1:1, but this is not limiting.

[0066] In the above-described embodiment, the facial image learning device and the facial image recognition device are each described as having a conversion unit, but the two conversion units may be integrated. Also, in the above-described embodiment, the facial image learning device and the facial image recognition device are described as being separate devices, but the two devices may be integrated. Also, in the above-described embodiment, the memory unit is described as being a device independent from the facial image learning device and the facial image recognition device, but the facial image learning device or the facial image recognition device may be provided with a memory unit.

[0067] In the above embodiment, it has been explained that one person has one face image for registration, but one person may have two or more face images for registration. In this case, multiple feature vectors of the face images for registration are registered for one person, and recognition is performed using each feature vector.

[0068] In the above embodiment, the convolutional neural network is described as being ResNeXt, but is not limited to this. For example, convolutional neural networks such as LeNet, AlexNet, and VGG can be used.

[0069] In the above-described embodiment, the facial image learning device and the facial image recognition device are described as independent pieces of hardware, but the present invention is not limited to this. For example, the present invention can also be realized by a program that causes hardware resources such as a CPU, memory, and hard disk of a computer to function as the facial image learning device or the facial image recognition device. This program may be distributed via a communication line or written to a recording medium such as a CD-ROM or flash memory. [Explanation of symbols]

[0070] 1. Facial recognition device 2. Facial image learning device 3. Facial image recognition device 4 Storage section 20 Conversion unit 21 Learning Department 30 Conversion unit 31 Registration Department 32 Recognition part

Claims

1. A facial image learning device that generates a deep learning model for recognizing a person from a face image for recognition in which a part of the face is occluded by an attachment, using a face image for training in which the face is not occluded by an attachment, comprising: a conversion unit that preliminarily sets a processing target area where the attachment is expected to be present, and converts the processing target area of ​​the learning face image into a state with fewer image features; a learning unit that generates the deep learning model by deep learning the training face image converted by the conversion unit; Equipped with The facial image learning device is characterized in that the conversion unit replaces pixels included in the processing target area of ​​the learning facial image with the same pixel value or random pixel values.

2. The facial image learning device according to claim 1, characterized in that the conversion unit expands or contracts the processing target area so that the converted training facial image has a predetermined aspect ratio, and converts the processing target area of ​​the training facial image expanded or contracted to the predetermined aspect ratio into a state with fewer image features.

3. the conversion unit vertically slides the learning face images by different amounts, and converts the processing target areas of the learning face images of a plurality of patterns with different slide amounts into a state with fewer image features; 3. The facial image learning device according to claim 1, wherein the learning unit generates the deep learning model for each pattern of the training facial image.

4. A facial image recognition device that recognizes a person from a recognition facial image in which a part of the face is occluded by an attachment, using a deep learning model generated by the facial image learning device according to claim 1, a conversion unit that preliminarily sets a processing target area where the attachment is expected to be present, and converts the processing target area of ​​the face image for registration and the face image for recognition, in which the face is not occluded by the attachment, into a state with fewer image features; a registration unit that inputs the facial image for registration converted by the conversion unit into the deep learning model to acquire a feature vector of the facial image for registration, and registers the acquired feature vector of the facial image for registration for each person; a recognition unit that acquires a feature vector of the face image for recognition by inputting the face image for recognition converted by the conversion unit into the deep learning model, and recognizes the person based on a comparison result between the feature vector of the acquired face image for recognition and the feature vector for each person registered by the registration unit; Equipped with The facial image recognition device is characterized in that the conversion unit replaces pixels included in the processing target areas of the facial image for registration and the facial image for recognition with the same pixel value or random pixel values.

5. 5. The facial image recognition device according to claim 4, wherein the conversion unit expands or contracts the processing target area so that the converted face image for registration and the face image for recognition have a predetermined aspect ratio, and converts the processing target area of ​​the face image for registration and the face image for recognition that have been expanded or contracted to the predetermined aspect ratio into a state with fewer image features.

6. the conversion unit vertically slides the registration face image and the recognition face image by different amounts, and converts processing target areas of the registration face image and the recognition face image of a plurality of patterns with different slide amounts into a state with fewer image features; the registration unit inputs the facial image for registration converted by the conversion unit into the deep learning model generated for each pattern according to claim 3, thereby acquiring a feature vector of the facial image for registration for each person and each pattern, and registering the acquired feature vector of the facial image for registration for each person and each pattern; The facial image recognition device according to claim 4 or 5, characterized in that the recognition unit obtains a feature vector of the facial image for recognition for each pattern by inputting the facial image for recognition converted by the conversion unit into a deep learning model generated for each pattern, calculates an inner product by combining the feature vectors of the facial image for registration and the facial image for recognition, and recognizes the person for whom the calculated inner product is largest.

7. A program for causing a computer to function as the facial image learning device according to any one of claims 1 to 3.

8. A program for causing a computer to function as the face image recognition device according to any one of claims 4 to 6.

Images