Image processing device, image processing method and program for image processing device

Abstract

To identify an object with high accuracy while reducing the amount of data of a feature vector even when part of the object is hidden.SOLUTION: An image processing apparatus comprises: calculation means that calculates a feature vector from an object of an input image; and identification means that performs identification of the object by using the feature vector. The feature vector is in an M-dimensional first format. A feature vector in an N-dimensional (N<M) second format being part of the feature vector in the M-dimensional first format is learned to be able to identify the object in which a particular part of the object is hidden.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to techniques for identifying objects in images. [Background technology]

[0002] In Patent Document 1, when a person is registered, both the feature vector of the entire face and the feature vector of the area not covered by clothing are acquired and registered. Then, when identifying the person, the presence or absence of clothing on the person in the input person image is determined, and based on the determination result, the feature vector of the entire face or the area not covered by clothing is acquired, and the feature vectors of the entire face or the area not covered by clothing are compared. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2007-280250 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the technology of Patent Document 1, when registering a person, it is necessary to acquire and register both the feature vector of the entire face and the feature vector of the area not covered by clothing, which results in a large amount of data to be stored.The present disclosure aims to enable high-accuracy object identification while suppressing the amount of feature vector data even when part of the object is hidden. [Means for solving the problem]

[0005] The image processing apparatus includes a calculation means for calculating a feature vector of a first format of M dimensions from an object of an input image, an identification means for identifying the object using the feature vector of the first format of M dimensions or a feature vector of a second format of N dimensions (N < M) which is a part of the feature vector of the first format of M dimensions, and a first detection means for detecting whether a specific part of the object is hidden, and the calculation means calculates the feature vector of the second format of N dimensions representing the features of the object in which the specific part is hidden. Learned to When it is detected by the first detection means that the specific part is not hidden, the identification means performs identification using the feature vector of the first format of M dimensions, and when it is detected by the first detection means that the specific part is hidden, the identification means performs identification using the feature vector of the second format of N dimensions.

Effect of the Invention

[0006] Even when a part of the object is hidden, the object can be identified with high accuracy while suppressing the data amount of the feature vector.

Brief Description of the Drawings

[0007] [Figure 1] It is a diagram showing an example of the hardware configuration of a monitoring system. [Figure 2] It is a diagram showing an example of the functional configuration of an image processing apparatus. [Figure 3] It is a diagram showing the data flow of an image processing apparatus. [Figure 4] It is a flowchart showing an example of an image processing method of an image processing apparatus. [Figure 5] It is a diagram showing an example of the configuration of a learning system for performing learning of a feature vector calculation unit. [Figure 6] It is a diagram showing an example of the configuration of a learning system for performing learning of a feature vector calculation unit. [Figure 7] It is a diagram showing an example of the configuration of a learning system for performing learning of a feature vector calculation unit. [Figure 8]FIG. 10 is a diagram illustrating an example of the configuration of a learning system that performs learning of a feature vector calculation unit. DETAILED DESCRIPTION OF THE INVENTION

[0008] (First embodiment) In recent years, the AI ​​field has seen an increase in products that use deep learning algorithms for object recognition, especially person and face recognition. For example, there are surveillance camera systems that display the identity of a person in the image from a pre-registered list, and access control systems that use facial recognition at entrance / exit gates to determine who is inside a facility.

[0009] In such a system, a feature vector is extracted from an input person image and compared with the feature vectors of many pre-registered people. The registered person with the most similar feature vector is then identified as the person in the input image (identification process). In the identification process, the similarity between each registered feature vector and the feature vector obtained from the input image is calculated (matching process).

[0010] In facial recognition devices for access control systems, people are usually registered without sunglasses, masks, or other accessories. However, at access gates and other locations, identification may be performed with the accessories on. In this case, the similarity between the feature vectors of the person without the accessories and the feature vectors of the person with the accessories is calculated. However, the characteristics of the area of ​​the accessories differ significantly, resulting in a low similarity and the person being identified as a different person. In contrast, when registering people with and without accessories, as in Patent Document 1, assuming that the feature vectors are the same size, the amount of data doubles. To ensure smooth identification, this data is expected to be stored in memory, which can be a significant difference when registering a large number of people. This embodiment reduces the amount of feature data to be registered in cases where accessories affect the comparison of feature vectors.

[0011] Fig. 1 is a diagram showing an example of the hardware configuration of a monitoring system according to the first embodiment. The monitoring system has an image processing device 100, an input unit 109, a monitor 110, a network 111, and an imaging device 112 as an imaging unit, and identifies people captured on the imaging device 112. Fig. 1 illustrates a case in which the monitoring system has three imaging devices 112, but the number of imaging devices 112 is not limited thereto. The image processing device 100 and the imaging devices 112 are connected to each other via the network 111.

[0012] In this embodiment, an example of human face recognition will be described. However, this embodiment is not limited to face recognition, and can also be applied to pet recognition such as dogs and cats, and general object recognition such as automobiles and bicycles.

[0013] The image processing device 100 is an object identification device, and includes a CPU 101, a ROM 102, a RAM 103, an external storage device 104, an input I / F 105, an output I / F 106, a communication I / F 107, and a system bus .

[0014] The CPU 101 is a central processing unit that controls the entire image processing device 100. The ROM 102 is a read-only memory that stores programs and parameters that do not require modification. The RAM 103 is a random access memory that temporarily stores programs and data supplied from an external storage device 104 or the like. The external storage device 104 is a storage device such as a hard disk or memory card that is fixedly installed in the image processing device 100. The external storage device 104 may include an optical disk such as a flexible disk (FD) or compact disk (CD), a magnetic or optical card, an IC card, or a memory card that is detachable from the image processing device 100. The functions and processes of the image processing device 100, which will be described later, are realized by the CPU 101 reading and executing programs stored in the ROM 102 or the external storage device 104.

[0015] The input I / F 105 is an interface with an input unit 109 such as a pointing device or keyboard that receives user operations and inputs data. The output device I / F 106 is an interface with a monitor 110 that displays data held by the image processing device 100 and supplied data. The communication I / F 107 is connected to a network 111 such as the Internet. The imaging device 112 is an imaging device that captures video, such as a surveillance camera, and is connected to the image processing device 100 via the network 111. The system bus 108 is a transmission path that connects each of the components 101 to 107 so that they can communicate with each other.

[0016] 2 is a diagram showing an example of the functional configuration of the image processing device 100 according to this embodiment. The image processing device 100 has an object detection unit 201, a feature vector calculation unit 202, an occlusion detection unit 203, a dimension selection unit 204, and a classification unit 205, and is realized by the CPU 101 executing a program.

[0017] 3 is a block diagram showing an example of a data flow of image processing performed by the image processing device 100 according to this embodiment. The imaging device 112 captures images to generate a surveillance video 301. The image processing device 100 acquires the surveillance video 301 generated by the imaging device 112 via the network 111 and the communication I / F 107. Here, the surveillance video 301 is assumed to be composed of images captured sequentially, but is not limited to this.

[0018] In the object identification process performed by the image processing device 100, the object detection unit 201 analyzes the acquired surveillance video 301 and extracts the face area of ​​a person appearing in the video. The method by which the object detection unit 201 extracts the face area is not limited. The object detection unit 201 can use, for example, a face detection method using a CNN (Convolutional Neural Network).

[0019] The feature vector calculation unit 202 calculates a feature vector 302 for a face region extracted by the object detection unit 201. The feature vector 302 is designed to accurately determine whether or not the face region is the same object. In face recognition, the feature vector 302 basically takes similar values ​​for the same person, regardless of facial expression, shooting direction, lighting, or the presence or absence of accessories. However, when the mouth or eyes, which are closely related to person identification, are covered by an accessory such as a mask or sunglasses, the authentication results are significantly affected by the characteristics of the mask or sunglasses themselves. Similarly, when a portion of a face is obscured by a pillar or ornament, or when a line of people is formed and the head of a person in front obscures part of the face of a person behind, the authentication results are similarly affected by occlusion. In this embodiment, to achieve authentication that is less susceptible to such influences, learning is performed so that features that identify an object when a specific portion of the object is obscured are concentrated in a specific dimension of the feature vector. In this embodiment, a case where a portion of the face is obscured by a mask is used as an example, but the present invention is not limited to masks. Furthermore, the method by which the feature vector calculation unit 202 calculates the feature vector of an object is not limited to the above method. An example of detailed learning for calculating feature vectors will be described later.

[0020] The occlusion detection unit 203 detects whether or not a mask is being worn for the face region extracted by the object detection unit 201. The method by which the occlusion detection unit 203 detects whether a mask is being worn is not limited. The occlusion detection unit 203 can use, for example, a mask detection method using CNN.

[0021] When the occlusion detection unit 203 detects that a mask is being worn, the dimension selection unit 204 selects the specific dimension described above from the feature vector 302, and generates a dimension-selected feature vector 303. Meanwhile, the dimension selection unit 204 acquires a registered feature vector 304 from the external storage device 104. Here, the registered feature vector 304 is a feature vector of the object to be identified. When the occlusion detection unit 203 detects that a mask is being worn, the dimension selection unit 204 selects a specific dimension from the registered feature vector 304, and generates a dimension-selected registered feature vector 305.

[0022] If the occlusion detection unit 203 does not detect the wearing of a mask, the identification unit 205 performs identification by comparing the feature vector 302 with the registered feature vector 304, and obtains an identification result 306. This identification result 306 is the registered feature vector 304 that has the highest similarity to the feature vector 302 among the registered feature vectors 304 and that has a similarity higher than a threshold value, as well as an image of the object and its name.

[0023] Furthermore, when the occlusion detection unit 203 detects that a mask is being worn, the identification unit 205 performs identification by comparing the dimension selected feature vector 303 with the dimension selected registered feature vector 305, and obtains an identification result 306. This identification result 306 is the dimension selected registered feature vector 305 that has the highest similarity to the dimension selected feature vector 303 among the dimension selected registered feature vectors 305 and that has a similarity higher than a threshold, as well as an image of the object and its name.

[0024] In this embodiment, the identification unit 205 performs identification using the selected feature vector in the step after the dimension selection unit 204 makes the selection, but it is also possible to calculate the similarity using (selecting) only a specific dimension during identification without dividing the steps.

[0025] The identification unit 205 then displays the identification result 306 on the monitor 110 via the output I / F 106. The identification unit 205 may not only display the face of the detected person, but may also alert the user of the detection by flashing part or the entire screen. Furthermore, the identification unit 205 may also alert the user by adding a warning light (not shown) to the monitoring system.

[0026] The external storage device 104 and the monitor 110 may be provided in a device separate from the image processing device 100, and the registered feature vector 304 and the classification result 306 may be transmitted and received via the communication I / F 107. The control program is not limited to being stored in the ROM 102, but may be stored in a separate device. For example, the object detection unit 201, the feature vector calculation unit 202, the occlusion detection unit 203, and the dimension selection unit 204 may be provided in the image capture device 112 or a separate server. The image processing device 100 may be configured to include only the dimension selection unit 204 and the classification unit 205. Such a configuration allows separate devices for video analysis processing and classification-related processing, allowing hardware specifications suitable for each process to be selected.

[0027] The above-described processing performed by the image processing device 100 will be described below with reference to a flowchart. The processing corresponding to each step of the flowchart described in this embodiment may be realized by software using the CPU 101, or may be realized by hardware such as an electronic circuit.

[0028] 4 is a flowchart showing an example of an image processing method of the image processing device 100 according to this embodiment. First, in step S401, the image processing device 100 acquires an image captured by the imaging device 112. Next, in step S402, the object detection unit 201 detects a face area from the image. Note that here, a flow when one face area is detected will be described. If multiple face areas are detected, the subsequent processes from step S403 to step S411 can be repeated, and a description thereof will be omitted here.

[0029] Next, in step S403, the feature vector calculation unit 202 calculates the feature vector 302 of the detected face region. Next, in step S404, the occlusion detection unit 203 detects whether or not the face region extracted by the object detection unit 201 is wearing a mask. Next, in step S405, the image processing device 100 acquires a plurality of registered feature vectors 304 from the external storage device 104.

[0030] Next, in step S406, if the occlusion detection unit 203 detects that a mask is being worn, the process proceeds to step S407, and if it detects that a mask is not being worn, the process proceeds to step S409.

[0031] In step S407, the dimension selection unit 204 selects a predetermined dimension of the feature vector 302 to obtain a dimension-selected feature vector 303. In step S408, the dimension selection unit 204 selects a predetermined dimension of the multiple registered feature vectors 304 to obtain multiple dimension-selected registered feature vectors 305.

[0032] Next, in step S409, if the occlusion detection unit 203 detects that a mask is being worn, the identification unit 205 compares the dimension selected feature vector 303 with multiple dimension selected registered feature vectors 305, calculates the similarity between them, and outputs the identification result 306. That is, the identification unit 205 identifies the dimension selected registered feature vector 305 with the highest similarity, and if the similarity is higher than a threshold, identifies the person in the video as the person in the dimension selected registered feature vector 305. If the above conditions are not met, the identification unit 205 determines that no person has been identified.

[0033] Furthermore, when the occlusion detection unit 203 detects that the subject is not wearing a mask, the identification unit 205 compares the feature vector 302 with a plurality of registered feature vectors 304, calculates the similarity between them, and outputs the identification result 306. That is, the identification unit 205 identifies the registered feature vector 304 with the highest similarity, and if the similarity is higher than a threshold, identifies the person appearing in the video as the person identified by the registered feature vector 304. If the above conditions are not met, the identification unit 205 determines that no person has been identified. Here, because the number of dimensions used in comparison differs when the dimension-selected feature vector 303 is used and when the feature vector 302 is used, different thresholds may be used.

[0034] Next, in step S410, if the identification unit 205 has identified a person, the process proceeds to step S411, and if the identification unit 205 has not identified a person, the process ends in Fig. 4. In step S411, the identification unit 205 displays the identification result 306 on the monitor 110 via the output I / F 106, and ends the process in Fig. 4.

[0035] 5(A) and 5(B) are diagrams showing an example of the configuration of a learning system that performs learning on the feature vector calculation unit 202 of the image processing device 100. In this embodiment, a learning method when the feature vector calculation unit 202 is configured as a deep neural network will be described. There are various methods for learning deep neural networks for object identification, but in this embodiment, the Siamese Network technique from the following document will be used as an example.

[0036] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Sackinger, Roopak Shah. “Signature Verification using a “Siamese” Time Delay Neural Network“. Advances in Neural Information Processing Systems. 1994; 6:737-744.

[0037] Figures 5(A) and (B) show examples of learning using image pairs. Figure 5(A) shows an example of learning system 5A when using image pair 51 (image 511 and image 512) in which no masks are worn. Images 511 and 512 are images in which no masks are worn. Figure 5(B) shows an example of learning system 5B when using image pair 55 (image 551 and image 552) in which one person is wearing a mask. Image 551 is an image in which no mask is worn. Image 552 is an image in which a mask is worn. Learning is performed by switching between learning system 5A in Figure 5(A) and learning system 5B in Figure 5(B) depending on whether the input image pair includes an image in which a mask is worn.

[0038] When using an image pair 51 (image 511 and image 512) without a mask, learning is performed on the entire feature vector and on a dimension-selected feature vector in which a predetermined dimension is selected from the entire feature vector. In other words, when no mask is worn, learning is performed so that both the entire feature vector and the dimension-selected feature vector in which a predetermined dimension is selected from the entire feature vector can be distinguished. This learning method is explained with reference to Figure 5(A).

[0039] 5(A), the DNN unit 52 has DNN 521 and DNN 522, which are the same deep neural network. The DNN 521 receives image 511 of the image pair 51 and calculates a feature vector 531. The DNN 522 receives image 512 of the image pair 51 and calculates a feature vector 532. The calculated pair of feature vectors 531 and 532 is a feature vector pair 53.

[0040] Here, feature vector 531 is an M-dimensional vector, part of which is N-dimensional feature vector 5311. Feature vector 532 is an M-dimensional vector, part of which is N-dimensional feature vector 5321. That is, feature vector 531 is a feature vector of the first type, and feature vector 5311 is a feature vector of the second type. Similarly, feature vector 532 is a feature vector of the first type, and feature vector 5321 is a feature vector of the second type.

[0041] Two loss functions, 541 and 542, are defined in the weight update unit 54. The loss function 541 uses the distance D between the feature vector 5311 and the feature vector 5321. The loss function 542 uses the distance D between the feature vector 531 and the feature vector 532. The distance D between the feature vectors uses the Euclidean distance. The following [Equation 1] is an example of the loss function 541 and the loss function 542 that use the contrastive loss function.

[0042]

number

[0043] Here, T is the label of the image pair 51, and takes the value 1 if they are the same person, and 0 if they are not. The margin is a hyperparameter that is adjusted through experience or as learning progresses. Here, margin = 1 is used as an example. The method of determining the margin value is widely known in the field of deep learning, but as mentioned above, it also depends on experience, so details will be omitted. L obtained in [Equation 1] is the value of loss function 541 or loss function 542.

[0044] Although Euclidean distance is used here as the distance D, other distance indices such as Manhattan distance may also be used. When using a similarity such as cosine similarity Sc, the similarity Sc may be converted into the distance D as in any of [Mathematical Expression 2].

[0045]

number

[0046] Next, the weight update unit 54 calculates an integrated loss function Ltotal when the loss function 541 of the feature vectors 5311 and 5321 is set to L1 and the loss function 542 of the feature vectors 531 and 532 is set to L2. The following [Equation 3] is an example of the integrated loss function Ltotal.

[0047]

number

[0048] Here, s is a parameter (weight) with 0≦s≦1. s does not need to be fixed throughout the learning period, and can be determined, for example, as follows: s is constant throughout the entire learning period. Also, in the early stages of learning, s is set to 1 or a value close to 1, and then s is gradually decreased to a predetermined value.

[0049] The weight update unit 54 uses the value of the loss function Ltotal by error propagation to update the weight for the DNN unit 52. Here, since the DNN 521 and the DNN 522 configure a Siamese Network, the weights of both are updated simultaneously.

[0050] The M-dimensional first-form feature vectors 531 and 532 are trained using a loss function Ltotal that is a weighted combination of loss functions L1 and L2. Loss function L1 is a loss function 541 for the N-dimensional second-form feature vectors 5311 and 5321. Loss function L2 is a loss function 542 for the M-dimensional first-form feature vectors 531 and 532.

[0051] Next, learning will be described for the case where an image pair 55 is used in which one of the face regions of the input images is wearing a mask. When an image pair 55 (image 551 and image 552) in which one of the face regions of the input images is wearing a mask is used, learning is performed on a dimension-selected feature vector in which a predetermined dimension is selected from all of the feature vectors. In other words, when at least one of the image pair 55 is wearing a mask, learning is performed so that the image can be identified using a dimension-selected feature vector in which a predetermined dimension is selected from all of the feature vectors. This learning method will be described with reference to FIG. 5(B). FIG. 5(B) shows the case of an image pair 55 in which only the face region of one of the images, image 552, is wearing a mask, but the same processing as in FIG. 5(B) is also performed for an image pair in which the face regions of both images are wearing masks.

[0052] 5(B), the DNN unit 52 is the same deep neural network as the DNN unit 52 in FIG. 5(A), and includes DNNs 521 and 522. The DNN 521 receives an image 551 from the image pair 55 and calculates a feature vector 531. The DNN 522 receives an image 552 from the image pair 55 in which a mask is worn and calculates a feature vector 532. The calculated pair of feature vectors 531 and 532 is the feature vector pair 53.

[0053] Here, the feature vector 531 is an M-dimensional vector, and a part of it is an N-dimensional feature vector 5311. The feature vector 532 is an M-dimensional vector, and a part of it is an N-dimensional feature vector 5321. That is, the feature vector 531 is a feature vector of the first form, and the feature vector 5311 is a feature vector of the second form. Similarly, the feature vector 532 is a feature vector of the first form, and the feature vector 5321 is a feature vector of the second form.

[0054] The weight update unit 54 uses only the loss function 541. The loss function 541 uses the distance D between the feature vector 5311 and the feature vector 5321, similar to the loss function 541 in FIG. 5(A). Then, the weight update unit 54 updates the weights of the DNN unit 52 using the loss function 541. The N-dimensional feature vectors 5311 and 5321 of the second form are learned using the loss function 541 of the N-dimensional feature vectors 5311 and 5321 of the second form.

[0055] The learning systems 5A and 5B perform learning such that features for identifying a face region when a specific part of the face region is hidden by a mask are concentrated in specific dimensions of a part of the feature vector by executing the learning in FIGS. 5(A) and 5(B).

[0056] Note that this embodiment is not limited to human face authentication and is applicable to object authentication. The object detection unit 201 detects an object from the input image. The feature vector calculation unit 202 calculates a feature vector 302 from the detected object. The feature vector is an M-dimensional feature vector of the first form. The N-dimensional (N < M) feature vector of the second form, which is a part of the M-dimensional feature vector of the first form, is a feature vector learned to be able to identify an object with a specific part of the object hidden. The object is, for example, a human face. The specific part of the object is, for example, hidden by a mask.

[0057] 5(A), the feature vector calculation unit 202 receives as input images 511 and 512 in which a specific portion of an object is not hidden, and learns M-dimensional first-type feature vectors 531 and 532. Also, as shown in FIG. 5(B), the feature vector calculation unit 202 receives as input at least one image 552 of multiple images 551 and 552 in which a specific portion of an object is hidden, and learns N-dimensional second-type feature vectors 5311 and 5321.

[0058] The occlusion detection unit 203 detects whether a specific part of an object in the input image is occluded. The dimension selection unit 204 selects an N dimension from the M-dimensional first-type feature vector 302 calculated by the feature vector calculation unit 202, and generates an N-dimensional second-type feature vector 303 that is a part of the M-dimensional first-type feature vector.

[0059] When the occlusion detection unit 203 detects that a specific portion is not occluded, the identification unit 205 performs identification using the M-dimensional first-type feature vector 302 calculated by the feature vector calculation unit 202. Furthermore, when the occlusion detection unit 203 detects that a specific portion is occluded, the identification unit 205 performs identification using the N-dimensional second-type feature vector 303 generated by the dimension selection unit 204.

[0060] As described above, in this embodiment, the learning systems 5A and 5B control the number of dimensions used in the loss function during learning by the feature vector calculation unit 202 depending on whether a mask is being worn. As a result, the learning systems 5A and 5B learn so that features for identifying a face when a mask is being worn are concentrated in some specific dimensions of the feature vector. Then, when performing identification or matching, the identification unit 205 switches between using the entire feature vector or some specific dimensions of the feature vector depending on whether a mask is being worn. As a result, the identification unit 205 can perform highly accurate identification and matching without having both feature vectors for when a mask is being worn and when a mask is not being worn.

[0061] (Second embodiment) In the first embodiment, when learning using the image pair 51 in which a mask is not worn, the learning system 5A performs learning using two loss functions 541 and 542. In the second embodiment, as shown in FIG. 6 , when learning using the image pair 51 in which a mask is not worn, the learning system 6 performs learning using only the overall loss function 542 of the feature vector. In the second embodiment, as in the first embodiment, the learning system 5B also performs learning so that features for identifying faces when a mask is worn are concentrated in some specific dimensions of the feature vector.

[0062] Fig. 6 shows an example of a learning system 6 of the feature vector calculation unit 202 in this embodiment. Compared to Fig. 5(A), Fig. 6 is configured without using a loss function 541. The learning systems 6 and 5B perform learning by switching between the processes shown in Fig. 6 and Fig. 5(B) depending on whether or not the input image pair includes an image with a mask attached.

[0063] Thereafter, the learning systems 6 and 5B perform learning in the same manner as in the first embodiment. As a result of this learning, as in the first embodiment, the learning system 5B learns that the features of the area that are not hidden even when the mask is worn are concentrated in a specific dimension of a part of the feature vector (an N-dimensional second-type feature vector).

[0064] 6, the weight update unit 54 calculates a loss function 542 using a distance D between a set of feature vector pairs 53 (a pair of feature vectors 531 and 532) calculated from the image pair 51. Then, the weight update unit 54 updates the weights for the DNN unit 52 using the loss function 542. The M-dimensional first-type feature vectors 531 and 532 are trained using the loss function 542 of the M-dimensional first-type feature vectors 531 and 532.

[0065] As described above, in this embodiment, when training is performed using image pairs 51 in which a mask is not worn, training is performed using only the loss function 542 for the entire feature vector. In this embodiment, as in the first embodiment, the learning system 5B trains so that features for identifying a face when a mask is worn are concentrated in a specific dimension of the feature vector. Then, when performing classification or matching, the classification unit 205 switches between using the entire feature vector or a specific dimension of the feature vector, depending on whether a mask is worn. This allows the classification unit 205 to perform high-precision classification and matching without having feature vectors for both when a mask is worn and when a mask is not worn.

[0066] (Third embodiment) In the first and second embodiments, the learning system 5B uses an image pair in which at least one person is wearing a mask to learn a dimension-selected feature vector in which a predetermined dimension is selected from the entire feature vector. In the third embodiment, as shown in FIG. 7, the learning system 7 uses an image pair 75 in which the area hidden when the mask is worn is masked to learn a dimension-selected feature vector in which a predetermined dimension is selected from the entire feature vector. As in the first and second embodiments, the third embodiment also allows the learning system 7 to learn so that features that identify a face when a mask is worn are concentrated in a specific part of the feature vector's dimensions.

[0067] FIG. 7 shows an example of a learning system 7 of the feature vector calculation unit 202 in this embodiment. Compared to FIG. 5(B), FIG. 7 uses image pair 75 instead of image pair 55. Image pair 75 is a pair of images 751 and 752. Images 751 and 752 are images in which the area that would be hidden when wearing a mask is masked. Depending on whether the input image pair includes a masked image, the learning system performs learning by switching between the processing shown in FIG. 5(A) or FIG. 6 and the processing shown in FIG. 7.

[0068] In FIG. 7, the learning system 7 generates images 751 and 752 in which regions that would be hidden when a mask is worn are masked, and inputs these to the DNNs 521 and 522, respectively. Then, similar to FIG. 5(B), the DNNs 521 and 522 calculate M-dimensional first-type feature vectors 531 and 532 based on the images 751 and 752, respectively. The weight update unit 54 calculates a loss function 541 using a distance D between N-dimensional second-type feature vectors 5311 and 5321 of the M-dimensional first-type feature vectors 531 and 532. Then, the weight update unit 54 updates the weights for the DNN unit 52 using the loss function 541.

[0069] 5(A) or 6, feature vector calculation unit 202 receives as input images 511 and 512 in which a specific portion of an object is not hidden, and learns M-dimensional first-type feature vectors 531 and 532. Also, in Fig. 7, feature vector calculation unit 202 receives as input images 751 and 752 in which a specific portion of an object is masked, and learns N-dimensional second-type feature vectors 5311 and 5321.

[0070] Thereafter, the learning system performs learning in the same manner as in the first and second embodiments. As a result of this learning, similar to the first and second embodiments, the learning system 7 can learn to concentrate the features of the area that is not hidden even when the mask is worn in a specific dimension of a part of the feature vector (an N-dimensional second-type feature vector).

[0071] When the face to be identified is wearing a mask, the feature vector calculation unit 202 in this embodiment generates an image in which the area hidden when the mask is worn is masked, as in the learning process, and calculates the feature vector 302. In this case, the identification unit 205 selects and identifies an N-dimensional second-type feature vector.

[0072] As described above, in this embodiment, the learning system 7 learns a dimension-selected feature vector in which a predetermined dimension is selected from the entire feature vector by using the area that is hidden when a mask is worn as a masking area. In this embodiment, as in the first and second embodiments, the learning system 7 learns so that features for identifying a face when a mask is worn are concentrated in a specific dimension of the feature vector. Then, when performing identification or matching, the identification unit 205 switches between using the entire feature vector or a specific dimension of the feature vector, depending on whether a mask is worn. This allows the identification unit 205 to perform highly accurate identification and matching without having both feature vectors for when a mask is worn and when a mask is not worn.

[0073] (Fourth embodiment) In the first to third embodiments, the learning system simultaneously learns the entire feature vector and some specific dimensions. In the fourth embodiment, the learning system first learns some specific dimensions of the feature vector, and then learns the entire feature vector. In this embodiment, as in the first to third embodiments, the learning system learns so that features for identifying faces when wearing a mask are concentrated in some specific dimensions of the feature vector. The learning system first determines the learning of feature vectors when a mask is worn, and then determines the learning of feature vectors when a mask is not worn. Since it is generally more difficult to identify faces when wearing a mask, the learning system assesses the learning and then learns feature vectors when a mask is not worn. In this embodiment, the learning system learns feature vectors when a mask is worn using image pairs in which at least one person is wearing a mask, as in the first and second embodiments.

[0074] In this embodiment, the learning system first performs the learning shown in FIG. 5(B) to determine N-dimensional second-type feature vectors 5311 and 5321. Then, the learning shown in FIG. 6 is performed to determine M-dimensional first-type feature vectors 531 and 532. In the learning shown in FIG. 6, the learning system fixes the N-dimensional second-type feature vectors 5311 and 5321 and performs the learning without changing them. In other words, the learning process learns the parts of the M-dimensional first-type feature vectors 531 and 532 excluding the N-dimensional second-type feature vectors 5321 and 5322.

[0075] The feature vector calculation unit 202 learns the N-dimensional second-type feature vectors 5311 and 5321, and then fixes the N-dimensional second-type feature vectors 5311 and 5321 and learns the M-dimensional first-type feature vectors 531 and 532.

[0076] As described above, in this embodiment, when training the feature vector calculation unit 202, the learning system first trains feature vectors of some specific dimensions, and then trains the entire feature vector. At that time, the learning system trains feature vectors for when a mask is worn using image pairs in which at least one person is wearing a mask. In this embodiment, as in the first to third embodiments, the learning system trains so that features for identifying a face when a mask is worn are concentrated in some specific dimensions of the feature vector. Then, when performing classification or matching, the identification unit 205 switches between using the entire feature vector or some specific dimensions of the feature vector, depending on whether a mask is worn. This enables the identification unit 205 to perform high-precision classification and matching without having feature vectors for when a mask is worn and when a mask is not worn.

[0077] (Fifth embodiment) In the first to third embodiments, the learning system simultaneously learns the entire feature vector and some specific dimensions. Furthermore, in the fourth embodiment, the learning system uses an image pair in which at least one person is wearing a mask to learn the feature vector when a mask is worn. In the fifth embodiment, similar to the fourth embodiment, the learning system first learns some specific dimensions, and then learns the entire feature vector. Furthermore, in the fifth embodiment, similar to the third embodiment, the learning system uses an image pair in which the area hidden when the mask is worn is masked to learn the feature vector when a mask is worn. In this embodiment, similar to the first to fourth embodiments, learning is performed so that the features that identify a face when a mask is worn are concentrated in some specific dimensions of the feature vector.

[0078] In this embodiment, the learning system first performs the learning shown in FIG. 7 to determine the learning of N-dimensional second-type feature vectors 5311 and 5321. Then, the learning system performs the learning shown in FIG. 6 to determine the learning of M-dimensional first-type feature vectors 531 and 532. In the learning shown in FIG. 6, as in the fourth embodiment, the learning system fixes the N-dimensional second-type feature vectors 5311 and 5321 and performs the learning without changing them. In other words, the learning process learns the parts of the M-dimensional first-type feature vectors 531 and 532 excluding the N-dimensional second-type feature vectors 5321 and 5322.

[0079] As described above, in this embodiment, when training the feature vector calculation unit 202, the learning system first trains feature vectors of a certain dimension, and then trains the entire feature vector. At that time, the learning system trains the feature vector when a mask is worn using an image pair in which the area hidden when the mask is worn is masked. In this embodiment, as in the first to fourth embodiments, the learning system trains so that features for identifying a face when a mask is worn are concentrated in a certain dimension of the feature vector. Then, when performing classification or matching, the identification unit 205 switches between using the entire feature vector or a certain dimension of the feature vector, depending on whether a mask is worn. This enables the identification unit 205 to perform high-precision classification and matching without having both feature vectors for when a mask is worn and when a mask is not worn.

[0080] (Sixth embodiment) In the first to fifth embodiments, the learning system performs learning using the entire feature vector, even when learning a specific dimension of a feature vector. In the present embodiment, when learning a specific dimension of a feature vector, the learning system calculates and learns a feature vector of a specific dimension of the feature vector using an image pair from which at least the area hidden by the mask is removed and the face area not hidden by the mask is extracted. Thereafter, the learning system performs learning of the entire feature vector using an image pair in which the subject is not wearing a mask. In this embodiment, as in the first to fifth embodiments, learning is performed so that features that identify a face when wearing a mask are concentrated in a specific dimension of the feature vector.

[0081] 8(A) and 8(B) show examples of feature vector learning systems 8A and 8B in this embodiment. Fig. 8(A) shows an example of learning system 8A that uses image pair 81 (image 811 and image 812) in which a face region not hidden by a mask has been cut out to train a DNN unit 82 that calculates an N-dimensional second-type feature vector. Image pair 81 includes image 811 and image 812. Image 811 and image 812 are images in which a face region not hidden by a mask has been cut out.

[0082] 8(B) shows an example of a learning system 8B that uses an image pair 85 (image 851 and image 852) without a mask to train a DNN unit 86 that calculates an M-dimensional first-type feature vector. The image pair 85 includes an image 851 and an image 852. The images 851 and 852 are images without a mask.

[0083] 8(A), the DNN unit 82 has DNN821 and DNN822, which are the same deep neural network. The DNN821 inputs image 811 of the image pair 81 and calculates an N-dimensional second-type feature vector 831. The DNN822 inputs image 812 of the image pair 81 and calculates an N-dimensional second-type feature vector 832. The calculated pair of feature vectors 831 and 832 is a feature vector pair 83.

[0084] The weight update unit 84 uses a loss function 841. As in the first embodiment, the loss function 841 uses the distance D between the feature vector 831 and the feature vector 832. Then, the weight update unit 84 updates the weight for the DNN unit 82 using the loss function 841. In the learning system 8A, by executing the learning of FIG. 8(A), learning is performed so that the feature for identifying a face when wearing a mask becomes the feature vector 831.

[0085] The learning system 8B in FIG. 8(B) performs learning after the learning system 8A in FIG. 8(A) completes learning. In FIG. 8(B), the DNN unit 86 has DNN861 and DNN862, which are the same deep neural networks. The DNN861 is configured to include the DNN821 trained in FIG. 8(A). The DNN862 is configured to include the DNN822 trained in FIG. 8(A). The DNN861 inputs image 851 of the image pair 85 and calculates an M-dimensional first-type feature vector 871. The DNN862 also inputs image 852 of the image pair 85 and calculates an M-dimensional first-type feature vector 872. The calculated pair of feature vectors 871 and 872 is the feature vector pair 87.

[0086] The weight update unit 88 uses a loss function 882. As in the first embodiment, the loss function 882 uses the distance D between the feature vector 871 and the feature vector 872. Then, the weight update unit 88 updates the weight for the DNN unit 86 using the loss function 882.

[0087] The DNN 861 or DNN 862 after the completion of learning in Fig. 8(B) becomes the feature vector calculation unit 202. After the completion of learning in Fig. 8(A), the learning in Fig. 8(B) is executed, thereby learning a feature vector in which the feature for identifying an object when a specific part of the object is hidden is part of the feature vector.

[0088] 8(A), the feature vector calculation unit 202 receives as input images 811 and 812 in which at least a specific portion of an object has been cropped, and learns N-dimensional second-type feature vectors 831 and 832. Thereafter, in FIG. 8(B), the feature vector calculation unit 202 receives as input images 851 and 852 in which a specific portion of the object is not occluded, and learns M-dimensional first-type feature vectors 871 and 872. The M-dimensional first-type feature vectors 871 and 872 include the N-dimensional second-type feature vectors 831 and 832 as parts thereof.

[0089] In this embodiment, when the face to be identified is wearing a mask, the identification unit 205 generates an image by cutting out the face area not hidden by the mask, calculates an N-dimensional second-type feature vector, and identifies the face, just as during learning.

[0090] As described above, in this embodiment, when the feature vector calculation unit 202 is trained, the learning system 8A first removes at least the area hidden by the mask and learns a feature vector of a specific dimension using, for example, image 811, which is an image of a face that is not hidden by the mask. Next, the learning system 8B learns the entire feature vector using, for example, image 851 of a face that is not wearing a mask. In this embodiment, as in the first to fifth embodiments, the learning system 8A trains so that features that identify a face when a mask is worn are concentrated in a specific dimension of the feature vector. Then, when performing identification or matching, the identification unit 205 switches between using the entire feature vector or a specific dimension of the feature vector, depending on whether a mask is worn. This allows the identification unit 205 to perform high-precision identification and matching without having both feature vectors for when a mask is worn and when a mask is not worn.

[0091] According to the first to sixth embodiments, the image processing device 100 can identify an object with high accuracy both when a part of the object to be identified is covered with an attachment and when it is not covered with an attachment. In this case, the image processing device 100 can identify the object with high accuracy without having both a feature vector when the object is partly covered with an attachment and a feature vector when it is not covered with an attachment.

[0092] The image processing device 100 can identify an object with high accuracy without increasing the amount of data in the feature vector, even when a part of the object to be identified is covered by an attachment or hidden by another object.

[0093] (Other embodiments) In the first to sixth embodiments, examples of face recognition have been described. However, the above embodiments are not limited to face recognition, and can also be applied to recognition of the entire human body, recognition of pets such as dogs and cats, and recognition of general objects such as cars and bicycles. Furthermore, in the above embodiments, wearing a mask has been described as an example of a factor that obscures the face. However, the above embodiments are not limited to wearing a mask, and can also be wearing sunglasses, etc., and are applicable to cases where part of the face is obscured by a pillar or ornament, or when people are lined up and the lower half of the face is obscured by the head of the person in front. Furthermore, when performing recognition of the entire human body, the method can also be applied to cases where the lower half of a human body is obscured, such as when people are overlapping each other in a surveillance camera image.

[0094] The present disclosure can also be realized by providing a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0095] The present disclosure is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the present disclosure. [Explanation of symbols]

[0096] 100 Image processing device, 201 Object detection unit, 202 Feature vector calculation unit, 203 Hidden detection unit, 204 Dimension selection unit, 205 Classification unit, 52, 82, 86 DNN unit, 54, 84, 88 Weight update unit

Claims

1. a calculation means for calculating an M-dimensional first type feature vector from an object in an input image; a classification means for classifying the object using the M-dimensional first-type feature vector or an N-dimensional (N<M) second-type feature vector which is a part of the M-dimensional first-type feature vector; a first detecting means for detecting whether a particular portion of the object is occluded; Equipped with the calculation means is trained to calculate the N-dimensional second type feature vector representing a feature of an object in which a specific portion of the object is occluded; The image processing device is characterized in that the identification means performs identification using the M-dimensional first-type feature vector when the first detection means detects that the specific part is not occluded, and performs identification using the N-dimensional second-type feature vector when the first detection means detects that the specific part is occluded.

2. 2. The image processing device according to claim 1, wherein the M-dimensional first-type feature vector is calculated by the calculation means trained using an image in which a specific portion of the object is not hidden as input, and the N-dimensional second-type feature vector is calculated by the calculation means trained using an image in which a specific portion of the object is hidden as input.

3. 2. The image processing device according to claim 1, wherein the M-dimensional first-type feature vector is calculated by the calculation means trained using an image in which a specific portion of the object is not hidden as input, and the N-dimensional second-type feature vector is calculated by the calculation means trained using an image in which a specific portion of the object is masked as input.

4. 4. The image processing device according to claim 1, wherein the N-dimensional second-type feature vector is a feature vector calculated by the calculation means that has been previously trained, and the M-dimensional first-type feature vector is a feature vector calculated by the calculation means that has been trained by fixing the N-dimensional second-type feature vector.

5. The image processing device described in claim 1, characterized in that the N-dimensional second-type feature vector is a feature vector calculated by the calculation means that has been trained using an image in which at least a specific portion of the object has been cut out as input, and the M-dimensional first-type feature vector is a feature vector calculated by the calculation means that has been trained using an image in which the specific portion of the object is not hidden as input, so as to include the N-dimensional second-type feature vector as a part.

6. 6. The image processing apparatus according to claim 1, wherein the calculation means performs training using a loss function of the M-dimensional feature vector of the first type.

7. 6. The image processing device according to claim 1, wherein the calculation means performs training using a loss function obtained by weighting and integrating a loss function of the M-dimensional first-type feature vector and a loss function of the N-dimensional second-type feature vector.

8. 8. The image processing apparatus according to claim 1, wherein the calculation means performs training using a loss function of the N-dimensional second-type feature vector.

9. a selection means for selecting an N-dimensional feature vector of a second type, which is a part of the M-dimensional feature vector of the first type, by selecting an N-dimensional feature vector from the M-dimensional feature vector of the first type calculated by the calculation means; 9. The image processing device according to claim 1, wherein the identification means performs identification using the N-dimensional second-type feature vector generated by the selection means when the first detection means detects that the specific part is hidden.

10. further comprising a second detection means for detecting an object from the input image; 10. The image processing apparatus according to claim 1, wherein the calculation means calculates the M-dimensional feature vector of the first type from the detected object.

11. the object is a human face, 11. The image processing device according to claim 1, wherein the specific portion of the object is hidden by a mask.

12. a calculation step in which a calculation means calculates an M-dimensional first type feature vector from the object of the input image; a classification step in which classification means classifies the object using the M-dimensional first type feature vector or an N-dimensional (N<M) second type feature vector which is a part of the M-dimensional first type feature vector; a first detection step for detecting whether a particular part of the object is occluded; Equipped with the calculation means is trained to calculate the N-dimensional second type feature vector representing a feature of an object in which a specific portion of the object is occluded; an image processing method for an image processing device, characterized in that the identification means performs identification using the M-dimensional first-type feature vector when it is detected in the first detection step that the specific part is not occluded, and performs identification using the N-dimensional second-type feature vector when it is detected in the first detection step that the specific part is occluded.

13. a calculation step of calculating an M-dimensional first type feature vector from the object of the input image; an identification step of identifying the object using the M-dimensional first-type feature vector or an N-dimensional (N<M) second-type feature vector which is a part of the M-dimensional first-type feature vector; a first detection step for detecting whether a particular part of the object is occluded; A program for causing a computer to execute the above, the computer is trained to calculate the N-dimensional second type feature vector representing features of the object when certain portions of the object are occluded; The computer performs classification using the M-dimensional first-type feature vector when it is detected in the first detection step that the specific part is not occluded, and performs classification using the N-dimensional second-type feature vector when it is detected in the first detection step that the specific part is occluded.

Images