Image processing device, learning method, and program

Abstract

To simultaneously train a feature extractor and a restorer which restores an ideal image from a feature quantity outputted from the feature extractor, so that they are totally optimized.SOLUTION: An image processing apparatus includes: acquisition means which acquires a first image including an object, a second image which includes the object and is different from the first image in condition relating to photographing, and identification information for identifying the object; extraction means which extracts a feature quantity from the second image; classification means which uses the feature quantity extracted by the extraction means to classify the object; restoration means which uses the feature quantity extracted by the extraction means to generate a restored image brought closer to the first image from the second image; and training means which trains the extraction means and the restoration means so that a value corresponding to a first difference relating to a relation between a classification result of the object obtained by the classification means and the identification information and a second difference relating to a relation between the restored image generated by the restoration means and the first image is reduced.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates in particular to an image processing apparatus, a learning method, and a program suitable for use in training a feature extractor that extracts feature quantities of an object. [Background technology]

[0002] In recent years, many feature extraction techniques have been proposed to process images of objects captured in photographs and extract useful information. Among these feature extraction techniques, many have been proposed that use multi-layered neural networks called deepnets (also known as deep neural networks or deep learning) to extract feature vectors of objects in images.

[0003] Feature extractors are trained using methods such as distance learning. In distance learning, the distance between feature vectors is first defined, and examples of distance definitions include Euclidean distance and cosine similarity. In distance learning, the feature extractor is trained to output feature vectors that are close in distance to images of identical objects or objects belonging to the same class, and feature vectors that are farther away from images of non-identical objects or objects belonging to non-identical classes. Examples of applications of distance learning include car identity determination and face recognition. In particular, a technology for realizing face recognition using distance learning is disclosed in Non-Patent Document 1.

[0004] On the other hand, even with images of the same person, there are ideal images where all shooting conditions are favorable (hereinafter referred to as ideal images) and non-ideal images where at least some of the shooting conditions are not favorable, such as not facing the camera directly or having obstructions (hereinafter referred to as non-ideal images). Therefore, the feature vectors of ideal images and non-ideal images may not be close to each other. In general, face recognition accuracy is high when only ideal images are used as trained registration images or matching images, and low when non-ideal images are used as registration images or matching images.

[0005] Non-Patent Document 2 discloses a restorer (BoostGAN) that reconstructs an ideal, frontal, unobstructed image from a non-ideal image of a profile face with part of the face obscured. In the technique described in Non-Patent Document 2, the loss during training of BoostGAN is designed to include information that represents the identity of the face in the reconstructed ideal image. This loss is defined by the error between the feature quantities obtained by inputting the reconstructed face image into a known feature extractor and the feature quantities obtained by inputting the ground truth ideal image into the same feature extractor. According to the method described in Non-Patent Document 2, using the face image reconstructed by BoostGAN for face recognition improves the performance of face recognition. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Deng, Jiankang, et al. "Arcface: Additive angular margin loss for deep face recognition." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019. [Non-Patent Document 2] Duan, Qingyan, and Lei Zhang. "Boostgan for occlusive profile face frontalization and recognition." arXiv preprint arXiv:1902.09782(2019). [Non-Patent Document 3] Dosovitskiy, Alexey, et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929(2020). [Non-Patent Document 4] Zhong, Yaoyao, and Weihong Deng. "Face Transformer for Recognition." arXiv preprint arXiv:2103.14803(2021). [Overview of the project] [Problems that the invention aims to solve]

[0007] However, training the restorer using the method described in Non-Patent Document 2 requires a pre-trained feature extractor. Therefore, it is not possible to train the feature extractor and the restorer simultaneously.

[0008] In view of the aforementioned problems, the present invention aims to enable a feature extractor and a restorer that reconstructs an ideal image from the features output by the feature extractor to be simultaneously trained to achieve overall optimization. [Means for solving the problem]

[0009] The image processing apparatus according to the present invention includes: an acquisition means for acquiring a first image containing an object, a second image containing the object but with different acquisition conditions from the first image, and identification information for identifying the object; an extraction means for extracting feature quantities from the second image; a classification means for classifying the object using the feature quantities extracted by the extraction means; a restoration means for generating a restored image from the second image that closely resembles the first image using the feature quantities extracted by the extraction means; and a learning means for learning the extraction means, the restoration means, and the classification means so that the values ​​corresponding to the first difference between the classification result of the object by the classification means and the identification information, and the second difference between the restored image generated by the restoration means and the first image become smaller. The extraction means includes: a division means for dividing the second image to generate a plurality of partial images; a transformation means for converting the partial images divided by the division means into tokens which are fixed-dimensional vectors; an addition means for adding class tokens having a fixed dimension corresponding to the tokens to the sequence of tokens transformed by the transformation means; and an encoding means for updating the sequence of tokens to which the class tokens have been added by the addition means based on the relationships between the tokens, and extracts the feature quantities of the object from the tokens updated by the encoding means. It is characterized by the following: [Effects of the Invention]

[0010] According to the present invention, it is possible to simultaneously learn a feature extractor and a restorer that restores an ideal image from the feature amounts output from the feature extractor so as to achieve global optimality.

Brief Description of Drawings

[0011] [Figure 1] It is a block diagram showing an example of the hardware configuration of an image processing apparatus. [Figure 2] It is a block diagram showing an example of the functional configuration of an image processing apparatus. [Figure 3] It is a flowchart showing an example of a learning processing procedure by an image processing apparatus. [Figure 4] It is a schematic diagram for explaining a learning processing procedure by an image processing apparatus. [Figure 5] It is a diagram for explaining a specific example of a method for collecting learning data. [Figure 6] It is a diagram for explaining an example of normalization processing of a face image. [Figure 7] It is a block diagram showing a detailed example of the functional configuration of a feature extraction unit. [Figure 8] It is a flowchart showing an example of a detailed processing procedure for obtaining a feature vector. [Figure 9] It is a schematic diagram for explaining a detailed processing procedure for obtaining a feature vector. [Figure 10] It is a block diagram showing an example of a configuration for performing one-time encoding processing by a Transformer Encoder. [Figure 11] It is a block diagram showing a detailed example of the configuration of a restoration unit. [Figure 12] It is a flowchart showing an example of an inference processing procedure by an image processing apparatus. [Figure 13] It is a schematic diagram for explaining an inference processing procedure by an image processing apparatus. [[ID=!]]

Embodiments for Carrying Out the Invention

[0012] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations. In the following description, a feature extractor that extracts feature vectors for identifying the identity of a face from an input image containing a human face (hereinafter referred to as a face image) will be used as an example. However, the object from which feature vectors are extracted is not limited to faces; it may be any object from which identity can be determined by the feature vectors.

[0013] Figure 1 is a block diagram showing an example of the hardware configuration of the image processing apparatus 100 according to this embodiment. The image processing apparatus 100 has a hardware configuration consisting of a control device 11, a storage device 12, an arithmetic unit 13, an input device 14, an output device 15, and an I / F device 16.

[0014] The control device 11 controls the entire image processing device 100. The storage device 12 holds the programs and data necessary for the operation of the control device 11. The arithmetic unit 13 performs the necessary arithmetic processing based on the control from the control device 11. The input device 14 is a human interface device, etc., which inputs user operations to the image processing device 100. The output device 15 is a display, etc., which presents the processing results of the image processing device 100 to the user.

[0015] The I / F device 16 is a wired interface such as a universal serial bus, local area network, or optical cable, or a wireless interface such as Wi-Fi or Bluetooth (registered trademark). The I / F device 16 also connects a camera or the like to input captured images to the image processing device 100, and transmits the processing results obtained by the image processing device 100 to an external device. Furthermore, the I / F device 10 inputs programs and various data necessary for the operation of the image processing device 100 to the image processing device 100 as needed.

[0016] Figure 2 is a block diagram showing an example of the functional configuration of the image processing apparatus 100 according to this embodiment. The image processing apparatus 100 has the following functional configuration: a data acquisition unit 201, a feature extraction unit 202, a face classification unit 203, a face classification loss calculation unit 204, a reconstruction unit 205, a reconstruction loss calculation unit 206, a learning unit 207, and an identity determination unit 208.

[0017] The data acquisition unit 201 acquires the data necessary during training and inference. For example, during training, it acquires samples (combinations of ideal images, non-ideal images, and person IDs) randomly selected from pre-collected training data, and during inference, it acquires input images (hereinafter referred to as "inferred images") in which the person to be recognized is pictured.

[0018] Here, we will explain the definitions of ideal and non-ideal images. An ideal image is an image taken under ideal shooting conditions, as follows: Coverage: Part or all of the face is not covered by a mask, sunglasses, or any other part of the body. Face orientation: The face is facing forward. Expression: No emotions or feelings are visible on the face; it is a neutral expression. Image Quality: High quality, allowing facial textures to be clearly visible. Lighting: No overexposure or underexposure. Background: Not chaotic, and no other faces exist. On the other hand, a non-ideal image refers to an image in which at least one of the above shooting conditions is deviated. In other words, a non-ideal image is an image of an object taken under conditions different from those of an ideal image.

[0019] The ideal and non-ideal images acquired by the data acquisition unit 201 during training are images containing only one face, and that face belongs to the same person. The person ID is the identification information of the person in the ideal and non-ideal images, and the person ID is assigned one of the numbers from 1 to C. The image to be inferred also contains only one face. The face image acquired by the data acquisition unit 201 may be an image stored in the storage device 12, or it may be a captured image received from the I / F device 16 via a network such as the Internet.

[0020] The feature extraction unit 202 extracts features from the face images acquired by the data acquisition unit 201 (non-ideal images during training, and images under inference during inference) to identify the identity of the faces in the face images. In the following, we will assume that a vector with a fixed dimension (hereinafter referred to as a feature vector) is extracted as the feature, but the form of the feature is not limited to this, and for example, a tensor of the second order or higher may also be used. Hereafter, the dimension of the feature vector will be D.

[0021] The face classification unit 203 receives the D-dimensional feature vector extracted by the feature extraction unit 202 and outputs a C-dimensional probability vector. The i-th component of the probability vector (where i is an integer between 1 and C, inclusive) represents the probability that the person ID of the face in the face image is i. The face classification loss calculation unit 204 calculates the face classification loss from the probability vector output by the face classification unit 203 and the person ID acquired by the data acquisition unit 201.

[0022] The reconstruction unit 205 receives the D-dimensional feature vector extracted by the feature extraction unit 202 and reconstructs an image that is as close as possible to the ideal image (hereinafter referred to as the reconstructed image). The reconstruction loss calculation unit 206 calculates the reconstruction loss from the reconstructed image reconstructed by the reconstruction unit 205 and the ideal image acquired by the data acquisition unit 201.

[0023] The learning unit 207 adjusts the parameters held by the feature extraction unit 202, the face classification unit 203, and the reconstruction unit 205 so that the weighted sum of the face classification loss calculated by the face classification loss calculation unit 204 and the reconstruction loss calculated by the reconstruction loss calculation unit 206 becomes small. The identity determination unit 208 receives the feature vectors extracted by the feature extraction unit 202 and determines whether the faces in the image belong to the same person or not.

[0024] Next, an example of the learning process flow by the image processing device 100 according to this embodiment will be explained using the flowchart in Figure 3 and the schematic diagram in Figure 4.

[0025] In S301, the learning unit 207 of the image processing device 100 initializes the parameters to be learned. Here, the parameters to be learned refer to the parameters held by the feature extraction unit 202, the face classification unit 203, and the reconstruction unit 205, respectively. Parameter initialization is performed by random initialization, zero initialization, etc. Alternatively, the framework of transfer learning may be used to initialize parameters that have already been learned in other tasks.

[0026] In S302, the data acquisition unit 201 acquires the data necessary for training. Specifically, it randomly acquires samples (combinations of ideal images, non-ideal images, and person IDs) from the training data collected in advance.

[0027] The following describes specific examples of methods for collecting training data. Below, we will describe an example where an ideal image is one in which the face is facing forward and not obscured, and a non-ideal image is one in which the face is not facing forward or at least part of the face is obscured. The first collection method is to collect only ideal images and artificially generate non-ideal images from them. This method will be explained below with reference to Figure 5(a). First, a large number of ideal images with known person IDs are collected. Image 500 in Figure 5(a) is an example of a collected ideal image.

[0028] Next, each collected ideal image is subjected to normalization. Hereafter, the normalized image will be referred to as a normalized face image. A normalized face image is an image extracted such that both the vertical and horizontal dimensions are a predetermined number of pixels, the face is not tilted to the left or right, the face size is approximately constant, and the relative position of the face to the image is approximately constant. The purpose of normalization is to reduce variations in the in-plane rotation angle and size of faces in the face image. Image 502 in Figure 5(a) is an example of the result of applying normalization to image 500. Details of the normalization process will be described later.

[0029] Next, the normalized ideal image is subjected to de-idealization. De-idealization refers to the process of artificially changing at least one of the shooting conditions, such as "occlusion" or "image quality," in a way that reduces the accuracy of face recognition. An example of de-idealization is occlusion of a part of the face. Occluding methods include filling random positions in the face image with a single color, or superimposing pre-prepared materials such as a mask image or sunglasses image onto the mouth and eye positions. This superposition can be achieved using known methods for estimating the positions of organ points such as the mouth and eyes.

[0030] Another example of de-idealization is processing that degrades the image quality of a face. For example, this could involve superimposing Gaussian noise onto a face, applying a blur effect to a face, compressing and then decompressing a face using a lossy compression method such as JPEG, or adjusting the contrast and brightness of the image to cause crushed whites or crushed blacks. Yet another example of de-idealization is processing that messes up the background of a face. For example, this could involve superimposing a complex texture onto the background or superimposing a part of someone else's face. Image 504 in Figure 5(a) is an example of the result of applying de-idealization to Image 502, and Image 504 is an example of superimposing a mask image onto the face shown in Image 502.

[0031] Here, an example of normalization processing of a face image will be explained with reference to Figure 6. Face image 600 is an image containing one face 610. In the normalization process, the positions of organ points such as the eyes, nose, and mouth of face 610 are estimated using a known organ point position estimation method for face image 600. In the example in Figure 6, it is assumed that the positions of the right eye center 601, the left eye center 602, the right corner of the mouth 603, and the left corner of the mouth 604 have been estimated. Next, based on the estimated positions of the organ points, a normalized face image 605 is extracted from face image 600. This extraction is performed, for example, by applying a geometric transformation such as an affine transformation to face image 600 so that the right eye center 601, the left eye center 602, the right corner of the mouth 603, and the left corner of the mouth 604 approach predetermined positions 606, 607, 608, and 609 of the normalized face image 605, respectively.

[0032] As described above, in the first data collection method, the training data is composed of the collected ideal images, non-ideal images artificially generated from the ideal images, and person IDs collected simultaneously with the ideal images.

[0033] The second method for collecting training data involves using pairs of ideal and non-ideal images. This method will be explained below with reference to Figure 5(b). For example, if the ideal image is a "frontal view" and the non-ideal image is a "profile view," it is difficult to generate a non-ideal image by applying a de-idealization process to the ideal image. This second collection method is used in such cases.

[0034] This method first collects a large number of pairs of ideal and non-ideal images for which the person ID is known. Images 510 and 520 in Figure 5(b) are examples of the collected pairs of ideal and non-ideal images. Both images 510 and 520 have the same person ID, but in image 510 the face is shown from the front, while in image 520 the face is shown at an angle.

[0035] Next, each ideal image and each non-ideal image collected is subjected to normalization. The detailed procedure for normalization is basically the same as described above, but if there are images with various face orientations, a geometric transformation may be performed by defining predetermined positions of different organ points according to the face orientation. For example, face orientations can be classified into five classes: left, left diagonal, front, right diagonal, and right, and predetermined positions of different organ points can be defined for each class. Images 512 and 522 are examples of the results of normalization applied to images 510 and 520, respectively.

[0036] As described above, in the second collection method, training data is constructed from the collected pairs of ideal and non-ideal images and their corresponding person IDs.

[0037] The data acquisition unit 201 randomly acquires samples (combinations of ideal images, non-ideal images, and person IDs) from the training data collected by one of the collection methods described above. Typically, deep networks acquire multiple samples to perform mini-batch learning. This is expected to speed up and stabilize the learning process.

[0038] Returning to the explanation of Figure 3, in S303, the feature extraction unit 202 extracts feature quantities (in this case, D-dimensional feature vectors) from the non-ideal images among the samples acquired by the data acquisition unit 201 to identify the identity of faces in the face images. Typical values ​​for D include 256, 512, and 1024.

[0039] Here, any deep network may be used for the feature extraction unit 202. For example, a deep network including a convolutional neural network can be used as the feature extraction unit 202. As an example of a deep network, a deep network called ResNet, as disclosed in Non-Patent Literature 1, can be used as the feature extraction unit 202. Alternatively, a deep network including a Transformer model may be used as the feature extraction unit 202. It is known that using the Transformer model, which has been widely used in the world of natural language processing, for image classification can achieve image classification performance equivalent to or better than that when using a convolutional neural network (see Non-Patent Literature 3). Furthermore, as disclosed in Non-Patent Literature 4, a neural network including a Transformer model may also be used as the feature extraction unit 202 for face recognition.

[0040] The following describes in detail the case where a Transformer model is used in the feature extraction unit 202. Figure 7 is a block diagram showing a detailed example of the functional configuration of the feature extraction unit 202. The feature extraction unit 202 has the following functional configuration: an image segmentation unit 701, a token conversion unit 702, a token addition unit 703, an encoding unit 704, and a feature determination unit 705.

[0041] The image splitting unit 701 receives a normalized face image, splits it into partial images with predetermined vertical and horizontal dimensions, and obtains a sequence of partial images. The token conversion unit 702 converts the sequence of partial images obtained by the image splitting unit 701 into a sequence of tokens. Here, a token is a vector with fixed dimension D1. The conversion from partial images to tokens is performed by flattening the partial images into vectors, and then applying a linear transformation to convert the dimension of the vectors to D1. Note that the linear transformation may include not only the process of multiplying the input by a weight matrix, but also the process of multiplying the input by a weight matrix and then adding a bias vector.

[0042] The token addition unit 703 adds D1-dimensional tokens called class tokens to the token sequence obtained by the token conversion unit 702. The encoding unit 704 performs an encoding process to update the token sequence obtained by the token addition unit 703 one or more times. Details of the encoding process will be described later. The final token sequence is called the encoded representation sequence.

[0043] The feature determination unit 705 extracts D-dimensional feature vectors as features from the encoded representation sequence acquired by the encoding unit 704. An example of the feature extraction flow of the feature extraction unit 202 will be explained using the flowchart in Figure 8 and the schematic diagram in Figure 9.

[0044] Figure 8 is a flowchart showing an example of a detailed processing procedure for S303 in Figure 3. Each process in the flowchart in Figure 8 will be explained below with reference to Figure 9. In S801, the image splitting unit 701 divides the normalized face image 900 into partial images, each having a predetermined number of pixels in both height and width, and obtains a sequence of partial images 901. Here, the image splitting unit 701 divides the normalized face image 900 so that every pixel belongs to exactly one partial image, that is, there are no gaps or overlaps between the partial images. Hereafter, the height of the normalized face image is H pixels, the width is W pixels, and the height of the partial image is P H Pixels, width is P WLet C be the number of pixels and channels. H is P H So, W is P W Each of these is divisible by (H / P), and the length N of the sub-image sequence is (H / P H )×(W / P W ) In the example in Figure 9, the length N of the partial image sequence 901 is 9. However, the division into partial images is not limited to the above, and the image may be divided so that the partial images overlap by a predetermined number of pixels, as disclosed in Non-Patent Document 4.

[0045] In S802, the token conversion unit 702 converts the partial image sequence 901 acquired by the image segmentation unit 701 into a token sequence 902. An example of conversion from partial images to tokens is described below. First, the partial image is flattened, and the number of dimensions is P H P W A vector C is obtained. Next, a linear transformation is applied to this vector to obtain a vector (token) with dimension D1. The parameters of the linear transformation are learned by the learning unit 207 in S308, which will be described later.

[0046] In S803, the token addition unit 703 adds one D1-dimensional token called a class token to the token sequence 902 obtained by the token conversion unit 702, thereby obtaining a new token sequence 903. This process makes the length of the token sequence N+1 (=N'). The parameters of the class token are learned by the learning unit 207 in S308, which will be described later. In the example in Figure 9, the token located at the leftmost end of the token sequence 903 is the class token.

[0047] Furthermore, the token addition unit 703 may add a D1-dimensional vector called a position embedding vector to each of the N' tokens. Here, the position embedding vector has different values ​​depending on the position in the token sequence 903 and holds information about which position in the normalized face image 900 each token corresponds to. The position embedding vector may be artificially designed, or it may be learned by the learning unit 207 in S308, which will be described later.

[0048] In S804, the encoding unit 704 performs encoding processing once or more on the token sequence 903 acquired by the token addition unit 703. Here, encoding processing refers to the process of updating each token based on the relationships between the tokens that make up the token sequence. As the encoding processing, for example, a method using a Transformer Encoder disclosed in Non-Patent Literature 3 is adopted.

[0049] The following explains how to perform encoding using a Transformer Encoder. Figure 10 is a block diagram showing an example of a configuration for performing a single encoding process by the Transformer Encoder 1001. The Transformer Encoder 1001 is configured to take a token sequence as input and output a token sequence. The Transformer Encoder 1001 includes a first layer normalization unit 1002, an attention unit 1003, and a first skip merging unit 1004. Furthermore, the Transformer Encoder 1001 includes a second layer normalization unit 1005, an MLP (Multi Layer Perceptron) unit 1006, and a second skip merging unit 1007.

[0050] The token sequence is first input to the first layer normalization unit 1002. The first layer normalization unit 1002 applies a normalization method called layer normalization to the token sequence, normalizing each token independently. The parameters required for layer normalization are learned by the learning unit 207 in S308, which will be described later.

[0051] The token sequence normalized by the first layer normalization unit 1002 is updated by the attention unit 1003. The attention unit 1003 has M attention mechanisms (where M is an integer greater than or equal to 1). Furthermore, if M > 1, each of the M attention mechanisms acquires different parameter weights through learning by the learning unit 207, allowing for the discovery of relationships between tokens from multiple perspectives.

[0052] Here, a method of updating a token sequence by one of the M attention mechanisms will be described. The token sequence is represented by an N'×D1 matrix. By multiplying this with the weight matrices U Q , U K , U V , query matrix Q, key matrix K, and value matrix V are obtained. Here, the weight matrices U Q , U K , U V all have a size of D1×D h , and the query matrix Q, key matrix K, and value matrix V all have a size of N'×D h . Note that D h is the dimension of the vector output by each attention mechanism, and is usually set as D1 / M.

[0053] Next, an attention matrix A is obtained using the softmax function according to the following formula (1). The attention matrix A has a size of N'×N', and reflects the degree of relevance between all tokens and all tokens. A = softmax(QK T / √D h ) ···(1)

[0054] By multiplying the attention matrix A by the value matrix V, a matrix with a size of N'×D h is obtained. This matrix can be interpreted as updated by taking the weighted sum of the value values of other tokens for each token according to the strength of the degree of relevance with other tokens.

[0055] By performing the above calculations for each of the M attention mechanisms, an N'×D hM matrices of size are obtained, and these are combined to form an N'×D1 matrix. By multiplying this matrix by a D1×D1 matrix obtained through learning by the learning unit 207, an N'×D1 matrix integrating the results of the M attention mechanisms is obtained. The token sequence represented by this matrix is ​​finally output from the attention unit 1003.

[0056] The first skip joiner 1004 adds the token sequence before it is input to the first layer normalization unit 1002 and the token sequence output from the attention unit 1003. The second layer normalization unit 1005 applies layer normalization to the token sequence output from the first skip joiner 1004. The layer normalization performed by the second layer normalization unit 1005 is the same as the process performed by the first layer normalization unit 1002.

[0057] The MLP unit 1006 independently performs a linear transformation, a nonlinear activation process, and another linear transformation on each token in the token sequence that has been layer-normalized by the second layer normalization unit 1005. The parameters for the linear transformation are learned by the learning unit 207 in S308, which will be described later. Examples of nonlinear activation processes include ReLU (Rectified Linear Unit) and GELU (Gaussian Error Linear Unit). The output of the MLP unit 1006 is a matrix of size N' × D1. The second skip junction unit 1007 adds the output of the first skip junction unit 1004 and the output of the MLP unit 1006.

[0058] As described above, the output of the second skip junction 1007 is the output obtained from one encoding process by the Transformer Encoder. As previously stated, the token sequence is updated one or more times using an encoder such as a Transformer Encoder, and the token sequence finally obtained is used as the encoded representation sequence.

[0059] Returning to the explanation of Figure 8, in S805, the feature determination unit 705 uses the encoded representation sequence 904 acquired by the encoding unit 704 to obtain a feature vector 907 for identifying the identity of faces in the face image. There are various methods for obtaining the feature vector. The following describes the methods for obtaining the feature vector.

[0060] One method for obtaining feature vectors is to use the encoded representation 905 corresponding to the class token directly. This method is applicable only when the dimension D of the feature vector to be extracted by the feature extraction unit 202 matches the dimension D1 of the tokens in the encoded representation sequence.

[0061] The second method for obtaining feature vectors is to input the encoded representation 905 corresponding to the class token into the feature vector transformation head 906. This head is typically a feedforward network (FFN) composed of one or more linear transformations. This FFN is designed to output a D-dimensional vector. The parameters of this FFN are learned by the learning unit 207 in S308, which will be described later.

[0062] The method for obtaining the feature vector 907 is not limited to these. For example, all encoded representations included in the encoded representation sequence 904 may be concatenated and input to a feature vector conversion head to obtain a D-dimensional vector. The process shown in Figure 8 above enables the feature extraction process of the feature extraction unit 202 in S303.

[0063] Returning to the explanation of Figure 3, in S304, the face classification unit 203 receives the feature vector extracted by the feature extraction unit 202 and outputs a C-dimensional probability vector. The processing of the face classification unit 203 can be implemented, for example, by the representative vector method disclosed in Non-Patent Literature 1. In the representative vector method, the face classification unit 203 outputs representative vectors V1 to V corresponding to each person ID (1 to C). CThe face classification unit 203 then takes the i-th component of the C-dimensional probability vector (where i is an integer between 1 and C) and combines it with the feature vector output by the feature extraction unit 202 and the representative vector V. i It is calculated using the dot product of the two terms.

[0064] Furthermore, by performing learning using the backpropagation method described later, when an image with a correct person ID j (where j is an integer between 1 and C) is given, the feature vector extracted from this image and the representative vector V j The learning unit 207 learns to minimize the angle with the vector. Furthermore, the representative vector V j The model is trained to maximize the angle between this vector and other representative vectors. This training method is disclosed in Non-Patent Document 1.

[0065] In S305, the face classification loss calculation unit 204 calculates the face classification loss from the probability vector output by the face classification unit 203 and the person ID acquired by the data acquisition unit 201. This face classification loss represents the error in the classification result, and is designed to be smaller as the probability of classifying the face into the correct person class increases, and larger as the probability decreases. The face classification loss can be calculated, for example, using a multi-class cross-entropy error function that takes a probability vector and a one-hot vector as inputs. Here, a one-hot vector is a C-dimensional vector such that, when the correct person ID of the face image is j (where j is an integer between 1 and C), only the j-th component is 1 and all other components are 0.

[0066] In S306, the reconstruction unit 205 receives the feature vectors extracted by the feature extraction unit 202 and reconstructs an image (reconstructed image) that is as close as possible to the ideal image. Any deep network may be used in the reconstruction unit 205; for example, a deep network including a convolutional neural network can be used in the reconstruction unit 205. The detailed configuration of the reconstruction unit 205 will be described below.

[0067] Figure 11 is a block diagram showing a detailed configuration example of the restoration unit 205. The linear transformation unit 1101 linearly transforms the feature vector extracted by the feature extraction unit 202 into a vector of a different dimension (e.g., 4096 dimensions). The shape transformation unit 1102 reshapes the feature vector linearly transformed by the linear transformation unit 1101 into a tensor with width, height, and channel direction (e.g., width 8, height 8, channels 64). The parameters for the linear transformation are learned by the learning unit 207 in S308, which will be described later.

[0068] The first upsampling unit 1103 applies a resolution expansion process to the tensor whose shape has been transformed by the shape transformation unit 1102, and obtains a tensor with transformed resolution (for example, with a width of 16, a height of 16, and 32 channels). The resolution expansion process is performed, for example, by deconvolution or bilinear interpolation. The first nonlinear activation processing unit 1104 applies a nonlinear activation process such as ReLU to the tensor whose resolution has been transformed by the first upsampling unit 1103.

[0069] The second upsampling unit 1105 applies a resolution expansion process to the tensor that has undergone nonlinear activation processing in the nonlinear activation processing unit 1104, as described above, and obtains a tensor with a further transformed resolution (for example, with a width of 32, a height of 32, and 16 channels). Then, the second nonlinear activation processing unit 1106 applies a nonlinear activation process such as ReLU to the tensor whose resolution has been transformed in the second upsampling unit 1105.

[0070] The third upsampling unit 1107 similarly applies resolution expansion processing to the tensor that has undergone nonlinear activation processing in the second nonlinear activation processing unit 1106, and obtains a tensor with a further transformed resolution (for example, with a width of 64, a height of 64, and 3 channels). This tensor is then output as the reconstructed image. Note that the resolution of the ideal image and the reconstructed image do not necessarily have to match.

[0071] Furthermore, the configuration of the reconstruction unit 205 is not limited to the configuration shown in Figure 11. For example, as mentioned above, a Transformer Encoder may be used in the feature extraction unit 202, and a module called a Transformer Decoder may be used in the reconstruction unit 205. In this configuration, the configuration may be modified so that the entire encoded representation sequence 904 is output to the reconstruction unit 205 and used for image reconstruction.

[0072] In S307, the restoration loss calculation unit 206 calculates the restoration loss from the restored image restored by the restoration unit 205 and the ideal image acquired by the data acquisition unit 201. The restoration loss is a value that represents the difference between the ideal image and the restored image, and is designed to be small when the difference between the ideal image and the restored image is small, and large when the difference is large. The restoration loss can be defined, for example, by the mean squared error at the pixel level between the ideal image (assuming it has been resized to the same resolution as the restored image) and the restored image. Specifically, the width of the restored image is W R Pixels, height H R If the number of pixels and channels is C, the recovery loss L1 can be calculated using the following equation (2).

[0073]

number

[0074] (2) In the formula, I R (w,h,c) represents the value of the c-th channel at the w-th pixel from the left and the h-th pixel from the top of the reconstructed image, and I GT (w,h,c) represents the value of the c-th channel at the w-th pixel from the left and the h-th pixel from the top of the ideal image. Note that the method of defining the reconstruction loss is not limited to the above. For example, as disclosed in Non-Patent Document 2, terms may be added so that the reconstruction loss decreases as the difference between adjacent pixels in the reconstruction image decreases, or terms may be added so that the reconstruction loss decreases as the reconstruction image is closer to symmetrical.

[0075] In S308, the learning unit 207 adjusts the parameters held by the feature extraction unit 202, the face classification unit 203, and the reconstruction unit 205 so that the weighted sum of the face classification loss calculated by the face classification loss calculation unit 204 and the reconstruction loss calculated by the reconstruction loss calculation unit 206 becomes small. This method will be explained in order. First, the learning unit 207 calculates the loss L according to equation (3) below by a weighted sum using a pre-set hyperparameter λ, assuming that the face classification loss calculated by the face classification loss calculation unit 204 is L0. L = L0 + λL1 ···(3)

[0076] Next, the learning unit 207 updates the parameters to be learned so as to reduce the loss L. The parameters are updated in small increments in the direction of reducing the loss L by using the backpropagation method, which is common in deep networks. By learning to reduce the loss L, the feature extraction unit 202, the face classification unit 203, and the reconstruction unit 205 can be trained to be optimal overall. Furthermore, the feature extraction unit 202 can be given the ability to output a feature vector that embeds the information necessary to reconstruct an image close to an ideal image from a non-ideal image, and the ability to output a feature vector for accurate face recognition. As a result, the feature extraction unit 202 can output feature vectors that are close to each other from ideal and non-ideal images of the same person.

[0077] In S309, the learning unit 207 determines whether to continue updating the parameters. The criteria for this determination are whether a sufficient number of learning iterations have been reached, or whether the model's performance has improved sufficiently. If either of these criteria is met, the unit determines to terminate the parameter update and proceeds to S310. Otherwise, it returns to S302 and continues learning.

[0078] In S310, the learning unit 207 stores the learned parameters in the storage device 12. The stored parameters are used when the image processing device 100 performs inference processing or when the image processing device 100 performs learning processing again.

[0079] Next, an example of the inference processing flow of the image processing apparatus 100 according to this embodiment will be explained using the flowchart in Figure 12 and the schematic diagram in Figure 13. The inference processing is performed by the data acquisition unit 201, the feature extraction unit 202, and the identity determination unit 208.

[0080] In S1201, the data acquisition unit 201 acquires multiple inferred images, each containing only one person who is the target of face recognition. These inferred images are subjected to the normalization process described in S302. In S1202, the feature extraction unit 202 extracts a D-dimensional feature vector from the inferred image acquired by the data acquisition unit 201 according to the procedure described above, in order to identify the identity of the faces in the face image.

[0081] In S1203, the identity determination unit 208 receives the feature vectors extracted by the feature extraction unit 202 and determines whether the faces in the inferred images belong to the same person or not. Below, an example of how to determine whether the faces belong to the same person or different people, given feature vectors f1 and f2 extracted from the first and second inferred images, respectively, is described. Here, it is assumed that the dimensionality of the feature vectors f1 and f2 is the same.

[0082] First, the similarity between the two feature vectors is calculated using a predetermined method. For example, when using cosine similarity, the similarity score is calculated using equation (4) below. Similarity score (f1, f2):=cos(θ) 12 ) =<f1,f2> ÷(|f1|·|f2|) ···(4)

[0083] (4) In the formula, θ 12 This is the angle between feature vectors f1 and f2.<x,y> |x| is the dot product of x and y, and |x| is the length of x. The identity determination unit 208 determines that the two individuals are the same person if the similarity score is below a predetermined threshold, and determines that they are different people if the score exceeds the threshold. The method for calculating similarity is not limited to the above, and any known method such as Euclidean distance may be used.

[0084] As described above, the image processing apparatus 100 of this embodiment updates the parameters held by the feature extraction unit 202, the face classification unit 203, and the reconstruction unit 205 so as to reduce the loss L. This allows the feature extractor and the reconstruction unit, which takes the feature vector output from the feature extractor as input and outputs a reconstructed image, to simultaneously learn to achieve overall optimization, thereby improving the performance of the feature extractor.

[0085] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0086] The embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various ways without departing from its technical concept or its main features. [Explanation of Symbols]

[0087] 201 Data acquisition unit, 202 Feature extraction unit, 203 Face classification unit, 205 Reconstruction unit, 207 Learning unit

Claims

1. An acquisition means for acquiring a first image containing an object, a second image containing the object but with different acquisition conditions from the first image, and identification information for identifying the object, An extraction means for extracting features from the second image, A classification means that classifies the object using the feature quantities extracted by the extraction means, A restoration means that generates a restored image from the second image that approximates the first image using the feature quantities extracted by the extraction means, A learning means that learns the extraction means, the restoration means, and the classification means so that the values ​​corresponding to the first difference between the classification result of the object by the classification means and the identification information, and the second difference between the restored image generated by the restoration means and the first image become smaller, It has, The extraction means is A division means for dividing the second image to generate multiple partial images, A transformation means that converts the partial image divided by the division means into a token which is a fixed-dimensional vector, An additional means for adding a class token having a fixed dimension corresponding to the token to the sequence of tokens converted by the conversion means, The system includes encoding means for updating a sequence of tokens to which the class token has been added by the aforementioned additional means, based on the relationships between the tokens. An image processing apparatus characterized by extracting feature quantities of the object from tokens updated by the encoding means.

2. The image processing apparatus according to claim 1, wherein the encoding means obtains a query matrix, a key matrix, and a value matrix from the column of tokens to which the class token has been added by the addition means, obtains an attention matrix that reflects the relationship between all tokens and all tokens based on the product of the query matrix and the key matrix, and updates the column of tokens based on the product of the attention matrix and the value matrix.

3. The image processing apparatus according to claim 1 or 2, wherein the additional means further adds a vector relating to the position of the partial image before division to the token.

4. The image processing apparatus according to any one of claims 1 to 3, further comprising a determination means for determining whether or not the objects included in the plurality of images are the same, based on feature quantities extracted from the plurality of images including an object by the extraction means.

5. A learning method implemented by a computer, The acquisition unit performs an acquisition step of acquiring a first image including an object, a second image including the object but with different acquisition conditions from the first image, and identification information for identifying the object. The feature extraction unit performs an extraction step in which it extracts feature quantities from the second image, A classification step in which a classification unit classifies the object using the feature quantities extracted in the extraction step, A restoration step in which the restoration unit generates a restored image that approximates the first image from the second image using the feature quantities extracted in the extraction step, A learning step in which the learning unit trains the extraction step, the restoration step, and the classification step to reduce the values ​​corresponding to the first difference between the classification result of the object in the classification step and the identification information, and the second difference between the restored image generated in the restoration step and the first image. It has, The extraction step is, A division step of dividing the second image to generate multiple partial images, A transformation step is performed to convert the divided subimages in the division step into tokens which are fixed-dimensional vectors, An additional step is to add a class token having a fixed dimension corresponding to the token to the sequence of tokens converted in the conversion step, The additional step includes an encoding step in which the class token is updated in the sequence of tokens to which the class token was added, based on the relationships between the tokens. A learning method characterized by extracting the feature quantities of the object from the tokens updated in the encoding step.

6. An acquisition step of acquiring a first image containing an object, a second image containing the object but with different acquisition conditions from the first image, and identification information for identifying the object, The extraction process involves extracting features from the second image described above, A classification step in which the object is classified using the feature quantities extracted in the extraction step, A restoration step is to generate a restored image that approximates the first image from the second image using the features extracted in the extraction step, A learning step that causes the extraction step, the restoration step, and the classification step to learn such that the values ​​corresponding to the first difference between the classification result of the object in the classification step and the identification information, and the second difference between the restored image generated in the restoration step and the first image become smaller, Have the computer run it, The extraction step is, A division step of dividing the second image to generate multiple partial images, A transformation step is performed to convert the divided subimages in the division step into tokens which are fixed-dimensional vectors, An additional step is to add a class token having a fixed dimension corresponding to the token to the sequence of tokens converted in the conversion step, The additional step includes an encoding step in which the class token is updated in the sequence of tokens to which the class token was added, based on the relationships between the tokens. A program characterized by extracting the feature quantities of the object from the tokens updated in the encoding process.

Images