Image processing apparatus, method of controlling the same, and program

Abstract

To further improve performance of a feature extractor using ViT.SOLUTION: A feature vector calculation unit includes: an image acquiring unit for acquiring an input image; a division unit for dividing the input image into partial image sequences as a plurality of partial images; a token conversion unit for converting the partial image into tokens having fixed dimensional vectors to convert the partial image sequences divided by the division unit into token sequences; an addition unit for adding a class token having the same dimensional vector as those of two or more tokens to the token sequences obtained by the toke conversion unit; an encoder unit for obtaining a final encoded expression sequence by updating the token sequences obtained by the addition unit based on correlation between tokens; a class toke encoded expression acquiring unit for acquiring, as a class token encoded expression sequence, an encoded expression corresponding to the class token among the encoded expression sequences obtained by the encoder unit; and an integrating unit for obtaining a feature vector of the input image by integrating the class token encoded expression sequences.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This invention relates to an image processing apparatus, a control method therefor, and a program. [Background technology]

[0002] Many feature extractors have been proposed that extract useful information from images as feature vectors to solve a target task. Among these, feature extractors using Convolutional Neural Networks (CNNs), a type of Deep Neural Network (DNN), have traditionally shown high performance.

[0003] In recent years, Non-Patent Document 1 proposes Vision Transformer (hereinafter referred to as ViT), which applies the DNN called Transformer (Patent Document 1), which has been widely used in the field of natural language processing, to image recognition. The feature extractor using ViT demonstrates performance that surpasses feature extractors using CNNs in image classification tasks. Furthermore, Non-Patent Document 2 demonstrates performance equivalent to feature extractors using CNNs when applying a feature extractor using ViT to a facial recognition task.

[0004] ViT calculates a feature vector by repeatedly performing the following processes: extracting features from various sub-regions of an image using multiple Self-Attention mechanisms, and focusing on certain sub-regions and aggregating the features obtained from them into a single vector called a class token. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Patent No. 6884871 [Non-patent literature]

[0006] [Non-Patent Document 1] Dosovitskiy, Alexey, et al. An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021: The Ninth International Conference on Learning Representations, 2021. [Non-Patent Document 2] Zhong, Y., & Deng, W. (2021). Face Transformer for Recognition. arXiv preprint arXiv:2103.14803. [Non-Patent Document 3] Deng, et. Al. ArcFace: Additive Angular Margin Loss for Deep Face Recognition. In CVPR, 2019 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] Ideally, each Self-Attention in ViT should learn to focus on different subregions when aggregating information into class tokens. This is because it allows for the extraction of information from the entire image, making it robust against changes in the position and occlusion of objects in the image, and the low redundancy between Self-Attentions allows for the effective utilization of ViT parameters.

[0008] However, the learning process does not always result in the ideal state described above. Sometimes, each Self-Attention learns to focus on the same subregion. In that case, the robustness to changes in the position and occlusion of objects in the image decreases, and performance degrades because the ViT parameters are not being used effectively.

[0009] In the present invention, it is intended to provide a technique for solving the above problems and improving the performance of a feature extractor using ViT.

Means for Solving the Problems

[0010] To solve this problem, for example, the image processing apparatus of the present invention has the following configuration. That is, image acquisition means for acquiring an input image; division means for dividing the input image into a sequence of partial images, which are a plurality of partial images; token conversion means for converting the partial images into tokens having fixed-dimensional vectors, thereby converting the sequence of partial images divided by the division means into a token sequence; addition means for adding class tokens having vectors of the same dimension as two or more of the tokens to the token sequence obtained by the token conversion means; encoding means for obtaining a final encoded representation sequence by updating the token sequence obtained by the addition means based on the relevance between tokens; class token encoded representation sequence acquisition means for acquiring, as a class token encoded representation sequence, the encoded representation corresponding to the class token from the encoded representation sequence obtained by the encoding means; and integration means for obtaining the feature vector of the input image by integrating the class token encoded representation sequence.

Effects of the Invention

[0011] According to the present invention, the performance of the feature extractor using ViT can be further improved.

Brief Description of the Drawings

[0012] [Figure 1] A block diagram showing a functional configuration example of the feature vector calculation unit according to the first embodiment. [Figure 2] A flowchart showing the procedure of the feature vector calculation process according to the first embodiment. [Figure 3]A schematic diagram of the Transformer Encoder of the first embodiment. [Figure 4] A schematic diagram of Self-Attention according to the first embodiment. [Figure 5] A block diagram showing an example of the functional configuration of the learning unit in the first embodiment. [Figure 6] A flowchart illustrating the procedure for the learning process in the first embodiment. [Figure 7] Hardware configuration diagram of the information processing device in the first embodiment. [Modes for carrying out the invention]

[0013] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0014] In the following section, we will explain the task of calculating feature vectors to identify the identity of a face from an image of a single person's face (hereinafter referred to as a face image) as an example.

[0015] [First Embodiment] Figure 7 is a hardware configuration diagram of an information processing device, such as a personal computer, to which this embodiment is applied. The simultaneous configuration and operation will be described below.

[0016] When the power to this device is turned ON, the CPU 701 executes the boot program stored in the ROM 702, performs hardware initialization, loads the OS (Operating System) stored in the HDD (Hard Disk) 704 into RAM, and transfers control to the OS (executes it), thereby enabling the device to function as an information processing device. In other words, the display unit 706, keyboard 707, and mouse 708 function as interfaces with the user. The I / F 705 is an interface for communicating with external devices, typically a network interface, USB (Universal Serial Bus), etc. The input image data to be processed in this embodiment is also input via this I / F 705. In the above configuration, when the user operates the keyboard 707 or mouse 708 to input a predetermined instruction, the CPU 701 loads an image processing application program from the HDD 704 into RAM 703 and executes it, enabling the device to function as an image processing device. The following describes the case when the device functions as an image processing device.

[0017] In this first embodiment, the process of calculating feature vectors from facial images and the learning process in an image processing application will be described.

[0018] Figure 1 is a block diagram showing an example of the functional configuration of the feature vector calculation unit in this embodiment. The parameter storage unit 108 shown corresponds to the HDD 704 in Figure 7, and the other reference numerals 101 to 107 should be understood as functioning when the CPU 701 executes an image processing application. Of course, instead of a program executed by the CPU 701, the dedicated circuit may be configured in hardware.

[0019] The feature vector calculation unit 100 includes an image acquisition unit 101, an image splitting unit 102, an image token conversion unit 103, a class token addition unit 104, an encoding unit 105, a class encoded representation sequence acquisition unit 106, a class token joining and integration unit 107, and a parameter storage unit 108.

[0020] The source of the images acquired by the image acquisition unit 101 may be, for example, an imaging device or a storage medium storing captured images, and the type of such medium is not specified.

[0021] Figure 2 is a flowchart showing the process flow for calculating feature vectors from a face image performed by the feature vector calculation unit 100 in this embodiment. The procedure for calculating feature vectors from a face image will be explained below using Figures 1 and 2.

[0022] First, in step S101, the image acquisition unit 101 acquires a face image. Here, the face image is defined as a pixel, with a width of W and a height of H (i.e., W × H pixels).

[0023] In step S102, the image division unit 102 divides the acquired face image into a set of partial images of a predetermined size. The width of the partial image is set to P. W , height P H Let's assume that N is the number of sub-images, then N = W × H / (P W ×P H ) Note that the partial image examples may partially overlap on the face image.

[0024] In step S103, the image token conversion unit 103 converts the partial images into image tokens, which are D-dimensional vectors, for all partial images. In other words, the image token conversion unit 103 converts the sequence of partial images into a sequence of tokens. The conversion to image tokens is done by first flattening the partial images. W P H It is converted to a dimensional vector, and then converted to a D-dimensional vector by a DNN. The DNN is composed of, for example, fully connected layers. W P H Any configuration that converts a dimensional vector to a D-dimensional vector is acceptable, and this embodiment is not limited to such configurations. The DNN is trained by the training process described later.

[0025] In step S104, the class token addition unit 104 adds class tokens, which are two or more D-dimensional vectors, to the token sequence. Class tokens are learnable parameters and are learned by the learning process described later. Further, as in Non-Patent Document 1, a D-dimensional vector, the Positional Embedding vector, may be added to each token included in the token sequence. The Positional Embedding vector has the effect of preserving the positional information of the partial image corresponding to each token on the face image.

[0026] In step S105, the encoding unit 105 obtains an encoded representation sequence by performing a process to update each token once or more times based on the relationships between tokens included in the token sequence using the DNN. The DNN has a multi-stage configuration of Transformer Encoder blocks, which are known from Non-Patent Document 1. A schematic diagram of the Transformer Encoder block is shown in Figure 3. The Transformer Encoder block consists of a Layer Normalization layer, which is a normalization layer, Multi-Head Self-Attention (hereinafter MHSA), which is known from Patent Document 1, a fully connected layer, and Gaussian Error Linear Units (hereinafter GeLU), which is a nonlinear activation function. The MLP block 301 in Figure 3 is repeated twice. The configuration given here is just one example, and Batch Normalization may be used as a normalization layer instead of Layer Normalization. Rectified Linear Units may be used as a nonlinear activation function instead of GeLU. The number of repetitions of the MLP block 301 is also arbitrary. Further convolutional layers may be added. This embodiment is not limited to these. DNNs are trained through the learning process described later.

[0027] Here, we will explain MHSA. Since MHSA is a derivative of the Self-Attention mechanism, we will first explain Self-Attention using the schematic diagram in Figure 4. For simplicity, let's assume that there are 4 image tokens and 2 class tokens. The input token matrix 401 is a matrix representation of the token sequence, with the row vectors being tokens. First, the input token matrix 401 is transformed into the query matrix 402, key matrix 403, and value matrix 404 by applying three fully connected layers. Next, the matrix product of the transposed matrix of the query matrix 402 and key matrix 403 is calculated, and the attention map 405 is calculated by dividing each element of the resulting matrix by √D and applying the Softmax function in the column direction. The attention map 405 is a token-number × token-number (6 × 6 in this case) matrix whose elements represent the relationships between each token, and is normalized so that the sum of the elements of each row vector is 1. Each row vector of the attention map 405 represents how much attention should be paid to each token in order to obtain the encoded representation of a given token. For image tokens, tokens containing a lot of information necessary to characterize the corresponding subimage are given high attention, while for class tokens, tokens containing information necessary to identify the person in the face image are given high attention. Finally, the output token matrix is ​​calculated by performing a matrix multiplication of the value matrix 404 and the attention map 405. The configuration described here is just one example; for example, a fully connected layer may be applied to the attention map 405, and this embodiment is not limited to these. MHSA is a method that divides the input token matrix 401 into H parts in the column direction (the dimension of the tokens is D / H), applies Self-Attention to each of the divided input token matrices, and then concatenates and integrates them in the column direction. This has the effect of preventing small features in each dimension from being easily ignored in high-dimensional vectors by dividing them into low-dimensional vectors. In MHSA, H attention maps are calculated.

[0028] In step S106, the class encoded representation sequence acquisition unit 106 acquires the encoded representation corresponding to the class token from the encoded representation sequence obtained by the encoding unit as the class token encoded representation sequence.

[0029] In step S107, the class token integration unit 107 integrates the class token encoded representation sequence using a DNN to output a final feature vector. The DNN consists of one fully connected layer. Using a DNN is merely an example; other methods such as weighted sums, vector mean, sum, or other basic statistics may also be used. The DNN may also consist of multiple fully connected layers. Furthermore, the class token integration unit 107 may be composed of a Transformer Encoder. Specifically, it consists of a feature vector token addition unit that adds a feature vector token, which is a single D-dimensional vector, to the class token encoded representation sequence; a second encoding unit that obtains a second encoded representation sequence from the class token encoded representation sequence obtained by the feature vector token addition unit; and a feature vector acquisition unit that obtains the feature vector encoded representation corresponding to the feature vector token from the second encoded representation sequence obtained from the second encoding unit as a feature vector. Any method that integrates a class encoded representation sequence consisting of multiple class encoded representations to output a single feature vector is acceptable, and this embodiment is not limited to these methods. The DNN is trained by the training process described later.

[0030] Here, we will explain a method for determining whether two facial images represent the same person or different people, based on the feature vectors calculated by the feature vector calculation unit 100.

[0031] First, the processes described in steps S101 to S107 are performed on each of the two images. This calculates the feature vectors for the first and second images. These feature vectors are denoted as f1 and f2.

[0032] Next, the similarity score between the two feature vectors is calculated. There are multiple known examples of the similarity used for the similarity score between feature vectors. Here, the angle between the feature vectors is used, similar to the method in Non-Patent Document 3. The similarity score S is calculated by the following formula (1).

[0033] [Number] Here, θ 12 represents the angle formed by the feature vectors f1 and f2. Also, <x, y> represents the inner product of the vectors x and y, and ||x|| represents the Euclidean norm of the vector x.

[0034] Finally, if the above similarity score S is less than or equal to a predetermined threshold, it is determined that they are the same person; otherwise, they are different persons.

[0035] FIG. 5 is a block diagram showing the basic functional configuration of the learning unit in the present embodiment. The learning unit 500 includes a learning data acquisition unit 501, a feature vector calculation unit 100, a loss calculation unit 502, a parameter update unit 503, a learning data storage unit 504, and a representative vector storage unit 505.

[0036] Hereinafter, the functions and operations of each functional component of the learning unit 500 will be described. The learning data acquisition unit 501 acquires the set of face images held by the learning data storage unit 504. Each image is provided with information on the ID of the person (for example, the person ID is the file name). The feature vector calculation unit 100 calculates feature vectors from the set of face images obtained by the learning data acquisition unit 501. The loss calculation unit 502 calculates a loss value from the set of face images obtained by the learning data acquisition unit 501, the feature vectors obtained by the feature vector calculation unit 100, and the representative vectors held by the representative vector storage unit. The parameter update unit 503 updates the parameters held by the parameter storage unit 109 of the feature vector calculation unit 100 and the representative vectors held by the representative vector storage unit 505 based on the loss value obtained by the loss calculation unit 502.

[0037] Figure 6 shows the flow of the learning process in this embodiment. The procedure for the learning process will be explained below with reference to Figures 1, 5, and 6. Here, learning is performed using the <Representative Vector Method>, which is publicly known in Non-Patent Literature 3. The Representative Vector Method is a facial recognition learning method that improves learning efficiency by setting a feature vector that represents each person and using it in combination. For details, please refer to Non-Patent Literature 3.

[0038] First, in step S201, the feature vector calculation unit 100 initializes the parameters of the image token conversion unit 103, the class token addition unit 104, the encoding unit 105, and the class token integration unit 107 based on the information held by the parameter storage unit 109. The multiple class tokens held by the class token addition unit 104 are initialized with random numbers that have different values ​​from each other. This is done so that each class token learns to aggregate different information from each other. To further enhance this effect, the class tokens may be initialized to be orthogonal to each other. In step S201, the representative vectors v1~v held by the representative vector storage unit 505 are also initialized. n The vector is initialized with random numbers, where 1 to n are the IDs of all individuals included in the training images. Each representative vector v is a d-dimensional vector (where d is a predetermined value).

[0039] Next, in step S202, the learning data acquisition unit 501 randomly selects face image sets I1~I from the learning data storage unit 504. m The data is retrieved. Each image is associated with the person's ID information.

[0040] Next, in step S203, the feature vector calculation unit 100 calculates each image I of the face image set. i Feature vector f from (i=1,…,m) i Calculate the feature vector f. i This is a d-dimensional vector.

[0041] Next, in step S204, the loss calculation unit 502 calculates the similarity (intraclass similarity) S of the features between each person image and the representative vector. IntraEquation (2a) shows the similarity (inter-class similarity) of the feature quantities of the representative vectors of each person and others. Inter Calculate (2b) according to equation (2b).

[0042]

number

[0043]

number

[0044] Next, in step S205, the parameter update unit 503 updates the parameters of the image token conversion unit 103, class token addition unit 104, encoding unit 105, and class token integration unit 107 held by the parameter storage unit 109, and the values ​​of the representative vectors held by the representative vector storage unit 505. By using the backpropagation method, which is common in DNNs, the values ​​are updated in small increments in the direction of reducing the loss value. As a result, the representative vectors are improved to function better as values ​​that represent the characteristics of each person, and the feature vectors output by the feature vector calculation unit 100 are improved so that they become similar to each other if they are feature vectors of the same person.

[0045] The learning unit 500 determines that the learning process described above has converged; otherwise, it repeats steps S201 onwards. Alternatively, instead of using convergence as the condition, the process may be executed until a predetermined number of iterations are reached.

[0046] Next, in step S207, the parameter storage unit 109 of the feature vector calculation unit 100 stores and saves the parameters it holds.

[0047] <Effects of the First Embodiment> In the first embodiment, ViT is trained to aggregate different information into each class token (a method to further enhance this effect will be described in the second embodiment). In order to aggregate different information into each class token, each Self-Attention must also focus on different image tokens. As a result, each of the multiple Self-Attentions in ViT is trained to focus on different subregions in the image. This allows for the extraction of information from the entire image, making it robust to changes in the position and occlusion of objects in the image, and reducing redundancy between Self-Attentions, thus allowing for effective utilization of ViT's parameters. Furthermore, the process of integrating class tokens with different information and outputting them as a feature vector is equivalent to ensembling feature vectors from different feature extractors, thus improving performance.

[0048] [Second Embodiment] In the first embodiment described above, the effect of learning to aggregate different information into each class token is enhanced by initializing each class token with random numbers that have different values ​​in step S201. In this second embodiment, the aforementioned effect is further enhanced by applying regularization in step S204 of the first embodiment to reduce the similarity between class tokens. An example of the functional configuration of this second embodiment is shown in Figures 1 and 5, similar to the first embodiment.

[0049] This section describes the method for calculating the regularization term used for regularization. Here, we will explain using a regularization term that uses three similarities: similarity between class tokens, similarity between class token encoded representations, and similarity between attention maps corresponding to class tokens. In this example, cosine similarity is used as the similarity measure, but other known similarities measures, such as Pearson's correlation coefficient, may also be used. This embodiment is not limited to these.

[0050] The i-th class token is x i CLS The class token encoded representation corresponding to the i-th class token output by the l-th Transformer Encoder is z li CLS The row vector corresponding to the i-th class token of the attention map calculated by the h-th Self-Attention of the MHSA possessed by the l-th Transformer Encoder is a lhi CLS Let's assume that.<x、y> ||x|| represents the dot product of vectors x and y, and ||x|| represents the Euclidean norm of vector x.

[0051] The regularization term R1, which uses the similarity between class tokens, is calculated using the following equation (4).

[0052]

number

[0053] The regularization term R2, which uses the similarity between class token encoded representations, is calculated by the following equation (5).

[0054]

number

[0055] The similarity of the class token encoded representations of some Transformer Encoders may be used, or a weighted sum of the similarities of the class token encoded representations of each Transformer Encoder may be used. This embodiment is not limited to these.

[0056] The regularization term R3, which uses the similarity between attention maps corresponding to class tokens, is calculated by the following equation (6).

[0057]

number

[0058] The similarity of the attention maps of some Transformer Encoders may be used, or the weighted sum of the similarity of the attention maps of each Transformer Encoder may be used. Alternatively, the similarity of some attention maps of the MHSA may be used, or the weighted sum of the similarity of the attention maps of each MHSA may be used. This embodiment is not limited to these methods.

[0059] The loss value L obtained by adding the regularization term described above to Equation 3 is used for training. The loss value L is calculated using the loss function shown in Equation (7).

[0060]

number

[0061] <Effects of the second embodiment> In the second embodiment, learning is performed to reduce the value of the regularization term, so the similarity between class tokens, the similarity between class token encoded representations, and the similarity between attention maps corresponding to class tokens are also reduced. As a result, ViT is learned to aggregate different information for each class token, further enhancing the effects of the first embodiment.

[0062] As explained above, according to the first and second embodiments of the present invention, in a feature extractor using ViT, different information is aggregated into multiple class tokens, and the combined form of these tokens is output as a feature vector. This results in improved performance because each of the multiple Self-Attentions of ViT focuses on different subregions in the image, and the process of integrating class tokens with different information and outputting them as a feature vector works equivalently to ensembling the feature vectors of different feature extractors.

[0063] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0064] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0065] 100...Feature vector calculation unit, 101...Image acquisition unit, 102...Image segmentation unit, 103...Image token conversion unit, 104...Class token addition unit, 105...Encoding unit, 106...Class token encoded representation acquisition unit, 107...Class token integration unit, 108...Parameter storage unit

Claims

1. An image acquisition means for acquiring an input image; A division means for dividing the input image into a sequence of partial images, which are a plurality of partial images; a token conversion means for converting the partial image sequence divided by the division means into a token sequence by converting the partial image sequence into a token sequence having a fixed dimension vector; adding means for adding, to the token string obtained by the token conversion means, two or more class tokens having vectors of the same dimension as the tokens; an encoding means for updating the token sequence obtained by the adding means based on the association between the tokens to obtain an encoded representation sequence; a class token encoded expression string acquisition means for acquiring class token encoded expressions corresponding to the two or more class tokens as class token encoded expression strings from the encoded expression strings obtained by the encoding means; a synthesis means for synthesizing the class token encoded representation sequence to obtain a feature vector of the input image; An image processing device comprising:

2. The integration means includes: A weighted sum of the class token encoded representation sequence is output as a feature vector of the input image.

2. The image processing device according to claim 1,

3. The integration means includes: The basic statistics of the class token encoded representation sequence are output as a feature vector of the input image.

2. The image processing device according to claim 1,

4. The integration means includes: The class token encoding sequence is integrated by a neural network to obtain a feature vector of the input image.

2. The image processing device according to claim 1,

5. The integration means includes: a feature vector token adding means for adding one feature vector token to the class token encoded representation sequence; a second encoding means for obtaining a second encoded representation string from the class token encoded representation string obtained by the feature vector token adding means; and a feature vector acquisition means for acquiring, as a feature vector of the input image, a feature vector encoded expression corresponding to a feature vector token in the second encoded expression string obtained from the second encoding means.

2. The image processing device according to claim 1,

6. The learning is performed by a loss function including a regularization term that reduces at least one of the similarity between the two or more class tokens and the similarity between the class token encoded representations corresponding to each of the two or more class tokens.

6. The image processing device according to claim 1,

7. The encoding means calculates an attention map which is a matrix of number of tokens by number of tokens, the matrix having relationships between each token as elements; The learning is performed by a loss function including a regularization term that reduces at least one of the similarity between the two or more class tokens, the similarity between the class token encoded representations corresponding to each of the two or more class tokens, and the similarity between the attention maps corresponding to each of the two or more class tokens.

6. The image processing device according to claim 1,

8. Each of the two or more class tokens is trained using common training data.

8. The image processing device according to claim 1,

9. Each of the two or more class tokens is initialized with a random value and trained.

9. The image processing device according to claim 1,

10. Each of the two or more class tokens is initialized and trained to be orthogonal to each other.

10. The image processing device according to claim 9,

11. an image acquisition step of acquiring an input image; A division step of dividing the input image into a sequence of partial images, which are a plurality of partial images; a token conversion step of converting the partial image sequence divided by the division step into a token sequence by converting the partial image into a token having a fixed dimensional vector; an addition step of adding two or more class tokens having vectors of the same dimension as the tokens to the token string obtained by the token conversion step; an encoding step of updating the token sequence obtained by the adding step based on the associations between the tokens to obtain an encoded representation sequence; a class token encoded expression sequence acquisition step of acquiring class token encoded expressions corresponding to the two or more class tokens as class token encoded expression sequences from the encoded expression sequence obtained by the encoding step; a class token integration step of integrating the class token encoded representation sequence to obtain a feature vector of the input image; 13. A method for controlling an image processing device, comprising:

12. A program that, when read and executed by a computer, causes the computer to execute each step of the method according to claim 11.

Images