Image processing device, control method and program for image processing device

Abstract

To provide an image processing apparatus which has improved feature extraction performance with multi-task learning by using a feature extractor employing a Transformer model, and a control method and program thereof.SOLUTION: An image processing apparatus 100 includes: an image acquisition unit 201 which acquires an image including an object; an image division unit 203 which divides the acquired image to generate a plurality of partial images; a token conversion unit 204 which converts the divided partial images to tokens being fixed-dimensional vectors; a token addition unit 205 which adds class tokens having fixed dimensions corresponding to the tokens, to a string of the converted tokens; an encoding unit 206 which updates the string of the tokens having the class tokens added thereto on the basis of relations between the tokens; a feature extraction unit 207 which extracts a feature quantity of the object from encoded representations corresponding to updated class tokens; and an attribute determination unit 208 which determines an attribute of the object from the encoded representations corresponding to the updated class tokens.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention particularly relates to an image processing device that performs multitask learning in machine learning, a control method for an image processing device, and a program. [Background technology]

[0002] In recent years, many feature extraction technologies have been proposed that extract useful information from images of objects captured in photographed images through advanced processing. Among the feature extraction technologies, many have been proposed, particularly those related to feature extractors that use multi-layered neural networks called deep nets (also called deep neural nets or deep learning) to extract feature vectors of objects in images.

[0003] The feature extractor is trained, for example, by a technique called distance learning. In distance learning, the distance between feature vectors is first defined, and examples of the definition of distance include Euclidean distance and cosine similarity. In distance learning, the feature extractor is trained so that feature vectors with close distances are output for images showing the same object or objects belonging to the same class, and feature vectors with far distances are output for images showing different objects or objects belonging to different classes. Known applications of distance learning include determining the identity of cars and face recognition. In particular, a technology for achieving face recognition using distance learning is disclosed in Non-Patent Document 1.

[0004] Furthermore, in machine learning, a technique called multi-task learning is known, in which two or more related tasks are simultaneously learned. In multi-task learning, one model is trained to simultaneously solve one main task and one or more subtasks, which often improves the predictive performance of the main task. Non-Patent Document 2 discloses a technique using a neural network that simultaneously learns a face recognition task and a face attribute determination task, thereby improving the accuracy of the main task, that is, face recognition. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Deng, Jiankang, et al. "Arcface: Additive angular margin loss for deep face recognition." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019. [Non-patent document 2] Wang, Zhanxiong, et al. "Multi-task deep neural network for joint face recognition and facial attribute prediction." Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval. 2017. [Non-patent document 3] Dosovitskiy, Alexey, et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929(2020). [Non-patent document 4] Zhong, Yaoyao, and Weihong Deng. "Face Transformer for Recognition." arXiv preprint arXiv:2103.14803(2021). Summary of the Invention [Problem to be solved by the invention]

[0006] In recent years, feature extractors using Transformer models, which can achieve accuracy equal to or better than that of feature extractors using convolutional neural networks, have rapidly begun to be used. However, when using Transformer models for face recognition and other applications, methods for improving feature extraction performance by incorporating a multi-task learning framework have not been fully investigated.

[0007] In view of the above-mentioned problems, the present invention aims to improve the performance of feature extraction through multi-task learning using a feature extractor that employs a Transformer model. [Means for solving the problem]

[0008] The image processing device of the present invention is an image processing device that performs multi-task learning of a main task of extracting features of an object and a sub-task of determining the attributes of the object, and is characterized by having: an acquisition means for acquiring an image including the object; a division means for dividing the image acquired by the acquisition means to generate a plurality of partial images; a conversion means for converting the partial images divided by the division means into tokens which are vectors of fixed dimension; an addition means for adding class tokens having fixed dimensions corresponding to the tokens to the sequence of tokens converted by the conversion means; an encoding means for updating the sequence of tokens to which the class tokens have been added by the addition means based on the relevance between the tokens; an extraction means for extracting features of the object from the class tokens updated by the encoding means; and a determination means for determining the attributes of the object from the class tokens updated by the encoding means. [Effects of the Invention]

[0009] According to the present invention, it is possible to improve the performance of feature extraction through multi-task learning using a feature extractor that employs a Transformer model. [Brief explanation of the drawings]

[0010] [Figure 1]FIG. 2 is a block diagram illustrating an example of a hardware configuration of an image processing apparatus. [Figure 2] 1 is a block diagram illustrating an example of a functional configuration of an image processing apparatus according to a first embodiment. [Figure 3] 10 is a flowchart illustrating an example of an inference processing procedure according to the first embodiment. [Figure 4] FIG. 10 is a diagram illustrating an example of normalization processing of a face image. [Figure 5] FIG. 10 is a block diagram showing an example of a configuration for performing one encoding process by a Transformer Encoder. [Figure 6] 5 is a flowchart illustrating an example of a learning process procedure according to the first embodiment. [Figure 7] FIG. 2 is a schematic diagram for explaining an inference process in the first embodiment. [Figure 8] 10 is a flowchart illustrating an example of an inference processing procedure according to the second embodiment. [Figure 9] FIG. 10 is a schematic diagram for explaining an inference process in the second embodiment. [Figure 10] 11 is a flowchart illustrating an example of an inference processing procedure according to the third embodiment. [Figure 11] FIG. 11 is a schematic diagram for explaining an inference process in the third embodiment. [Figure 12] 13 is a flowchart showing an example of an inference processing procedure according to the fourth embodiment. [Figure 13] FIG. 13 is a schematic diagram for explaining an inference process in the fourth embodiment. [Figure 14] 13 is a flowchart illustrating an example of a learning process procedure according to the fourth embodiment. [Figure 15] FIG. 10 is a block diagram illustrating an example of the functional configuration of an image processing apparatus according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the configurations shown in the drawings. The following describes an example of a feature extractor that extracts feature vectors for identifying the identity of a face from an input image containing a human face (hereinafter referred to as a face image). However, the object from which the feature vector is extracted is not limited to a face, and any object whose identity can be determined by the feature vector may be used. Furthermore, in the following embodiments, an example of multi-task learning will be described in which the main task is a feature extraction task and the subtask is a face attribute determination task.

[0012] (First embodiment) 1 is a block diagram showing an example of the hardware configuration of an image processing device 100 according to this embodiment. The image processing device 100 has, as its hardware configuration, a control device 11, a storage device 12, a calculation device 13, an input device 14, an output device 15, and an I / F device 16.

[0013] The control device 11 controls the entire image processing device 100. The storage device 12 stores programs and data necessary for the operation of the control device 11. The arithmetic device 13 executes necessary arithmetic processing based on control from the control device 11. The input device 14 is a human interface device or the like, which inputs user operations to the image processing device 100. The output device 15 is a display or the like, which presents the processing results of the image processing device 100 to the user.

[0014] The I / F device 16 is a wired interface such as a universal serial bus, a local area network, or an optical cable, or a wireless interface such as Wi-Fi or Bluetooth (registered trademark). The I / F device 16 also connects a camera or the like to input captured images to the image processing device 100, and transmits processing results obtained by the image processing device 100 to the outside. Furthermore, the I / F device 16 inputs programs, various data, and the like required for the operation of the image processing device 100 to the image processing device 100 as necessary.

[0015] 2 is a block diagram showing an example of the functional configuration of an image processing device 100 according to this embodiment. The functional configuration of the image processing device 100 includes an image acquisition unit 201, an image normalization unit 202, an image division unit 203, a token conversion unit 204, a token addition unit 205, an encoding unit 206, a feature extraction unit 207, an attribute determination unit 208, and a learning unit 209. Details of the learning unit 209 will be described later.

[0016] The image acquisition unit 201 acquires a facial image. The facial image acquired by the image acquisition unit 201 may be an image stored in the storage device 12, or may be a captured image received from the I / F device 16 via a network such as the Internet.

[0017] The image normalization unit 202 acquires a normalized face image by performing normalization processing on the face image acquired by the image acquisition unit 201. The normalized face image is an image extracted so that both the vertical and horizontal widths have a predetermined number of pixels, the face is not tilted to the left or right, the size of the face is approximately constant, and the relative position of the face with respect to the normalized face image is approximately constant. The purpose of the normalization processing is to reduce variations in the in-plane rotation angle and size of the face shown in the face image.

[0018] The image dividing unit 203 divides the normalized face image normalized by the image normalization unit 202 into partial images, each having a predetermined number of pixels in both height and width, to obtain a sequence of partial images. The token conversion unit 204 converts the sequence of partial images obtained by the image dividing unit 203 into a sequence of tokens. Here, a token is a vector having a certain fixed dimension (hereinafter referred to as D). The conversion from a partial image to a token is performed by flattening the partial image to convert it into a vector, and then performing a linear transformation to convert the number of dimensions of the vector into D. Note that the linear transformation may include not only a process of multiplying the input by a weighting matrix, but also a process of multiplying the input by a weighting matrix and then adding a bias vector.

[0019] The token adding unit 205 adds D-dimensional tokens called class tokens to the token sequence acquired by the token converting unit 204. The encoding unit 206 performs encoding processing to update the token sequence acquired by the token adding unit 205 one or more times. Details of the encoding processing will be described later. The finally obtained token sequence is called an encoded representation sequence.

[0020] The feature extraction unit 207 acquires a feature vector for identifying the identity of the face in the facial image by using an encoded expression corresponding to a class token from the encoded expression sequence acquired by the encoding unit 206. The acquired feature vector is, for example, a feature used by the face recognition unit 210 to determine whether objects included in two images are the same object. The face recognition unit 210 determines that the objects (people) included in the two images are the same if the distance between the feature amounts extracted from the two images is less than a predetermined value, and determines that the objects are different if the distance between the feature amounts is equal to or greater than the predetermined value. The attribute determination unit 208 determines the attribute of the face in the facial image by using an encoded expression corresponding to a class token from the encoded expression sequence acquired by the encoding unit 206.

[0021] Next, an example of the flow of the inference process of the image processing device 100 according to this embodiment will be described with reference to the flowchart of FIG. 3 and the schematic diagram of FIG. In S301, the image acquisition unit 201 acquires a face image 400. For ease of explanation, it is assumed that the face image 400 contains only one face 410 as shown in FIG.

[0022] In S302, the image normalization unit 202 performs normalization processing on the face image 400 acquired by the image acquisition unit 201, thereby acquiring a normalized face image 405. Here, an example of the face image normalization processing will be described with reference to FIG. 4. The face image 400 is an image including one face 410. In the normalization processing, the positions of organ points of the face 410, such as the eyes, nose, and mouth, are estimated using a known organ point position estimation method for the face image 400. In the example of FIG. 4, it is assumed that the positions of a right eye center 401, a left eye center 402, a right mouth edge 403, and a left mouth edge 404 have been estimated. Next, a normalized face image 405 is extracted from the face image 400 based on the estimated organ point positions. This extraction is performed by applying a geometric transformation such as an affine transformation to the face image 400 so that the right eye center 401, left eye center 402, right mouth edge 403, and left mouth edge 404 approach predetermined positions 406, 407, 408, and 409, respectively, of the normalized face image 405.

[0023] In S303, the image dividing unit 203 divides the normalized face image 405 acquired by the image normalization unit 202 into partial images each having a predetermined number of pixels in both height and width, and acquires a sequence of partial images 701. Here, the image dividing unit 203 divides the normalized face image 405 so that every pixel in the normalized face image 405 belongs to exactly one partial image, that is, so that there are no gaps or overlaps between the partial images. In the following, the height of the normalized face image is H pixels, the width is W pixels, and the height of the partial images is P H Pixels, vertical width is P W Let C be the number of pixels and channels. H is P H So, W is P W The length N of the partial image sequence 701 is (H / P H )×(W / P W 7, the length N of the partial image sequence 701 is 9. However, the division into partial images is not limited to the above, and as disclosed in Non-Patent Document 4, the image may be divided so that the partial images overlap each other by a predetermined number of pixels.

[0024] In S304, the token conversion unit 204 converts the partial image sequence 701 acquired by the image division unit 203 into a token sequence 702. An example of the conversion process from partial images to tokens will be described below. First, the partial images are flattened to reduce the number of dimensions to P H P W A vector of C is obtained. Next, a linear transformation is performed on this vector to obtain a vector (token) with a dimensionality of D. Note that the parameters of the linear transformation are parameters learned by the learning unit 209, as will be described later.

[0025] In S305, the token adding unit 205 adds one D-dimensional token called a class token to the token sequence 702 acquired by the token converting unit 204, thereby obtaining a new token sequence 703. This process makes the length of the token sequence N+1 (=N'). Note that the parameters of the class token are also parameters learned by the learning unit 209. In the example of FIG. 7, the token located at the left end of the token sequence 703 is the class token.

[0026] Furthermore, the token adding unit 205 may add a D-dimensional vector called a position embedding vector to each of the N' tokens. Here, the position embedding vector has a different value depending on the position in the token sequence 703, and holds information on which position each token corresponds to in the normalized face image 405. The position encoding vector may be artificially designed, or may be learned in advance by the learning unit 209.

[0027] In S306, the encoding unit 206 performs encoding processing one or more times on the token sequence 703 acquired by the token adding unit 205. Here, encoding processing refers to processing for updating each token based on the relationship between the tokens that make up the token sequence. For example, a method using a Transformer Encoder disclosed in Non-Patent Document 3 is adopted as the encoding processing.

[0028] The following describes a method for performing encoding using a Transformer Encoder. 5 is a block diagram showing an example of a configuration for performing one encoding process by a Transformer Encoder 501. The Transformer Encoder 501 is configured to input a token sequence and output a token sequence. The Transformer Encoder 501 has a first layer normalization unit 502, an attention unit 503, and a first skip combination unit 504. The Transformer Encoder 501 further has a second layer normalization unit 505, an MLP (Multi Layer Perceptron) unit 506, and a second skip combination unit 507.

[0029] The token sequence is first input to the first layer normalization unit 502. The first layer normalization unit 502 performs normalization called layer normalization on the token sequence, and each token is normalized independently. The parameters required for layer normalization are parameters learned by the learning unit 209, as will be described later.

[0030] The token sequence normalized by the first layer normalization unit 502 is updated by the attention unit 503. The attention unit 503 has M attention mechanisms (M is an integer equal to or greater than 1). When M>1, the M attention mechanisms each acquire different parameter weights through learning by the learning unit 209, and associations between tokens can be discovered from multiple perspectives.

[0031] Here, we will explain how a token sequence is updated by one of the M attention mechanisms. The token sequence is expressed as an N' × D matrix. A weight matrix U Q , U K , U V By multiplying these, we obtain the query matrix Q, the key matrix K, and the value matrix V. Here, the weight matrix U Q , U K , U VAll are DxD h The query matrix Q, key matrix K, and value matrix V all have a size of N' × D h It has the size of D h is the number of dimensions of the vector output by each attention mechanism, which is usually set as D / M.

[0032] Next, the attention matrix A is obtained using the softmax function according to the following equation (1). The attention matrix A has a size of N' x N' and reflects the relevance between all tokens. A=softmax(QK T / √D h ) ···(1)

[0033] By multiplying the attention matrix A by the value matrix V, we get N' × D h This matrix can be interpreted as updating the value of each token by taking a weighted sum of the values ​​of other tokens according to the strength of their association with each other token.

[0034] By performing the above calculation for each of the M attention mechanisms, we obtain N' × D h M matrices of size N' are obtained, and by combining these, an N'×D matrix is ​​obtained. By multiplying this matrix by a matrix of size D×D obtained through learning by the learning unit 209, an N'×D matrix is ​​obtained that integrates the results of the M attention mechanisms. The token sequence represented by this matrix is ​​the final output from the attention unit 503.

[0035] The first skip combiner 504 adds the token sequence before being input to the first layer normalizer 502 and the token sequence output from the attention unit 503. The second layer normalizer 505 performs layer normalization on the token sequence output from the first skip combiner 504. The layer normalization performed by the second layer normalizer 505 is similar to the processing performed by the first layer normalizer 502.

[0036] The MLP unit 506 performs linear transformation, nonlinear activation processing (such as ReLU or GELU), and linear transformation, in that order, independently on each token in the token sequence layer-normalized by the second layer normalization unit 505. Note that the parameters of the linear transformation are also parameters learned by the learning unit 209. The output of the MLP unit 506 is a matrix of size N' x D. The second skip combiner 507 adds the output of the first skip combiner 504 and the output of the MLP unit 506.

[0037] As described above, the output of the second skip combiner 507 is the output obtained by one encoding process using a Transformer Encoder. As described above, the process of updating the token sequence is performed one or more times using an encoder such as a Transformer Encoder, and the final token sequence obtained is used as the encoded representation sequence.

[0038] Returning to the explanation of Fig. 3, in S307, the feature extraction unit 207 uses the encoded expression 705 corresponding to the class token among the encoded expression sequence 704 acquired by the encoding unit 206 to acquire a feature vector for identifying the identity of the face appearing in the facial image. There are various methods for acquiring the feature vector. Below, the methods for acquiring the feature vector will be described.

[0039] The first method for obtaining a feature vector is to directly use the encoded representation 705 corresponding to the class token. In this case, the number of dimensions of the feature vector is D. The second method for acquiring a feature vector is to input the encoded representation 705 corresponding to the class token to a head 706 for converting the feature vector. This head is typically a feedforward network (FFN) configured by one or more linear transformations. The number of dimensions of the feature vector is equal to the output dimension of this head. The parameters of this FFN are learned by a learning unit 209, as will be described later.

[0040] Next, an example of a method for determining the identity of faces using this feature vector will be described. For example, when two face images are given, the face recognition unit 210 determines whether the faces belong to the same person.

[0041] First, feature vectors f1 and f2 are extracted from the first and second facial images, respectively. Here, it is assumed that the feature vectors f1 and f2 have the same number of dimensions. Next, the similarity between the two feature vectors is calculated using a predetermined method. For example, when using cosine similarity, the similarity score is calculated using the following formula (2). Similarity score(f1,f2):=cos(θ 12 ) =<f1,f2> ÷(|f1|·|f2|) ···(2)

[0042] (2) In the formula, θ 12 is the angle between feature vectors f1 and f2,<x,y> is the dot product of x and y, and |x| is the length of x. If the similarity score is below a predetermined threshold, it is determined that they are the same person, and if it exceeds the threshold, it is determined that they are different people. The method of calculating similarity is not limited to the above, and any known method such as Euclidean distance may be used.

[0043] In S308, the attribute determination unit 208 determines the attributes of the face in the face image using the encoded expression 705 corresponding to the class token from the encoded expression sequence 704 acquired by the encoding unit 206. The determined face attributes include whether or not the face is wearing a mask, whether or not the face is wearing glasses (sunglasses), the face direction, whether the eyes are open or closed, facial expression (whether the face is smiling or angry, etc.), age, race, sex, etc. There are also various methods for determining attributes.

[0044] The first method for determining attributes is to prepare a unique head for each attribute determination task. This head is typically an FFN consisting of one or more linear transformations and activation functions such as a sigmoid function or a softmax function. The parameters of this FFN are also trained by the training unit 209.

[0045] For example, the task head 707 that determines whether a face mask is being worn can be configured with a neural network that solves binary classification problems. One example is a neural network that connects one or more linear transformation and activation processes with a sigmoid function. Here, the final linear transformation has one unit, meaning that the output is a scalar value. This output is input to the sigmoid function, which outputs a continuous value between 0 and 1. This value can be interpreted as the probability that the face 410 in the face image 400 is wearing a mask.

[0046] On the other hand, for example, the task head 708 for determining age can be configured as a neural network for solving regression problems if age is treated as a continuous value, or as a neural network for solving multi-class classification problems if age is treated as a class divided into 10-year age increments. For example, the multi-class classification problem is defined as classifying age into 10 classes in 10-year increments: 0-9 years, 10-19 years, ..., 80-89 years, and 90 years and above. In this case, the task head 708 can be configured as a neural network connected to one or more linear transformation and activation processes and a softmax function. Here, the number of units in the final linear transformation is set to 10. The output of this function is input to the softmax function. The output of the softmax function can be interpreted as the probability that the age of face 410 in face image 400 belongs to one of the 10 classes.

[0047] In the above, the inference processing of the image processing device 100 according to this embodiment has been described as performing both the acquisition of feature vectors in S307 and the attribute determination in S308, but it is also possible to perform only one of these. By performing only the processing for the necessary tasks, the calculation time can be reduced.

[0048] Next, a learning method for the image processing device 100 according to this embodiment will be described. As described above, multi-task learning, which simultaneously learns two tasks, a feature extraction task and a facial attribute determination task, can derive better performance than learning each task individually. Furthermore, prior to multi-task learning, it is necessary to collect learning data in advance. Therefore, a method for creating learning data will be described.

[0049] First, a large number of face images with known person IDs are collected. The number of different person IDs in the learning data is C, and person IDs 1 to C are assigned in order. Next, normalization processing is performed on the face images to obtain normalized face images. The method for generating normalized face images is the same as the processing procedure in S302. Note that if estimation of the positions of organ points fails, that face image is not included in the learning data.

[0050] Next, correct answer data for the attributes of the facial images is created. For example, when performing two facial attribute determination tasks, determining whether or not a face is wearing a mask and determining age, correct answer values ​​for those attributes are assigned to all collected facial images. Correct answer values ​​can be assigned manually, or they can be assigned automatically using an existing attribute classifier and then manually corrected.

[0051] Furthermore, when learning a task to extract features, a classification layer that inputs feature vectors and outputs a C-dimensional probability vector is connected after the processing of the feature extraction unit 207. The i-th component (i is an integer between 1 and C) of the C-dimensional probability vector represents the probability that the person ID of the face in the facial image is i.

[0052] In particular, the feature extraction unit 207 may be configured to be able to learn by focusing on the angle formed between the feature vectors output by the feature extraction unit 207. Specifically, for images in which the same person appears, the feature extraction unit 207 is trained so that the angle formed between the feature vectors output by the feature extraction unit 207 becomes smaller. Also, for images in which different people appear, the feature extraction unit 207 is trained so that the angle formed between the feature vectors output by the feature extraction unit 207 becomes larger. Such learning can be achieved by the representative vector method disclosed in Non-Patent Document 1. In the representative vector method, representative vectors V1 to V2 corresponding to each person ID (1 to C) are C The classification layer stores the i-th component (i is an integer between 1 and C) of the C-dimensional probability vector output by the classification layer. The i-th component (i is an integer between 1 and C) of the C-dimensional probability vector output by the classification layer is stored in ... i It is calculated by the dot product of

[0053] In addition, by performing learning using the backpropagation method described later, when an image with correct person ID j (j is an integer between 1 and C) is given, the feature vector extracted from this image and the representative vector V j The angle between the representative vector V and the j The learning method is performed so that the angle between the representative vector and other representative vectors becomes as large as possible. This learning method is disclosed in Non-Patent Document 1.

[0054] Hereinafter, the classification layer and representative vectors connected to the feature extraction unit 207 only during learning will be collectively referred to as an additional block. The additional block is used only during learning, and is not used during inference.

[0055] Next, an example of the flow of the learning process of the image processing device 100 according to this embodiment will be described with reference to the flowchart of FIG. In S601, the learning unit 209 of the image processing device 100 initializes parameters to be learned. Here, the parameters to be learned include linear transformation parameters of the token conversion unit 204, parameters of class tokens added by the token addition unit 205, and position encoding vectors. Furthermore, the parameters to be learned also include parameters used in the encoding process of the encoding unit 206, linear transformation parameters of the feature extraction unit 207, parameters of additional blocks of the feature extraction unit 207, and attribute determination parameters of the attribute determination unit 208. The parameters are initialized by random initialization, zero initialization, or the like. Furthermore, the parameters may be initialized using parameters that have already been learned in another task using a transfer learning framework.

[0056] In S602, the image processing device 100 acquires a face classification result and a face attribute determination result from the normalized face image of the learning data. This method will be explained in order. First, the image processing device 100 acquires a face feature vector and a face attribute determination result from the normalized face image by the processing procedure from S301 to S308 in FIG. 3. Next, an additional block in the feature extraction unit 207 acquires a probability vector indicating to which class of person IDs (1 to C) the face in the normalized face image belongs. This probability vector is called the face classification result. The processing in S602 is usually performed as mini-batch learning on multiple normalized face images randomly selected from the learning data at once. This is expected to speed up and stabilize learning.

[0057] In S603, the learning unit 209 calculates a first loss from the face classification result obtained in S602 and the correct person ID. This first loss represents an error in the classification result, and is designed to be smaller the higher the probability that the face is classified into the correct person class and larger the lower the probability. The first loss can be calculated, for example, using a multi-class cross-entropy error function that takes as input the face classification result output from the classification layer connected to the feature extraction unit 207 and a One-Hot vector. Here, the One-Hot vector is a C-dimensional vector in which, when the correct person ID of the face image is j (j is an integer between 1 and C), only the j-th component is 1 and the other components are 0.

[0058] In S604, the learning unit 209 calculates a second loss from the face attribute determination result acquired in S602 and the correct attribute. This second loss is calculated by selecting an appropriate function depending on the problem to which the attribute determination has been reduced. For example, if the attribute determination problem has been reduced to binary class classification, the second loss is calculated using a binary cross-entropy error function. Also, if the attribute determination problem has been reduced to multi-class classification, the second loss is calculated using a multi-class cross-entropy error function. Also, if the attribute determination problem has been reduced to a regression problem, the second loss is calculated using a squared error function, absolute error function, or the like.

[0059] In S605, the learning unit 209 adjusts the parameters of the learning target so as to reduce the weighted sum of the first loss and the second loss. This method will be explained in order. First, let us assume that the first loss is L0, the number of attribute determination problems is A, and the second losses of each attribute determination problem are L1 to L2. A In this case, the learning unit 209 calculates the loss L by using a pre-set hyperparameter λ i (i is an integer between 1 and A) Specifically, the loss L is calculated according to the following formula (3).

[0060]

number

[0061] Next, the learning unit 209 updates the parameters to be learned so as to reduce the loss L. The parameters are minutely updated in a direction that reduces the loss L by using the backpropagation method that is common in deep networks. The parameters are learned so that the weighted sum of the loss in the feature extraction task and the loss in the face attribute determination task is simultaneously reduced. In this way, the feature extraction unit 207 is trained to have the ability to output feature vectors that can simultaneously extract face feature vectors and determine face attributes with high accuracy.

[0062] In S606, the learning unit 209 determines whether to continue updating the parameters. The criteria for this determination are whether the number of learning iterations has reached a sufficient number, or whether the model performance has improved sufficiently. If either of these criteria is met, it is determined that the parameter update should be terminated, and the process proceeds to S607. On the other hand, if not, the process returns to S602 and continues learning.

[0063] In S607, the learning unit 209 stores the learned parameters in the storage device 12. When the image processing device 100 performs an inference process or when the image processing device 100 performs a learning process again, the stored parameters are used.

[0064] As described above, according to the image processing device 100 of this embodiment, it is possible to improve the performance of a feature extractor that uses a Transformer model by incorporating a multitask learning framework.

[0065] (Second embodiment) In this embodiment, a configuration using two class tokens will be described. Explanation of the same parts as in the first embodiment will be omitted, and only the differences will be described.

[0066] In the first embodiment, a configuration was described in which one class token is added to a token sequence obtained from a partial image sequence, and both feature vector acquisition and facial attribute determination are performed from the encoded representation corresponding to the class token. Consider a case where, for example, two facial images of the same person are taken, one of which is a facial image (image a) of a face wearing a mask and the other is a facial image (image b) of a face not wearing a mask. Since the two facial images are of the same person, it is desirable that the distance between the feature vectors extracted from each image is as close as possible. Meanwhile, the correct attribute determination result for the two facial images is that image a has a mask and image b does not have a mask. Thus, for two facial images of the same person, the feature extraction task must extract a feature vector that accurately represents the person's identity regardless of changes in the appearance of the facial image or changes over time. Meanwhile, the attribute determination task must obtain different determination results for each image.

[0067] In this embodiment, a configuration will be described that takes into account the different properties between the feature extraction task and the attribute determination task as described above, and that uses two class tokens to enable more accurate inference in the feature extraction task and the attribute determination task. An example of the flow of the inference processing of the image processing device 100 according to this embodiment will be described below using the flowchart in FIG. 8 and the conceptual diagram in FIG. 9. Note that the internal configuration of the image processing device 100 is the same as in the first embodiment, so a description will be omitted and only the parts that differ from the inference processing in FIG. 3 will be described.

[0068] 8 are the same as S301 to S304 in FIG. 3, respectively. In S801, the token adding unit 205 adds two D-dimensional class tokens A and B to the token sequence 702 acquired by the token conversion unit 204, thereby obtaining a new token sequence 903. This process makes the length of the token sequence N+2 (=N'). As in the first embodiment, the parameters of the class tokens are parameters learned by the learning unit 209. In the example of FIG. 9, the token located at the left end of the token sequence 903 is class token A, and the token located second from the left is class token B. Thereafter, the encoding unit 206 performs the same process as S306 in FIG. 3.

[0069] In S802, the feature extraction unit 207 acquires a feature vector for identifying the identity of the face in the facial image by using the encoded expression 905 corresponding to the class token A from the encoded expression sequence 904 acquired by the encoding unit 206. The method for acquiring the feature vector is the same as the method described in S307 of FIG.

[0070] In S803, the attribute determination unit 208 determines the attribute of the face appearing in the face image by using the encoded expression 906 corresponding to class token B from the encoded expression sequence 904 acquired by the encoding unit 206. The method of determining the attribute is the same as the method described in S308 of FIG.

[0071] An example of the flow of the inference process of the image processing device 100 according to this embodiment has been described above. The number of tokens N' after class tokens are assigned is N+1 in the first embodiment, whereas it is N+2 in this embodiment. The order of the time complexity required for inference is O(N' 2 ) (O is the Landau symbol), the order of the time complexity does not change significantly even if one class token is added. The flow of the learning process of the image processing device 100 according to this embodiment is the same as the learning flow of the first embodiment, except that the number of class tokens is now two. That is, in this embodiment, in S602, the facial feature vector and facial attribute determination result are obtained according to the procedure shown in FIG. 8.

[0072] As described above, according to this embodiment, separate class tokens are provided for the feature extraction task and the attribute determination task. This allows each class token to be specialized to output an optimal encoded representation for each task, enabling more accurate inference in the feature extraction task and the attribute determination task.

[0073] (Third embodiment) In the second embodiment, one class token is added to the token sequence obtained from the partial image sequence for each of the feature extraction task and the attribute determination task. Then, from the encoded representation corresponding to each class token, both feature vectors are acquired and facial attributes are determined. A use case exists in which facial images of the same person are observed over a long period of time using a fixed camera installed in a certain location, and the attribute determination task is repeated. For example, attributes such as "whether or not a mask is worn," "whether or not sunglasses are worn," "face direction," "blink," "facial expression," and "age" are likely to change, while attributes such as "race" and "gender" are less likely to change. Therefore, by dividing attributes into multiple classes according to the variability of the attribute task and assigning a different class token to each class, it is expected that an optimal encoded representation according to the variability of the attribute can be obtained.

[0074] Therefore, in this embodiment, a configuration will be described that takes into account the differences in the properties of the attribute determination tasks as described above and uses three or more class tokens to enable more accurate inference in the feature extraction task and the attribute determination task. An example of the flow of the inference processing of the image processing device 100 according to this embodiment will be described below using the flowchart in Fig. 10 and the conceptual diagram in Fig. 11. Note that since the internal configuration of the image processing device 100 is the same as in the first embodiment, a description will be omitted and only the parts that differ from the inference processing in Fig. 3 will be described.

[0075] In S1001, the token adding unit 205 adds three D-dimensional class tokens A, B, and C to the token sequence 702 acquired by the token conversion unit 204, thereby obtaining a new token sequence 1103. This process makes the length of the token sequence N+3 (=N'). As in the first embodiment, the parameters of the class tokens are parameters learned by the learning unit 209. In the example of FIG. 11, the class token C, class token B, and class token A are arranged in this order from the left end of the token sequence 1103. Thereafter, the encoding unit 206 performs the same process as S306 in FIG. 3.

[0076] In S1002, the feature extraction unit 207 acquires a feature vector for identifying the identity of the face in the facial image by using the encoded expression 1105 corresponding to the class token A from the encoded expression sequence 1104 acquired by the encoding unit 206. The method for acquiring the feature vector is the same as the method described in S307 of FIG.

[0077] In S1003, the attribute determination unit 208 determines the attributes of the face appearing in the face image by using the encoded expression 1106 corresponding to class token B from the encoded expression sequence 1104 acquired by the encoding unit 206. Here, the target attributes are assumed to be gender, race, etc., which are unlikely to change from image to image if the face images are of the same person. The method of determining the attributes is the same as the method described in S308 of FIG. 3.

[0078] In S1004, the attribute determination unit 208 determines the attributes of the face shown in the face image by using the encoded expression 1107 corresponding to the class token C from the encoded expression sequence 1104 acquired by the encoding unit 206. Here, the target attributes are assumed to be things that tend to change from image to image, even for face images of the same person, such as whether or not a mask is worn, age, etc. The method of determining the attributes is the same as the method described in S308 of FIG. 3.

[0079] An example of the flow of inference processing by the image processing device 100 according to this embodiment has been described above. The number of tokens N' after class tokens are assigned is N+1 in the first embodiment, whereas it is N+3 in this embodiment. As in the second embodiment, increasing the number of class tokens by two does not significantly change the order of time complexity. In the above example, attributes are divided into two classes based on their variability, and one class token is prepared for each class. Alternatively, attributes may be divided into P classes (P is an integer greater than or equal to 3 and less than the number of attribute tasks) based on their variability, and one class token may be prepared for each class. The flow of learning processing by the image processing device 100 according to this embodiment is the same as that of the first embodiment, except that the number of class tokens is increased to three or more. That is, in this embodiment, in S602, facial feature vectors and facial attribute determination results are acquired according to the procedure shown in FIG. 10.

[0080] As described above, according to this embodiment, one class token is provided for the feature extraction task, and two or more class tokens are provided for the attribute determination task, focusing on the variability of attributes. This allows each class token to be specialized to output an optimal encoded representation for each task, enabling more accurate inference in the feature extraction task and the attribute determination task.

[0081] (Fourth embodiment) In this embodiment, a configuration will be described in which attributes are determined in a lower layer of the encoder, and the weights of the remaining layers of the encoder are changed depending on the result. In this embodiment, the same feature extraction is performed up to a certain point regardless of the face attribute determination result, and feature extraction parameters are changed depending on the face attribute determination result from that point on, thereby enabling feature extraction with high accuracy and a small amount of calculation. Below, a description of the same parts as in the first to third embodiments will be omitted, and only the differences will be described.

[0082] 15 is a block diagram showing an example of the functional configuration of the image processing device 100 according to this embodiment. Only the parts that differ from FIG. 2 will be described below. The common encoding unit 1501 performs encoding processing to update the token sequence one or more times on the token sequence acquired by the token adding unit 205. Hereinafter, the token sequence finally obtained after repeating the encoding processing is referred to as the first encoded expression sequence.

[0083] The switching attribute determining unit 1502 uses a part of the first encoded expression sequence acquired by the common encoding unit 1501 to determine the attribute of the face appearing in the face image. The individual encoding unit 1503 switches weight parameters for the remaining encoding processes according to the facial attribute determination result by the switching attribute determination unit 1502. Furthermore, the individual encoding unit 1503 uses the switched weight parameters to perform one encoding process to update the token sequence on the first encoded expression sequence 1302. The token sequence finally obtained after repeating the encoding process is called the second encoded expression sequence.

[0084] The feature extraction unit 1504 uses a part of the second encoded expression sequence acquired by the individual encoding unit 1503 to acquire a feature vector for identifying the identity of the face appearing in the facial image. The attribute determining unit 1505 uses a part of the second encoded expression sequence acquired by the individual encoding unit 1503 to determine the attribute of the face appearing in the face image. The learning unit 1506 performs the overall processing related to the learning process.

[0085] Next, an example of the flow of the inference processing of the image processing device 100 according to this embodiment will be described with reference to the flowchart of Fig. 12 and the conceptual diagram of Fig. 13. Only the parts that differ from the inference processing of Fig. 3 will be described below.

[0086] In S1201, the token adding unit 205 adds three D-dimensional class tokens A, B, and C to the token sequence 702 acquired by the token conversion unit 204, thereby obtaining a new token sequence 1301. This process makes the length of the token sequence N+3 (=N'). As in the first embodiment, the parameters of the class tokens are learned in advance through the learning phase. In the example of FIG. 13, class token C, class token B, and class token A are arranged in order from the left end of the token sequence 1301.

[0087] In S1202, the common encoding unit 1501 performs encoding processing to update the token sequence 1301 acquired by the token adding unit 205 one or more times to acquire a first encoded representation sequence 1302. Details of the encoding processing are the same as the method described in S306 of Fig. 3. This encoding processing is always performed regardless of the face attributes.

[0088] In S1203, the switching attribute determination unit 1502 determines the attributes of the face in the face image using the encoded expression 1303 corresponding to the class token C from the first encoded expression sequence 1302 acquired by the common encoding unit 1501. Hereinafter, the switching attribute determination unit 1502 will be described as determining whether or not a mask is being worn, but the attribute determination items are not limited to this and the unit may determine any one or more types of attributes. For example, the unit may determine a face orientation attribute or a race attribute. The method for determining attributes from the encoded expression is the same as the method described in S308 of FIG. 3.

[0089] In S1204, the individual encoding unit 1503 switches weight parameters for the remaining encoding processes based on the attribute determination result of the switching attribute determination unit 1502 in S1203. For example, if it is determined in S1203 that the face 410 is not wearing a mask, the individual encoding unit 1503 switches to weight parameters optimized for a face image without a mask. If it is determined that the face 410 is wearing a mask, the individual encoding unit 1503 switches to weight parameters optimized for a face image with a mask.

[0090] Next, in S1204, the individual encoding unit 1503 performs encoding processing to update the token sequence on the first encoded expression sequence 1302 one or more times using the switched weight parameters, thereby obtaining a second encoded expression sequence 1304. Details of the encoding processing are the same as those described in S306 of FIG.

[0091] In S1205, the feature extraction unit 1504 acquires a feature vector for identifying the identity of the face appearing in the facial image by using the encoded expression 1305 corresponding to the class token A from the second encoded expression sequence 1304 acquired by the individual encoding unit 1503. The method for acquiring the feature vector is the same as the method described in S307 of FIG.

[0092] In S1206, the attribute determination unit 1505 determines the attribute of the face appearing in the face image by using the encoded expression 1306 corresponding to class token B from the second encoded expression sequence 1304 acquired by the individual encoding unit 1503. The method of determining the attribute is the same as the method described in S308 of FIG.

[0093] Next, the flow of the learning process of the image processing device 100 according to this embodiment will be described with reference to the flowchart of FIG. In S1401, the learning unit 1506 initializes the parameters to be learned. The parameter initialization method is the same as the method described in S601 of FIG.

[0094] In S1402, the learning unit 1506 learns the parameters to be learned with the weight parameter switching process described in S1204 turned off. In other words, the parameters to be learned are learned without switching to the weight parameters used by the individual encoding unit 1503. Here, the parameter learning method will be briefly explained. First, the loss is defined as a weighted sum of the loss of the attribute determination unit for switching 1502, the loss of the feature extraction unit 1504, and the loss of the attribute determination unit 1505. Here, the coefficient of the weighted sum is a hyperparameter. Next, the parameters to be learned are updated so that this loss is reduced. The parameters are updated minutely in a direction that reduces the loss by using the backpropagation method commonly used in DNNs.

[0095] In S1403, the learning unit 1506 determines whether to continue updating the parameters. The criteria for this determination are whether the number of learning iterations has reached a sufficient number, or whether the model performance has improved sufficiently. If either of these criteria is met, it is determined that the parameter update should be terminated, and the process proceeds to S1404. On the other hand, if not, the process returns to S1402 and continues learning.

[0096] In S1404, the learning unit 1506 fixes the parameters up to the common encoding unit 1501 and excludes them from the learning target. Then, the individual encoding unit 1503 prepares copies of the weights for the individual encoding unit 1503 for the number of attribute classes determined by the switching attribute determination unit 1502. For example, if the switching attribute determination unit 1502 determines whether or not a mask is worn as a face attribute, the individual encoding unit 1503 makes two copies of the weight parameters for the individual encoding unit 1503 learned in S1402, one for when a mask is not worn and one for when a mask is worn. Then, the learning unit 1506 updates one of the copied weight parameters based on the determination result of the switching attribute determination unit 1502.

[0097] In S1405, the learning unit 1506 determines whether to continue updating the parameters. As in S1403, the criteria for this determination are whether the number of learning iterations has reached a sufficient number or whether the model performance has improved sufficiently. If either of these criteria is met, it is determined that the parameter update should be terminated, and the process proceeds to S1406. On the other hand, if not, the process returns to S1404 and continues learning.

[0098] In S1406, the learning unit 1506 stores the learned parameters in the storage device 12. When the image processing device 100 performs an inference process or when the image processing device 100 performs a learning process again, the stored parameters are used.

[0099] As described above, according to this embodiment, some of the facial attributes are determined in the lower layers of the encoder, and the weight parameters of the remaining layers of the encoder can be changed according to the determination results. As a result, the calculations in the lower layers of the encoder are common, and it is possible to reduce the amount of calculation while performing optimal feature extraction according to the facial attribute determination results.

[0100] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0101] The above-described embodiments are merely examples of specific implementations of the present invention, and should not be construed as limiting the technical scope of the present invention. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features. [Explanation of symbols]

[0102] 201 image acquisition unit, 203 image division unit, 204 token conversion unit, 205 token addition unit, 206 encoding unit, 207 feature extraction unit, 208 attribute determination unit

Claims

1. An image processing device that performs multitask learning of a main task of extracting a feature amount of an object and a subtask of determining an attribute of the object, an acquisition means for acquiring an image including the object; a dividing means for dividing the image acquired by the acquiring means to generate a plurality of partial images; a conversion means for converting the partial images divided by the division means into tokens, which are vectors of fixed dimensions; adding means for adding a class token having a fixed dimension corresponding to the token to the sequence of tokens converted by the converting means; an encoding means for updating the sequence of tokens to which the class token has been added by the adding means based on the association between the tokens; extraction means for extracting features of the object from the class tokens updated by the encoding means; a determination means for determining attributes of the object from the class tokens updated by the encoding means; 1. An image processing device comprising:

2. the adding means adds one class token having a fixed dimension corresponding to the token to the sequence of tokens converted by the converting means; the extraction means extracts a feature of the object from the updated one class token; 2. The image processing apparatus according to claim 1, wherein said determining means determines the attribute of said object from the same class token as the class token used by said extracting means.

3. the adding means adds a first class token and a second class token as the class tokens to the string of tokens converted by the converting means; the extraction means extracts a feature of the object from the updated first class token; 2. The image processing apparatus according to claim 1, wherein said determining means determines the attribute of said object from said updated second class token.

4. the adding means adds, as the class tokens, first to Nth class tokens (N is an integer of 3 or more) to the string of tokens converted by the converting means; the extraction means extracts a feature of the object from the updated first class token; 2. The image processing apparatus according to claim 1, wherein said determining means determines the attribute of said object from one of said updated second class token to N-th class token.

5. 5. The image processing apparatus according to claim 4, wherein said determining means changes a class token used to determine the attribute of said object in accordance with the variability of the attribute.

6. The image processing device described in any one of claims 1 to 5, characterized in that the encoding means calculates a query matrix, a key matrix, and a value matrix from the string of tokens to which the class token has been added by the adding means, calculates an attention matrix reflecting the degree of relevance between all tokens based on the product of the query matrix and the key matrix, and updates the string of tokens based on the product of the attention matrix and the value matrix.

7. The image processing device according to any one of claims 1 to 6, characterized in that the multitasking learning is performed by updating parameters used in the conversion means, the addition means, the encoding means, the extraction means, and the determination means so as to reduce a weighted sum of an error from the correct data of the feature extracted by the extraction means and an error from the correct data of the determination result by the determination means.

8. An image processing device that performs multitask learning of a main task of extracting a feature amount of an object and a subtask of determining an attribute of the object, an acquisition means for acquiring an image including the object; a dividing means for dividing the image acquired by the acquiring means to generate a plurality of partial images; a conversion means for converting the partial images divided by the division means into tokens, which are vectors of fixed dimensions; adding means for adding a class token having a fixed dimension corresponding to the token to the sequence of tokens converted by the converting means; a first encoding means for updating the sequence of tokens to which the class token has been added by the adding means based on the relevance between the tokens; a first determination means for determining a first attribute of the object from the class token updated by the first encoding means; a second encoding means for further updating the sequence of tokens updated by the first encoding means based on the association between the tokens in accordance with the determination result by the first determination means; extraction means for extracting features of the object from the class tokens updated by the second encoding means; a second determination means for determining a second attribute of the object from the class token updated by the second encoding means; 1. An image processing device comprising:

9. the object is a person's face, The image processing device according to any one of claims 1 to 8, characterized in that the attributes are one or more of: whether or not a mask is worn, whether or not glasses are worn, facial direction, blinking, facial expression, age, race, and gender.

10. 10. The image processing device according to claim 1, wherein the conversion means converts the image data into tokens by performing a conversion process including a linear conversion.

11. 11. The image processing apparatus according to claim 1, wherein the adding means further adds a vector relating to the position of the partial image before division to the token.

12. 12. The image processing device according to claim 1, further comprising a recognition unit that determines whether or not objects included in two images are the same based on the feature amount extracted by the extraction unit.

13. A control method for an image processing device that performs multitask learning of a main task of extracting a feature amount of an object and a subtask of determining an attribute of the object, comprising: acquiring an image including the object; a division step of dividing the image acquired in the acquisition step to generate a plurality of partial images; a conversion step of converting the partial images divided in the division step into tokens, which are vectors of fixed dimensions; an adding step of adding a class token having a fixed dimension corresponding to the token to the sequence of tokens converted in the converting step; an encoding step of updating the sequence of tokens to which the class token has been added in the adding step based on the relevance between the tokens; an extraction step of extracting features of the object from the class tokens updated in the encoding step; a determination step of determining attributes of the object from the class tokens updated in the encoding step; 1. A method for controlling an image processing apparatus, comprising:

14. A control method for an image processing device that performs multitask learning of a main task of extracting a feature amount of an object and a subtask of determining an attribute of the object, comprising: acquiring an image including the object; a division step of dividing the image acquired in the acquisition step to generate a plurality of partial images; a conversion step of converting the partial images divided in the division step into tokens, which are vectors of fixed dimensions; an adding step of adding a class token having a fixed dimension corresponding to the token to the sequence of tokens converted in the converting step; a first encoding step of updating the sequence of tokens to which the class token has been added in the adding step based on the relevance between the tokens; a first determination step of determining a first attribute of the object from the class token updated in the first encoding step; a second encoding step of further updating the sequence of tokens updated in the first encoding step based on the relevance between the tokens in accordance with the determination result in the first determination step; an extraction step of extracting features of the object from the class tokens updated in the second encoding step; a second determination step of determining a second attribute of the object from the class token updated in the second encoding step; 1. A method for controlling an image processing apparatus, comprising:

15. A program for controlling an image processing device that performs multitask learning of a main task of extracting a feature amount of an object and a subtask of determining an attribute of the object, acquiring an image including the object; a division step of dividing the image acquired in the acquisition step to generate a plurality of partial images; a conversion step of converting the partial images divided in the division step into tokens, which are vectors of fixed dimensions; an adding step of adding a class token having a fixed dimension corresponding to the token to the sequence of tokens converted in the converting step; an encoding step of updating the sequence of tokens to which the class token has been added in the adding step based on the relevance between the tokens; an extraction step of extracting features of the object from the class tokens updated in the encoding step; a determination step of determining attributes of the object from the class tokens updated in the encoding step; A program that causes a computer to execute the following.

16. A program for controlling an image processing device that performs multitask learning of a main task of extracting a feature amount of an object and a subtask of determining an attribute of the object, acquiring an image including the object; a division step of dividing the image acquired in the acquisition step to generate a plurality of partial images; a conversion step of converting the partial images divided in the division step into tokens, which are vectors of fixed dimensions; an adding step of adding a class token having a fixed dimension corresponding to the token to the sequence of tokens converted in the converting step; a first encoding step of updating the sequence of tokens to which the class token has been added in the adding step based on the relevance between the tokens; a first determination step of determining a first attribute of the object from the class token updated in the first encoding step; a second encoding step of further updating the sequence of tokens updated in the first encoding step based on the relevance between the tokens in accordance with the determination result in the first determination step; an extraction step of extracting features of the object from the class tokens updated in the second encoding step; a second determination step of determining a second attribute of the object from the class token updated in the second encoding step; A program that causes a computer to execute the following.

Images