Learning method, re-identification device, re-identification method, and program

Abstract

To provide a re-identification method capable of enhancing accuracy regarding re-identification of a specific object captured in image data.SOLUTION: In a re-identification method according to the present disclosure, a machine learning model includes a plurality of feature extraction layers, each sequentially connected from one to another, and a plurality of embedded layers, each connected to either one of the plurality of feature extraction layers. The re-identification method according to the present disclosure comprises: acquiring first image data and second image data in both of which a specific object is captured; acquiring a plurality of first output data and a plurality of second output data by inputting the first image data and the second image data into the machine learning model; calculating a plurality of distances between each of the plurality of first output data and each of the plurality of second output data; and determining that the specific object captured in the first image data and the one in the second image data are identical when a predetermined number or more of the distances among the plurality of distances are equal to or less than a predetermined threshold.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a technique for re-identifying a specific object appearing in image data using a machine learning model, and also to a technique for training a machine learning model for the purpose of re-identification. [Background technology]

[0002] Patent Document 1 discloses a method for re-identifying an object, which includes applying a convolutional neural network (CNN) to a pair of images representing an object, and calculating a positive pair probability as to whether the pair of images represents the same object. Patent Document 1 further discloses that the CNN includes a first convolutional layer, a first max pooling layer for obtaining a feature map of each image, a cross-input neighborhood difference layer for generating a neighborhood difference map, a patch summary layer for generating a patch summary feature map, a first fully connected layer for generating a feature vector, a second fully connected layer for generating two scores representing a positive pair class and a negative pair class, and a softmax layer for generating a positive pair probability and a negative pair probability.

[0003] Patent document 2 discloses a method for identifying item categories, which includes the steps of acquiring an image of an identified item and extracting edge mask information of the identified item image, cutting the identified item image according to the edge max information, identifying the category of the identified item image according to the cut-processed identified item image and a preset item category identification model, and outputting the identification result. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-506788 A [Patent Document 2] JP 2021-117969 A Summary of the Invention [Problem to be solved by the invention]

[0005] In recent years, there has been progress in the development of techniques for re-identification, which identifies a specific object captured in one image data with a specific object captured in another image data. Re-identification is useful for tracking objects and recognizing the surrounding environment.

[0006] In re-identification, machine learning models are generally used. However, it is assumed that the specific objects captured in each image data differ in terms of viewpoint, lighting conditions, occurrence of occlusion, resolution, etc. For this reason, re-identification is one of the difficult tasks in machine learning. In particular, person re-identification, where the specific object is a person, is a more difficult task because, while higher accuracy is required, differences in clothing are expected and occlusion occurs frequently.

[0007] As for re-identification, various techniques have been proposed for the configuration of a machine learning model, a re-identification method using a machine learning model, and a learning method for a machine learning model, as disclosed in Patent Document 1 or Patent Document 2. On the other hand, it is expected that the appropriate machine learning technique differs depending on the learning environment and the format of input data. For this reason, there is a demand for proposals for further techniques that can be expected to improve the accuracy of re-identification.

[0008] One objective of the present disclosure is to propose a technique for improving accuracy in re-identifying a specific object that appears in image data. [Means for solving the problem]

[0009] The first disclosure relates to a method for learning a machine learning model, wherein the machine learning model includes a plurality of feature extraction layers each connected in sequence, and a plurality of embedding layers each connected to any one of the plurality of feature extraction layers, each of the plurality of feature extraction layers being configured to extract a feature map for input data, and each of the plurality of embedding layers being configured to map the feature map to a feature vector in an embedding space of a predetermined dimension, and output the feature vector.

[0010] The learning method according to the first disclosure includes obtaining a plurality of training data, each of which is associated with a correct answer label, inputting the plurality of training data into the machine learning model, obtaining a plurality of output data sets which are outputs of the plurality of embedding layers, calculating a loss function based on the plurality of output data sets, and learning the machine learning model in a direction in which the loss function decreases. Here, each of the plurality of output data sets is an output set for the input of the plurality of training data for one of the plurality of embedding layers. The loss function also includes a plurality of distance learning terms given corresponding to each of the plurality of output data sets, and each of the plurality of distance learning terms is given to have a smaller value as the distance in the embedding space between outputs for training data associated with the same correct answer label among the plurality of training data is shorter and the distance in the embedding space between outputs for training data associated with different correct answer labels among the plurality of training data is longer for the corresponding output data set.

[0011] The second disclosure relates to a learning method that further has the following characteristics in addition to the learning method according to the first disclosure.

[0012] Each of the plurality of training data is image data showing a specific object, and the ground truth label is a classification of the specific object.

[0013] The third disclosure relates to a learning method that further has the following characteristics in comparison with the learning method according to the second disclosure.

[0014] The particular object is a person and the classification is a designation of an individual.

[0015] The fourth disclosure relates to a re-identification device.

[0016] A re-identification device according to a fourth disclosure includes at least one processor and a memory coupled to the at least one processor and storing a plurality of executable instructions and a machine learning model. Here, the machine learning model includes a plurality of feature extraction layers each connected in sequence, and a plurality of embedding layers each connected to any one of the plurality of feature extraction layers. Each of the plurality of feature extraction layers is configured to extract a feature map for input data, and each of the plurality of embedding layers is configured to map the feature map to a feature vector in an embedding space of a predetermined dimension and output the feature vector. The executable instructions are configured to cause the at least one processor to execute a process of acquiring first image data and second image data in which a specific object is captured, a process of acquiring a plurality of first output data that are outputs of the plurality of embedding layers using the first image data as an input of the machine learning model, a process of acquiring a plurality of second output data that are outputs of the plurality of embedding layers using the second image data as an input of the machine learning model, and an identification process of identifying the specific object captured in the first image data and the specific object captured in the second image data based on the plurality of first output data and the plurality of second output data. The identification process also includes calculating a plurality of distances in the embedding space between each of the plurality of first output data and each of the plurality of second output data, and determining that the specific object captured in the first image data and the second image data are the same when a predetermined number or more of the plurality of distances are equal to or less than a predetermined threshold.

[0017] The fifth disclosure relates to a re-identification device having the following characteristics in addition to the re-identification device according to the fourth disclosure.

[0018] The particular object is a person.

[0019] The sixth disclosure relates to a re-identification device having the following characteristics in addition to the re-identification device according to the fourth or fifth disclosure.

[0020] The machine learning model is trained by the training method according to the first disclosure.

[0021] The seventh disclosure relates to a re-identification method for re-identifying a specific object appearing in image data using a machine learning model, wherein the machine learning model includes a plurality of feature extraction layers each connected in sequence, and a plurality of embedding layers each connected to any one of the plurality of feature extraction layers, each of the plurality of feature extraction layers being configured to extract a feature map for input data, and each of the plurality of embedding layers being configured to map the feature map to a feature vector in an embedding space of a predetermined dimension, and output the feature vector.

[0022] The re-identification method according to the seventh disclosure includes: acquiring first image data and second image data in which the specific object appears; acquiring a plurality of first output data that are outputs of the plurality of embedding layers using the first image data as an input of the machine learning model; acquiring a plurality of second output data that are outputs of the plurality of embedding layers using the second image data as an input of the machine learning model; and performing identification of the specific object appearing in the first image data and the specific object appearing in the second image data based on the plurality of first output data and the plurality of second output data. Here, performing the identification includes calculating a plurality of distances in the embedding space between each of the plurality of first output data and each of the plurality of second output data; and determining that the specific object appearing in the first image data and the second image data is the same when a predetermined number or more of the plurality of distances are equal to or less than a predetermined threshold.

[0023] The re-identification method according to the eighth disclosure relates to a re-identification method having the following additional features compared to the re-identification method according to the seventh disclosure.

[0024] The particular object is a person.

[0025] The ninth disclosure relates to a re-identification method according to the seventh or eighth disclosure, further having the following characteristics.

[0026] The machine learning model is trained by the training method according to the first disclosure.

[0027] The tenth disclosure relates to a program for causing a computer to execute learning of a machine learning model, wherein the machine learning model includes a plurality of feature extraction layers each connected in sequence, and a plurality of embedding layers each connected to any one of the plurality of feature extraction layers, each of the plurality of feature extraction layers being configured to extract a feature map for input data, and each of the plurality of embedding layers being configured to map the feature map to a feature vector in an embedding space of a predetermined dimension, and output the feature vector.

[0028] The program according to the tenth disclosure is configured to cause a computer to execute the following processes: acquiring a plurality of training data, each of which is associated with a correct answer label; inputting the plurality of training data into the machine learning model; acquiring a plurality of output data sets that are outputs of the plurality of embedding layers; calculating a loss function based on the plurality of output data sets; and learning the machine learning model in a direction in which the loss function decreases. Here, each of the plurality of output data sets is an output set for the input of the plurality of training data for one of the plurality of embedding layers. The loss function also includes a plurality of distance learning terms that are given corresponding to each of the plurality of output data sets, and each of the plurality of distance learning terms is given to have a smaller value as the distance in the embedding space between outputs for training data associated with the same correct answer label among the plurality of training data is shorter and the distance in the embedding space between outputs for training data associated with different correct answer labels among the plurality of training data is longer for the corresponding output data set.

[0029] The eleventh disclosure relates to a program for causing a computer to execute re-identification of a specific object appearing in image data using a machine learning model. Here, the machine learning model includes a plurality of feature extraction layers each connected in sequence, and a plurality of embedding layers each connected to any one of the plurality of feature extraction layers. Each of the plurality of feature extraction layers is configured to extract a feature map for input data, and each of the plurality of embedding layers is configured to map the feature map to a feature vector in an embedding space of a predetermined dimension, and output the feature vector.

[0030] The program according to the eleventh disclosure is configured to cause a computer to execute a process of acquiring first image data and second image data in which the specific object is reflected, a process of acquiring a plurality of first output data which are outputs of the plurality of embedding layers using the first image data as an input of the machine learning model, a process of acquiring a plurality of second output data which are outputs of the plurality of embedding layers using the second image data as an input of the machine learning model, and an identification process of identifying the specific object reflected in the first image data and the specific object reflected in the second image data based on the plurality of first output data and the plurality of second output data. Here, the identification process includes calculating a plurality of distances in the embedding space between each of the plurality of first output data and each of the plurality of second output data, and determining that the specific object reflected in the first image data and the second image data is the same when a predetermined number or more of the plurality of distances are equal to or less than a predetermined threshold. Effect of the Invention

[0031] According to the present disclosure, the output of the machine learning model is a plurality of feature vectors output by a plurality of embedding layers. Then, identification of a specific object appearing in the image data is performed based on whether or not a predetermined number of distances among a plurality of distances for the plurality of feature vectors are equal to or less than a predetermined threshold. In this way, re-identification is performed by measuring the similarity between a plurality of feature maps with different scales. As a result, the accuracy of re-identification can be improved. [Brief description of the drawings]

[0032] [Figure 1] FIG. 1 is a conceptual diagram for explaining person re-identification performed in person tracking. [Diagram 2] FIG. 1 is a block diagram showing a schematic configuration of a process for person re-identification using a machine learning model. [Diagram 3] FIG. 1 is a block diagram showing a schematic configuration of a typical machine learning model for comparison with this embodiment. [Figure 4] FIG. 2 is a block diagram showing the configuration of a machine learning model according to the present embodiment. [Diagram 5] FIG. 2 is a block diagram illustrating an example of a machine learning model according to the present embodiment. [Figure 6] FIG. 1 is a conceptual diagram showing an example of multiple feature vectors output by a machine learning model according to this embodiment. [Figure 7] FIG. 1 is a conceptual diagram showing a preferred example of multiple feature vectors output by the machine learning model according to this embodiment when multiple image data are input. [Figure 8] 4 is a flowchart for explaining a learning method according to the present embodiment. [Figure 9] FIG. 13 is a conceptual diagram showing examples of multiple training data. [Figure 10] FIG. 13 is a conceptual diagram for explaining a plurality of distance learning terms. [Figure 11] 1 is a flowchart for explaining a re-identification method according to the present embodiment. [Figure 12]10 is a conceptual diagram for explaining a plurality of distances calculated in the re-identification method according to the present embodiment. FIG. [Figure 13] 1 is a block diagram showing a configuration of a re-identification device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] Hereinafter, the embodiments of the present disclosure will be described with reference to the drawings. However, when the number, quantity, amount, range, etc. of each element is mentioned in the embodiments shown below, the idea of ​​the present disclosure is not limited to the mentioned number unless it is specifically stated or the number is clearly specified in principle. Furthermore, the configurations etc. described in the embodiments shown below are not necessarily essential to the idea of ​​the present disclosure unless it is specifically stated or the number is clearly specified in principle. In addition, the same reference numerals are attached to the same or corresponding parts in each drawing, and the duplicated explanation is appropriately simplified or omitted.

[0034] 1. Person re-identification The re-identification method and the re-identification device according to the present embodiment use a machine learning model to re-identify a specific object that appears in image data. In the following, a case where the re-identification method and the re-identification device are applied to person re-identification where the specific object is a person will be described.

[0035] Person re-identification is useful, for example, in tracking a person. FIG. 1 is a conceptual diagram for explaining person re-identification performed in tracking a person 1. FIG. 1 shows a case in which a person 1 moves along an arrow. Here, a camera 3 is installed at each of two different points 2a and 2b on the movement path of the person 1. That is, the person 1 is imaged by the camera 3 at each of the two different points 2a and 2b. The camera 3 is, for example, a surveillance camera installed on a sidewalk.

[0036] In the example shown in FIG. 1, tracking of person 1 is performed in image data captured by camera 3 (particularly, video as a collection of image data). However, the imaging range 4 of one camera is limited. Therefore, it has been considered to track person 1 across videos captured by multiple cameras 3. This makes it possible to expand the tracking range. On the other hand, installing multiple cameras 3 so that their imaging ranges 4 overlap is costly and undesirable. Furthermore, the imaging ranges 4 of existing surveillance cameras usually do not overlap.

[0037] If the imaging ranges 4 do not overlap, tracking of the person 1 will be performed in image data that is spatially and temporally discontinuous. This makes it necessary to re-identify the person. That is, identification is performed between the person captured in image data captured by one camera 3 and the person captured in image data captured by another camera 3. This makes it possible to continue tracking of the person 1 in the image data captured by one camera 3 in the image data captured by the other camera 3.

[0038] In the example shown in Fig. 1, image data 10a captured by a camera 3 installed at a point 2a and image data 10b captured by a camera 3 installed at a point 2b are shown. The same person 1 is captured in each of the image data 10a and 10b. Therefore, in person re-identification, it is required to determine that the person captured in the image data 10a and the person captured in the image data 10b are the same person 1. If person re-identification is performed appropriately, tracking of the person 1 can be continued from the point 2a to the point 2b.

[0039] 2 is a block diagram showing a schematic configuration of a process related to person re-identification using a machine learning model 110.

[0040] The machine learning model 110 receives image data as input and outputs features of the image data. The machine learning model 110 is realized, for example, as a program, and is stored in the memory of a computer that performs person re-identification. Here, the image data is image data in which a person 1 appears. In particular, the image data may be image data that has been cropped so that the person 1 appears prominently (see image data 10a and 10b shown in FIG. 1). For example, the image data input to the machine learning model 110 is image data in which person detection is performed on raw image data captured by the camera 3, and thus the person 1 appearing in the raw image data has been cropped. In this case, a known suitable technology may be adopted for the person detection.

[0041] The format of the feature output by the machine learning model 110 is determined based on the configuration of the machine learning model 110, and is a subject of consideration regarding the re-identification method. A trained model is used as the machine learning model 110. The training method of the machine learning model 110 is also one of the subjects of consideration.

[0042] The database 200 manages a plurality of image data. The database 200 is realized, for example, by a database server configured to be able to communicate with a computer that performs person re-identification. The database 200 is configured, for example, by sequentially acquiring image data captured by a plurality of cameras 3. Each of the plurality of image data managed in the database 200 may be image data cropped so that the person 1 is prominently shown, as described above. In particular, in the database 200, information specifying the individual person 1 shown in the image data is associated with each of the plurality of image data. For example, ID information assigned to each individual is associated. Furthermore, each of the plurality of image data managed in the database 200 may be associated with a feature amount obtained by inputting the image data to the machine learning model 110. In this case, it is sufficient to input each of the plurality of image data to the machine learning model 110 in advance and acquire a feature amount.

[0043] Person re-identification is typically performed by inputting image data showing the person 1 to be re-identified and identifying the image data with multiple pieces of image data managed in database 200. In this sense, the image data showing the person 1 to be re-identified can also be called a "query," and the multiple pieces of image data managed in database 200 can also be called a "gallery." These terms will be used appropriately below.

[0044] The identification processing unit 132 performs identification of the person 1 appearing in the image data based on the feature amount output by the machine learning model 110. In particular, the identification processing unit 132 performs identification of the person 1 appearing in the query image data and the person 1 appearing in the gallery image data. This realizes re-identification of the person 1 appearing in the query image data. The identification processing unit 132 is realized, for example, as a part of a program. The processing result output by the identification processing unit 132 may be the image data of the gallery that is determined to show the same person 1, or may be information (for example, ID information) that specifies an individual determined to be the same as the person 1 appearing in the query image data. Alternatively, when identifying the query image data with one of the image data of the gallery, the processing result may be a determination of whether they are the same or not.

[0045] The identification processing unit 132 performs identification by measuring the similarity with the feature amount of the image data of the query (hereinafter, also simply referred to as the "query feature amount"). That is, the feature amount of the query is compared with the feature amount of the image data of the gallery. Then, it is determined that the person 1 appearing in the image data of the query is the same as the person 1 appearing in the image data having a feature amount similar to the query feature amount. Here, the identification processing unit 132 may acquire the feature amount of the image data of the gallery as an output of the machine learning model 110, or may acquire the feature amount by referring to the database 200. In the former case, each image data of the gallery is input to the machine learning model 110 each time, and the feature amount is acquired. In the latter case, as described above, it is sufficient that the feature amount is associated with each of the multiple image data managed in the database 200.

[0046] The index of similarity and the method of determining similarity are determined based on the configuration of the identification processing unit 132, and are the subject of consideration regarding the re-identification method.

[0047] As described above, person re-identification is performed using the machine learning model 110. Incidentally, it is assumed that the image data of the query and the gallery are captured in different environments, captured at different times, captured by different cameras 3, and the like. In addition, it is assumed that the person 1 captured in each image data is different in viewpoint, lighting conditions, occurrence of occlusion, resolution, clothing, and the like, even if the image data is a pair of image data in which the same person 1 is captured. For this reason, person re-identification is one of the difficult tasks in machine learning.

[0048] In order to improve the accuracy of person re-identification, the re-identification method according to this embodiment is characterized by the configuration of the machine learning model 110 and the processing executed in the identification processing unit 132. The learning method of the machine learning model 110 for implementing the re-identification method according to this embodiment is also characterized. Below, the machine learning model 110 according to this embodiment, the learning method of the machine learning model 110, and the re-identification method and re-identification device according to this embodiment will be described.

[0049] 2. Machine learning models First, for comparison with this embodiment, a schematic configuration of a typical machine learning model 110 is shown in Fig. 3. The machine learning model 110 shown in Fig. 3 is composed of a CNN. In particular, in the machine learning model 110 shown in Fig. 3, four CNNs are connected in sequence, and a multilayer perceptron (MLP) is connected to the final-stage CNN. The MLP is typically a fully connected layer.

[0050] As is well known, CNNs can extract appropriate feature maps for image data. In particular, it is known that when multiple CNNs are connected in sequence, the later CNNs output feature maps that represent more abstract features for the image data. This can be said to mean that the feature maps output by each of the multiple CNNs have different scales. This is usually because the data sizes of the feature maps output by each of the multiple CNNs are different from each other. In addition, multiple feature maps with different scales are also called "multi-scale" feature maps.

[0051] The MLP shown in Fig. 3 takes as input the feature map output by the final-stage CNN. The output of the MLP can be regarded as a vector (feature vector) in which the values ​​of neurons in the output layer are arranged. That is, in the machine learning model 110 shown in Fig. 3, the feature value output by the machine learning model 110 is the feature vector output by the MLP. Also, considering that the dimension of the feature vector is determined by the configuration of the MLP, the MLP can be regarded as a mapping that maps the feature map to a feature vector in an embedding space of a predetermined dimension.

[0052] Consider performing person re-identification using this typical machine learning model 110. In this case, it is required to perform learning of the machine learning model 110 so that image data showing the same person 1 output feature vectors whose positions in the embedding space are close to each other, and image data showing different people 1 output feature vectors whose positions in the embedding space are far from each other. If learning can be performed in this way, it is expected that the person 1 shown in the query image data is the same as the person 1 shown in the image data having a feature vector whose position is close to that of the query feature vector. That is, in this case, in the identification processing unit 132, the similarity is measured by the distance in the embedding space between the feature vectors.

[0053] It is expected that the above-mentioned learning of the typical machine learning model 110 can be achieved by a known learning method. However, even if sufficient learning is performed on the training data by a known learning method, person re-identification with sufficient accuracy cannot be achieved. This is considered to be because, as described above, the person 1 shown in each image data is influenced by various differences even in a pair of image data showing the same person 1. For this reason, even if the number of training data is increased, there is a concern of overlearning on the training data, and sufficient effects cannot be expected.

[0054] In response to this, the inventors of the present disclosure came up with the idea that in person re-identification, it is effective to perform identification by measuring the similarity of multiple feature maps with different scales. This is because it is considered effective to make a comprehensive judgment on various features in order to determine whether or not a person is identical to the person 1 captured in the query image data in a robust manner in the face of various differences in factors. In other words, multiple feature maps with different scales represent different features from each other, and each feature is expected to be useful for distinguishing different individuals.

[0055] The machine learning model 110 according to this embodiment is configured based on the above idea. The machine learning model 110 according to this embodiment will be described below. Fig. 4 is a block diagram showing the configuration of the machine learning model 110 according to this embodiment.

[0056] The machine learning model 110 according to the present embodiment includes a plurality of feature extraction layers 111 that are sequentially connected to each other, and a plurality of embedding layers 112 that are each connected to one of the plurality of feature extraction layers 111. In the example shown in FIG. 4, the machine learning model 110 includes four feature extraction layers 111 (#1, #2, #3, #4) and four embedding layers 112 (#1, #2, #3, #4) that are connected to each of the four feature extraction layers 111. However, the number of the plurality of feature extraction layers 111 and the plurality of embedding layers 112 may be appropriately determined according to the environment to which the present embodiment is applied. In addition, the plurality of embedding layers 112 may be configured to be connected to a part of the plurality of feature extraction layers 111. For example, in FIG. 4, the machine learning model 110 may be configured not to include the embedding layer 112 of #1 and the embedding layer 112 of #3. In this case, the number of the plurality of feature extraction layers 111 is four, and the number of the plurality of embedding layers 112 is two.

[0057] Each of the multiple feature extraction layers 111 is configured to extract a feature map for input data. Here, the input data of the first feature extraction layer 111 is image data, and the input data of the subsequent feature extraction layers 111 are feature maps output by the preceding feature extraction layer 111. Therefore, the feature maps output by each of the multiple feature extraction layers 111 have different scales. Furthermore, when configured in this way, the feature maps output by each of the multiple feature extraction layers 111 usually have different data sizes.

[0058] As one example, the feature extraction layer 111 can be realized by a CNN. As another example, the feature extraction layer 111 can be realized by a patch division layer and an encoder layer based on a Transformer, particularly a Vision Transformer (ViT). In this case, the patch division layer divides input data into a plurality of patches, and the encoder layer takes the plurality of patches as input and outputs a feature map.

[0059] Each of the multiple embedding layers 112 receives as input a feature map output by the connected feature extraction layer 111. Then, the feature map is mapped to a feature vector in an embedding space of a predetermined dimension, and the feature vector is output. Here, the multiple embedding layers 112 are configured so that the dimensions of the feature vectors output by each of them are equal to each other. In other words, the feature vectors output by each of the multiple embedding layers 112 are vectors in the same embedding space.

[0060] As an example, the embedding layer 112 can be realized by an MLP. The MLP may typically be a fully connected layer. In this case, in order to make the dimensions of the feature vectors output by each layer equal to each other, the number of neurons in the output layer of the MLP may be equal for each of the multiple embedding layers 112.

[0061] FIG. 5 shows an example of a machine learning model 110 in which the feature extraction layer 111 is realized by a patch division layer and an encoder layer, and the embedding layer 112 is realized by an MLP.

[0062] In this way, according to the machine learning model 110 of this embodiment, it is possible to obtain multiple feature vectors in the same embedding space for multiple feature maps with different scales. Therefore, the feature values ​​output by the machine learning model 110 of this embodiment are multiple feature vectors output by the multiple embedding layers 112. Each of the multiple feature extraction layers 111 may have a different structure and independent parameters. For example, each of the multiple feature layers 111 may have a different layer depth. Similarly, each of the multiple embedding layers 112 may have a different structure and independent parameters.

[0063] Fig. 6 shows an example of multiple feature vectors output when image data is input to the machine learning model 110 shown in Fig. 4. In the example shown in Fig. 4, the positions of four feature vectors 21a, 21b, 21c, and 21d output from each of the four embedding layers 112 (#1, #2, #3, #4) in the embedding space 20 are shown by specific figures. Here, the dimension of the embedding space 20 is assumed to be two-dimensional for simplicity.

[0064] 3. Learning Methods The learning method according to this embodiment will be described below.

[0065] 7 shows a preferred example of feature vectors output by the machine learning model 110 shown in FIG. 4 when three image data, namely, two image data showing the same person 1 and image data showing a person 1 different from the other two, are input to the machine learning model 110. FIG. 7 is a diagram similar to FIG. 6. In the example shown in FIG. 7, feature vector sets 22a and 22b are output data when two image data showing the same person 1 are input. On the other hand, feature vector set 22c is output data when image data showing a person 1 different from the other two is input.

[0066] As shown in Fig. 7, when person re-identification is performed using the machine learning model 110 according to this embodiment, it is required to train the machine learning model 110 so that the positions of the multiple feature vectors of image data showing the same person 1 are close to each other in the embedding space 20, while the positions of the multiple feature vectors of image data showing different people 1 are far from each other in the embedding space 20. By training in this way, the re-identification method according to this embodiment can perform identification taking into account each of multiple feature maps with different scales. The re-identification method according to this embodiment will be described later.

[0067] The learning method according to this embodiment achieves learning of a machine learning model 110 that outputs features as shown in Fig. 7. Fig. 8 is a flowchart for explaining the learning method according to this embodiment. Each process of the flowchart shown in Fig. 8 is executed at each predetermined processing cycle.

[0068] In step S100, a plurality of training data for learning the machine learning model 110 is Here, a correct label is associated with each of the multiple training data. FIG. 9 shows an example of the multiple training data. FIG. 9 shows three image data 10a, 10b, and 10c in which a person 1 appears, in particular, as the multiple training data. Correct labels 11a, 11b, and 11c are associated with the three image data 10a, 10b, and 10c, respectively. The correct label is information that specifies an individual for the person 1 appearing in the image data. That is, FIG. 9 shows that the same person 1 appears in the image data 10a and the image data 10b.

[0069] Referring again to Figure 8, after step S100, the process proceeds to step S110.

[0070] In step S110, the multiple pieces of training data acquired in step S100 are input to the machine learning model 110.

[0071] After step S110, the process proceeds to step S120.

[0072] In step S120, the output of the machine learning model 110 for the input in step S120 is obtained. In particular, a plurality of output datasets that are the output of the plurality of embedding layers 112 are obtained. Here, each of the plurality of output datasets is an output set for an input for one of the plurality of embedding layers 112. Thus, each of the plurality of output datasets is a set of feature vectors for a feature map at a certain scale. For example, when the machine learning model 110 shown in FIG. 4 is taken as an example, one of the plurality of output datasets is an output set for an input for the embedding layer 112 of #1. That is, in this case, four output datasets for each of the embedding layers 112 of #1, #2, #3, and #4 are obtained.

[0073] After step S120, the process proceeds to step S130.

[0074] In step S130, a loss function is calculated based on the multiple output data sets acquired in step S120. In the learning method according to this embodiment, the configuration of the loss function is characteristic. The loss function according to this embodiment includes multiple distance learning terms that are provided corresponding to each of the multiple output data sets. Here, each of the multiple distance learning terms is provided so that, for the corresponding output data set, the shorter the distance in the embedding space 20 between outputs (feature vectors) for training data associated with the same correct answer label among the multiple training data, and the longer the distance in the embedding space 20 between outputs (feature vectors) for training data associated with different correct answer labels among the multiple training data, the smaller the value.

[0075] FIG. 10 is a conceptual diagram for explaining a plurality of distance learning terms. FIG. 10 is a diagram similar to FIG. 6. In particular, FIG. 10 shows a case where two image data are input as training data to the machine learning model 110 shown in FIG. 4. As shown in FIG. 10, at this time, four output data sets 23a, 23b, 23c, and 23d are acquired. FIG. 10 shows distances d1, d2, d3, and d4 in the embedding space 20 between the outputs (feature vectors) for each of the four output data sets. That is, in the example shown in FIG. 10, when the correct answer labels associated with the two image data are the same, the four distance learning terms are given so that the shorter the distances d1, d2, d3, and d4 are, the smaller the values ​​become. On the other hand, when the correct answer labels associated with the two image data are different, the longer the distances d1, d2, d3, and d4 are, the smaller the values ​​become.

[0076] The loss function calculated in the learning method according to this embodiment can be expressed by Loss in the following formula (1). Here, Li (i=1, 2, . . . , n) represents each of the multiple distance learning terms, and n corresponds to the number of multiple output data sets acquired in step S120. other is a term of the loss function that is appropriately given to achieve other learning objectives. L other is not an essential configuration for the learning method according to this embodiment.

[0077]

number

[0078] Li can be realized by, for example, contrastive loss or triplet loss. Contrastive loss and triplet loss are well known, and detailed description will be omitted. In addition, a suitable configuration may be adopted as the distance learning term.

[0079] Any suitable format may be used as the distance in the embedding space 20. For example, the distance in the embedding space 20 may be exemplified by Euclidean distance or cosine similarity.

[0080] Referring again to Figure 8, after step S130, the process proceeds to step S140.

[0081] In step S140, the machine learning model 110 is trained in a direction that reduces the loss function calculated in step S130. Typically, the parameters of the machine learning model 110 are updated by the backpropagation method in a direction that reduces the loss function.

[0082] The loss function includes the above-mentioned multiple distance learning. Therefore, the direction in which the loss function decreases is the direction in which the distance in the embedding space 20 between the outputs (feature vectors) for training data associated with the same correct answer label becomes shorter. Alternatively, it is the direction in which the distance in the embedding space 20 between the outputs (feature vectors) for training data associated with different correct answer labels becomes longer.

[0083] After step S140, if the termination condition is met (step S150; Yes), the learning of the machine learning model 110 ends. If the termination condition is not met (step S150; No), the process returns to step S100 and is repeated. Here, the termination condition is, for example, that learning has been completed for all image data to be prepared as training data, that the loss function calculated after step S140 is equal to or less than a predetermined threshold, etc.

[0084] In step S100, the acquisition of training data may be performed for all image data prepared as training data for learning. Then, in step S110, a part of the training data acquired in step S100 (for example, in units of batches or epochs) may be input to the machine learning model 110. In this case, if the termination condition is not satisfied after step S140 (step S150; No), the process may be configured to return to step S110 and repeat the process.

[0085] As described above, according to the learning method of this embodiment, the loss function is configured to include the above-mentioned multiple distance learning terms. Then, learning of the machine learning model 110 is performed in a direction to decrease the loss function. This achieves learning of the machine learning model 110 that results in an output as shown in FIG. 7. The learning method of this embodiment can also be applied to a program that causes a computer to execute processing related to the learning of the machine learning model 110.

[0086] It should be noted here that the feature vectors output by each of the multiple embedding layers 112 are vectors in the same embedding space 20. This allows each of the multiple distance learning terms to be given by the same type of distance in the same embedding space 20. Furthermore, by configuring the loss function as shown in equation (1), it is possible to equally evaluate each of the multiple feature maps with different scales during learning.

[0087] 4. Re-identification method The re-identification method according to this embodiment will be described below.

[0088] Fig. 11 is a flowchart for explaining the re-identification method according to this embodiment. Each process of the flowchart shown in Fig. 11 is executed at each predetermined processing cycle. In the following description, it is assumed that the machine learning model 110 has been trained by the above-mentioned training method.

[0089] In step S200, first image data and second image data are acquired as image data to be subjected to person re-identification. Typically, query image data and gallery image data are acquired.

[0090] In step S210, output data is obtained by using the image data obtained in step S200 as input to the machine learning model 110. That is, a plurality of first output data which are outputs (feature vectors) of the plurality of embedding layers 112 when the first image data is input, and a plurality of second output data which are outputs (feature vectors) of the plurality of embedding layers 112 when the second image data is input are obtained.

[0091] After step S210, the person 1 appearing in the first image data and the person 1 appearing in the second image data are identified based on the first output data and the second output data acquired in step S210 (step S220). Step S220 is a process executed in the identification processing unit 132. The re-identification method according to this embodiment is characterized by the processes executed in the identification processing unit 132 (steps S221 to S224).

[0092] In step S221, a plurality of distances in the embedding space 20 between each of the plurality of first output data and each of the plurality of second output data are calculated. For example, as shown in FIG. 12, a plurality of first output data 22s and a plurality of second output data 22t are acquired as outputs of the machine learning model 110. In this case, the plurality of distances calculated in step S221 are d1, d2, d3, and d4 shown in FIG. 12. However, the format of the distance in the embedding space 20 is equivalent to the format of the distance adopted in learning the machine learning model 110.

[0093] Referring again to Fig. 11, in step S222, it is determined whether a predetermined number or more of the distances calculated in step S221 are equal to or less than a predetermined threshold. Here, the predetermined number and the predetermined threshold may be optimally determined through experimentation. For example, the predetermined number may be set to half of the distances.

[0094] If a predetermined number of the distances are equal to or smaller than a predetermined threshold (step S222; Yes), the person 1 in the first image data and the second image data is determined to be the same (step S223), and the process ends. If a predetermined number of the distances are not equal to or smaller than a predetermined threshold (step S222; No), the person 1 in the first image data and the second image data is determined to be not the same, and the process ends (step S224).

[0095] In other words, in the re-identification method according to the present embodiment, if a predetermined number or more of features shown in multiple feature maps with different scales are similar, the person 1 shown in the first image data and the second image data is determined to be the same. This allows a comprehensive judgment of various features in person re-identification.

[0096] Note that step S210 may be performed in advance for the first image data or the second image data. For example, when the first image data is query image data and the second image data is gallery image data, output data may be acquired in advance for the second image data. In other words, the output data acquired in step S210 may be associated in advance with the gallery image data.

[0097] Furthermore, if it is determined that the person 1 appearing in the first image data is not the same as the person 1 appearing in the second image data (step S224), the flowchart shown in Fig. 11 may be repeatedly executed. For example, when the first image data is query image data and the second image data is gallery image data, if it is determined that the person 1 appearing in the first image data is not the same as the person 1 appearing in the second image data, the second image data may be reacquired from the gallery image data and the process may be executed again.

[0098] As described above, according to the re-identification method of this embodiment, if a predetermined number or more of the distances calculated in step S221 are equal to or less than a predetermined threshold, the person 1 captured in the first image data and the person 1 captured in the second image data are determined to be the same person. This allows identification to be performed taking into account each of a plurality of feature maps with different scales. As a result, the accuracy of person re-identification can be improved.

[0099] Here, it should be noted that in the re-identification method according to this embodiment, the feature vectors output from each of the multiple embedding layers 112 are vectors in the same embedding space 20. This allows multiple feature maps with different scales to be evaluated equally during identification.

[0100] 5. Re-identification device The re-identification device according to this embodiment will be described below.

[0101] 13 is a block diagram showing the configuration of a re-identification device 100 according to this embodiment. The re-identification device 100 is a computer including a memory 101, a processor 102, and a communication interface 103. The memory 101 is coupled to the processor 102, and stores a plurality of executable instructions 131, a machine learning model 110, and various data 120 required for executing processing. The instructions 131 are provided by a program 130. In this sense, the memory 101 can also be called a "program memory."

[0102] The communication interface 103 transmits and receives information to and from devices external to the re-identification device 100. For example, the re-identification device 100 connects to a database 200 via the communication interface 103. The communication interface 103 acquires image data, stores or updates the machine learning model 110, notifies the results of processing, and the like. Information acquired via the communication interface 103 is stored in the memory 101 as data 120.

[0103] The instructions 131 are configured to cause the processor 102 to execute processing related to the re-identification method shown in Fig. 11. In other words, the processor 102 operates in accordance with the instructions 131 to realize the execution of processing related to the re-identification method shown in Fig. 11 based on the machine learning model 110 and the data 120.

[0104] 6.Effects As described above, according to this embodiment, the feature values ​​output by the machine learning model 110 are multiple feature vectors output by multiple embedding layers 112. Then, the person 1 appearing in the image data is identified based on whether or not a predetermined number of distances among multiple distances for the multiple feature vectors are equal to or less than a predetermined threshold. In this way, re-identification is performed by measuring the similarity between multiple feature maps with different scales. As a result, the accuracy of re-identification can be improved.

[0105] In this embodiment, the case where the method is applied to person re-identification has been described, but the method can also be applied to re-identification where the specific object is not a person. For example, the method can also be applied to re-identification of a dog that appears in image data. In this case, learning can be performed by the learning method according to the embodiment, with the correct answer label being the classification of the specific object. In particular, in this embodiment, the fineness of the classification can be arbitrary. For example, when the method is applied to re-identification of a dog, the classification can specify an individual as in person re-identification, or can specify a dog breed.

[0106] In addition, the re-identification method and the re-identification device according to the present embodiment can be implemented as part of a function or device. For example, the re-identification method according to the present embodiment can be implemented as part of a tracking function. [Explanation of symbols]

[0107] 1 person 20 Embedding Space 100 Re-identification device 101 Memory 102 processors 103 Communication Interface 110 Machine Learning Models 111 Feature Extraction Layer 112 Embedding Layer 120 Data 130 Programs 131 Instructions 132 Identification processing section 200 Databases

Claims

1. A method for training a machine learning model, comprising: The machine learning model is A plurality of sequentially connected feature extraction layers; a plurality of embedding layers, each of which is connected to one of the plurality of feature extraction layers; Including, Each of the plurality of feature extraction layers is configured to extract a feature map for input data; Each of the plurality of embedding layers is configured to map the feature map to a feature vector in an embedding space of a predetermined dimension and output the feature vector; The learning method includes: Obtaining a plurality of training data, each of which is associated with a correct answer label; inputting the plurality of training data into the machine learning model; obtaining a plurality of output datasets that are outputs of the plurality of embedding layers; and calculating a loss function based on the plurality of output data sets; training the machine learning model in a direction in which the loss function decreases; Including, each of the plurality of output data sets is a set of outputs for one of the plurality of embedding layers for the plurality of training data inputs; the loss function includes a plurality of distance learning terms provided corresponding to each of the plurality of output data sets; Each of the plurality of distance learning terms is given to have a smaller value as the distance in the embedding space between outputs for training data associated with the same correct answer label among the plurality of training data is shorter, and as the distance in the embedding space between outputs for training data associated with different correct answer labels among the plurality of training data is longer, for the corresponding output data set. A learning method comprising:

2. 2. The learning method according to claim 1, Each of the plurality of training data is image data showing a specific object, The ground truth label is the classification of the particular object. A learning method comprising:

3. The learning method according to claim 2, the specific object is a person, The classification is a personal designation A learning method comprising:

4. At least one processor; a memory coupled to the at least one processor and configured to store a plurality of executable instructions and a machine learning model; Equipped with The machine learning model is A plurality of sequentially connected feature extraction layers; a plurality of embedding layers, each of which is connected to one of the plurality of feature extraction layers; Including, Each of the plurality of feature extraction layers is configured to extract a feature map for input data; Each of the plurality of embedding layers is configured to map the feature map to a feature vector in an embedding space of a predetermined dimension and output the feature vector; The plurality of executable instructions may be configured to cause the at least one processor to: A process of acquiring first image data and second image data showing a specific object; A process of obtaining a plurality of first output data that are outputs of the plurality of embedding layers using the first image data as an input of the machine learning model; A process of obtaining a plurality of second output data that are outputs of the plurality of embedding layers using the second image data as an input of the machine learning model; an identification process for identifying the specific object captured in the first image data and the specific object captured in the second image data based on the plurality of first output data and the plurality of second output data; configured to execute The identification process includes: calculating a plurality of distances in the embedding space between each of the plurality of first output data and each of the plurality of second output data; determining that the specific object shown in the first image data and the specific object shown in the second image data are the same when a predetermined number of the distances among the plurality of distances are equal to or smaller than a predetermined threshold value; Includes A re-identification device characterized by:

5. The re-identification device according to claim 4, The particular object is a person. A re-identification device characterized by:

6. The re-identification device according to claim 4 or claim 5, The machine learning model is trained by the training method according to claim 1. A re-identification device characterized by:

7. A re-identification method for re-identifying a specific object appearing in image data using a machine learning model, comprising: The machine learning model is A plurality of sequentially connected feature extraction layers; a plurality of embedding layers, each of which is connected to one of the plurality of feature extraction layers; Including, Each of the plurality of feature extraction layers is configured to extract a feature map for input data; Each of the plurality of embedding layers is configured to map the feature map to a feature vector in an embedding space of a predetermined dimension and output the feature vector; The re-identification method includes: acquiring first image data and second image data in which the specific object is captured; Obtaining a plurality of first output data that are outputs of the plurality of embedding layers using the first image data as an input of the machine learning model; Obtaining a plurality of second output data that are outputs of the plurality of embedding layers using the second image data as an input of the machine learning model; Identifying the specific object appearing in the first image data and the specific object appearing in the second image data based on the plurality of first output data and the plurality of second output data; Including, Conducting the identifying includes: calculating a plurality of distances in the embedding space between each of the plurality of first output data and each of the plurality of second output data; determining that the specific object shown in the first image data and the specific object shown in the second image data are the same when a predetermined number of the distances among the plurality of distances are equal to or smaller than a predetermined threshold value; Includes A re-identification method comprising:

8. The re-identification method according to claim 7, The particular object is a person. A re-identification method comprising:

9. The re-identification method according to claim 7 or 8, The machine learning model is trained by the training method according to claim 1. A re-identification method comprising:

10. A program for causing a computer to execute learning of a machine learning model, The machine learning model is A plurality of sequentially connected feature extraction layers; a plurality of embedding layers, each of which is connected to one of the plurality of feature extraction layers; Including, Each of the plurality of feature extraction layers is configured to extract a feature map for input data; Each of the plurality of embedding layers is configured to map the feature map to a feature vector in an embedding space of a predetermined dimension and output the feature vector; The program is A process of acquiring a plurality of training data, each of which is associated with a correct answer label; inputting the plurality of training data into the machine learning model; obtaining a plurality of output datasets, the output of the plurality of embedding layers; Calculating a loss function based on the plurality of output data sets; A process of training the machine learning model in a direction in which the loss function decreases; configured to cause a computer to execute each of the plurality of output data sets is a set of outputs for one of the plurality of embedding layers for the plurality of training data inputs; the loss function includes a plurality of distance learning terms provided corresponding to each of the plurality of output data sets; Each of the plurality of distance learning terms is given to have a smaller value as the distance in the embedding space between outputs for training data associated with the same correct answer label among the plurality of training data is shorter, and as the distance in the embedding space between outputs for training data associated with different correct answer labels among the plurality of training data is longer, for the corresponding output data set. A program characterized by:

11. A program for causing a computer to perform re-identification of a specific object appearing in image data using a machine learning model, The machine learning model is A plurality of sequentially connected feature extraction layers; a plurality of embedding layers, each of which is connected to one of the plurality of feature extraction layers; Including, Each of the plurality of feature extraction layers is configured to extract a feature map for input data; Each of the plurality of embedding layers is configured to map the feature map to a feature vector in an embedding space of a predetermined dimension and output the feature vector; The program is A process of acquiring first image data and second image data in which the specific object is captured; A process of obtaining a plurality of first output data that are outputs of the plurality of embedding layers using the first image data as an input of the machine learning model; A process of obtaining a plurality of second output data that are outputs of the plurality of embedding layers using the second image data as an input of the machine learning model; an identification process for identifying the specific object captured in the first image data and the specific object captured in the second image data based on the plurality of first output data and the plurality of second output data; configured to cause a computer to execute The identification process includes: calculating a plurality of distances in the embedding space between each of the plurality of first output data and each of the plurality of second output data; determining that the specific object shown in the first image data and the specific object shown in the second image data are the same when a predetermined number of the distances among the plurality of distances are equal to or smaller than a predetermined threshold value; Includes A program characterized by:

Images