Model adaptation device, model adaptation method, and model adaptation program

Abstract

This model adaptation device, after training of a person collation model, uses data (query) that has been acquired from an operation environment of the person collation model to adapt the person collation model to the operation environment. For example, when having calculated the degree of similarity between a feature amount of an image of a query and a feature amount of each of images in a gallery by the person collation model, the model adaptation device updates the parameter of a feature extractor of the person collation model so that the degree of similarity between the feature amount of any image among the images in the gallery and the feature amount of the image of the query is increased. Then, after the update of the parameter of the feature extractor of the person collation model, the model adaptation device discards the query that has been used for the adaptation of the operation environment.

Description

Model adaptation device, model adaptation method, and model adaptation program

[0001] The present invention relates to a model adaptation device, a model adaptation method, and a model adaptation program for adapting a model to an operational environment.

[0002] To identify a person captured on a surveillance camera, there is a technology (person matching technology) that uses an image of the person to be identified as a query and finds images of the same person from a database (gallery) of images of people. In recent years, the use of deep learning has made it possible to match people with high accuracy.

[0003] Ge Yixiao, Dapeng Chen, and Hongsheng Li, “MUTUAL MEAN-TEACHING:PSEUDO LABEL REFINERY FOR UNSUPERVISED DOMAIN ADAPTATION ON PERSON RE-IDENTIFICATION“, ICLR2020

[0004] However, conventional person matching technologies do not take into account changes in the operating environment of the person matching model. Therefore, if the operating environment of the person matching model differs from the learning environment of the person matching model, the accuracy of person matching may decrease. For example, if the brightness, image quality, etc. of the image of the query person differs from the learning environment of the person matching model, the accuracy of person matching may decrease.

[0005] Here, in order to prevent a decrease in the accuracy of person matching even when the operating environment changes, it is also possible to perform additional training of the person matching model using a new dataset collected in the operating environment of the person matching model.

[0006] However, collecting new datasets in the operational environment for additional training of the person matching model is costly. In addition, since the operational environment is constantly changing, the person matching model must be continuously adapted to the operational environment. Therefore, additional training of the person matching model must also be continuously performed.

[0007] Furthermore, since the images of people used in training the person matching model are personal information, it is preferable to discard them after training from the viewpoint of protecting privacy. However, the collected data (images of people) cannot be immediately discarded for additional training of the person matching model.

[0008] Therefore, in order to solve the above-mentioned problems, the present invention aims to adapt a person matching model to an operating environment without increasing costs, to enable continuous adaptation to the operating environment, and to enable the data (images of people) used to adapt the person matching model to the operating environment to be discarded.

[0009] In order to solve the above-mentioned problems, the present invention is characterized by comprising an image acquisition unit that acquires an image of a target object from an operating environment of a matching model that predicts an image of an object that is the same as an object that appears in the image of a target object, and a parameter update unit that uses a feature extractor to extract features of the image of the target object and features of each image stored in a memory unit, and updates parameters of the feature extractor of the matching model so as to increase the confidence of the prediction based on the similarity between the extracted features of the image of the target object and the features of each image stored in the memory unit.

[0010] According to the present invention, when adapting a matching model to an operating environment, the cost is not increased and the model can be continuously adapted to the operating environment. Furthermore, data used to adapt the matching model to the operating environment can be discarded.

[0011] FIG. 1 is a diagram showing an example of person matching using a person matching model. FIG. 2 is a diagram showing an example of person matching using a person matching model. FIG. 3 is a diagram for explaining parameter updating of the person matching model in the model adaptation device. FIG. 4 is a diagram for explaining an overview of the model adaptation device. FIG. 5 is a diagram showing an example of the configuration of the model adaptation device. FIG. 6 is a diagram showing an example of training of the person matching model. FIG. 7 is a flowchart showing an example of a processing procedure executed by the model adaptation device. FIG. 8 is a flowchart showing an example of a calculation process of the confidence factor of a prediction of the person matching model executed by the model adaptation device. FIG. 9 is a diagram showing an example of corruption inflicted on an image of test data. FIG. 10 is a diagram showing evaluation results of the person matching model adapted to the operating environment by the model adaptation device of this embodiment. FIG. 11 is a diagram showing a computer executing a model adaptation program.

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a description will be given of an embodiment of the present invention with reference to the drawings, but the present invention is not limited to the embodiment.

[0013] In the following description, the matching model to be applied by the model adaptation device is an example of a person matching model that predicts an image in which the same person as the person in the query image appears.

[0014] First, an overview of the person matching model will be described with reference to Figures 1 and 2. As shown in Figure 1, the person matching model finds an image of a person to be matched (query) from a database (gallery) that matches the image of the person.

[0015] 2, the person matching model includes a feature extractor that extracts features from each of a query image and a gallery image. The person matching model then uses the query feature and the gallery feature extracted by the feature extractor to extract gallery images that have features similar to the query. In this way, the person matching model finds an image of a person who is the same as the person in the query.

[0016] Here, the model adaptation device of this embodiment updates the parameters of the feature extractor of the person matching model so that, for example, the feature of one of the gallery images is similar to the feature of the query image ( FIG. 3 ), thereby improving the accuracy of person matching when the person matching model is used.

[0017] An overview of the model adaptation device will be explained using Fig. 4. The model adaptation device first learns a person matching model using training data (learning the person matching model in a learning environment). The model adaptation device then adapts the learned person matching model to the operating environment.

[0018] For example, the model adaptation device acquires an image of a person to be matched acquired in the operating environment as a query, and then uses the trained person matching model to predict the probability that each person image in the gallery is the same as the query person (performs person matching).

[0019] Here, the model adaptation device updates the parameters of the person matching model (parameters of the feature extractor of the person matching model) in a direction that increases the confidence level of the prediction of the person matching model.

[0020] For example, the model adaptation device updates the parameters of the feature extractor of the person matching model so that, in the distribution of probabilities calculated by the person matching model that the query matches each of the gallery images, only images from one of the galleries have a high probability and images from other galleries have a low probability (see Figure 3).

[0021] In this way, when the learning environment of the person matching model differs from the operating environment, the model adaptation device can adapt the person matching model to the operating environment using a query in the operating environment. This eliminates the need to acquire large amounts of data when adapting the person matching model to the operating environment. As a result, the model adaptation device can easily adapt the person matching model continuously and at low cost.

[0022] Furthermore, after applying the person matching model to the operational environment, the model adaptation device can discard the query since it is no longer needed, thereby protecting the personal information of people appearing in the query image used to adapt the person matching model.

[0023] [Configuration Example] Next, a configuration example of the model adaptation device 10 will be described with reference to Fig. 5. The model adaptation device 10 includes, for example, an input / output unit 11, a storage unit 12, and a control unit 13.

[0024] The input / output unit 11 is an interface that controls the input and output of various data. The input / output unit 11 receives, for example, training data for the person matching model, a query acquired from the operation environment of the person matching model, and an input of a gallery.

[0025] The storage unit 12 stores data, programs, etc. that are referenced when the control unit 13 executes various processes. The storage unit 12 is realized by a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0026] For example, the storage unit 12 stores the gallery, parameters of the person matching model, etc. received by the input / output unit 11. The gallery is a collection of images of people who are the targets of person matching.

[0027] The control unit 13 controls the entire model adaptation device 10. The functions of the control unit 13 are realized, for example, by a CPU (Central Processing Unit) executing a program stored in the storage unit 12.

[0028] The control unit 13 includes, for example, a learning unit 131 and a model adaptation unit 132. The model adaptation unit 132 includes a query acquisition unit 133, a person matching unit 134, a parameter update unit 135, and a query discard unit 136. Note that the following equation 1 shows the definitions of the symbols that appear in each equation.

[0029]

[0030] [Learning Unit] The learning unit 131 receives learning data (learning images {xi t} i＝1 nt ) to train a person matching model. For example, the training unit 131 trains the person matching model by supervised learning in which a person label is assigned to each image of the training data. Then, the training unit 131 stores parameters of the trained person matching model in the storage unit 12.

[0031] The person matching model includes, for example, a feature extractor that extracts feature amounts of an image of a person to be matched, and a classifier that uses the extracted feature amounts of the image to classify the image as to which person the image belongs to.

[0032] For example, the learning unit 131 learns the person matching model so that images of the same person are brought closer to each other and images of different people are separated from each other in the feature space, as shown in Fig. 6. Furthermore, the learning unit 131 learns the person matching model so as to minimize errors in person classification based on the feature amounts of each image.

[0033] The person matching model may be learned by unsupervised learning or by a combination of supervised and unsupervised learning.

[0034] [Model Adaptation Unit] Returning to Fig. 5, the query acquisition unit 133 acquires a query image from the operating environment of the person matching model. The person matching unit 134 extracts, from the gallery images, images of a person who is predicted to be the same person as the person in the query image using the person matching model.

[0035] For example, the person matching unit 134 first extracts a feature from a feature extractor f θt For example, the person matching unit 134 calculates feature vectors of the gallery image and the query image shown in the following Expression 3 from the gallery image and the query image shown in the following Expression 2 (see Expression (1)).

[0036]

[0037] The person matching unit 134 uses the initial feature extractor f θ0may be used to calculate feature vectors of the gallery images and the query image.

[0038] Thereafter, the person matching unit 134 extracts images of people from the gallery images that are predicted to be the same person as the person in the query image using a classifier of the person matching model based on the feature vectors of the gallery images and the query image.

[0039] For example, the person matching unit 134 calculates the similarity between the feature amount of the extracted query image and the feature amount of each gallery image. For example, the person matching unit 134 calculates the similarity between the feature amount of the extracted query image and the feature amount of each gallery image based on the following equation (2): q and the feature vector z for each gallery image i g The cosine similarity between i The similarity s i is the feature vector z of the query image q and the feature vector z for each gallery image i g It may be calculated by the Euclidean distance, inner product, etc.

[0040]

[0041] Thereafter, the person matching unit 134, for example, i The n gallery images (see Equation 6 below) are extracted in descending order of σ as images of the same person as the person in the query image.

[0042]

[0043] Returning to the description of Fig. 5, the parameter update unit 135 calculates the distribution of probabilities that the query image matches each of the gallery images, based on the similarity between the feature amount of the query image calculated by the person matching unit 134 and the feature amount of each of the gallery images. Then, the parameter update unit 135 calculates the confidence level of the prediction by the person matching model from the calculated probability distribution.

[0044] For example, the parameter update unit 135 extracts k similarities (see Equation 8 below) in descending order of similarity from the similarities (see Equation 7 below) between the feature amounts of the query image and the feature amounts of each of the gallery images.

[0045]

[0046] Next, the parameter update unit 135 uses the extracted k similarities to determine whether the query image is b j Probability of matching the gallery image p j Calculate.

[0047]

[0048] In the above example, the parameter update unit 135 updates the probability p j The softmax function was used to calculate the similarity {s b1,…, s bk} are each greater than or equal to 0 and the sum is 1. k ], any function other than the softmax function may be used.

[0049] The parameter update unit 135 updates the above probability distribution [p1, ..., p k ] to increase the confidence of the prediction of the person matching model calculated from t Update.

[0050] For example, the parameter update unit 135 calculates the probability distribution [p1, . . . , p k ]Calculate the entropy H of

[0051]

[0052] Here, the probability distribution [p1,…,p k ], a specific p j Only larger other p j The smaller the value of p1,…,p k ] is closer to a uniform distribution (=the confidence level of the prediction is lower), the entropy H becomes larger. Therefore, the parameter update unit 135 updates the parameter θ t Update.

[0053] For example, the parameter update unit 135 uses a gradient method or the like to update the parameter θ of the feature extractor in a direction that reduces the entropy H. t In this case, the parameter update unit 135 updates the parameter θ based on the following equation (5): t Update.

[0054]

[0055] The parameter update unit 135 updates the parameters θ of the feature extractor of the person matching model as described above. t By updating the similarity between the feature of the query image and the feature of each gallery image, the accuracy of person matching of the person matching model can be improved.

[0056] In addition to the above, the confidence level of the person matching model prediction can also be calculated using {p1,...,p k}, the maximum probability value of {p1,...,p k} and the second highest probability (margin). k}, the maximum probability value of {p1,...,p k}, the larger these values ​​are, the higher the confidence level of the prediction of the person matching model is. Therefore, the parameter update unit 135 updates the parameter θ t Update.

[0057] Furthermore, the parameter update unit 135 updates the parameters θ of the feature extractor of the person matching model using an equation to which a regularization term is added, such as the following equation (6): t In addition, λ in the formula (6) is a constant indicating the strength of regularization.

[0058]

[0059] The parameter update unit 135 updates the parameter θ of the feature extractor of the person matching model based on the above equation (6). t By updating the parameter θ of the feature extractor of the person matching model, t This update prevents the feature extractor from forgetting what it has learned in the learning environment.

[0060] The query discarding unit 136 updates the parameter θ of the feature extractor of the person matching model by the parameter updating unit 135. t After is updated, discard the query image.

[0061] According to the model adaptation device 10 described above, the person matching model can be adapted to the operating environment continuously without increasing costs. Furthermore, the model adaptation device 10 discards the query image (query) after adapting the person matching model to the operating environment. As a result, the privacy of the data used to adapt the person matching model to the operating environment can be protected.

[0062] [Example of Processing Procedure] Next, an example of processing procedure executed by the model adaptation device 10 will be described with reference to Fig. 7. First, the learning unit 131 of the model adaptation device 10 learns a person matching model using training data (S1).

[0063] Then, the query acquisition unit 133 acquires a query (S2). Then, the person matching unit 134 performs person matching processing by inputting the query to the person matching model (S3: Execute person matching processing using the query). Here, the parameter update unit 135 updates the parameter θ of the feature extractor so as to increase the confidence of the prediction of the person matching model in the person matching processing. t (S4). After that, the query discard unit 136 discards the query (S5).

[0064] Next, an example of calculating the confidence level of the prediction of the person matching model in S4 of Fig. 7 will be described with reference to Fig. 8. For example, the parameter update unit 135 extracts k similarities between the query and the gallery calculated by the person matching unit 134 (S41). For example, the parameter update unit 135 extracts the k similarities between the query and the gallery calculated by the person matching unit 134 in descending order of the similarities.

[0065] After S41, the parameter update unit 135 uses the k similarities extracted in S41 to determine whether the query is b j The probability of matching the th gallery (p j) (S42). As a result, the parameter update unit 135 calculates the distribution [p1,...,p k ] is obtained.

[0066] After S42, the parameter update unit 135 updates the probability distribution [p1, . . . , p k ] (S43). For example, the parameter update unit 135 calculates the confidence level of the prediction of the person matching model from the probability distribution [p1, ..., p k ]Calculate the entropy H of

[0067] Thereafter, the parameter update unit 135 updates the parameter θ of the feature extractor in a direction that increases the accuracy of prediction of the person matching model using a gradient method or the like. t Update.

[0068] In the above embodiment, the matching model adapted by the model adaptation device 10 is a person matching model, but this is not limiting. For example, the matching model may be a model for matching automobiles, etc.

[0069] [Experimental Results] Next, the evaluation results of the person matching model adapted to the operating environment by the model adaptation device 10 of this embodiment will be described.

[0070] The dataset used in the experiment was Market-1501. In this experiment, a person matching model was trained using the Market-1501 training dataset, and then person matching was performed using the trained person matching model. The gallery images used for person matching were the Market-1501 test dataset. In this experiment, an image from the test dataset was corrupted and used as the query image (assuming it was a query image in the production environment). Using the trained person matching model, features from the query image and each of the gallery images were extracted, and person matching was performed.

[0071] The corruption applied to the images in the test dataset was gradually increased. The corruption types were Brightness (which changes the brightness of the image), Gaussian-blur (which blurs the image), and Pixelate (which reduces the image quality) (see Figure 9).

[0072] The experimental results are shown in Figure 10. The horizontal axis of each graph in Figure 10 indicates the number of updates to the person matching model, and the vertical axis indicates the person matching accuracy (Top-1 CMC accuracy) of the person matching model. The vertical dashed lines in each graph indicate the timing when the strength of the corruption applied to the image was switched. The comparative example is a person matching model that was not adapted to the operating environment.

[0073] As shown in FIG. 10, it was confirmed that in a situation where the operating environment is continuously changing, the person matching model adapted to the operating environment by the model adaptation device 10 of this embodiment improves the accuracy of person matching.

[0074] [System Configuration, etc.] The components of each unit shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program executed by the CPU, or can be realized as hardware using wired logic.

[0075] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0076] [Program] The model adaptation device 10 can be implemented by installing a program (model adaptation program) as package software or online software on a desired computer. For example, by having the model adaptation device execute the program, the model adaptation device can function as the model adaptation device 10. The model adaptation device referred to here includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as terminals such as PDAs (Personal Digital Assistants).

[0077] 11 is a diagram showing an example of a computer that executes a model adaptation program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0078] The memory 1010 includes a read-only memory (ROM) 1011 and a random access memory (RAM) 1012. The ROM 1011 stores a boot program such as a basic input / output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0079] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define the processes executed by the model adaptation device 10 are implemented as program modules 1093 in which computer-executable codes are written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, the program modules 1093 for executing processes similar to those of the functional configuration of the model adaptation device 10 are stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced with an SSD (Solid State Drive).

[0080] Data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. The CPU 1020 then reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary and executes them.

[0081] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may also be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0082] REFERENCE SIGNS LIST 10 Model adaptation device 11 Input / output unit 12 Storage unit 13 Control unit 131 Learning unit 132 Model adaptation unit 133 Query acquisition unit 134 Person matching unit 135 Parameter update unit 136 Query discard unit

Claims

1. A model adaptation device comprising: an image acquisition unit that acquires an image of a target object from an operating environment of a matching model that predicts an image of an object that is the same as an object captured in the image of the target object; and a parameter update unit that uses a feature extractor to extract features of the image of the target object and features of each image stored in a memory unit, and updates parameters of the feature extractor of the matching model so as to increase the certainty of the prediction based on the similarity between the extracted features of the image of the target object and the features of each image stored in the memory unit.

2. The model adaptation device according to claim 1, further comprising a discarding unit that discards the image to be matched after updating parameters of the feature extractor of the matching model.

3. A model adaptation method executed by a model adaptation device, comprising: a step of acquiring an image of a target object from an operating environment of a matching model that predicts an image of an object identical to an object captured in the image of a target object; and a step of extracting, by a feature extractor, features of the image of the target object and features of each of the images stored in a memory unit, and updating parameters of the feature extractor of the matching model so as to increase the certainty of the prediction based on the degree of similarity between the extracted features of the image of the target object and the features of each of the images stored in the memory unit.

4. A model adaptation program for causing a computer to execute the steps of: acquiring an image of a target object from an operating environment of a matching model that predicts an image of an object identical to an object captured in the image of a target object; and extracting, by a feature extractor, features of the image of the target object and features of each of the images stored in a memory unit, and updating parameters of the feature extractor of the matching model so as to increase the confidence of the prediction based on the degree of similarity between the extracted features of the image of the target object and the features of each of the images stored in the memory unit.

Images