Specific program, specific method, and information processing device

Abstract

To prevent an erroneous identification of a person.SOLUTION: An information processing device obtains a plurality of pair images of an identical person that are obtained from an area where images captured by a plurality of cameras are overlapped. The information processing device generates a digraph having a node associated with a person feature amount that is obtained from each of a plurality of person images included in the obtained plurality of pair images. The information processing device calculates the weight of a link of the generated digraph, based on the number of person images for which the person feature amounts are similar between nodes. The information processing device determines a combination pattern of person feature amounts in which the total number of person images for which the person feature amounts are similar is the maximum in the whole of the plurality of pair images, based on the calculated weight of the link.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates to a specific program, a specific method, and an information processing device. [Background technology]

[0002] Due to changes in lifestyles and labor shortages, the use of in-store surveillance cameras for analyzing purchasing behavior is being employed to automate and streamline store operations. Examples of purchasing behavior analysis include estimating consumer purchasing characteristics through behavioral analysis of how customers browse within a store, and detecting suspicious behavior in stores with self-checkout counters, thereby achieving new customer acquisition and improved store operational efficiency. Behavioral analysis based on browsing behavior refers to analyzing which products a target consumer purchases within the store, while detection of suspicious behavior refers to whether a customer leaves the store without scanning items placed in their shopping basket.

[0003] In recent years, person tracking technology using multiple surveillance cameras installed in stores has been used to analyze purchasing behavior within various stores. One known method of person tracking is a technique that combines a person detection model and a person identification model to track the same person. For example, in this technique, the person detection model detects bounding boxes from the images of each surveillance camera, and the person identification model identifies whether the bounding boxes in each frame of each surveillance camera belong to the same person. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2019-29021 [Patent Document 2] Japanese Patent Publication No. 2018-61114 [Overview of the project] [Problems that the invention aims to solve]

[0005] However, with the above technology, the image characteristics of the training data for each model used in the person tracking technology often differ from the image characteristics of the image data captured at the store where the person tracking technology is actually applied. This leads to a decrease in the inference accuracy of the person identification model and results in misidentification of people.

[0006] For example, the field of view and brightness of surveillance cameras differ from store to store, and furthermore, the customer base differs in terms of seasonal changes, fashion trends, age, race, and backgrounds such as the color and pattern of merchandise shelves, floors, and pillars. The number of combinations of such image characteristics is enormous, and it is not practical to train the system for all of them.

[0007] Furthermore, since it is impractical to prepare a separate training dataset for each store, publicly available datasets are often used for training each model.

[0008] For example, a person detection model is constructed using deep learning or similar methods to take image data as input, estimate the location of a person within the image data, and output the bounding box for that area. Similarly, a person identification model is constructed using deep learning or similar methods to take image data with bounding boxes for two people as input and output the feature vectors for those people. In the following discussion, image data with specified bounding boxes may be referred to as a "bounding box image."

[0009] Thus, for each model, it is preferable to obtain bounding box images of the same person taken from various angles, for a large number of people. However, obtaining training datasets in a real environment is extremely costly. Furthermore, it is difficult to cover the image characteristics of various stores using publicly available datasets.

[0010] One aspect of this project is to provide a specific program, identification method, and information processing device that can suppress misidentification of individuals. [Means for solving the problem]

[0011] In the first proposal, a specific program is instructed by a computer to acquire multiple pairs of images of the same person obtained from overlapping regions of images captured by multiple cameras, generate a directed graph having nodes corresponding to the person features obtained from each of the multiple person images included in the acquired pairs of images, calculate the link weights of the generated directed graph based on the number of person images with similar person features between the nodes, and, based on the calculated link weights, identify the combination pattern of person features that maximizes the total number of person images with similar person features across all of the multiple pairs of images. [Effects of the Invention]

[0012] According to one embodiment, misidentification of individuals can be suppressed. [Brief explanation of the drawing]

[0013] [Figure 1] Figure 1 shows an example of the overall system configuration according to Example 1. [Figure 2] Figure 2 is a diagram illustrating reference technology 1 for person tracking technology. [Figure 3] Figure 3 illustrates the generation of training data using actual store footage. [Figure 4] Figure 4 is a diagram illustrating reference technology 2 for person tracking technology. [Figure 5] Figure 5 is a diagram illustrating reference technology 3 for person tracking technology. [Figure 6] Figure 6 illustrates an example of pseudo-label generation. [Figure 7] Figure 7 illustrates an example of pseudo-label generation. [Figure 8] Figure 8 illustrates an example of selecting human features. [Figure 9] Figure 9 illustrates an example of selecting human features. [Figure 10]Figure 10 illustrates the generation of a person identification model used in the person tracking technology according to Example 1. [Figure 11] Figure 11 is a functional block diagram showing the functional configuration of the information processing device according to Embodiment 1. [Figure 12] Figure 12 illustrates the generation of a person detection model. [Figure 13] Figure 13 illustrates the calculation of the projection transformation coefficient. [Figure 14] Figure 14 illustrates the detection of a person's bounding box. [Figure 15] Figure 15 is a diagram illustrating coordinate transformation. [Figure 16] Figure 16 illustrates the extraction of identical pairs of individuals. [Figure 17] Figure 17 illustrates the generation of a directed graph. [Figure 18] Figure 18 is a diagram illustrating the calculation of link weights. [Figure 19] Figure 19 illustrates the formulation of a combinatorial optimization problem. [Figure 20] Figure 20 illustrates the identification of combination patterns of human features. [Figure 21] Figure 21 is a diagram illustrating the inference process. [Figure 22] Figure 22 is a flowchart showing the pre-processing flow. [Figure 23] Figure 23 is a flowchart showing the process for extracting identical pairs of individuals. [Figure 24] Figure 24 is a flowchart showing the flow of the training data generation process. [Figure 25] Figure 25 is a flowchart showing the flow of machine learning processing for a person identification model. [Figure 26] Figure 26 is a flowchart showing the flow of the inference process. [Figure 27] Figure 27 illustrates the effects of Example 1. [Figure 28] Figure 28 illustrates an example of a hardware configuration. [Modes for carrying out the invention]

[0014] The following describes in detail, with reference to the drawings, embodiments of the specific program, specific method, and information processing device disclosed in this application. However, the invention is not limited by these embodiments. Furthermore, each embodiment can be combined as appropriate within a non-consistent range. [Examples]

[0015] [Overall structure] Figure 1 is a diagram showing an example of the overall configuration of the system according to Embodiment 1. As shown in Figure 1, this system includes a store 1, which is an example of a space, a plurality of cameras 2 installed in different locations within store 1, and an information processing device 10.

[0016] Each of the multiple cameras 2 is an example of a surveillance camera that captures a predetermined area within store 1, and transmits the captured video data to the information processing device 10. In the following explanation, the video data may be referred to as "video data." The video data also contains multiple image frames in chronological order. Each image frame is assigned a frame number in ascending order of chronological order. A single image frame is image data of a still image captured by camera 2 at a certain point in time.

[0017] The information processing device 10 is an example of a computer that analyzes image data captured by each of the multiple cameras 2. Each of the multiple cameras 2 and the information processing device 10 are connected using various networks, such as the internet or dedicated lines, whether wired or wireless. Furthermore, the store 1 is equipped with regular cash registers, self-checkout registers, etc., and the store staff carry terminals such as smartphones.

[0018] In recent years, various stores (especially those that have introduced self-checkout systems) have been using person tracking technology with multiple surveillance cameras installed within the store to analyze purchasing behavior within the store. Figure 2 is a diagram illustrating reference technology 1 of person tracking technology. As shown in Figure 2, person tracking technology is a technology for tracking the same person by combining a person detection model 50 and a person identification model 60.

[0019] The person detection model 50 detects a person bounding box (Bbox) indicating the location of a person in response to the image data input from each camera 2, and outputs it as an output result. The person identification model 60, in response to the input of two person bounding boxes detected from the image data of each camera, outputs a determination result of whether or not the people are the same person by evaluating the similarity or distance of the feature quantities (feature vectors) of those people.

[0020] However, in actual operation, if the image characteristics of the training data used for machine learning (training) of the person identification model differ from the image characteristics of the actual image data captured by each camera 2, the accuracy of the person identification model 60 will decrease. Also, because the installation positions of each camera 2 are different, the camera's field of view, brightness, background, etc., will also differ, so the accuracy of the person identification model 60 will decrease when the environment of the training data does not match the environment of actual operation.

[0021] In other words, if there is a mismatch in image characteristics between the training data for person identification and the actual stores being inferred, the distribution of person features will fluctuate, reducing the accuracy of person feature inference and leading to misidentification of individuals. Such misidentification makes it difficult to track the same person using image data captured by camera 2, preventing accurate analysis of purchasing behavior.

[0022] Therefore, in Reference Technology 2 for person tracking, since the floor map and camera placement of store 1 can be obtained, the overlapping areas of the shooting regions of multiple cameras are utilized, and the characteristic that person bounding boxes at the same position captured by each camera 2 at the same time represent the same person is focused on, and training data for person identification of the target store is obtained. By performing machine learning of the person identification model using the training data obtained in this way, the influence of image characteristics is reduced, and misidentification of people is suppressed.

[0023] Figure 3 illustrates the generation of training data using actual video footage from store 1. As shown in Figure 3, each camera 2 installed in store 1 captures images from different positions and in different directions, but the areas they capture are partially common (overlapped). For example, the image data captured by camera A shows person A and person B, and the image data captured by camera B shows person A, person B, and person D, with person A and person B being captured in common by each camera. Therefore, although it is not possible to identify "who person A and person B are," it can be identified that they are the same person. Also, person A and person B are captured from different directions and are not the same image data.

[0024] In other words, by using the video data from camera 2 inside the store, it is possible to collect multiple image data of the same person, captured from different angles. Reference technology 2 described above uses each of these image data of the same person captured from different angles as training data to perform machine learning on a person identification model.

[0025] Figure 4 illustrates Reference Technique 2 for person tracking technology. As shown in Figure 4, Reference Technique 2 obtains training data in which image data and ground truth data (person labels) are associated from commonly used public datasets. Then, Reference Technique 2 inputs the image data into a first machine learning model, for example, composed of a convolutional neural network, obtains the output result, and trains the first machine learning model so that the output result matches the ground truth data. In other words, Reference Technique 2 generates the first machine learning model by machine learning a multi-class classification problem using training data related to multiple people.

[0026] Subsequently, Reference Technique 2 generates a second machine learning model using the input and hidden layers of the trained first machine learning model and a new output layer. Furthermore, Reference Technique 2 generates training data with identical person labels (ground truth data) using first and second image data, which are image data of the same person generated from store image data. Then, Reference Technique 2 inputs the first and second image data of the training data generated from store image data into the second machine learning model to obtain an output result including the identity determination result, and trains the second machine learning model so that the output result matches the ground truth data. In other words, Reference Technique 2 generates a second machine learning model by machine learning a two-class classification problem using training data about a given person. Hereafter, the first and second image data, which are image data of the same person generated from store image data, may be referred to as "pair images of the same person" or "pair images". Furthermore, hereafter, each of the first and second image data elements included in the pair images of the same person may be referred to as "sample".

[0027] According to Reference Technology 2, by performing person identification using the second machine learning model generated in this way, person features suitable for the target store are learned, improving the accuracy of person tracking and enabling accurate purchasing behavior analysis.

[0028] An example of an application of Reference Technique 2 is Reference Technique 3, which is described below. Figure 5 is a diagram illustrating Reference Technique 3 for person tracking technology. As shown in Figure 5, Reference Technique 3 is similar to Reference Technique 2, which was explained using Figure 4, in that it generates a first machine learning model by training on a multi-class classification problem using training data obtained from publicly available datasets that are generally used. On the other hand, Reference Technique 3 differs from Reference Technique 2 in that a third machine learning model, which is different from the second machine learning model, is generated using the input layer and hidden layer of the trained first machine learning model and a new output layer.

[0029] To explain this point, in Reference Technology 3, a distance matrix M1 of person features is generated using a dataset of N paired images, which includes pairs of bounding box images of the same person extracted based on the overlapping areas of the shooting regions of multiple cameras 2 in store 1.

[0030] For example, each sample contained in each of the N paired images is input into a convolutional neural network (NN) whose input and hidden layers are repurposed from a first trained machine learning model. The convolutional NN, with the samples input in this way, outputs person features (feature vectors) obtained by embedding the samples into the feature space. This yields person features for each of the 2N samples contained in the dataset of N paired images.

[0031] In Reference Technique 3, a distance matrix M1 is generated by calculating the distance (such as cosine distance) between human features for each combination of 2N samples. In Reference Technique 3, based on a dataset of paired images, the human features of one sample from a pair of samples included in the same paired image are selected, and the human features of the other sample that was not selected are replaced with the human features of the selected sample. After this human feature correction, which unifies the human features between samples in paired images, the distance matrix M1 is generated by calculating the distance between human features for each combination of 2N samples. Note that while an example where distance is calculated is given here, the similarity between samples may be calculated instead of distance.

[0032] Subsequently, in Reference Technique 3, 2N samples contained in a dataset of N paired images are clustered based on the distance matrix M1.

[0033] For example, Figure 5 illustrates a case where clustering is performed using a 5x5 distance matrix M1, which shows five samples from i=1 to i=5. In this case, if the distance between two clusters is less than or equal to the threshold Th1(0.2), the two clusters are merged.

[0034] In this case, the sample i=1 does not have any other samples whose distance is the threshold Th1, so it is not merged with other clusters, and the sample i=1 remains as a single cluster. On the other hand, each of the four samples i=2 to i=5, as shown by the hatching, has a distance of the threshold Th1 in all combinations, so they are merged into a single cluster.

[0035] In Reference Technique 3, of the two clusters obtained as a result of clustering, a pseudo-label A is generated for the cluster to which the sample i=1 belongs, and a pseudo-label B is generated for the cluster to which the four samples i=2 to i=5 belong. Based on these generated pseudo-labels, training data is created in which the person features of 2N samples contained in the dataset of N paired images are associated with the pseudo-labels.

[0036] Furthermore, in Reference Technique 3, a third machine learning model is trained that uses the human features of 2N samples as explanatory variables, pseudo-labels as the target variable, and includes a convolutional neural network (NN) which repurposes the input and hidden layers of a pre-trained first machine learning model, as well as a new output layer.

[0037] Specifically, the sample human features are input into a third machine learning model to obtain output results containing class labels corresponding to the pseudo-label categories, and the third machine learning model is trained so that the output results match the ground truth pseudo-labels. In other words, the parameters of the third machine learning model are updated by optimizing the objective function based on the output results of the third machine learning model and the ground truth pseudo-labels. For example, the parameters of the third machine learning model can be updated based on a loss function that minimizes the loss obtained from the output results and the ground truth pseudo-labels.

[0038] However, in the above reference technique 3, the correction of human features, which unifies the human features between paired image samples, may be corrected to the human features of the inappropriate sample, which can lead to a deterioration in the accuracy of pseudo-labels and misidentification of individuals.

[0039] Figures 6 and 7 illustrate examples of pseudo-label generation. Figures 6 and 7 show paired images, including the pair of samples i=2 and i=5 shown in Figure 5, as examples of paired images of the same person. Furthermore, Figure 6 shows an OK example where the selection of the corrected sample from the paired images is successful, resulting in successful pseudo-label generation. On the other hand, Figure 7 shows an NG example where the selection of the corrected sample from the paired images fails, resulting in unsuccessful pseudo-label generation.

[0040] Hereafter, we will continue our explanation assuming that four of the five samples i=1 to i=5 shown in Figures 5 to 7 are bounding box images of the same person.

[0041] For example, in the example shown in Figure 6, the sample with i=2 is selected as the sample to be used for correction from the two samples in the paired image. In this case, the person features of the sample with i=5, which was not selected as the sample to be used for correction, are corrected to the person features of the sample with i=2, which was selected as the sample to be used for correction.

[0042] After these corrections are performed, a distance matrix M1 is generated by calculating the distance between the human features for each combination of five samples from i=1 to i=5. When clustering of the five samples from i=1 to i=5 is performed based on this distance matrix M1, pseudo-labels are successfully generated, similar to the example shown in Figure 5. Specifically, pseudo-label A is generated for the cluster to which the sample i=1 belongs, and pseudo-label B is generated for the cluster to which the four samples i=2 to i=5 belong.

[0043] In this way, if the human feature of the sample i=2 is selected as the sample to be used for correction, the same pseudo-label B can be assigned to the four samples i=2 to i=5, and thus the generation of pseudo-labels is successful.

[0044] On the other hand, in the example shown in Figure 7, the sample with i=5 is selected as the sample to be corrected from the two samples in the paired image. In this case, the person features of the sample with i=2, which was not selected as the sample to be corrected, are corrected to the person features of the sample with i=5, which was selected as the sample to be corrected.

[0045] After these corrections are performed, a distance matrix M2 is generated by calculating the distance between the human features of each of the five samples i=1 to i=5 for each combination of samples. When clustering of the five samples i=1 to i=5 is performed based on this distance matrix M2, the results are as follows.

[0046] In other words, the sample i=1 does not have any other samples at a distance of threshold Th1, so it is not merged with other clusters, and the sample i=1 remains as a single cluster. Also, the person feature of the sample i=2 has been replaced with the person feature of the sample i=5, so the distance between the person feature of the sample i=2 and the person feature of the sample i=5 is zero. Aside from this sample i=5, the sample i=2 does not have any other samples at a distance of threshold Th1. Therefore, the samples i=2 and i=5 are classified as the same cluster. Also, the sample i=3 has a sample i=4 at a distance of threshold Th1. For this reason, the samples i=3 and i=4 are classified as the same cluster.

[0047] Of the three clusters obtained as a result of this clustering, the cluster to which the sample i=1 belongs is assigned a pseudo-label A. Furthermore, the clusters to which the samples i=2 and i=5 belong are assigned a pseudo-label B. Furthermore, the clusters to which the samples i=3 and i=4 belong are assigned a pseudo-label C.

[0048] If the human feature of sample i=5 is selected as the sample to be used for correction in this way, the same pseudo-label cannot be assigned to the four samples i=2 to i=5, and therefore the generation of pseudo-labels fails.

[0049] To avoid the example shown in Figure 7, randomly selecting the samples used for correction, or fixing one of them, would inevitably lead to selection errors in each of the N paired images, and the inclusion of inappropriate human features would also be unavoidable.

[0050] Therefore, in Example 1, when selecting representative person features from a pair of images of the same person, we focus on the characteristic of the number of similar samples for which the distance between the person features of the samples is below a threshold.

[0051] In this context, "characteristics" refers, in one aspect, to the observation that samples not belonging to the same category's pseudo-labels—that is, bounding box images—tend to have fewer similar samples. In other words, selecting the person feature with a larger number of similar samples can facilitate its incorporation into the same category's pseudo-labels.

[0052] Figures 8 and 9 illustrate examples of human feature selection. Figure 8 shows the number of similar samples in the OK example, where the selection of the sample to be used for correction from the two samples of a paired image is successful, while Figure 9 shows the number of similar samples in the NG example, where the selection of the sample to be used for correction from the two samples of a paired image fails.

[0053] As shown in Figure 8, in the successful example of selecting a sample for correction, the sample with i=2 is selected from the pair of images of the same person. In this case, the sample with i=2 is similar to the samples with i=3 and i=4, so the number of similar samples is "2". On the other hand, as shown in Figure 9, in the unsuccessful example of selecting a sample for correction, the sample with i=5 is selected from the pair of images of the same person. In this case, there are no similar samples for the sample with i=5, so the number of similar samples is "0".

[0054] As shown in the examples in Figures 8 and 9, it is clear that samples belonging to the same category of pseudo-label tend to have a large number of similar samples, while samples not belonging to the same category of pseudo-label tend to have a small number of similar samples. From this, it is obvious that selecting the person feature with a larger number of similar samples will promote its incorporation into the same category of pseudo-label.

[0055] Based on this idea, Example 1 identifies a combination pattern of human features that maximizes the total number of similar samples across N paired images. The samples are clustered using a distance matrix calculated based on the identified combination pattern of human features, and each sample is assigned a pseudo-label based on the results to perform machine learning on a human identification model.

[0056] This allows for the correction of person features, which unifies the person features between paired image samples, by using the person features of the appropriate sample for correction, thereby suppressing the deterioration of false label accuracy and reducing misidentification of people.

[0057] Figure 10 illustrates the generation of a person identification model used in the person tracking technology according to Example 1. As shown in Figure 10, the person tracking technology according to Example 1 is similar to Reference Technology 3, explained using Figure 5, in that it generates a first machine learning model by training on a multi-class classification problem using training data obtained from publicly available datasets that are generally used. On the other hand, Example 1 differs in that, when determining the combination pattern of samples to be used for correction for each pair of images, it identifies the combination pattern of samples that maximizes the total number of similar samples across all N paired images.

[0058] Here, when calculating the total number of similar samples, it is affected by the selection results of other pairs of identical individuals. Therefore, instead of comparing the numbers for each individual pair of identical individuals, it is necessary to calculate the combination that maximizes the overall number.

[0059] For example, when calculating the total number of similar samples for all possible combinations of sample selections for each pair of images of the same person, the computational complexity increases as the number of pairs of images of the same person increases. That is, if there are N pairs of images of the same person, there are 2 possible sample choices for each pair of images, so the computational complexity is 2 N Therefore, in Example 1, an approximate solution method is applied to identify combination patterns of human features that achieves the same accuracy in generating pseudo-labels as when calculating the total number of similar samples for all combination patterns, and reduces the computational cost.

[0060] More specifically, in Example 1, a directed graph is defined for each pair of images, in which two nodes corresponding to the human features of the two samples contained in that pair of images are arranged sequentially from the starting point to the ending point. In this directed graph, each child node of the two nodes in one pair of images corresponds to the two nodes in another pair of images, and each of the two parent nodes and each of the two child nodes are connected by a link. The directed graph defined in this way represents a path from the starting point to the ending point, taking one sample from each pair of images.

[0061] Under such a directed graph, in Example 1, the number of similar samples reflecting the selection result of the person feature of the preceding node is calculated, and this value is used as the weight of each link. For example, the number of similar samples between the nodes included in the path to the preceding node connected to the starting point of the link and the following node connected to the ending point of the link is calculated as the link weight.

[0062] Furthermore, in Example 1, the problem is formulated as a combinatorial optimization problem to identify the combination of node weights that maximizes the sum of the link weights on a directed graph, i.e., the combination of human features, according to an algorithm such as Dijkstra's algorithm. This is equivalent to a combinatorial optimization problem to find the optimal combination from among the possible combinations of the sum of the weights of paths that can be taken on a directed graph. By performing the calculation of this formulated optimization problem, the combination of human features that maximizes the total number of similar samples across N paired images is identified. Moreover, when Dijkstra's algorithm is applied, the computation time required to identify the combination of human features is N 2 It can be reduced to that extent.

[0063] According to the combination patterns of person features identified in this way, for each of the N paired images, the person features to be used for correction are selected from the two sample person features.

[0064] Therefore, as shown in the example in Figure 8, clustering is performed based on the distance matrix M1 calculated after successfully selecting the person features to be used for correction. Consequently, as shown in the example in Figure 6, the sample i=1 is assigned a pseudo-label A, and the same pseudo-label B is assigned to the four samples i=2 to i=5, thus the generation of pseudo-labels is successful.

[0065] Based on the training data to which the pseudo-labels successfully generated in this way are assigned to the samples, a third machine learning model is trained, which includes a convolutional neural network that repurposes the input and hidden layers of the trained first machine learning model, and a new output layer.

[0066] By using the third machine learning model trained in this manner, the information processing device 10 can perform person identification, thereby suppressing the deterioration of pseudo-label accuracy, improving person tracking accuracy, and enabling accurate purchasing behavior analysis.

[0067] [Functional Configuration] Figure 11 is a functional block diagram showing the functional configuration of the information processing device 10 according to Embodiment 1. As shown in Figure 11, the information processing device 10 has a communication unit 11, a storage unit 12, and a control unit 20.

[0068] The communication unit 11 is a processing unit that controls communication with other devices, and is implemented, for example, by a communication interface. For example, the communication unit 11 receives video data from the camera 2 and transmits the processing results from the control unit 20 to the store clerk's terminal or the like.

[0069] The memory unit 12 is a processing unit that stores various data and programs executed by the control unit 20, and is implemented using memory or a hard disk. The memory unit 12 stores the video data DB 13, the public dataset 14, the store dataset 15, the person detection model 16, and the person identification model 17.

[0070] The video data DB13 is a database that stores video data captured by each of the multiple cameras 2 installed in store 1. For example, the video data DB13 stores video data for each camera 2, or for each time period in which the video was captured.

[0071] The public dataset 14 stores pre-collected training data. Specifically, the public dataset 14 stores training data used for machine learning of the person detection model 16 and training data used for machine learning of the multi-class classification problem of the person identification model 17.

[0072] For example, the training data used for machine learning in the person detection model 16 is data that associates image data containing people with bounding boxes indicating the location of the people in the images. In other words, the image data is the explanatory variable, and the bounding boxes are the target variable (ground truth data).

[0073] Furthermore, the training data for multi-class classification problems consists of data where bounding boxes for individuals are associated with individual labels indicating who the individual is. In other words, the bounding boxes are the explanatory variables, and the individual labels are the dependent variables (ground truth data).

[0074] The store dataset 15 stores training data used for machine learning of the person identification model 17. Specifically, the store dataset 15 stores training data generated by the control unit 20 (described later) using video data from camera 2 in store 1. The training data stored here is data in which person bounding box images and pseudo-labels are associated. That is, the person bounding box images are explanatory variables, and the same person labels are the target variables (ground truth data).

[0075] The person detection model 16 is a machine learning model having an input layer, an intermediate layer, and an output layer that detects the bounding box of a person in the image data in response to the input image data. For example, the person detection model 16 can employ a convolutional neural network.

[0076] The person identification model 17 is a machine learning model having an input layer, an intermediate layer, and an output layer that identifies which person a person bounding box represents based on that bounding box input. For example, the person identification model 17 can employ a convolutional neural network.

[0077] The control unit 20 is a processing unit that oversees the entire information processing device 10 and is implemented by, for example, a processor. This control unit 20 includes a detection model generation unit 21, a pre-processing unit 22, a learning data generation unit 23, an identification model generation unit 24, and an inference execution unit 25. The detection model generation unit 21, the pre-processing unit 22, the learning data generation unit 23, the identification model generation unit 24, and the inference execution unit 25 are implemented by electronic circuits and processes executed by the processor.

[0078] The detection model generation unit 21 is a processing unit that generates a person detection model 16 using machine learning. Specifically, the detection model generation unit 21 generates the person detection model 16 by updating various parameters such as weights of the person detection model 16 so that it can detect person bounding boxes from the input training data.

[0079] Figure 12 illustrates the generation of the person detection model 16. As shown in Figure 12, the detection model generation unit 21 obtains training data from the public dataset 14, in which input image data and ground truth data with specified person bounding boxes are associated. The detection model generation unit 21 then inputs the image data into the person detection model 16 and obtains the output result of the person detection model 16. Subsequently, the detection model generation unit 21 performs machine learning on the person detection model 16, such as backpropagation, to reduce the error between the output result and the ground truth data.

[0080] The preprocessing unit 22 comprises an image acquisition unit 22a and a conversion processing unit 22b, and is a processing unit that performs preprocessing to generate training data for the person identification model 17 from image data captured at store 1. Specifically, the preprocessing unit 22 estimates the projection transformation coefficients of the shooting areas of each camera 2 with respect to the floor map of store 1, which is the target of inference.

[0081] The video acquisition unit 22a is a processing unit that acquires video data from each camera 2 and stores it in the video data DB 13. For example, the video acquisition unit 22a may acquire video from each camera 2 as needed, or it may acquire video periodically.

[0082] The conversion processing unit 22b is a processing unit that estimates projection transformation coefficients for converting image coordinates, which are the coordinates of image data captured by each camera 2 and differ for each camera 2, into floor map coordinates, which are the coordinates of the floor map of store 1 and are common to all cameras. Since the cameras and floor configuration are generally fixed, the estimation of projection transformation (homography) coefficients only needs to be performed once.

[0083] Figure 13 illustrates the calculation of projection transformation coefficients. As shown in Figure 13, the transformation processing unit 22b specifies arbitrary corresponding points (corresponding points) between the camera image (image coordinate system) and the floor map (floor map coordinate system). For example, the transformation processing unit 22b identifies points (x1,y1), (x2,y2), (x3,y3), and (x4,y4) from the image coordinate system. Similarly, the transformation processing unit 22b identifies points (X1,Y1), (X2,Y2), (X3,Y3), and (X4,Y4) from the floor map coordinate system. Subsequently, the transformation processing unit 22b calculates the projection transformation coefficient a from the image coordinate system (x,y) to the floor map coordinate system (X,Y). i (i=1-8) is calculated by solving the system of equations shown in equation (1) of Figure 13. Note that the corresponding points may be specified by the user, or points in the same location may be identified through image analysis.

[0084] The learning data generation unit 23 comprises a detection unit 23a, an extraction unit 23b, a generation unit 23c, a calculation unit 23d, and a identification unit 23e. It is a processing unit that performs person detection and coordinate calculation to generate learning data for the person identification model 17 from image data of the camera 2.

[0085] The detection unit 23a is a processing unit that detects human bounding boxes from image data captured by each camera 2 using a trained human detection model 16. Figure 14 is a diagram illustrating the detection of human bounding boxes. As shown in Figure 14, the detection unit 23a inputs the image data captured by the camera 2 to the human detection model 16 and obtains output results in which human bounding boxes with ID=a, ID=b, ID=c, and ID=d are detected.

[0086] In this way, the detection unit 23a performs person detection on various image data captured in different directions by each camera 2 installed at different locations, acquires output results including the detected person bounding box, and stores them in the storage unit 12 or the like.

[0087] The extraction unit 23b is a processing unit that calculates the floor map coordinates of the person bounding box detected by the detection unit 23a and extracts pairs of images of the same person.

[0088] First, the extraction unit 23b uses the projection transformation coefficient calculated by the preprocessing unit 22 to convert the person bounding box in the image coordinate system detected by the detection unit 23a into the floor map coordinate system. Figure 15 is a diagram illustrating the coordinate transformation. As shown in Figure 15, the extraction unit 23b uses the image coordinates (x,y) of the center of the lower end of each person bounding box as the person's position and calculates the person's position in floor map coordinates (X,Y).

[0089] For example, the extraction unit 23b converts the points (x1,y1), (x2,y2), (x3,y3), and (x4,y4) that indicate the location of a person detected in the image coordinate system into points (X1,Y1), (X2,Y2), (X3,Y3), and (X4,Y4) that indicate the location of a person in the floor map coordinate system, using the conversion formula shown in equation (2) in Figure 15. In this way, the extraction unit 23b represents the person bounding boxes in the image coordinate system specific to each camera 2, which are captured in the image data of each camera 2, in the floor map coordinate system common to all cameras.

[0090] Next, the extraction unit 23b acquires a dataset of a pair of person bounding box images located at equivalent floor map coordinates between two cameras. That is, the extraction unit 23b extracts a pair of (paired) person bounding boxes of the same person using the person bounding boxes of a plurality of image data captured at the same time among the image data of each camera 2.

[0091] FIG. 16 is a diagram for explaining the extraction of the same person pair. As shown in FIG. 16, the extraction unit 23b acquires image data A captured by camera A at time t and image data B captured by camera B at the same time t. Then, the extraction unit 23b converts the person bounding box in the image coordinate system detected from the image data A of camera A into a person bounding box in the floor map coordinate system using Equation (2) of FIG. 15. Similarly, the extraction unit 23b converts the person bounding box in the image coordinate system detected from the image data B of camera B into a person bounding box in the floor map coordinate system using Equation (2) of FIG. 15.

[0092] Then, the extraction unit 23b calculates the range of floor map coordinates where the imaging ranges of each camera overlap. For example, as shown in FIG. 16, the imaging range of camera A is such that the X-axis is in the range of X A in or X A out and the Y-axis is in the range of Y A in or Y A out Within this range, the person positions (X A a , Y A a ) and (X A b , Y A b ) are detected. Also, the imaging range of camera B is such that the X-axis is in the range of X B in or X B out and the Y-axis is in the range of Y B inKa Y B out It is within the range, and within that range, the position of the person is (X B a ,Y B a ), (X B b ,Y B b ), (X B c ,Y B c ), (X B d ,Y B d ) has been detected. Note that the position of each person is the image coordinates of the center of the lower edge of the detected person bounding box, as described above.

[0093] Here, the extraction unit 23b extracts the range of the floor map coordinates of camera A (X A ,Y A ) and the range of the floor map coordinates of camera B (X B ,Y B The overlapping range (X AB ,Y AB ) is calculated. Note that as shown in equation 3 of Figure 16, X AB The range is "X A in or X B in The maximum value among " and "X A out Or X B out It is less than or equal to the minimum value among "Y AB The range is "Y A in or Y B in The maximum value among "Y" and A out Or Y B out It is less than or equal to the minimum value among them.

[0094] Next, the extraction unit 23b is the overlapping range (X AB ,Y ABFor the groups of people in each camera in (), equivalent-position person pairs are extracted. Specifically, the extraction unit 23b uses a method such as minimum weighted matching based on Euclidean distance to extract combinations of neighboring pairs, and among the neighboring pairs, pairs with a Euclidean distance smaller than a predetermined threshold are regarded as the same person pairs. At this time, since the extraction unit 23b will obtain a large amount of almost the same pair data for each frame extraction, it can also be thinned out by sampling.

[0095] In the example of FIG. 16, the extraction unit 23b detects that in the overlapping range of camera A and camera B, the person Aa (X A a , Y A a ) and the person Ab (X A b , Y A b ) are detected in the shooting range on the camera A side, and the person Ba (X B a , Y B [[ID=;21]] a ) and the person Bd (X B d , Y B d ) are detected in the shooting range on the camera B side. Subsequently, the extraction unit 23b calculates the Euclidean distance between the person Aa (X A a , Y A a ) and the person Ba (X B a , Y B a ) and the Euclidean distance between the person Aa (X A a , Y A a ) and the person Bd (X B d , Y B d ) are calculated. Similarly, the extraction unit 23b calculates the Euclidean distance between the person Ab (X A b , Y A b ) and the person Ba (X B a , Y B a) Euclidean distance and person Ab(X A b ,Y A b ) and person Bd(X B d ,Y B d Calculate the Euclidean distance between the two points.

[0096] Subsequently, the extraction unit 23b identifies person Aa(X) as a pair of people whose Euclidean distance is less than a predetermined threshold. A a ,Y A a ) and person Ba(X B a ,Y B a ), person Ab(X A b ,Y A b ) and person Bd(X B d ,Y B d Extract each pair from the list.

[0097] In this way, the extraction unit 23b extracts pairs of images of the same person (person bounding box) from the image data of each camera captured at the same time. This yields N sets of images of the same person.

[0098] The generation unit 23c is a processing unit that generates a directed graph having nodes corresponding to the person features of multiple samples included in a pair of images of the same person.

[0099] Figure 17 illustrates the generation of a directed graph. As shown in Figure 17, first, the generation unit 23c inputs each of the samples contained in each of the N paired images into a convolutional neural network, which is a repurposed input layer and hidden layer of a first machine learning model that has been trained. This yields the person features for each of the 2N samples contained in the dataset of N paired images.

[0100] Then, the generation unit 23c arranges two nodes corresponding to the two sample person features contained in each of the N pairs of images of the same person, in order from the starting point to the ending point. At this time, the child nodes of each of the two nodes in one pair of images become the two nodes in another pair of images, and each of the two parent nodes and each of the two child nodes are connected by a link.

[0101] For example, the starting node v of a directed graph S Next, the first pair of person features f 1 i Corresponding node and person feature f 1 j The corresponding nodes are arranged. Following this, the second pair of person features f 2 i Corresponding node and person feature f 2 j The corresponding node, ..., the Nth pair of person features f N i Corresponding node and person feature f N j The corresponding nodes are arranged in order. After these Nth pairs of nodes, the endpoint node v T They are lined up.

[0102] In such a directed graph, the child nodes of each of the two nodes in a pair of images are the two nodes in another pair of images, and each of the two parent nodes and each of the two child nodes are connected by a link.

[0103] For example, taking the link between the first pair and the second pair as an example, the following four links are defined: namely, person feature f 1 i and person feature f 2 i Links connecting, person feature f 1 i and person feature f 2 j Links connecting, person feature f 1 j and person feature f2 i Links connecting, person feature f 1 j and person feature f 2 j A link is defined to connect them.

[0104] A directed graph, as defined in this way, represents a path from a starting point to an endpoint, taking one sample from each pair of images.

[0105] The calculation unit 23d is a processing unit that calculates the weights of links included in the directed graph. For example, for each link included in the directed graph, the calculation unit 23d calculates the number of similar samples that reflect the selection result of the person feature of the preceding node, and sets that value as the weight of the link.

[0106] Figure 18 illustrates the calculation of link weights. Figure 18 shows examples of the number of similar samples for links indicated by thick solid lines and the number of similar samples for links indicated by thick dashed lines.

[0107] For example, in the case of a link shown by a thick solid line, the number of similar samples is calculated between the person feature of sample i=2, which is the immediate next node connected to the endpoint of the link, and the person features of the nodes included in the path from the immediate next node connected to the starting point of the link, such as samples i=1, i=3, and i=4. In this case, the distance between the person feature of sample i=2 and the person features of samples i=3 and i=4 is less than or equal to the threshold Th1, so the number of similar samples is calculated to be "2". This calculated number of similar samples "2" is then set as the weight of the link shown by the thick solid line.

[0108] Furthermore, in the case of links indicated by thick dashed lines, the number of similar samples is calculated between the person feature of sample i=5, which is the immediate next node connected to the endpoint of the link, and the person features of the nodes included in the path from the immediate next node connected to the starting point of the link, for example, samples i=1, i=3, and i=4. In this case, there are no person features whose distance to the person feature of sample i=5 is less than or equal to the threshold Th1, so the number of similar samples is calculated to be "0". This calculated number of similar samples "0" is then set as the weight of the link indicated by the thick dashed line.

[0109] Note that Figure 18 shows examples of the person features of samples i=1, i=3, and i=4 that are common between the two links, as an example of nodes included in the path to the previous node. However, different person features may be included for each node in the previous pair.

[0110] The identification unit 23e is a processing unit that identifies a combination pattern of person features that maximizes the total number of similar samples across all paired images of the same person. For example, the identification unit 23e is formulated as a combinatorial optimization problem that identifies a combination pattern of person features, i.e., a combination pattern of person features that maximizes the sum of the weights of links on a directed graph according to Dijkstra's algorithm.

[0111] Figure 19 illustrates the formulation of a combinatorial optimization problem. For example, the elements related to the directed graph G=(V,E) shown in Figure 19 are represented as follows: that is, the edges are e ij =(v i ,v j )∈E, node v i ∈V, and further, edge e ij The weight of a ij The number of similar samples is denoted as |R(i,J)|. Node v i This corresponds to the person features of sample i. The number of similar samples |R(i,J)| corresponds to the number of similar samples of node j, taking into account the person features of the preceding node i.

[0112] Here, the starting node v of the directed graph G shown in Figure 19.S From the endpoint node v T The optimization problem of finding the path z that maximizes the total number of similar samples can be formulated as algorithm Ag according to Dijkstra's algorithm. This algorithm Ag represents a combinatorial optimization problem in which the optimal combination is found from among the combinations of the sum of weights of possible paths z in a directed graph G. When Dijkstra's algorithm is applied in this way, the computational complexity for identifying combination patterns of person features is N. 2 It can be reduced to this extent.

[0113] The specific unit 23e identifies the combination pattern of human features that maximizes the total number of similar samples across all N paired images by performing calculations on the optimization problem formulated in algorithm Ag.

[0114] Figure 20 illustrates the identification of combination patterns of human features. As shown in Figure 20, by performing the calculation of the optimization problem formulated in algorithm Ag shown in Figure 19, the path of links shown in thick lines is identified as the combination pattern of human features that maximizes the total number of similar samples across all N paired images. This identified combination pattern of human features allows for the successful selection of human features to be used for correction. For example, the path obtained from the optimization problem calculation allows for the selection of human features j from among human features i and j included in the second paired image that promotes incorporation into the same pseudo-label.

[0115] After the combination patterns of human features are identified in this way, the training data generation unit 23 performs a correction for each pair of images of the same person, unifying the human features of the two samples to match the combination pattern of human features identified by the identification unit 23e. Through this correction, for each pair of images of the same person, any human features that were not selected in the combination pattern of human features are replaced with the human features that were selected in the combination pattern of human features.

[0116] Subsequently, as shown in Figure 10, the training data generation unit 23 calculates a distance matrix M1 by calculating the distance between the person features of each sample using the corrected person features for each combination of 2N samples. Then, based on the distance matrix M1, the training data generation unit 23 clusters the 2N samples contained in the dataset of N paired images and generates a pseudo-label for each cluster obtained as a result of the clustering. After that, the training data generation unit 23 generates training data in which the person features of the 2N samples contained in the dataset of N paired images are the explanatory variables and the pseudo-labels are the target variables, and stores it in the store dataset 15.

[0117] Returning to Figure 11, the identification model generation unit 24 has a first machine learning unit 24a and a second machine learning unit 24b, and is a processing unit that performs machine learning on the person identification model 17.

[0118] The first machine learning unit 24a performs machine learning on a multi-class classification problem using the public dataset 14 to generate a first machine learning model. As explained with reference to Figure 10, the first machine learning unit 24a generates a first machine learning model by performing machine learning on a multi-class classification problem that identifies the person in each input training data, depending on the input training data in which the same person is depicted in different ways. The first machine learning model may include a convolutional neural network with an input layer and hidden layers, and an output layer.

[0119] For example, the first machine learning unit 24a inputs various bounding boxes of person A included in the public dataset 14 into a convolutional neural network and obtains each classification result (output result) from the output layer. Then, the first machine learning unit 24a updates the parameters of the convolutional neural network and the output layer so that the error between each classification result and the person label (person A) is small, in other words, so that it is classified as person A.

[0120] Similarly, the first machine learning unit 24a inputs various bounding boxes of person B included in the public dataset 14 into a convolutional neural network and obtains each classification result from the output layer. The first machine learning unit 24a then updates the parameters of the convolutional neural network and the output layer to minimize the error between each classification result and the person label (person B).

[0121] Once machine learning using the publicly available dataset is complete, the second machine learning unit 24b generates a person identification model 17, which is an example of a third machine learning model, by performing machine learning using the store dataset 15.

[0122] Specifically, the second machine learning unit 24b sets the parameters of the layer structure of the person identification model 17 using a convolutional neural network that includes the input layer and hidden layer of the first machine learning model which has been trained, and a new output layer which has not been trained. Then, the second machine learning unit 24b generates the person identification model 17 by performing machine learning that classifies person bounding box images into pseudo-label classes using the training data stored in the store dataset 15.

[0123] For example, as shown in Figure 10, the second machine learning unit 24b trains a third machine learning model that uses the person features of 2N samples as explanatory variables, pseudo-labels as the target variable, and includes a convolutional neural network that repurposes the input and hidden layers of the trained first machine learning model, as well as a new output layer.

[0124] Specifically, the sample human features are input into a third machine learning model to obtain output results containing class labels corresponding to the pseudo-label categories, and the third machine learning model is trained so that the output results match the ground truth pseudo-labels. In other words, the parameters of the third machine learning model are updated by optimizing the objective function based on the output results of the third machine learning model and the ground truth pseudo-labels. For example, the parameters of the third machine learning model can be updated based on a loss function that minimizes the loss obtained from the output results and the ground truth pseudo-labels.

[0125] Returning to Figure 11, the inference execution unit 25 is a processing unit that uses the person identification model 17 generated by the identification model generation unit 24 to identify the people captured in each image data captured by the camera 2 in the physical store. In other words, the inference execution unit 25 uses the person identification model 17 to associate the people in the image data captured by each camera 2.

[0126] Figure 21 is a diagram illustrating the inference process. As shown in Figure 21, the inference execution unit 25 inputs each image data captured by each camera 2 in the store into the trained person detection model 16 and obtains an output result that includes the detected person bounding box. For example, the inference execution unit 25 obtains the person bounding box for "ID=xx" and the person bounding box for "ID=yy" which are included in different output results.

[0127] Then, the inference execution unit 25 inputs the bounding box of the person with "ID=xx" into the person identification model 17 and obtains person features from the layer immediately preceding the output layer of the person identification model 17. Similarly, the inference execution unit 25 inputs the bounding box of the person with "ID=yy" into the person identification model 17 and obtains person features from the layer immediately preceding the output layer of the person identification model 17.

[0128] Subsequently, the inference execution unit 25 calculates the similarity of each feature, and if the similarity is high, it infers that the bounding box of person "ID=xx" and the bounding box of person "ID=yy" are the same person. On the other hand, if the similarity of each feature is low, the inference execution unit 25 infers that the bounding box of person "ID=xx" and the bounding box of person "ID=yy" are not the same person.

[0129] For example, the inference execution unit 25 calculates the similarity of each feature, such as the Euclidean distance, cosine similarity, or squared error of each feature element, and infers that they are the same person if the calculated similarity is greater than or equal to a threshold.

[0130] By tracking the bounding boxes of each individual inferred to be the same person in this way, it can be used to analyze that person's behavior and purchases within the store.

[0131] [Process flow] Next, we will explain the processes performed by each of the above-mentioned processing units. Here, we will describe the preprocessing, the extraction of identical person pairs, the training data generation process, the machine learning process, and the inference process.

[0132] (Pre-processing) Figure 22 is a flowchart showing the pre-processing flow. As shown in Figure 22, the pre-processing unit 22 acquires video data from each camera 2 (S101) and acquires a pre-designed store floor map (S102).

[0133] Then, the preprocessing unit 22 identifies corresponding points, which are arbitrary points in the image data from camera 2 and the floor map (S103), and estimates the projection transformation coefficient using equation (1) in Figure 13 (S104).

[0134] (Process for extracting identical pairs of people) Figure 23 is a flowchart showing the flow of the process for extracting identical pairs of people. As shown in Figure 23, the learning data generation unit 23 acquires video data from each camera 2 from the video data DB 13 (S201) and obtains the projection transformation coefficient estimated by the preprocessing unit 22 (S202).

[0135] Next, the learning data generation unit 23 inputs each image data from the video data of each camera 2 into the person detection model 16 and performs person detection (S203), detecting the person bounding box (S204).

[0136] Then, the learning data generation unit 23 calculates the floor map coordinates of each person's bounding box using projection transformation coefficients (S205). In other words, the learning data generation unit 23 converts the image coordinate system of each person's bounding box to floor map coordinates.

[0137] Subsequently, the learning data generation unit 23 calculates the overlapping area of ​​the floor map coordinate system for the image data from the two cameras (S206). Then, the learning data generation unit 23 extracts pairs of people at equivalent locations from the image data captured by the two cameras at the same time (S207).

[0138] (Training data generation process) Figure 24 is a flowchart showing the flow of the training data generation process. As shown in Figure 24, the training data generation unit 23 generates a directed graph G for each pair of images of the same person, in which two nodes corresponding to the two samples of person features contained in the pair of images are arranged in order from the starting point to the ending point (S211).

[0139] In step S211 described above, the child nodes of each of the two nodes in one pair of images are made to be the two nodes in another pair of images, and links connecting each of the two parent nodes and each of the two child nodes are set up in the directed graph G.

[0140] Then, the learning data generation unit 23 executes a loop process 1 that repeats the processes from step S212 to step S213 below a number of times corresponding to the number of links K included in the directed graph G generated in step S211.

[0141] In other words, the training data generation unit 23 calculates the number of similar samples between the nodes included in the path to the node immediately preceding link k and the node immediately following link k (S212). Then, the training data generation unit 23 sets the number of similar samples calculated in step S212 as the weight of link k (S213).

[0142] As this loop process 1 is repeated, the weight of each of the K links in the directed graph G is calculated.

[0143] Then, the learning data generation unit 23 calculates an optimization problem to find the optimal combination from among the possible combinations of the sum of weights of paths z in the directed graph G, thereby identifying the combination pattern of nodes that maximizes the total number of similar samples across all N paired images, i.e., the combination pattern of human features (S214).

[0144] Next, the learning data generation unit 23 performs a correction (S215) to unify the person features of the two samples into person features corresponding to the combination pattern of person features identified in step S214 for each pair of images of the same person (N).

[0145] Subsequently, the learning data generation unit 23 calculates a distance matrix by calculating the distance between the person features of each sample using the corrected person features for each combination of 2N samples (S216).

[0146] Then, the learning data generation unit 23 clusters 2N samples contained in the dataset of N paired images based on the distance matrix, and generates pseudo-labels for each cluster obtained as a result of the clustering (S217 and S218).

[0147] Then, the training data generation unit 23 generates training data in which the person features of 2N samples contained in the dataset of N paired images are used as explanatory variables and the pseudo-labels are used as the target variable, and stores this data in the store dataset 15 (S219).

[0148] (Machine learning processing) Figure 25 is a flowchart showing the flow of the machine learning process for the person identification model. As shown in Figure 25, the identification model generation unit 24 acquires existing training data pre-stored in the public dataset 14 (S301), and uses the existing training data to perform machine learning on the first machine learning model as a multi-class classification problem (S302).

[0149] Next, the identification model generation unit 24 acquires training data for the target store, which is generated using the image data of the store stored in the store dataset 15 (S303), and performs machine learning of the person identification model 17 using the training data for the target store (S304).

[0150] (Inference processing) Figure 26 is a flowchart showing the flow of the inference process. As shown in Figure 26, the inference execution unit 25 acquires each image data from each camera 2 (S401), inputs each image data into the person detection model 16, and detects the person bounding box (S402).

[0151] The inference execution unit 25 then inputs the two person bounding boxes into the person identification model 17 (S403) and obtains the feature quantities of each person bounding box from the layer immediately preceding (the layer before) the output layer of the person identification model 17 (S404). Subsequently, the inference execution unit 25 calculates the similarity of the feature quantities of each person bounding box and performs person identification (S405).

[0152] [effect] As described above, when the information processing device 10 selects person features representing two samples from a pair of images of the same person, it focuses on the characteristic of the number of similar samples where the distance between the person features of the samples is below a threshold, and identifies a combination pattern of person features that maximizes the total number of similar samples across all N paired images. Based on the training data to which pseudo-labels successfully generated according to this identified combination pattern of person features are assigned, a person identification model 17 is generated. Therefore, by performing person identification using the person identification model 17 thus generated, the deterioration of the accuracy of the pseudo-labels is suppressed, the accuracy of person tracking is improved, and accurate purchasing behavior analysis can be achieved. Furthermore, since training data for person identification of the target store can be obtained, training data for analysis can be obtained automatically, and the accuracy of person identification can be continuously improved.

[0153] Figure 27 illustrates the effects of Example 1. Figure 27 shows a comparison of the inference accuracy of person identification between Reference Technology 3 and the technology from Example 1 (proposed technology). Here, datasets A, B, and C, each with different person image characteristics (season, background, etc.), are used as examples. Furthermore, the inference accuracy of person identification between Reference Technology 3 and the proposed technology from Example 1 is compared for a total of four patterns: pattern P1 where training is performed on dataset A and inference on dataset B, pattern P2 where training is performed on dataset B and inference on dataset A, pattern P3 where training is performed on dataset B and inference on dataset C, and pattern P4 where training is performed on dataset A and inference on dataset C.

[0154] As shown in Figure 27, the inference accuracy is compared using cumulative matching characteristics, which is the proportion of individuals identified as the same person within a specific rank within a large dataset of people. For example, in Reference Technology 3, the correction of person features, which unifies person features between paired image samples, may be corrected to the person features of the inappropriate sample, resulting in insufficient inference accuracy. On the other hand, in the method according to Example 1, the correction of person features, which unifies person features between paired image samples, can use the person features of the appropriate sample for correction, thus improving inference accuracy. For example, comparing the precision for the highest rank, in pattern P1, the inference accuracy of Reference Technology 3 is "93.2", while that of Example 1 is improved to "94.0". Furthermore, in pattern P2, the inference accuracy of Reference Technology 3 is "81.9", while that of Example 1 is improved to "82.3". Furthermore, in pattern P3, the inference accuracy of Reference Technology 3 is "60.6", while that of Example 1 is improved to "65.1". Furthermore, in pattern P4, while the inference accuracy of Reference Technology 3 is "64.0", the inference accuracy of Example 1 has improved to "66.9". Thus, in all four patterns from P1 to P4, the inference accuracy of Example 1 is superior to that of Reference Technology 3, clearly demonstrating the improvement in inference accuracy of Example 1.

[0155] In this way, the information processing device 10 learns person features suitable for the target store, improving the accuracy of person tracking and enabling accurate purchasing behavior analysis. The information processing device 10 can track browsing behavior and suspicious behavior by accurately identifying people from multiple surveillance cameras within the store. The information processing device 10 can acquire and learn person identification data for the target store from overlapping information of the shooting areas of multiple cameras. [Examples]

[0156] Now, although embodiments of the present invention have been described, the present invention may be implemented in various other forms besides those described above.

[0157] [Numerical values, etc.] The number of cameras, numerical examples, training data examples, machine learning models, and coordinate examples used in the above embodiment are merely examples and can be changed at will. Furthermore, the processing flow described in each flowchart can be modified as appropriate within a consistent range. Each model can be a model generated by various algorithms, such as neural networks. In the above embodiment, the second machine learning unit 24b was described as constructing a person identification model 17 using a convolutional neural network including the input and hidden layers of the first machine learning model (which has already been trained) and a new, untrained output layer. However, this is not limiting, and the person identification model 17 can also be constructed using some layers of the first machine learning model. In this case, it is preferable to exclude the output layer of the first machine learning model. Also, coordinate transformation can be performed on an image data basis or on a person bounding box basis.

[0158] [system] Unless otherwise specified, the processing procedures, control procedures, specific names, and various data and parameters shown in the above documents and drawings may be changed at will.

[0159] Furthermore, the specific forms of distribution and integration of the components of each device are not limited to those shown in the diagram. For example, the pre-processing unit 22 and the learning data generation unit 23 may be integrated. In other words, all or part of the components may be functionally or physically distributed and integrated in any unit depending on various loads and usage conditions. Moreover, all or any part of the processing functions of each device may be implemented by a CPU and a program executed for analysis on that CPU, or as hardware using wired logic.

[0160] [Hardware] Figure 28 illustrates an example of hardware configuration. As shown in Figure 28, the information processing device 10 includes a communication device 10a, an HDD (Hard Disk Drive) 10b, memory 10c, and a processor 10d. Furthermore, the components shown in Figure 19 are interconnected by a bus or the like.

[0161] The communication device 10a is a network interface card or the like, and communicates with other devices. The HDD 10b stores programs and databases that operate the functions shown in Figure 11.

[0162] The processor 10d operates a process that performs the functions described in Figure 11 by reading a program that performs the same processing as each processing unit shown in Figure 11 from the HDD 10b or the like and loading it into memory 10c. For example, this process performs the same functions as each processing unit of the information processing device 10. Specifically, the processor 10d reads a program that has the same functions as the detection model generation unit 21, pre-processing unit 22, learning data generation unit 23, identification model generation unit 24, inference execution unit 25, etc. from the HDD 10b or the like. Then, the processor 10d executes a process that performs the same processing as the detection model generation unit 21, pre-processing unit 22, learning data generation unit 23, identification model generation unit 24, inference execution unit 25, etc.

[0163] Thus, the information processing device 10 operates as an information processing device that executes an information processing method by reading and executing a program. Furthermore, the information processing device 10 can also achieve the same functionality as the embodiment described above by reading the program from the recording medium using a media reader and executing the read program. Note that the program referred to in this other embodiment is not limited to being executed by the information processing device 10. For example, the above embodiment may also be applied to cases where another computer or server executes the program, or where these computers or servers collaborate to execute the program.

[0164] This program may be distributed via a network such as the Internet. Alternatively, this program may be recorded on a computer-readable recording medium such as a hard disk, flexible disk (FD), CD-ROM, MO (Magneto-Optical disk), or DVD (Digital Versatile Disc), and executed by being read from the recording medium by a computer. [Explanation of Symbols]

[0165] 1 store 2 cameras 10 Information Processing Devices 11 Communications Department 12 Storage section 13 Video Data Database 14 Public Datasets 15 store dataset 16. Person detection models 17. Person Identification Model 20 Control Unit 21 Detection Model Generation Unit 22 Pre-processing 22a Video acquisition unit 22b Conversion Processing Unit 23. Training Data Generation Unit 23a Detection unit 23b Extraction part 23c Generator 23d Calculation Unit 23e Specific part 24 Identification Model Generation Unit 24a 1st Machine Learning Department 24b Second Machine Learning Department 25 Inference Execution Unit

Claims

1. On the computer, By obtaining multiple pairs of images of the same person from the overlapping areas of images captured by each of multiple cameras, A directed graph is generated that has nodes corresponding to the person features obtained from each of the multiple person images contained in the multiple paired images acquired. Based on the number of human images with similar human features between the nodes, the link weights of the generated directed graph are calculated. Based on the calculated link weights, identify the combination pattern of person features that maximizes the total number of person images with similar person features across the multiple paired images. A specific program characterized by executing a process.

2. The aforementioned generation process is, The specific program according to claim 1, characterized in that for each of the plurality of paired images, two nodes corresponding to each of the person features of the two person images contained in the paired image are arranged in order from the starting point to the ending point, and a directed graph is generated by linking each of the two parent nodes of the first paired image and each of the two child nodes of the second paired image among the plurality of paired images.

3. The process to be identified is, The identification program according to claim 1, characterized in that it searches for a combination of nodes that maximizes the sum of the link weights along the path from the starting point to the ending point of the directed graph, according to Dijkstra's algorithm, to identify a combination of person features that maximizes the total number of person images with similar person features across the multiple paired images.

4. For each of the aforementioned paired images, the person feature quantity of the two person images that is not included in the combination pattern is corrected to the person feature quantity of the person included in the combination pattern. For each combination of the aforementioned person images, a distance matrix is ​​calculated by calculating the distance between the person features of the person images using the corrected person features. Based on the calculated distance matrix, the human images included in the dataset of multiple paired images are clustered. A pseudo-label is generated for each cluster obtained as a result of clustering. As training data to be used in machine learning for a machine learning model that classifies the aforementioned person images into the classes of the aforementioned pseudo-labels, the training data is generated with the person images included in the dataset of multiple paired images as explanatory variables and the correct pseudo-labels as the target variable. A specific program according to any one of claims 1 to 3, characterized in that it causes the computer to perform the processing.

5. Computers By obtaining multiple pairs of images of the same person from the overlapping areas of images captured by each of multiple cameras, A directed graph is generated that has nodes corresponding to the person features obtained from each of the multiple person images contained in the multiple paired images acquired. Based on the number of human images with similar human features between the nodes, the link weights of the generated directed graph are calculated. Based on the calculated link weights, identify the combination pattern of person features that maximizes the total number of person images with similar person features across the multiple paired images. A method for identifying a specific entity, characterized by performing a process.

6. By obtaining multiple pairs of images of the same person from the overlapping areas of images captured by each of multiple cameras, A directed graph is generated that has nodes corresponding to the person features obtained from each of the multiple person images contained in the multiple paired images acquired. Based on the number of human images with similar human features between the nodes, the link weights of the generated directed graph are calculated. Based on the calculated link weights, identify the combination pattern of person features that maximizes the total number of person images with similar person features across the multiple paired images. An information processing device characterized by having a control unit.

Images