Machine learning program, machine learning method, and information processing device

Abstract

To suppress misidentification of a person.SOLUTION: An information processing apparatus inputs learning data related to a plurality of persons to a first machine learning model and executes machine learning of the first machine learning model so as to identify a person. The information processing apparatus extracts a part of layers of the first machine learning model generated by the machine learning and generates a new second machine learning model by using the extracted part of the layers. The information processing apparatus inputs multiple pieces of learning data related to a predetermined person to the second machine learning model and executes machine learning of the second machine learning model so as to determine whether the person is identical with the person included in each of the multiple pieces of learning data.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a machine learning program, a machine learning method, and an information processing device. [Background technology]

[0002] Due to changes in lifestyles and labor shortages, purchasing behavior analysis using in-store surveillance cameras is being used to automate and streamline store operations. Examples of purchasing behavior analysis include estimating consumer purchasing characteristics through behavioral analysis based on shopping patterns within the store, and detecting suspicious behavior in stores with self-checkouts, which can lead to new customer acquisition and more efficient store operations. Behavioral analysis based on shopping patterns refers to analyzing which products a target consumer purchases in the store, while suspicious behavior detection refers to whether a consumer leaves the store without scanning items they have added to their shopping cart.

[0003] In recent years, people tracking technology using multiple surveillance cameras installed in various stores has been used to analyze purchasing behavior in stores. A known type of person tracking technology is a technology for tracking the same person that combines a person detection model and a person identification model. For example, in the same person tracking technology, a person detection model is used to detect bounding boxes from images captured by each surveillance camera, and a person identification model is used to identify whether the bounding boxes of people in each frame captured by each surveillance camera represent the same person. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-29021 [Patent Document 2] Japanese Patent Application Publication No. 2018-61114 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the above technology, the image characteristics of the training data for each model used in the person tracking technology often differ from the image characteristics of image data captured in stores where the person tracking technology is actually applied, reducing the inference accuracy of the person identification model and resulting in misidentification of people.

[0006] For example, each store to which the system is applied has different surveillance camera angles and brightness, as well as different customer demographics (e.g., age and race), clothing trends (depending on the season and fashion), and different backgrounds (e.g., colors and patterns of shelves, floors, and pillars).The number of combinations of such image characteristics is enormous, and it is not realistic to train the system for all of them.

[0007] In addition, since it is practically impractical to prepare a dataset of learning data for each store to be used to train each model, publicly available datasets are often used.

[0008] For example, a person detection model is constructed using deep learning or the like to input image data, estimate the location of a person in the image data, and output that area (bounding box). A person identification model is constructed using deep learning or the like to input image data in which the bounding boxes of two people are specified, and output the feature quantities (feature vectors) of those people. Hereinafter, image data in which bounding boxes are specified may be referred to as a "bounding box image."

[0009] As such, it is desirable to acquire a large number of bounding box images of the same person taken from various angles as training data for each model, but acquiring training datasets in real environments is extremely costly, and it is difficult to cover the image characteristics of various stores with publicly available datasets.

[0010] In one aspect, an object of the present invention is to provide a machine learning program, a machine learning method, and an information processing device that can reduce erroneous person identification. [Means for solving the problem]

[0011] In a first proposal, the machine learning program causes a computer to execute the following processes: inputting training data related to multiple people into a first machine learning model; executing machine learning of the first machine learning model to identify people; extracting a portion of a layer of the first machine learning model generated by the machine learning; generating a new second machine learning model using the extracted portion of the layer; inputting multiple training data related to a specified person into the second machine learning model; and executing machine learning of the second machine learning model to identify whether the person is the same as the person included in each of the multiple training data. [Effects of the Invention]

[0012] According to one embodiment, it is possible to reduce erroneous person identification. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of a system according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating a reference technique for person tracking technology. [Figure 3] FIG. 3 is a diagram illustrating the generation of learning data using actual store footage. [Figure 4] FIG. 4 is a diagram illustrating generation of a person identification model used in the person tracking technology according to the first embodiment. [Figure 5] FIG. 5 is a functional block diagram of the information processing apparatus according to the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating the generation of a person detection model. [Figure 7] FIG. 7 is a diagram illustrating calculation of the projective transformation coefficients. [Figure 8] FIG. 8 is a diagram illustrating the detection of a person bounding box. [Figure 9] FIG. 9 is a diagram illustrating coordinate transformation. [Figure 10]FIG. 10 is a diagram for explaining extraction of pairs of the same person. [Figure 11] FIG. 11 is a diagram illustrating the generation of learning data. [Figure 12] FIG. 12 is a diagram illustrating the generation of a person identification model. [Figure 13] FIG. 13 is a diagram illustrating the inference process. [Figure 14] FIG. 14 is a flowchart showing the flow of the pre-processing. [Figure 15] FIG. 15 is a flowchart showing the flow of the data collection process. [Figure 16] FIG. 16 is a flowchart showing the flow of machine learning processing of a person identification model. [Figure 17] FIG. 17 is a flowchart showing the flow of the inference process. [Figure 18] FIG. 18 is a diagram illustrating the effect of the first embodiment. [Figure 19] FIG. 19 is a diagram illustrating an example of a hardware configuration. DETAILED DESCRIPTION OF THE INVENTION

[0014] The following describes in detail the embodiments of the machine learning program, the machine learning method, and the information processing device disclosed in the present application, with reference to the accompanying drawings. Note that the present invention is not limited to these embodiments. Furthermore, the embodiments can be combined as appropriate within a consistent range. [Example]

[0015] [Overall configuration] Fig. 1 is a diagram illustrating an example of the overall configuration of a system according to Example 1. As illustrated in Fig. 1, the system includes a store 1, which is an example of a space, a plurality of cameras 2 installed in different locations in the store 1, and an information processing device 10.

[0016] Each of the multiple cameras 2 is an example of a surveillance camera that captures an image of a specific area within the store 1, and transmits the captured image data to the information processing device 100. In the following description, the image data may be referred to as "image data." The image data includes multiple image frames in chronological order. Each image frame is assigned a frame number in ascending chronological order. One image frame is image data of a still image captured by the camera 2 at a certain timing.

[0017] The information processing device 10 is an example of a computer that analyzes each image data captured by each of the multiple cameras 2. Each of the multiple cameras 2 and the information processing device 10 are connected via various networks, such as the Internet or a dedicated line, whether wired or wireless. In addition, regular cash registers, self-checkouts, etc. are installed in the store 1, and store clerks carry terminals such as smartphones.

[0018] In recent years, various stores (especially stores that have introduced self-checkouts) have been using person tracking technology using multiple surveillance cameras installed in the store to analyze purchasing behavior within the store. Figure 2 is a diagram explaining a reference technology for person tracking technology. As shown in Figure 2, the person tracking technology is a technology for tracking the same person that combines a person detection model 50 and a person identification model 60.

[0019] The person detection model 50 detects a person bounding box (Bbox) indicating the location of a person in response to input image data from each camera, and outputs the result as an output. The person identification model 60, in response to input of two person bounding boxes detected from image data from each camera, outputs a determination result as to whether the people are the same person by evaluating the similarity of the feature amounts (feature vectors) of those people.

[0020] However, in actual operation, if the image characteristics of the learning data (training data) used for machine learning (training) of the person identification model differ from the image characteristics of the actual image data captured by each camera 2, the accuracy of the person identification model 60 will decrease. Furthermore, since the installation positions of each camera 2 are different, the camera angle of view, brightness, background, etc. will also differ, and therefore the accuracy of the person identification model 60 will decrease in a situation where the environment of the learning data does not match the environment of actual operation.

[0021] In other words, if there is a mismatch in image characteristics between the learning data for person identification and the actual store being inferred, the distribution of person features will fluctuate, reducing the accuracy of inferring person features and leading to misidentification of people. Such misidentification makes it difficult to track the same person using image data captured by camera 2, making it impossible to accurately analyze purchasing behavior.

[0022] Therefore, in Example 1, since the floor map and camera layout of store 1 can be obtained, overlapping areas of the shooting areas of multiple cameras are used, and learning data for person identification of the store to be inferred is acquired by focusing on the characteristic that person bounding boxes at the same position captured by each camera 2 at the same time represent the same person. The learning data acquired in this way is used to perform machine learning of the person identification model, thereby reducing the impact of image characteristics and suppressing erroneous person identification.

[0023] FIG. 3 is a diagram illustrating the generation of learning data using actual footage of a store 1. As shown in FIG. 3, each camera 2 installed in the store 1 captures images from different positions and in different directions, but some of the captured areas are common (overlapping). For example, image data captured by camera A captures person A and person B, and image data captured by camera B captures person A, person B, and person D, with person A and person B being captured in common by each camera. Therefore, although it is not possible to identify "who" person A and person B are, it is possible to identify that they are the same person. Furthermore, person A and person B are captured from different directions, and are not the same image data.

[0024] That is, by using the video data from the camera 2 in the store, it is possible to collect a plurality of image data of the same person captured from different directions. The information processing device 10 of the first embodiment uses each of such image data of the same person captured from different directions as learning data to perform machine learning of a person identification model.

[0025] FIG. 4 is a diagram illustrating generation of a person identification model used in the person tracking technology according to the first embodiment. As shown in FIG. 4, the information processing device 10 acquires training data in which image data and correct answer data (person labels) are associated with each other from a publicly available dataset or the like. Then, the information processing device 10 inputs the image data into a first machine learning model configured by, for example, a convolutional neural network to acquire an output result, and trains the first machine learning model so that the output result matches the correct answer data. That is, the information processing device 10 generates a first machine learning model by machine learning a multi-class classification problem using training data related to multiple people.

[0026] The information processing device 10 then generates a second machine learning model using the input layer and intermediate layer of the trained first machine learning model and a new output layer. The information processing device 10 also generates training data to which a same-person label (correct answer data) is assigned, using first image data and second image data, which are image data of the same person generated from store image data. The information processing device 10 then inputs the first image data and second image data of the training data generated from the store image data into the second machine learning model to obtain an output result including a determination result of identity, and trains the second machine learning model so that the output result matches the correct answer data. That is, the information processing device 10 generates the second machine learning model by machine learning a two-class classification problem using training data related to a specific person.

[0027] By performing person identification using the second machine learning model generated in this manner, the information processing device 10 learns person features suitable for the store to be inferred, improving the accuracy of person tracking and enabling accurate purchasing behavior analysis.

[0028] [Function Configuration] 5 is a functional block diagram illustrating a functional configuration of the information processing device 10 according to Example 1. As illustrated in FIG.

[0029] The communication unit 11 is a processing unit that controls communication with other devices, and is realized by, for example, a communication interface, etc. For example, the communication unit 11 receives video data from the camera 2 and transmits the processing results by the control unit 20 to a store clerk's terminal, etc.

[0030] The storage unit 12 is a processing unit that stores various data and programs executed by the control unit 20, and is realized by a memory, a hard disk, etc. The storage unit 12 stores a video data DB 13, a public dataset 14, a store dataset 15, a person detection model 16, and a person identification model 17.

[0031] The video data DB 13 is a database that stores video data captured by each of the multiple cameras 2 installed in the store 1. For example, the video data DB 13 stores video data for each camera 2 or for each time period during which the video was captured.

[0032] The public dataset 14 stores training data collected in advance. Specifically, the public dataset 14 stores training data used for machine learning of the person detection model 16 and training data used for machine learning of the person identification model 17 for a multi-class classification problem.

[0033] For example, the learning data used for machine learning of the person detection model 16 is data in which image data containing a person is associated with a person bounding box indicating the location of the person in the image. That is, the image data is an explanatory variable, and the person bounding box is a target variable (correct data).

[0034] Furthermore, training data for multi-class classification problems is data in which person bounding boxes are associated with person labels that indicate who the person is. That is, the person bounding boxes are explanatory variables, and the person labels are target variables (correct answer data).

[0035] The store dataset 15 stores training data used for machine learning of a two-class classification problem of the person identification model 17. Specifically, the store dataset 15 stores training data generated by the control unit 20, which will be described later, using video data from the camera 2 of the store 1. The training data stored here is data in which two person bounding boxes are associated with a same person label indicating whether the people are the same person. In other words, the two person bounding boxes are explanatory variables, and the same person label is a target variable (correct answer data).

[0036] The person detection model 16 is a machine learning model that has an input layer, an intermediate layer, and an output layer, and detects a person bounding box in image data in response to input image data. For example, the person detection model 16 can employ a convolutional neural network.

[0037] The person identification model 17 is a machine learning model that has an input layer, an intermediate layer, and an output layer, and that identifies the person to which the person bounding box corresponds in response to an input of the person bounding box. For example, a convolutional neural network can be adopted for the person identification model 17.

[0038] The control unit 20 is a processing unit that controls the entire information processing device 10, and is realized by, for example, a processor. The control unit 20 has a detection model generation unit 21, a pre-processing unit 22, a data collection unit 23, an identification model generation unit 24, and an inference execution unit 25. The detection model generation unit 21, the pre-processing unit 22, the data collection unit 23, the identification model generation unit 24, and the inference execution unit 25 are realized by electronic circuits included in the processor, processes executed by the processor, etc.

[0039] The detection model generation unit 21 is a processing unit that generates the person detection model 16 through machine learning. Specifically, the detection model generation unit 21 generates the person detection model 16 by updating various parameters such as weights of the person detection model 16 so as to detect a person bounding box from input learning data.

[0040] Fig. 6 is a diagram illustrating the generation of the person detection model 16. As shown in Fig. 6, the detection model generation unit 21 acquires, from the public dataset 14, training data in which input image data is associated with ground truth data in which a person bounding box is specified. Then, the detection model generation unit 21 inputs the image data to the person detection model 16 and acquires an output result of the person detection model 16. Thereafter, the detection model generation unit 21 performs machine learning on the person detection model 16 by error backpropagation or the like so as to reduce the error between the output result and the ground truth data.

[0041] The pre-processing unit 22 has an image acquisition unit 22a and a transformation processing unit 22b, and is a processing unit that executes pre-processing to generate learning data for a two-class classification problem from image data captured in the store 1. That is, the pre-processing unit 22 estimates projective transformation coefficients of the image capture area of ​​each camera 2 with respect to the floor map of the store 1, which is the inference target.

[0042] The video acquisition unit 22a is a processing unit that acquires video data from each camera 2 and stores the data in the video data DB 13. For example, the video acquisition unit 22a may acquire the video data from each camera 2 at any time or periodically.

[0043] The transformation processing unit 22b is a processing unit that estimates projective transformation coefficients for transforming image coordinates, which are coordinates of image data captured by each camera 2 and which differ for each camera 2, into floor map coordinates, which are coordinates of a floor map of the store 1 and which are common to all cameras. Note that, since the camera and floor configurations are generally fixed, it is sufficient to estimate the projective transformation (homography) coefficients only once.

[0044] FIG. 7 is a diagram illustrating calculation of the projection transformation coefficient. As shown in FIG. 7, the transformation processing unit 22b specifies any corresponding points (corresponding points) between the camera image (image coordinate system) and the floor map (floor map coordinate system). For example, the transformation processing unit 22b identifies points (x1, y1), (x2, y2), (x3, y3), and (x4, y4) from the image coordinate system. Similarly, the transformation processing unit 22b identifies points (X1, Y1), (X2, Y2), (X3, Y3), and (X4, Y4) from the floor map coordinate system. Thereafter, the transformation processing unit 22b calculates projection transformation coefficients a from the image coordinate system (x, y) to the floor map coordinate system (X, Y). i (i=1-8) is calculated by solving the simultaneous equations shown in equation (1) in Fig. 7. Note that the corresponding points may be specified by the user, or points at the same location may be identified by image analysis.

[0045] The data collection unit 23 has a detection unit 23a and a learning data generation unit 23b, and is a processing unit that performs person detection and coordinate calculation to generate learning data for a two-class classification problem from the image data of the camera 2.

[0046] The detection unit 23a is a processing unit that detects a person bounding box from image data captured by each camera 2 using a trained person detection model 16. FIG. 8 is a diagram illustrating the detection of a person bounding box. As shown in FIG. 8, the detection unit 23a inputs image data captured by the camera 2 to the person detection model 16, and obtains an output result in which the person bounding box of ID=a, the person bounding box of ID=b, the person bounding box of ID=c, and the person bounding box of ID=d have been detected.

[0047] In this way, the detection unit 23a performs person detection on various image data captured in different directions by each camera 2 installed in different positions, obtains output results including the detected person bounding boxes, and stores them in the memory unit 12, etc.

[0048] The learning data generation unit 23b is a processing unit that calculates floor map coordinates of the person bounding box detected by the detection unit 23a, extracts paired images of the same person, and generates learning data for a two-class classification problem.

[0049] First, the learning data generation unit 23b transforms the person bounding boxes in the image coordinate system detected by the detection unit 23a into the floor map coordinate system using the projective transformation coefficients calculated by the pre-processing unit 22. Fig. 9 is a diagram illustrating the coordinate transformation. As shown in Fig. 9, the learning data generation unit 23b determines the image coordinates (x, y) of the center of the bottom edge of each person bounding box as the person position, and calculates the person position in the floor map coordinates (X, Y).

[0050] For example, the learning data generation unit 23b converts points (x1, y1), (x2, y2), (x3, y3), and (x4, y4) that indicate the position of a person detected in the image coordinate system into points (X1, Y1), (X2, Y2), (X3, Y3), and (X4, Y4) that indicate the position of the person in the floor map coordinate system, using the conversion formula shown in formula (2) in Fig. 9. In this way, the learning data generation unit 23b expresses the person bounding box in the image coordinate system specific to each camera 2 that is captured in the image data of each camera 2, in the floor map coordinate system that is common to each camera.

[0051] Next, the learning data generation unit 23b acquires a data set of paired person bounding box images located at equivalent floor map coordinates between the two cameras. That is, the learning data generation unit 23b extracts pairs of person bounding boxes representing the same person (pairs) using person bounding boxes of multiple image data captured at the same time from the image data of each camera 2.

[0052] FIG. 10 is a diagram illustrating extraction of identical person pairs. As shown in FIG. 10, the learning data generation unit 23b acquires image data A captured by camera A at time t and image data B captured by camera B at the same time, time t. Then, the learning data generation unit 23b converts the person bounding box in the image coordinate system detected from the image data A of camera A into a person bounding box in the floor map coordinate system using equation (2) in FIG. 9. Similarly, the learning data generation unit 23b converts the person bounding box in the image coordinate system detected from the image data B of camera B into a person bounding box in the floor map coordinate system using equation (2) in FIG. 9.

[0053] Then, the learning data generating unit 23b calculates the range of floor map coordinates where the imaging ranges of the cameras overlap. For example, as shown in FIG. 10, the imaging range of camera A is A in or X A out The range is Y and the Y axis is Y A in or Y A out Within this range, the position of the person is (X A a ,Y A a ) and (X A b ,Y A b ) is detected. The imaging range of camera B is X axis. B in or X B out The range is Y and the Y axis is Y B in or YB out Within this range, the position of the person is (X B a ,Y B a ), (X B b ,Y B b ), (X B c ,Y B c ), (X B d ,Y B d ) is detected. As described above, each person position is the image coordinate of the center of the bottom edge of the detected person's bounding box.

[0054] Here, the learning data generating unit 23b calculates the floor map coordinate range (X A ,Y A ) and the range of floor map coordinates of camera B (X B ,Y B ) overlapping range (X AB ,Y AB ) is calculated. As shown in Equation 3 of Figure 10, X AB The range of "X A in or X B in " and "X A out Or X B out " is less than or equal to the smallest value of Y AB The range of "Y A in or Y B in " and "Y A out Or Y B out " is less than or equal to the smallest value of ".

[0055] Next, the learning data generating unit 23b calculates the overlapping range (X AB ,Y AB) for a group of people captured by each camera, pairs of people at similar positions are extracted. Specifically, the learning data generation unit 23b extracts combinations of nearby pairs using a method such as minimum weighted matching based on Euclidean distance, and among the nearby pairs, pairs whose Euclidean distance is smaller than a predetermined threshold are considered to be identical person pairs. In this case, the learning data generation unit 23b will acquire a large amount of nearly identical pair data if it extracts data for every frame, so it can also thin out the data by sampling.

[0056] In the example of FIG. 10, the learning data generating unit 23b generates a person Aa (X A a ,Y A a ) and person Ab(X A b ,Y A b ) person is detected, and person Ba(X B a ,Y B a ) and person Bd(X B d ,Y B d ) has been detected. Then, the learning data generation unit 23b A a ,Y A a ) and person Ba(X B a ,Y B a ) and the Euclidean distance of person Aa(X A a ,Y A a ) and person Bd(X B d ,Y B d Similarly, the learning data generation unit 23b calculates the Euclidean distance of person Ab(X A b ,Y A b ) and person Ba(X B a ,Y Ba ) and the Euclidean distance of person Ab(X A b ,Y A b ) and person Bd(X B d ,Y B d ) to calculate the Euclidean distance.

[0057] Thereafter, the learning data generation unit 23b selects person Aa (X A a ,Y A a ) and person Ba(X B a ,Y B a ), person Ab(X A b ,Y A b ) and person Bd(X B d ,Y B d ) to extract each pair.

[0058] In this way, the learning data generation unit 23b extracts pairs of people (person bounding boxes) included in the image data captured by each camera at the same time that represent the same person, and generates learning data for a two-class classification problem.

[0059] 11 is a diagram illustrating the generation of training data. As shown in FIG. 11, the training data generation unit 23b generates training data in which the person bounding boxes corresponding to the positions of the people extracted as the same person pair are used as explanatory variables, and the labels indicating that the person bounding boxes are the same person (same person=0 or not the same person=1) are used as objective variables, and stores the training data in the store dataset 15.

[0060] In the example of FIG. 11, the learning data generating unit 23b generates the learning data of a person Aa (X A a ,Y A a) is the bounding box of the person Ba(X B a ,Y B a ) as the second image data, and the objective variable is the same person label (same person=0) indicating that person Aa and person Ba are the same person.

[0061] That is, the training data generation unit 23b uses person bounding boxes of the same person captured at the same time but from different directions in the store 1 to be inferred as training data for a two-class classification problem. The correct answer information (label) of the training data generated here is not a person label indicating an individual person, such as which person it is, but a same person label indicating whether they are the same person or not. Note that even for pairs determined to be different people, if the error between the predefined threshold and the Euclidean distance is equal to or less than a second threshold and they are determined to be somewhat similar, a label of different people can be added to the training data. This makes it possible to train pairs of confusable person bounding boxes with small errors to be determined to be different people.

[0062] 5, the identification model generation unit 24 has a first machine learning unit 24a and a second machine learning unit 24b, and is a processing unit that executes machine learning of the person identification model 17. Specifically, the identification model generation unit 24 executes machine learning of the person identification model 17 by using a combination of a multi-class classification problem and a two-class classification problem.

[0063] The first machine learning unit 24a performs machine learning on a multi-class classification problem using the public dataset 14 to generate a first machine learning model. FIG. 12 is a diagram illustrating the generation of a person identification model 17. As shown in FIG. 12, the first machine learning unit 24a generates a first machine learning model by machine learning on a multi-class classification problem in response to input of each piece of training data in which the same person is depicted in different ways, and identifies the person depicted in each piece of input training data. The first machine learning model is composed of a convolutional neural network including an input layer and an intermediate layer, and an output layer.

[0064] For example, the first machine learning unit 24a inputs various person bounding boxes of person A included in the public dataset 14 into a convolutional neural network and acquires each identification result (output result) from the output layer. Then, the first machine learning unit 24a updates the parameters of the convolutional neural network and the output layer so as to reduce the error between each identification result and the person label (person A), in other words, so as to identify person A.

[0065] Similarly, the first machine learning unit 24a inputs various person bounding boxes of person B included in the public dataset 14 into the convolutional neural network and acquires each classification result from the output layer. Then, the first machine learning unit 24a updates the parameters of the convolutional neural network and the output layer so as to reduce the error between each classification result and the person label (person B).

[0066] Once machine learning using the public dataset is completed, the second machine learning unit 24b performs machine learning using a two-class classification problem using the store dataset 15 to generate a person identification model 17, which is an example of a second machine learning model.

[0067] Specifically, the second machine learning unit 24b uses a convolutional neural network including an input layer and an intermediate layer of the trained first machine learning model and a new untrained output layer to construct the person identification model 17. Then, the second machine learning unit 24b generates the person identification model 17 by machine learning using the training data stored in the store dataset to identify binary labels in which 0 indicates the same person and 1 indicates a different person.

[0068] 11, the second machine learning unit 24b inputs each person bounding box of a pair extracted as a positive example (same person) to a convolutional neural network and obtains a classification result (output result) from the output layer. Then, the second machine learning unit 24b updates the parameters of the convolutional neural network and the output layer so as to reduce the error between each classification result and the same person label (same person=0), in other words, so as to classify the pairs as the same person.

[0069] The second machine learning unit 24b inputs a pair of one person bounding box included in the pair extracted as a positive example (same person) and a randomly extracted person bounding box of another person into the convolutional neural network, and acquires a classification result from the output layer. The second machine learning unit 24b then updates the parameters of the convolutional neural network and the output layer so as to reduce the error between each classification result and the same person label (non-same person=1), in other words, so as to classify the people as non-same people.

[0070] In this way, the identification model generation unit 24 generates a first machine learning model that performs multi-class classification, and generates a person identification model 17 that performs two-class classification, using a convolutional neural network of the first machine learning model.

[0071] 5, the inference execution unit 25 is a processing unit that uses the person identification model 17 generated by the identification model generation unit 24 to identify people appearing in each piece of image data captured by the cameras 2 in the brick-and-mortar store. That is, the inference execution unit 25 uses the person identification model 17 to link people in the image data captured by each camera 2.

[0072] 13 is a diagram illustrating the inference processing. As shown in FIG. 13, the inference execution unit 25 inputs each image data captured by each camera 2 in the store into the trained person detection model 16 and obtains an output result including a detected person bounding box. For example, the inference execution unit 25 obtains a person bounding box for "ID=xx" and a person bounding box for "ID=yy" that are included in different output results.

[0073] Then, the inference execution unit 25 inputs the person bounding box of "ID=xx" into the person identification model 17, and acquires person features from the layer immediately before the output layer of the person identification model 17. Similarly, the inference execution unit 25 inputs the person bounding box of "ID=yy" into the person identification model 17, and acquires person features from the layer immediately before the output layer of the person identification model 17.

[0074] The inference execution unit 25 then calculates the similarity of each feature, and if the similarity is high, infers that the person bounding box for "ID=xx" and the person bounding box for "ID=yy" are the same person. On the other hand, if the similarity of each feature is low, the inference execution unit 25 infers that the person bounding box for "ID=xx" and the person bounding box for "ID=yy" are not the same person.

[0075] For example, the inference execution unit 25 calculates the similarity of each feature, such as the Euclidean distance or cosine similarity of each feature, or the squared error of the elements of each feature, and if the calculated similarity is greater than or equal to a threshold, it infers that the people are the same person.

[0076] By tracking the bounding boxes of each person inferred to belong to the same person in this way, it is possible to analyze that person's behavior in the store and the items they purchase.

[0077] [Processing flow] Next, the processes executed by the above-mentioned processing units will be described. Here, the pre-processing, data collection process, machine learning process, and inference process will be described.

[0078] (Pre-processing) Fig. 14 is a flowchart showing the flow of the pre-processing. As shown in Fig. 14, the pre-processing unit 22 acquires video data from each camera 2 (S101), and acquires a floor map of the store that has been designed in advance (S102).

[0079] Then, the pre-processing unit 22 identifies corresponding points, which are any points that correspond to each other, between the image data of the camera 2 and the floor map (S103), and estimates projective transformation coefficients using equation (1) in FIG. 7 (S104).

[0080] (Data collection and processing) Fig. 15 is a flowchart showing the flow of the data collection process. As shown in Fig. 15, the data collection unit 23 acquires the video data of each camera 2 from the video data DB 13 (S201), and acquires the projective transformation coefficients estimated by the pre-processing unit 22 (S202).

[0081] Next, the data collection unit 23 executes human detection by inputting each image data in the video data of each camera 2 into the human detection model 16 (S203), and detects a human bounding box (S204).

[0082] Then, the data collecting unit 23 calculates floor map coordinates of the person bounding box of each person using the projective transformation coefficients (S205). That is, the data collecting unit 23 converts the image coordinate system of the person bounding box of each person into floor map coordinates.

[0083] Thereafter, the data collecting unit 23 calculates an overlapping area in the floor map coordinate system for the image data of the two cameras (S206). Then, the data collecting unit 23 extracts person pairs at equivalent positions from the image data captured at the same time by the two cameras (S207). The extracted person pairs and the same person labels are generated as learning data.

[0084] (machine learning processing) Fig. 16 is a flowchart showing the flow of machine learning processing of a person identification model. As shown in Fig. 16, the identification model generation unit 24 acquires existing training data stored in advance in the public dataset 14 (S301), and executes machine learning of the first machine learning model as a multi-class classification problem using the existing training data (S302).

[0085] Next, the identification model generation unit 24 acquires learning data for the target store generated using image data of the store stored in the store dataset 15 (S303), and performs machine learning of the person identification model 17 as a two-class classification problem using the learning data for the target store (S304).

[0086] (inference processing) Fig. 17 is a flowchart showing the flow of the inference process. As shown in Fig. 17, the inference execution unit 25 acquires each image data from each camera 2 (S401), inputs each image data to the person detection model 16, and detects a person bounding box (S402).

[0087] Then, the inference execution unit 25 inputs the two person bounding boxes to the person identification model 17 (S403), and acquires the feature amounts of each person bounding box from the layer immediately before (one layer before) the output layer of the person identification model 17 (S404). After that, the inference execution unit 25 calculates the similarity between the feature amounts of each person bounding box and executes person identification (S405).

[0088] [effect] As described above, the information processing device 10 can acquire learning data for person identification of the inference target store by focusing on the characteristic that person bounding boxes at the same position captured by each camera 2 at the same time represent the same person. Here, the information processing device 10 learns using insufficient label information (same person label) that cannot be used in the reference technology, as the learning data acquired in Example 1 does not include person labels. Therefore, the information processing device 10 can automatically acquire learning data to be analyzed, and can continuously improve the accuracy of person identification.

[0089] Furthermore, although a two-class classification problem has a smaller amount of label information than a multi-class classification problem, the method according to the first embodiment can automatically acquire a large amount of identical person pair data that contributes to improving accuracy by utilizing the overlapping area of ​​the camera 2. Therefore, the information processing device 10 can eliminate the limitation on the amount of label information by the amount of data.

[0090] Fig. 18 is a diagram for explaining the effect of Example 1. Fig. 18 shows a comparison of the inference accuracy of person identification between the reference technology and the technology according to Example 1 (proposed technology). Here, dataset A and dataset B with different person image characteristics (season, background, etc.) were prepared, and learning was performed using dataset A and inference was performed using dataset B. Note that in the method according to Example 1, dataset B was also used for learning (however, only for the same person label).

[0091] As shown in FIG. 18, when comparing inference accuracy based on cumulative matching characteristics, which is the rate at which the same person is identified within a certain rank among a large amount of person data, the reference technology has sufficient inference accuracy for the same data set, but does not achieve sufficient inference accuracy for different data sets due to different image characteristics. On the other hand, the method according to Example 1 can incorporate the image characteristics of the inference data into the learning model, thereby improving inference accuracy. For example, when comparing the top-ranked precision, the reference technology has a precision of "0.437," while Example 1 has improved to "0.603." Furthermore, when comparing the top 10 precision, the reference technology has a precision of "0.693," while Example 1 has improved to "0.842."

[0092] In this way, the information processing device 10 learns person feature amounts suitable for the store to be inferred, improving person tracking accuracy and enabling highly accurate purchasing behavior analysis. The information processing device 10 can track shopping spree behavior, suspicious behavior, etc. by accurately identifying people from multiple surveillance cameras in the store. The information processing device 10 can acquire and learn person identification data for the store to be inferred from overlap information on the shooting areas of multiple cameras. [Example]

[0093] Although the embodiments of the present invention have been described above, the present invention may be embodied in various different forms other than the above-described embodiments.

[0094] [Numbers, etc.] The number of cameras, numerical examples, learning data examples, machine learning models, coordinate examples, and the like used in the above embodiments are merely examples and can be changed as desired. The process flow described in each flowchart can also be changed as appropriate within a consistent range. Each model can be generated using various algorithms, such as a neural network. In the above embodiment, the second machine learning unit 24b configures the person identification model 17 using a convolutional neural network including an input layer and an intermediate layer of a trained first machine learning model and a new untrained output layer. However, this is not limited to this example, and the person identification model 17 can also be configured using some layers of the first machine learning model. In this case, it is preferable to exclude the output layer of the first machine learning model.

[0095] Furthermore, the coordinate transformation can be performed on an image data basis or on a person bounding box basis. The person bounding box is an example of person data, and the person detection model is an example of a third machine learning model. The image coordinate system is an example of a first index and a first coordinate system, and the floor map coordinate system is an example of a second index and a second coordinate system. Furthermore, the image data of the floor map landmark is an example of transformed image data.

[0096] [system] The information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings may be changed arbitrarily unless otherwise specified.

[0097] Furthermore, the specific form of distribution or integration of the components of each device is not limited to that shown in the figure. For example, the pre-processing unit 22 and the data collection unit 23 may be integrated. In other words, all or some of the components may be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions of each device may be realized by a CPU and a program analyzed and executed by the CPU, or may be realized as hardware using wired logic.

[0098] [Hardware] Fig. 19 is a diagram illustrating an example of a hardware configuration. As shown in Fig. 19, an information processing device 10 includes a communication device 10a, an HDD (Hard Disk Drive) 10b, a memory 10c, and a processor 10d. The components shown in Fig. 19 are connected to each other via a bus or the like.

[0099] The communication device 10a is a network interface card or the like, and communicates with other devices. The HDD 10b stores programs and DBs that operate the functions shown in FIG.

[0100] The processor 10d reads out from the HDD 10b or the like a program that executes the same processes as the respective processing units shown in Fig. 5 and loads it into the memory 10c, thereby operating a process that executes each function described in Fig. 5 or the like. For example, this process executes the same functions as the respective processing units of the information processing device 10. Specifically, the processor 10d reads out from the HDD 10b or the like a program that has the same functions as the detection model generation unit 21, the pre-processing unit 22, the data collection unit 23, the identification model generation unit 24, the inference execution unit 25, and the like. Then, the processor 10d executes a process that executes the same processes as the detection model generation unit 21, the pre-processing unit 22, the data collection unit 23, the identification model generation unit 24, the inference execution unit 25, and the like.

[0101] In this way, the information processing device 10 operates as an information processing device that executes an information processing method by reading and executing a program. The information processing device 10 can also realize functions similar to those of the above-described embodiment by reading the program from a recording medium using a medium reading device and executing the read program. Note that the program in these other embodiments is not limited to being executed by the information processing device 10. For example, the above-described embodiment may also be applied in the same way to cases where another computer or server executes the program, or where these execute the program in cooperation with each other.

[0102] This program may be distributed via a network such as the Internet. Alternatively, this program may be recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, a magneto-optical disk (MO), or a digital versatile disk (DVD), and may be read out from the recording medium and executed by a computer. [Explanation of symbols]

[0103] 1 store 2 Cameras 10. Information processing equipment 11 Communications Department 12 Storage section 13 Video Data DB 14 Public Datasets 15 store dataset 16 Person Detection Model 17 Person Identification Model 20 Control Unit 21 Detection model generation unit 22 Pre-processing section 22a Image acquisition unit 22b Conversion processing section 23 Data Collection Department 23a Detector 23b Training data generation unit 24 Identification model generation unit 24a Machine Learning Department 1 24b Second Machine Learning Department 25 Inference execution unit

Claims

1. On the computer, generating a plurality of first training data sets in which image data and ground truth data indicating the identity of people appearing in the image data are associated with each other from a publicly available public dataset; inputting the plurality of first learning data into a first machine learning model, and executing machine learning of the first machine learning model that performs multi-class classification to identify people appearing in image data; extracting a partial layer of the first machine learning model generated by the machine learning; generating a new, untrained second machine learning model using the extracted trained part of the layers; generating a plurality of second learning data sets from an image dataset captured by a plurality of cameras installed in the store, in which pairs of image data showing the same person are associated with correct answer data indicating whether or not the images show the same person; inputting the plurality of second learning data into the second machine learning model, and executing machine learning of the second machine learning model that performs two-class classification to identify whether or not individuals included in each of the plurality of second learning data are the same individual; A machine learning program characterized by executing processing.

2. The process to be performed is:

2. The machine learning program according to claim 1, wherein the first machine learning model is generated by machine learning a multi-class classification problem that identifies people depicted in the input first training data in which the same person is depicted in different ways.

3. The process to be performed is:

3. The machine learning program according to claim 2, wherein the second machine learning model is generated by machine learning a two-class classification problem that identifies whether people appearing in the plurality of second learning data are the same person or not, in response to input of the plurality of second learning data.

4. inputting person data relating to a plurality of people identified based on a plurality of image data captured at the same time by a plurality of cameras installed in a predetermined space into the second machine learning model that has already been trained, and acquiring each feature amount from the layer immediately preceding the output layer of the second machine learning model; linking each person depicted in each of the person data based on each of the feature amounts corresponding to each of the person data; 4. The machine learning program according to claim 1, wherein the program causes the computer to execute a process.

5. The acquiring process includes: inputting the plurality of image data into a third machine learning model that is generated by machine learning using learning data related to people so as to extract area information indicating the location of the people, and acquiring area information for each of the plurality of people; The linking process includes: inputting the domain information into the second machine learning model that has already been trained, and acquiring each feature amount from a layer immediately before the output layer of the second machine learning model; The machine learning program according to claim 4 , wherein the people are linked based on the feature amounts acquired based on the area information corresponding to each person.

6. the extracting process extracts an input layer and an intermediate layer of the first machine learning model generated by the machine learning; 6. The machine learning program according to claim 1, wherein the generating process generates the second machine learning model composed of the extracted input layer and intermediate layer and a new output layer.

7. The computer generating a plurality of first training data sets in which image data and ground truth data indicating the identity of people appearing in the image data are associated with each other from a publicly available public dataset; inputting the plurality of first learning data into a first machine learning model, and executing machine learning of the first machine learning model that performs multi-class classification to identify people appearing in image data; extracting a partial layer of the first machine learning model generated by the machine learning; generating a new, untrained second machine learning model using the extracted trained part of the layers; generating a plurality of second learning data sets from an image dataset captured by a plurality of cameras installed in the store, in which pairs of image data showing the same person are associated with correct answer data indicating whether or not the images show the same person; inputting the plurality of second learning data into the second machine learning model, and executing machine learning of the second machine learning model that performs two-class classification to identify whether or not individuals included in each of the plurality of second learning data are the same individual; A machine learning method characterized by performing processing.

8. A method for generating a plurality of first learning data sets in which image data and ground truth data indicating the identity of a person appearing in the image data are associated with each other from a publicly available public dataset; inputting the plurality of first learning data into a first machine learning model, and executing machine learning of the first machine learning model that performs multi-class classification to identify people appearing in image data; extracting a partial layer of the first machine learning model generated by the machine learning; generating a new, untrained second machine learning model using the extracted trained part of the layers; generating a plurality of second learning data sets from an image dataset captured by a plurality of cameras installed in the store, in which pairs of image data showing the same person are associated with correct answer data indicating whether or not the images show the same person; inputting the plurality of second learning data into the second machine learning model, and executing machine learning of the second machine learning model that performs two-class classification to identify whether or not individuals included in each of the plurality of second learning data are the same individual; An information processing device comprising a control unit.

Images