Information processing program, information processing method, and information processing device

Abstract

The present invention improves accuracy of locus data.　In the present invention, a computer acquires: commodity data that indicates the display position of each of a plurality of commodities in a store; a plurality of locus data items that correspond to a plurality of persons and that respectively indicate movement paths in the store inferred from image data from a camera; and a plurality of settlement data items that correspond to the plurality of persons and that respectively indicate commodities purchased in the store from among the plurality of commodities. The computer specifies a commodity indicated by first settlement data of a person corresponding to first locus data, and searches for second locus data indicating the locus passing from the display position of the specified commodity through a fixed range . The computer corrects the first locus data by using the second locus data.

Description

Information processing program, information processing method, and information processing device

[0001] The present invention relates to an information processing program, an information processing method, and an information processing device.

[0002] Stores that display and sell a variety of products often need to understand the paths people take within the store. Analyzing paths can be useful for designing a store to facilitate smooth movement of people and for determining product placement to promote product sales. An information processing device can analyze image data from cameras installed in the store to detect people from the image data and generate trajectory data that indicates the paths people take.

[0003] For example, a surveillance system has been proposed that analyzes image data from multiple surveillance cameras installed within a store to identify the area of ​​the store through which a person has passed. The proposed surveillance system detects people from the image data of each surveillance camera and determines the appearance characteristics of the detected people. The surveillance system compares the appearance characteristics between the multiple surveillance cameras and considers people with similar appearance characteristics to be the same person, thereby identifying the store area to which the person has passed.

[0004] At this time, the monitoring system receives purchase history data from a POS (Point of Sales) terminal installed in the store and associates the person detected from image data of the area around the POS terminal with the purchase history data. If there is a missing period in which the store area where the person was located is unknown, the monitoring system identifies the product shelf where the product indicated by the purchase history data is displayed and presumes that the person detected from image data of the area around the identified product shelf is the same person.

[0005] International Publication No. 2017 / 163283

[0006] As described above, an information processing device may determine the identity of a person based on the appearance characteristics of the person captured in image data. On the other hand, multiple people with similar appearance characteristics may be present in a store at the same time. In this case, the information processing device may erroneously combine the movement paths of different people, generating erroneous trajectory data. Therefore, in one aspect, the present invention aims to improve the accuracy of trajectory data.

[0007] In one aspect, an information processing program is provided that causes a computer to execute the following process. The computer acquires product data indicating the display positions of multiple products in a store, multiple trajectory data corresponding to multiple people, each indicating a movement path within the store estimated from camera image data, and multiple payment data corresponding to the multiple people, each indicating a product purchased from the multiple products at the store. The computer identifies, from the multiple payment data, a product indicated by first payment data of a person corresponding to first trajectory data in the multiple payment data, and searches, from the multiple trajectory data, for second trajectory data that passes within a certain range from the display position of the identified product. The computer uses the second trajectory data to modify the first trajectory data.

[0008] In one aspect, there is provided an information processing method executed by a computer, and an information processing device having a storage unit and a processing unit.

[0009] In one aspect, the accuracy of the trajectory data is improved. These and other objects, features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings illustrating preferred embodiments of the present invention by way of example.

[0010] 1 is a diagram for explaining an information processing apparatus according to a first embodiment; FIG. 2 is a diagram illustrating an example of an information processing system according to a second embodiment; FIG. 3 is a block diagram illustrating an example of hardware of an image analyzing apparatus; FIG. 4 is a diagram illustrating an example of generating a trajectory from image data of a plurality of surveillance cameras; FIG. 5 is a diagram illustrating an example of erroneous recognition of a truck included in a trajectory; FIG. 6 is a diagram illustrating an example of a store map; FIG. 7 is a diagram illustrating an example of a trajectory table; FIG. 8 is a diagram illustrating an example of a POS data table; FIG. 9 is a diagram illustrating an example of a product table; FIG. 10 is a diagram illustrating an example of truck selection based on POS data; FIG. 11 is a diagram illustrating an example of calculation of a score for a trajectory; FIG. 12 is a diagram illustrating an example of a matching table; FIG. 13 is a diagram illustrating an example of a count of candidate purchase items viewed from a truck; FIG. 14 is a diagram illustrating an example of trajectory correction; FIG. 15 is a diagram illustrating an example of complementing a gap period; FIG. 16 is a block diagram illustrating an example of functions of an image analyzing apparatus; FIG. 17 is a flowchart illustrating an example of a trajectory generation procedure; FIG. 18 is a flowchart illustrating an example of a trajectory generation procedure (continued).

[0011] The present embodiment will be described below with reference to the drawings. [First Embodiment] The first embodiment will be described.

[0012] FIG. 1 is a diagram illustrating an information processing device according to a first embodiment. The information processing device 10 according to the first embodiment analyzes trajectory data indicating a movement path within a store estimated from camera image data, and corrects the trajectory data that may contain errors. The original trajectory data may be generated by the information processing device 10 or by another information processing device. The information processing device 10 may be a client device or a server device. The information processing device 10 may also be called a computer or an image analysis device.

[0013] The information processing device 10 includes a storage unit 11 and a processing unit 12. The storage unit 11 may be a volatile semiconductor memory such as a random access memory (RAM), or may be a non-volatile storage such as a hard disk drive (HDD) or a flash memory.

[0014] The processing unit 12 is, for example, a processor such as a central processing unit (CPU), a graphics processing unit (GPU), or a digital signal processor (DSP). However, the processing unit 12 may also include an electronic circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The processor executes a program stored in a memory such as a RAM (which may be the storage unit 11). A set of processors may be called a multiprocessor or simply a "processor."

[0015] The memory unit 11 stores product data 13, multiple pieces of trajectory data including trajectory data 14a and 14b, and multiple pieces of payment data including payment data 15a and 15b. The product data 13 indicates the display position of each of multiple products in a store. The store is equipped with display shelves, and each product has a specific display position. Examples of stores include retail stores such as supermarkets, convenience stores, and home improvement centers. A shopper enters a store, moves around the store on foot, etc., and selects products to purchase. The shopper passes a payment position in the store, pays for the purchased products, and leaves the store. A POS terminal, for example, is installed at the payment position. The display position is specified, for example, using coordinates on a planar map of the store viewed from above.

[0016] The multiple trajectory data correspond to multiple shoppers. Each trajectory data indicates the movement path of a person within a store and is estimated from image data from a camera. For example, the movement path tracks the person's location from after entering the store until before payment, and the trajectory data includes multiple points, each of which corresponds to a coordinate on a flat map and a time. The camera is, for example, a surveillance camera installed in the store. Multiple surveillance cameras may be installed in the store.

[0017] For example, the information processing device 10 detects people from each image frame included in the image data and generates feature information indicating the characteristics of the detected people. An image recognition model, which is a trained machine learning model, may be used to detect people. The image recognition model may be a neural network such as a convolutional neural network. The image recognition model may extract a bounding box, which is a rectangular area in which a person appears, from the image frame. The feature information indicates the person's external characteristics, such as clothing, age, and physique. The feature information may use a class determined by the machine learning model or a numerical vector calculated when detecting a person.

[0018] For example, the information processing device 10 compares the positions in the images where people are detected and the generated feature information between adjacent image frames. The information processing device 10 infers that people with similar positions and feature information in the images are the same person. The information processing device 10 also converts the coordinates in the images where people are detected into coordinates on a planar map. The coordinate conversion method is determined in advance, taking into account, for example, the positions of cameras in the store and the imaging characteristics of the cameras.

[0019] For example, the information processing device 10 adds the capture time of the image frame to the coordinates on the planar map. The information processing device 10 associates multiple coordinates of people who are estimated to be the same person. When multiple cameras are used, the information processing device 10 associates the coordinates of people detected from image data from different cameras based on the similarity of feature information and time. This generates trajectory data. However, trajectory data may also be generated by another information processing device.

[0020] As described above, a person's movement path is tracked based on feature information indicating their appearance. However, when multiple people with similar appearances are in a store at the same time, it can be difficult to accurately distinguish between them. As a result, erroneous trajectory data that mixes the movement paths of different people can be generated.

[0021] The multiple payment data correspond to multiple shoppers. Each payment data indicates a purchased item among multiple displayed items. The payment data is generated, for example, by a POS terminal installed in the store. Each payment data is associated with one of the trajectory data.

[0022] The information processing device 10 may associate the payment data with the trajectory data by referencing image data of the area around the payment location. The information processing device 10 may also associate the payment data with the trajectory data based on the similarity between the time of the last point included in the trajectory data and the time of payment. For example, payment data 15a indicates the purchase of product A and is associated with trajectory data 14a. Payment data 15b indicates the purchase of product B and is associated with trajectory data 14b.

[0023] The processing unit 12 determines the possibility that at least one of the plurality of trajectory data has been confused with a different person having similar appearance characteristics by referring to the payment data. If there is a high possibility that a different person has been confused, the processing unit 12 corrects the trajectory data using another trajectory data. Here, we consider the case where trajectory data 14a is to be corrected.

[0024] The processing unit 12 identifies the purchased product indicated by the payment data 15a corresponding to the trajectory data 14a. In the example of FIG. 1, the purchased product is product A. The processing unit 12 identifies the display position of product A by referring to the product data 13. The processing unit 12 searches for trajectory data that passes through a certain range (for example, within one meter) from the identified display position among the multiple pieces of trajectory data that have been generated. Trajectory data that passes through a certain range is, for example, trajectory data that includes points whose distance from the display position is equal to or less than a threshold. In the example of FIG. 1, trajectory data 14b is the corresponding trajectory data.

[0025] When there are multiple pieces of trajectory data passing through a certain range, the processing unit 12 may select, from the trajectory data, the trajectory data whose feature information is most similar to that of the trajectory data 14a. Furthermore, the processing unit 12 may select, from among the trajectory data whose feature information similarity exceeds a threshold, the trajectory data that is closest to the display position of the product A. The feature information of the trajectory data 14a may be generated from image data that is closest to the payment position. Furthermore, the processing unit 12 may select trajectory data within a certain time period from the time of another section of the trajectory data 14a.

[0026] The processing unit 12 corrects the trajectory data 14a using the trajectory data 14b. For example, the processing unit 12 replaces the movement path of a corresponding section included in the trajectory data 14a with the movement path of a section that passes near the product A among the sections included in the trajectory data 14b.

[0027] In the example of FIG. 1 , the movement paths indicated by the trajectory data 14a and 14b intersect. The display position of product A is in a direction closer to the payment position from the intersection position. Although payment data 15a indicates product A, the trajectory data 14a does not pass near product A, while the trajectory data 14b passes near product A. If the external features of the two people corresponding to the trajectory data 14a and 14b are similar, there is a high possibility that tracking of the person at the intersection position has failed, resulting in the latter half of the movement paths of the trajectory data 14a and 14b being swapped. In this case, the processing unit 12 modifies the trajectory data 14a and 14b so that the latter half of the movement paths are swapped.

[0028] As described above, the information processing device 10 of the first embodiment acquires product data 13, multiple pieces of trajectory data corresponding to multiple people, and multiple pieces of payment data corresponding to multiple people. The product data 13 indicates the display positions of multiple products in a store. Each piece of trajectory data indicates a movement path within the store estimated from camera image data. Each piece of payment data indicates a product purchased at the store. The information processing device 10 identifies the product indicated by the payment data 15a of the person corresponding to the trajectory data 14a, searches for trajectory data 14b that passes within a certain range from the display position of the identified product, and modifies the trajectory data 14a using the trajectory data 14b.

[0029] This allows the camera image data to be used to understand the movement paths of people within a store, providing information useful for store design and product placement decisions.In addition, erroneous trajectory data that connects the movement paths of multiple people with similar appearances can be corrected, improving the accuracy of the trajectory data.

[0030] The information processing device 10 may replace the movement route of a first section corresponding to the second section included in the trajectory data 14a with the movement route of a second section that passes through a certain range among the sections included in the trajectory data 14b, thereby reproducing the movement route of the person corresponding to the trajectory data 14a when correctly tracking the person.

[0031] Furthermore, time information may be added to each of the trajectory data and the payment data. Information processing device 10 may identify multiple candidates for the person's trajectory data corresponding to payment data 15a based on the time information, and may calculate an evaluation value for each candidate based on the distance between the product display position indicated by payment data 15a and each candidate. Information processing device 10 may select one of the multiple candidates as trajectory data 14a based on the evaluation value.

[0032] This allows for correlation between trajectory data 14a and payment data 15a, even in cases where it is difficult to correlate trajectory data and payment data one-to-one using time information alone due to multiple people around the payment location, and therefore products likely to have been purchased by the person corresponding to trajectory data 14a can be identified with high accuracy.

[0033] Furthermore, the information processing device 10 may determine whether there are overlaps between the purchased items in the payment data 15a and other payment data within a certain time period after the payment data 15a, and may further use the overlaps to select one of multiple candidates. This improves the accuracy of identifying the purchased items of the person corresponding to the trajectory data 14a.

[0034] Second Embodiment Next, a second embodiment will be described. Fig. 2 is a diagram illustrating an example of an information processing system according to the second embodiment.

[0035] The information processing system of the second embodiment includes one or more POS terminals including POS terminal 32, multiple surveillance cameras including surveillance cameras 33, 34, and 35, and image analysis device 100. The one or more POS terminals and multiple surveillance cameras are installed in store 31. Store 31 is a retail store such as a supermarket, convenience store, or home improvement center. Store 31 sells multiple different types of products, which are displayed on shelves. The multiple surveillance cameras capture images of different sales areas.

[0036] One or more POS terminals, multiple surveillance cameras, and image analysis device 100 are connected to network 30. Network 30 may include a wide area communication network such as the Internet, or may include a local area network (LAN). POS terminal 32 generates POS data indicating shopper payments and transmits the POS data to image analysis device 100. Surveillance cameras 33, 34, and 35 each capture an image of the sales floor area and generate image data, which are then transmitted to image analysis device 100.

[0037] The image analyzing device 100 is a computer that analyzes image data and POS data received from the store 31 and estimates the trajectory of shoppers' movements within the store. The image analyzing device 100 may be a client device or a server device. The image analyzing device 100 corresponds to the information processing device 10 of the first embodiment.

[0038] 3 is a block diagram showing an example of hardware of an image analysis device. The image analysis device 100 has a CPU 101, RAM 102, HDD 103, GPU 104, input interface 105, media reader 106, and communication interface 107, all connected via a bus. The CPU 101 corresponds to the processing unit 12 in the first embodiment. The RAM 102 or HDD 103 corresponds to the storage unit 11 in the first embodiment.

[0039] The CPU 101 is a processor that executes program instructions. The CPU 101 loads programs and data stored in the HDD 103 into the RAM 102 and executes the programs. The image analysis device 100 may have multiple processors.

[0040] RAM 102 is a volatile semiconductor memory that temporarily stores programs executed by CPU 101 and data used in calculations by CPU 101. Image analysis device 100 may also include a type of volatile memory other than RAM.

[0041] The HDD 103 is a non-volatile storage that stores software programs such as an operating system (OS), middleware, and application software, as well as data. The image analysis device 100 may also include other types of non-volatile storage, such as flash memory or a solid-state drive (SSD).

[0042] The GPU 104 performs image processing in cooperation with the CPU 101 and outputs the image to a display device 111 connected to the image analysis device 100. The display device 111 is, for example, a CRT (Cathode Ray Tube) display, a liquid crystal display, an organic EL (Electro Luminescence) display, or a projector. Other types of output devices, such as a printer, may also be connected to the image analysis device 100.

[0043] Furthermore, the GPU 104 may be used as a general purpose computing on graphics processing unit (GPGPU). The GPU 104 may execute a program in response to an instruction from the CPU 101. The image analysis device 100 may include a volatile semiconductor memory other than the RAM 102 as a GPU memory.

[0044] The input interface 105 receives an input signal from an input device 112 connected to the image analysis apparatus 100. The input device 112 is, for example, a mouse, a touch panel, or a keyboard. A plurality of input devices may be connected to the image analysis apparatus 100.

[0045] The medium reader 106 is a reading device that reads programs and data recorded on the recording medium 113. The recording medium 113 is, for example, a magnetic disk, an optical disk, or a semiconductor memory. Magnetic disks include flexible disks (FDs) and HDDs. Optical disks include compact discs (CDs) and digital versatile discs (DVDs). The medium reader 106 copies the programs and data read from the recording medium 113 to other recording media such as the RAM 102 or the HDD 103. The read programs may be executed by the CPU 101.

[0046] The recording medium 113 may be a portable recording medium. The recording medium 113 may be used to distribute programs and data. The recording medium 113 and the HDD 103 may also be referred to as computer-readable recording media.

[0047] The communication interface 107 communicates with other information processing devices via the network 30. The communication interface 107 may be a wired communication interface connected to a wired communication device such as a switch or a router, or may be a wireless communication interface connected to a wireless communication device such as a base station or an access point.

[0048] Next, generation of a trajectory from image data will be described. FIG. 4 is a diagram showing an example of generating a trajectory from image data of multiple surveillance cameras. The image analysis device 100 receives image data from the surveillance camera 33 and detects a person from each of multiple image frames included in the image data. For example, the image analysis device 100 detects a person from each of image frames 141, 142, and 143. The image frames 141, 142, and 143 are fixed-point camera images of a specific product shelf and the surrounding aisle, and are arranged in ascending order of capture time. Shoppers walking down the aisle are detected from image frames 141, 142, and 143. The image analysis device 100 uses a trained machine learning model, such as a convolutional neural network, to detect people.

[0049] The machine learning model extracts a bounding box representing a rectangular area containing a person from each image frame. Multiple bounding boxes representing different people may be extracted from a single image frame. The machine learning model adds feature information representing the external characteristics of the detected person as a label to the bounding box. The image analysis device 100 compares the bounding boxes and feature information between adjacent image frames. If the percentage of overlapping area between two bounding boxes exceeds a threshold and the similarity of the feature information exceeds a threshold, the image analysis device 100 estimates that the two bounding boxes represent the same person. A series of consecutive bounding boxes representing the same person within a short time period is sometimes called a tracklet.

[0050] The feature information is, for example, a list of appearance attributes of the detected person. Appearance attributes include, for example, gender, age group, physique, clothing, and clothing color. These attributes are determined by a machine learning model. In this case, the feature information may have multiple dimensions corresponding to various attribute values ​​and be expressed as a binary vector in which the dimension corresponding to the relevant attribute value is set to 1 and the dimension corresponding to the inapplicable attribute value is set to 0. However, the feature information may also be a real vector calculated internally by the machine learning model to extract a bounding box. The similarity of the feature information is expressed, for example, by the cosine similarity or Euclidean distance between vectors.

[0051] The image analysis device 100 converts the coordinates of the bounding boxes on the image frames into coordinates on a store map that shows a top-down view of the store. The coordinate conversion method is determined in advance based on the location and imaging direction of the surveillance cameras 33 and the imaging characteristics of the surveillance cameras 33. The image analysis device 100 assigns the imaging time at which the image frame was generated to the converted coordinates on the store map. As a result, points indicating the location and time of the detected person are plotted on the store map. The image analysis device 100 generates a track 145 that shows the movement path of the detected person by connecting multiple points corresponding to multiple bounding boxes determined to be the same person.

[0052] Image data from a single surveillance camera may generate one track per person, or two or more separate tracks. Possible causes of track separation include a sudden change in a person's posture, a person temporarily hiding behind an obstruction, a person temporarily entering the blind spot of the surveillance camera, or incorrect recognition of feature information due to the influence of light.

[0053] Image analysis device 100 adds feature information 144 to track 145. Feature information 144 may indicate appearance features common to multiple image frames included in the period of track 145, or may indicate the mode or average value of determined appearance features. Furthermore, feature information 144 may indicate appearance features determined from the first or last image frame of the period of track 145. Feature information for each time point included in the period of track 145 may also be saved.

[0054] Image analysis device 100 performs the same processing on image data from surveillance cameras 34 and 35 as on image data from surveillance camera 33. Image analysis device 100 detects people from each of multiple image frames included in the image data from surveillance camera 34, and generates track 146. Image analysis device 100 also detects people from each of multiple image frames included in the image data from surveillance camera 35, and generates track 147.

[0055] The image analysis device 100 compares the characteristic information and time of the generated tracks and connects two or more tracks that are likely to represent the same person. This generates a trajectory that shows the entire movement path of a person. The image analysis device 100 searches for other tracks that start within a certain time from the end time of a certain track and have a similarity of characteristic information that exceeds a threshold, and connects the other tracks behind the track. For example, the image analysis device 100 connects tracks 145, 146, and 147 to generate trajectory 148.

[0056] However, there are limits to the accuracy of identifying the same person based only on feature information and time. The trajectory generated at this stage does not necessarily connect tracks without gaps throughout the entire period from store entry to payment, and there may be gaps where tracks are missing. Also, tracks of different people with similar feature information may be mistakenly linked.

[0057] FIG. 5 is a diagram showing an example of erroneous recognition of tracks included in a trajectory. In one trajectory, tracks 151, 155, and 153 are connected in order. Track 151 shows a short-term movement route of a certain person. Following track 151, track 152, which is the same person as track 151, is generated. Tracks 151 and 152 are generated, for example, from image data from different surveillance cameras. However, track 155, which is a different person, is mistakenly connected after track 151 instead of track 152. If the feature information of tracks 152 and 155 is similar and the coordinates and times of tracks 152 and 155 are relatively close, erroneous tracks like this may be connected. Track 153, which is the same person as track 151, is correctly connected after track 155. In this way, erroneous tracks may be connected only in the middle of a trajectory.

[0058] Furthermore, the person in tracks 151 and 153 temporarily disappears from any of the surveillance cameras after track 153. This can occur when a person is temporarily hidden behind an obstruction as seen from a surveillance camera, or when a person temporarily enters a blind spot of a surveillance camera. Track 154, which is the same person as tracks 151 and 153, is generated for the time after the person leaves the obstruction or the blind spot.

[0059] However, if there is a gap that is not captured by the surveillance camera, it may be difficult for the image analysis device 100 to recognize that tracks 153 and 154 represent the same person. Therefore, track 154 may be mistakenly recognized as representing a different person from track 153. In this case, track 154 is not connected to the trajectory. In this way, a portion of the trajectory may become a gap period (blank period) that is not filled by any track.

[0060] Note that track 153 may be at the end of a trajectory. Also, after a gap period, tracks 151 and 153 may be linked to tracks by the same person. Also, after a gap period, tracks 151 and 153 may be linked to track 156 by a different person. Also, tracks 152 and 154 may be mistakenly included in different trajectories.

[0061] In this way, estimating the tracks of the same person based only on feature information and time may result in an incomplete trajectory. Therefore, the image analysis device 100 of the second embodiment corrects the trajectory by referring to POS data. For example, the image analysis device 100 replaces track 155 included in this trajectory with track 152. The image analysis device 100 also complements the gap period included in this trajectory with track 154.

[0062] Next, correction of a trajectory generated from image data will be described. FIG. 6 is a diagram showing an example of a store map. The image analysis device 100 stores a store map 131. The store map 131 is a map showing a sales floor where products are displayed as viewed from above. Positions on the store map 131 are expressed by two-dimensional coordinates including X and Y coordinates. The store map 131 shows multiple product shelves. The display positions of each product are determined in advance. Shoppers move along aisles between the product shelves. The shopper's movement path is expressed by a curve on the store map 131.

[0063] Store 31 is equipped with one or more POS terminals, such as POS terminal 32. The locations of the POS terminals may be indicated on store map 131. Store 31 is also equipped with multiple surveillance cameras, such as surveillance cameras 33, 34, and 35. The locations of the surveillance cameras may be indicated on store map 131. Among the multiple surveillance cameras, there may be a surveillance camera that captures shoppers making payments at the POS terminal, or a surveillance camera that captures shoppers lining up at the POS terminal. If there is no such surveillance camera, it is preferable that among the multiple surveillance cameras, there is a surveillance camera that captures an area as close as possible to the POS terminal.

[0064] FIG. 7 is a diagram showing an example of a trajectory table. The image analysis device 100 stores a trajectory table 132. The trajectory table 132 associates a track ID, time, coordinates, feature information, and trajectory ID. The track ID identifies the track. For one track ID, multiple pairs of time and coordinates are listed. The time is the time when a person was detected. The coordinates indicate a position on the store map 131.

[0065] The feature information indicates the external features of the detected person. The feature information is expressed as a vector. The feature information may be saved for each track or for each time point. The trajectory ID identifies a trajectory that includes multiple tracks. It can also be said that the trajectory ID identifies a person. Note that the image analysis device 100 does not need to distinguish between the track ID and the trajectory ID as identifiers. For example, when two tracks are linked, the image analysis device 100 may modify one of the track IDs so that the track IDs of the two tracks are the same.

[0066] FIG. 8 is a diagram showing an example of a POS data table. The image analysis device 100 stores the POS data table 133. The POS data table 133 associates terminal IDs, payment IDs, product names, categories, and payment times. The terminal ID identifies the POS terminal. The POS data table 133 may contain a mixture of POS data from different POS terminals. The payment ID identifies the shopper's payment procedure. A collection of records with the same payment ID may be called a receipt.

[0067] The product name is the name of the product purchased by the shopper. The category indicates the classification of the purchased product. The display location of the product is specified by the combination of the product name and category. Two or more products may be purchased in one payment procedure. The payment time is the time when payment is made via the POS terminal 32, and is associated with the payment ID.

[0068] 9 is a diagram showing an example of a product table. The image analysis device 100 stores a product table 134. The product table 134 indicates the display positions of products in the store 31. The product table 134 associates product names, categories, and coordinates. The product name is the name of the product. The category indicates the classification of the product. The product names and categories correspond to those included in the POS data table 133. The coordinates are positions on the store map 131 that indicate the positions where the products are displayed.

[0069] FIG. 10 is a diagram showing an example of truck selection based on POS data. The image analyzing device 100 refers to the POS data table 133 and the product table 134 to correct the trajectory recorded in the trajectory table 132. If a person purchases a specific product, it is assumed that the person passed by the display location of that product. If the trajectory corresponding to that person does not pass by the display location of that product, there is a high possibility that the trajectory contains an error. The image analyzing device 100 utilizes this property to correct the trajectory.

[0070] As an example, image analyzing device 100 detects from POS data table 133 that person 161 has purchased three items, and searches product table 134 for the display positions of those three purchased items. Similarly, image analyzing device 100 detects that person 162 has purchased one item, and searches for the display position of that purchased item. Furthermore, image analyzing device 100 detects that person 163 has purchased one item, and searches for the display position of that purchased item.

[0071] Here, attention is focused on the trajectory estimated for person 161. The display position of the second product purchased by person 161 is represented by point 164 on store map 131. Trucks 165 and 166 are generated from image data depicting the area around point 164. Truck 165 passes by point 164, whereas truck 166 does not. In this case, even if the feature information of trucks 165 and 166 is similar, image analyzing device 100 can determine that truck 165 is more likely to represent person 161 than truck 166.

[0072] If the estimated trajectory for person 161 is incomplete, image analysis device 100 modifies the trajectory to include track 165. If the current trajectory includes another track (e.g., track 166) with the same time as track 165 but that track is farther from point 164, image analysis device 100 replaces the other track with track 165. If the time of track 165 in the current trajectory belongs to a gap period, image analysis device 100 fills the gap period with track 165.

[0073] Here, when correcting the trajectory by referring to the purchased items, the image analysis device 100 determines the correspondence between the trajectory and the payment data. The image analysis device 100 uses time to determine the correspondence. The image analysis device 100 compares the payment time included in a certain payment data with the end time of each of the multiple trajectories. The image analysis device 100 associates, with the payment data, any trajectory for which the time difference obtained by subtracting the end time from the payment time is less than a threshold value (for example, approximately ten to several tens of seconds). This associates the trajectory with a set of purchased items.

[0074] However, if there are many shoppers around POS terminal 32, it may be difficult to associate one-to-one correspondence between trajectories and payment data based on time alone. In this case, image analyzing device 100 estimates the correspondence by referring to the purchased items.

[0075] 11 is a diagram showing an example of how scores are calculated for trajectories. For a given payment data, image analysis device 100 searches for all trajectories for which the time difference is less than the threshold. If there are two or more matching trajectories, image analysis device 100 refers to the set of products indicated by the payment data and calculates the score for each matching trajectory.

[0076] Trajectory 170 is a candidate trajectory to be associated with payment data. Note that there is a gap period in trajectory 170. The payment data indicates products A, B, and E as purchased products. Point 171 indicates the installation location of POS terminal 32. Point 172 indicates the display location of product A. Point 173 indicates the display location of product B. Point 175 indicates the display location of product E.

[0077] The image analysis device 100 calculates the shortest distance between each of products A, B, and E and the trajectory 170. The distance is, for example, the Euclidean distance on the store map 131. Point 176 is the point included in the trajectory 170 that is the shortest distance from point 172. Therefore, the shortest distance for product A is the distance between points 172 and 176. Point 177 is the point included in the trajectory 170 that is the shortest distance from point 173. The shortest distance for product B is the distance between points 173 and 177. Point 178 is the point included in the trajectory 170 that is the shortest distance from point 175. The shortest distance for product E is the distance between points 175 and 178.

[0078] The image analysis device 100 counts the number of nearby products whose distance is less than a threshold (e.g., 1 meter). The image analysis device 100 also sums the distances of the nearby products. The image analysis device 100 calculates the sum of the number of nearby products and the total distance as the score of the trajectory 170. For certain payment data, the image analysis device 100 selects up to K trajectories with the highest scores as candidates. K is an integer greater than 1, and is, for example, several to about ten.

[0079] FIG. 12 is a diagram showing an example of a matching table. The image analysis device 100 generates the matching table 135 by the above scoring. The matching table 135 associates a payment ID, a purchased item, and a truck ID. The payment ID identifies the payment data. The purchased item is the item listed in the payment data. The truck ID identifies the last truck included in the trajectory. However, a trajectory ID that identifies the trajectory may be used instead of the truck ID. A maximum of K truck IDs can be associated with one payment ID.

[0080] For each payment data, the image analyzing device 100 may select the trajectory with the highest score from among multiple candidates. This allows the trajectory and the set of purchased items to be matched so that they are most consistent. However, the trajectory at this point may contain errors. Therefore, the image analyzing device 100 may analyze whether there are overlaps in purchased items between payment data with similar payment times and narrow down the trajectories by taking this overlap into account.

[0081] 13 is a diagram showing an example of a count of candidate purchase items from the perspective of trucks. The image analyzing device 100 generates a table 136 that associates purchased items with candidate truck IDs. For example, truck candidates representing people who have completed payment procedures for items A, B, and C are trucks #11 and #14. The same truck ID may be included in different records in the table 136. For each truck ID, the image analyzing device 100 counts the number of times that the truck ID appears in the table 136.

[0082] Furthermore, image analyzing device 100 lists, for each truck ID, purchased products associated with that truck ID in table 136. As a result, image analyzing device 100 generates table 137 in which truck IDs, appearance counts, and potential purchased products are associated with each other. For example, truck #11 appears 12 times in table 136, and may represent someone who purchased products A, B, and C, or may represent someone who purchased products A, B, and D.

[0083] The image analyzing device 100 counts the number of times the same product appears in the purchase product candidates in the table 137 for each truck ID. As a result, the image analyzing device 100 generates a table 138 that associates truck IDs, the number of times the truck ID appears, and the product count. The product count indicates the number of times each of multiple products appears. A count of "2" or more for a certain product indicates that multiple people who were close to each other at the time of payment purchased that product. The higher the count for a product, the more likely it is that the person represented by that truck ID purchased that product.

[0084] For each truck ID, the image analysis device 100 determines the consistency between the movement path of the trajectory including the truck and the product count. When there are multiple candidate trajectories corresponding to certain payment data, the image analysis device 100 selects one trajectory by referring to the product count. For example, if one trajectory passes near the display location of a commonly purchased item with a count of 2 or more, and the other trajectory does not pass near the display location of such a commonly purchased item, the image analysis device 100 preferentially selects the former trajectory. Furthermore, for example, if both trajectories pass near the display location of a commonly purchased item with a count of 2 or more, the image analysis device 100 preferentially selects the trajectory with the higher score. However, the image analysis device 100 may also preferentially select the trajectory with the larger commonly purchased item count.

[0085] FIG. 14 is a diagram showing an example of trajectory correction. In the above manner, image analysis device 100 identifies purchased products corresponding to a certain trajectory. As an example, purchased products corresponding to trajectory 170 include products A and C. Point 174 indicates the display position of product C. Image analysis device 100 replaces some of the tracks included in trajectory 170, taking points 172 and 174 into consideration. Furthermore, image analysis device 100 selects tracks that fill gap periods included in trajectory 170, taking points 172 and 174 into consideration.

[0086] Image analysis device 100 searches various tracks generated from image data of various surveillance cameras for tracks including a point whose distance from point 172 is less than a threshold value (e.g., 1 meter). The track searched for may currently be incorporated into another trajectory. Image analysis device 100 calculates the similarity of feature information between the relevant track and trajectory 170. As the feature information of trajectory 170, image analysis device 100 may use feature information of the starting point of trajectory 170, feature information of the end point of trajectory 170, or feature information at the same time as the relevant track.

[0087] The image analysis device 100 selects as the correct track a track whose distance from point 172 is less than a threshold and whose similarity in feature information exceeds a threshold (e.g., 0.8 in the case of cosine similarity). The image analysis device 100 replaces, among the tracks included in the trajectory 170, tracks that share a time with the selected track with the selected correct track. If there are two or more tracks whose distance from point 172 is less than the threshold, the image analysis device 100 may select the track whose similarity in feature information is the highest. Alternatively, the image analysis device 100 may select the track whose distance from point 172 is the smallest among the tracks whose similarity exceeds the threshold.

[0088] For example, around point 172 are tracks 181, 182, and 183. Track 182 is included in current trajectory 170. Tracks 181 and 183 are not included in current trajectory 170. Image analysis device 100 selects track 181 and replaces track 182, which is included in trajectory 170, with track 181.

[0089] Similarly, image analysis device 100 searches for a track including a point whose distance from point 174 is less than a threshold. Image analysis device 100 calculates the similarity of feature information between the relevant track and trajectory 170. Image analysis device 100 selects a track whose distance from point 174 is less than the threshold and whose similarity of feature information exceeds the threshold. If the start time of the selected track is within a certain time from the time immediately before the gap period of trajectory 170, image analysis device 100 inserts the selected track into trajectory 170 to fill the gap period.

[0090] For example, around point 174 are tracks 184, 185, and 186. None of tracks 184, 185, and 186 is included in current trajectory 170. Image analysis device 100 selects track 184 and inserts track 184 into trajectory 170. At this time, image analysis device 100 may rewrite the identifier of track 184 so that the identifier assigned to track 184 is the same as that of trajectory 170.

[0091] FIG. 15 is a diagram showing an example of complementing a gap period. As described above, the image analyzing device 100 complements a gap period by integrating tracks that pass near the display positions of purchased products into the trajectory 170. However, there may still be gap periods in which tracks are missing because they are far from the display positions of any purchased products. Therefore, the image analyzing device 100 complements the remaining gap periods based on the feature information and the time. Completion of the remaining gap periods is performed repeatedly, starting from the earliest time.

[0092] Image analysis device 100 searches for tracks whose starting point coordinates are within a certain distance from the coordinates of the point immediately preceding the gap period and whose starting point time is within a certain time from the time of the point immediately preceding the gap period. Image analysis device 100 calculates the similarity of feature information between each of the corresponding tracks and trajectory 170, and inserts tracks whose similarity exceeds a threshold into trajectory 170. If there are two or more tracks whose similarity exceeds the threshold, image analysis device 100 may insert the track with the highest similarity into trajectory 170. Image analysis device 100 determines whether there is still a gap period. If there is still a gap period, image analysis device 100 repeats the above process using the end point of the inserted track as the base point.

[0093] For example, image analyzing device 100 detects tracks 191, 192, and 193 whose starting points match the point immediately before the gap period. Of tracks 191, 192, and 193, image analyzing device 100 selects track 191, which has the highest similarity in feature information, and inserts it into trajectory 170. Even after track 191 is inserted, a gap period still remains. Therefore, image analyzing device 100 detects tracks 194 and 195 whose starting points match the end point of track 191. Of tracks 194 and 195, image analyzing device 100 selects track 194, which has the highest similarity in feature information, and inserts it into trajectory 170.

[0094] Even after track 194 is inserted, a gap period still remains. Therefore, image analyzing device 100 detects tracks 196 and 197 whose starting points match the end point of track 194. Of tracks 196 and 197, image analyzing device 100 selects track 196, which has the highest similarity in feature information, and inserts it into trajectory 170. By inserting track 196, no gap period remains in trajectory 170. This completes the interpolation process.

[0095] Next, the functions and processing procedures of the image analysis device 100 will be described. FIG. 16 is a block diagram showing an example of the functions of the image analysis device. The image analysis device 100 has an image storage unit 121, a store data storage unit 122, a POS data storage unit 123, and a trajectory storage unit 124. These storage units are implemented using, for example, RAM 102 or HDD 103. The image analysis device 100 also has an image recognition unit 125, a tracking unit 126, a matching unit 127, and a correction unit 128. These processing units are implemented using, for example, CPU 101 and a program.

[0096] Image storage unit 121 stores image data received by image analysis device 100 from multiple surveillance cameras installed in store 31, such as surveillance cameras 33, 34, and 35. The image data includes multiple image frames, each with an image capture time stamp.

[0097] The store data storage unit 122 stores map data that indicates a store map 131. The store data storage unit 122 also stores a product table 134 that indicates the display positions of multiple products. The store data storage unit 122 also stores information on the positions of each surveillance camera and a conversion method for converting coordinates on the image frame of each surveillance camera into coordinates on the store map 131.

[0098] The POS data storage unit 123 stores a POS data table 133 in which POS data is registered. The POS data is received by the image analysis device 100 from one or more POS terminals installed in the store 31, such as the POS terminal 32.

[0099] The trajectory storage unit 124 stores a trajectory table 132 in which the trajectories of multiple shoppers within the store 31 are registered. The trajectories registered in the trajectory table 132 are generated by the tracking unit 126 and corrected by the correcting unit 128.

[0100] The image recognition unit 125 reads image data from the image storage unit 121 and detects people from each of multiple image frames included in the image data. A machine learning model such as a convolutional neural network is used for image recognition. The image recognition unit 125 stores a trained machine learning model. The training of the machine learning model may be performed by the image analysis device 100 or by another information processing device. The image recognition unit 125 extracts a bounding box indicating the area of ​​the detected person from the image frame and generates feature information indicating the external features of the detected person. The feature information may be called identification information (ID).

[0101] The tracking unit 126 arranges bounding boxes extracted from image frames from the same surveillance camera in chronological order and determines the identity of a person between the bounding boxes. The tracking unit 126 also converts the coordinates of the bounding boxes on the image frames into coordinates on the store map 131. The tracking unit 126 generates tracks by connecting multiple points on the store map 131 that correspond to multiple bounding boxes determined to represent the same person. The tracking unit 126 determines the identity of a person between multiple tracks generated from image data from multiple surveillance cameras based on the similarity of feature information and time continuity. The tracking unit 126 generates a trajectory by connecting multiple tracks determined to represent the same person.

[0102] The matching unit 127 reads the POS data from the POS data storage unit 123 and determines the correspondence between the multiple trajectories generated by the tracking unit 126 and the payment data of multiple people included in the POS data. At this time, the matching unit 127 references the display positions of products registered in the store data storage unit 122 and determines the consistency between the trajectories and the display positions of purchased products. The matching unit 127 associates the payment data with the trajectories, taking into account the consistency between the trajectories and the display positions, as well as the time difference between the payment time and the trajectory time.

[0103] The correction unit 128 corrects at least one of the multiple trajectories registered in the trajectory storage unit 124 based on the display position of the purchased product. The correction unit 128 identifies the display position of the purchased product and searches for tracks whose distance from the display position is less than a threshold. The correction unit 128 selects tracks whose distance is less than the threshold and whose similarity of feature information exceeds a threshold. If a track at the same time is already included in the trajectory, the correction unit 128 replaces the track. If the relevant time belongs to a gap period, the correction unit 128 inserts the selected track into the trajectory.

[0104] 17 is a flowchart showing an example of a procedure for generating a trajectory. (S10) The image recognition unit 125 detects people from the image frames of each monitoring camera. The image recognition unit 125 determines the external features of the detected people and generates feature information.

[0105] (S11) The tracking unit 126 converts the coordinates on the image frames of the person detected in step S10 into coordinates on the store map 131. (S12) The tracking unit 126 tracks the same person between image frames from the same surveillance camera based on the coordinates on the image frames and feature information of the detected person. The tracking unit 126 generates a track by connecting the coordinates of the same person on the store map 131.

[0106] (S13) The tracking unit 126 compares the feature information of the multiple tracks generated from the image data of the multiple surveillance cameras. The tracking unit 126 connects two or more tracks with similar feature information to generate a trajectory.

[0107] (S14) The matching unit 127 searches for one or more trajectories from the generated trajectories whose end points are within a certain time period from the payment time of the payment data. (S15) The matching unit 127 identifies the display position of each purchased item indicated by the payment data. For each of the one or more trajectories that meet the search criteria of step S14, the matching unit 127 calculates a score based on the distance between the display position and the trajectory.

[0108] (S16) The matching unit 127 associates up to K trajectories with the payment data as candidates, starting from the highest score calculated in step S15. Note that steps S14 to S16 are performed for each piece of payment data included in the POS data.

[0109] (S17) For each trajectory that appears in step S16, the matching unit 127 tallies one or more purchased items of payment data that may correspond to that trajectory. This counts the number of occurrences of items that may have been purchased by the person represented by the trajectory.

[0110] (S18) The matching unit 127 narrows down the candidates for the trajectory that correspond to the payment data based on the product count calculated in step S17, the movement path of the trajectory, and the score calculated for the trajectory. This determines the payment data that corresponds to the trajectory.

[0111] 18 is a flowchart (continued) showing an example of the procedure for generating a trajectory. (S19) The correction unit 128 selects one trajectory to be corrected. The correction unit 128 identifies the display position of the purchased item indicated by the payment data corresponding to the trajectory to be corrected. The correction unit 128 searches for trucks that pass within a certain range from the display position.

[0112] (S20) The correction unit 128 calculates the similarity of the feature information between each of the tracks found in step S19 and the trajectory to be corrected. (S21) The correction unit 128 selects tracks from the found tracks whose similarity exceeds a threshold. For example, the correction unit 128 selects the track with the highest similarity.

[0113] (S22) The correction unit 128 determines whether the time of the track selected in step S21 belongs to a gap period in which a track is missing from the trajectory to be corrected. If it belongs to the gap period, the process proceeds to step S24; otherwise, the process proceeds to step S23.

[0114] (S23) The correction unit 128 identifies an existing track that has a common time with the track selected in step S21 from among the existing tracks included in the trajectory to be corrected. The correction unit 128 replaces the existing track with the identified track. Then, the process proceeds to step S25.

[0115] (S24) The correction unit 128 integrates the track selected in step S21 into the trajectory to be corrected, thereby complementing at least a part of the gap period. Note that if there are two or more purchased products, steps S19 to S24 are executed for each purchased product.

[0116] (S25) The correction unit 128 determines whether a gap period remains in the trajectory to be corrected. If a gap period remains, the process proceeds to step S26. If no gap period remains, the process proceeds to step S29.

[0117] (S26) The correction unit 128 searches for tracks whose distance from the point immediately preceding the gap period is within a certain range and whose time is within a certain time from the time of the immediately preceding point. (S27) The correction unit 128 calculates the similarity of the feature information between each of the tracks searched for in step S26 and the trajectory to be corrected.

[0118] (S28) The correction unit 128 selects tracks whose similarity exceeds a threshold value from among the searched tracks. For example, the correction unit 128 selects the track with the highest similarity. The correction unit 128 integrates the selected track into the trajectory. Note that steps S26 to S28 are repeated until all gap periods are completed.

[0119] (S29) The correction unit 128 outputs the corrected trajectory. The correction unit 128 may store the corrected trajectory in non-volatile storage, may display it on the display device 111, or may transmit it to another information processing device.

[0120] As described above, the image analyzing device 100 according to the second embodiment analyzes image data from multiple surveillance cameras installed in the store 31 and generates trajectories for each of multiple shoppers. This allows the image analyzing device 100 to provide information that is useful for determining store design and product placement without placing an additional burden on shoppers.

[0121] The image analyzing device 100 also identifies purchased items from the POS data and determines the consistency between the display locations of the purchased items and the generated trajectory. If the trajectory does not pass near the display location, the image analyzing device 100 searches for the correct track among tracks that pass near the display location and replaces some of the tracks included in the trajectory. This corrects the trajectory and improves its accuracy, even if tracks of people with similar appearances are mistakenly linked.

[0122] Furthermore, if the trajectory lacks a track that is thought to have passed near the display location, the image analysis device 100 integrates tracks that pass near the display location into the trajectory to fill in at least a portion of the gap period. This allows the accuracy of the trajectory to be improved by integrating the divided trajectories, even if the trajectory of the same person is divided due to an obstruction or a blind spot in the surveillance camera. Furthermore, the image analysis device 100 further searches for other tracks that follow the integrated track and integrates them into the trajectory. This eliminates the gap period.

[0123] Furthermore, when there are two or more trajectories that match the payment time of the payment data, the image analysis device 100 associates the payment data with the trajectories, taking into account the consistency between the display positions of the purchased items indicated by the payment data and the trajectories. This allows for accurate estimation of the purchased items of the shopper indicated by the trajectories, even when there are many shoppers around the POS terminal 32. Furthermore, the image analysis device 100 may narrow down the trajectories corresponding to the payment data by further considering the consistency between the trajectories and the overlapping status of purchased items between shoppers who made similar payments. This improves the accuracy of associating payment data with trajectories.

[0124] The foregoing merely illustrates the principles of the present invention. Further, since numerous modifications and changes will be apparent to those skilled in the art, the present invention is not limited to the exact construction and application shown and described above, and all corresponding modifications and equivalents are deemed to be within the scope of the present invention as defined by the appended claims and their equivalents.

[0125] REFERENCE SIGNS LIST 10 Information processing device 11 Storage unit 12 Processing unit 13 Product data 14a, 14b Trajectory data 15a, 15b Payment data

Claims

1. An information processing program that causes a computer to execute the following processes: acquire product data indicating the display position of each of a plurality of products in a store, a plurality of trajectory data corresponding to a plurality of people, each of which indicates a movement path within the store estimated from camera image data, and a plurality of payment data corresponding to the plurality of people, each of which indicates a product purchased at the store among the plurality of products; identify a product indicated by first payment data of a person corresponding to first trajectory data among the plurality of payment data; search for second trajectory data among the plurality of trajectory data that passes within a certain range from the display position of the identified product; and correct the first trajectory data using the second trajectory data.

2. An information processing program as described in claim 1, wherein the correction includes a process of replacing a travel route of a first section that corresponds to the second section included in the first trajectory data with a travel route of a second section that passes through the certain range included in the second trajectory data.

3. An information processing program as described in claim 1, wherein time information is added to the multiple trajectory data and the multiple payment data, and the search includes a process of identifying multiple candidates of human trajectory data corresponding to the first payment data from the multiple trajectory data based on the time information, calculating an evaluation value for each of the multiple candidates based on the distance between the display position of the product indicated by the first payment data and each of the multiple candidates, and selecting one of the multiple candidates as the first trajectory data based on the evaluation value.

4. An information processing program as described in claim 3, wherein the search includes a process of determining whether there is overlap of products between the first payment data and second payment data among the plurality of payment data that is within a certain time period from the first payment data, and further using the overlap to select one of the plurality of candidates.

5. An information processing method in which a computer executes the following processes: acquiring product data indicating the display position of each of a plurality of products in a store, a plurality of trajectory data corresponding to a plurality of people, each of which indicates a movement path within the store estimated from camera image data, and a plurality of payment data corresponding to the plurality of people, each of which indicates a product purchased at the store among the plurality of products; identifying a product indicated by first payment data of a person corresponding to first trajectory data among the plurality of payment data; searching for second trajectory data among the plurality of trajectory data that passes within a certain range from the display position of the identified product; and correcting the first trajectory data using the second trajectory data.

6. An information processing device having: a memory unit that stores product data indicating the display position of each of a plurality of products in a store, a plurality of trajectory data corresponding to a plurality of people, each of which indicates a movement path within the store estimated from camera image data, and a plurality of payment data corresponding to the plurality of people, each of which indicates a product purchased at the store among the plurality of products; and a processing unit that identifies, from the plurality of payment data, a product indicated by first payment data of a person corresponding to first trajectory data among the plurality of trajectory data, searches for second trajectory data from the plurality of trajectory data that passes within a certain range from the display position of the identified product, and modifies the first trajectory data using the second trajectory data.

Images