Information processing apparatus, method for controlling information processing apparatus, and program

Abstract

To appropriately reduce the number of persons who need to be tracked inside a shop to reduce a tracking load, and also to make it possible to accurately track a movement line of each customer in the shop even when the number of customers increases and the shop becomes crowded with many customers.SOLUTION: A tracking device 100 does not immediately start tracking a person (customer 303) who enters a shop 300, and on condition that sensors provided on showcases 309A detect that doors of the showcases are opened, starts tracking persons (customer 304) who are recognized to be in front of the showcases from a video from a TOF camera 301 as tracking targets. A person (customer 306) who have paid for commodities is excluded from the tracking targets.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device that tracks a person in a store or the like, and a control method and program for the information processing device. [Background technology]

[0002] In the past, there was a technology that tracked the movement of customers in a store captured by security cameras installed on the ceiling in order to digitize customer behavior in the store. Recently, the technology is being considered for use in walk-through unmanned stores where customers pick up the products they want in the store and pay automatically, as a technology to link customers in the store with the products they purchase.

[0003] Patent document 1 discloses a technology that uses images taken inside a store to match information about customers entering the store at the entrance / exit with information about the customer's movement within the store, and also matches information about the customer's movement from the payment area within the store to the entrance / exit with information about purchases made in the store. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-162256 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology of Patent Document 1 tracks the in-store behavior of all customers who enter the store from the entrance. Therefore, when the number of customers entering the store increases and the store becomes crowded with many customers, the tracking load increases, making it difficult to accurately track the movement of each customer who is moving around in the store. For example, there is a problem that the person being tracked may be replaced, making it easier for erroneous tracking to occur.

[0006] The present invention has been made to solve the above-mentioned problems. The present invention aims to provide a mechanism that can appropriately reduce the number of subjects that need to be tracked in a store and reduce the tracking load. [Means for solving the problem]

[0007] The present invention is characterized in that it has a tracking means that tracks people inside a store using video footage taken inside the store, and the tracking means is a sensor that detects a predetermined action taken by a person on a shelf displaying products installed in the store or a predetermined change in the shelf that occurs in response to the action, and, upon detection, begins tracking a person that is recognized from the video to be in front of the shelf as a tracking target, and does not track a person that has been recognized as entering the store until the person becomes the tracking target. Effect of the Invention

[0008] According to the present invention, it is possible to appropriately reduce the number of people who need to be tracked in a store and reduce the tracking load. As a result, even when the number of customers increases and the store is crowded with many customers, it is possible to accurately track the movement of each customer who is moving around in the store. For example, it is possible to prevent erroneous tracking from occurring due to the tracking person being replaced. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an overview of an information processing system according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a diagram illustrating an overview of an information processing system according to an embodiment of the present invention. [Diagram 3] FIG. 1 is a diagram showing an example of the configuration of an information processing system according to an embodiment of the present invention. [Figure 4] FIG. 1 is a block diagram showing an example of the configuration of a tracking device according to an embodiment of the present invention. [Diagram 5] 4 is a flowchart showing an example of the operation of the information processing system according to the present embodiment. [Figure 6]4 is a flowchart showing an example of the operation of the information processing system according to the present embodiment. [Figure 7] 3A to 3C are diagrams illustrating various tables used in the information processing system of the present embodiment. [Figure 8] FIG. 13 is a diagram showing an example of a product payment screen displayed on the product payment terminal. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. 1 and 2 are diagrams for explaining an overview of an information processing system according to an embodiment of the present invention. The information processing system of this embodiment is a system that tracks (also called "pursuit" or "tracking") customers within the store 300 using images captured by a TOF camera 308 installed on the ceiling or the like of the store 300.

[0011] In this system, even if a customer, such as customer 303, enters the store from store entrance / exit 302 through passage gate 315, tracking of that customer does not begin at that point (the customer is not yet considered a tracking target). Since there are many customers who simply enter the store and then leave without purchasing any products, not tracking such customers reduces the number of customers to be tracked and the tracking load.

[0012] When a customer, like customer 304, opens the door of a showcase 309 (309A, 309B, . . .), the system starts tracking the customer (the customer is the tracking target). This is because a customer who opens the door of showcase 309 is considered to have a high possibility of purchasing product 310 displayed in showcase 309. It should be noted that customer 305 corresponds to a customer that is already being tracked by this system.

[0013] Furthermore, when a customer, like customer 306, makes a payment using product payment terminal 312, the system will end tracking of that customer (remove them from the tracking target). This is because it is considered that a customer who has completed product payment is no longer required to be tracked. In this way, by removing customers who have made a payment or the like and no longer need to be tracked from the tracking target, it is possible to reduce the number of customers to be tracked and the tracking load.

[0014] In this way, the system of this embodiment reduces the number of customers who need to be tracked among those who enter the store, and as a result, makes it possible to accurately track the movement of each customer within the store even when a large number of customers enter the store and the store becomes crowded with many customers.

[0015] In addition, when a customer being tracked, such as customers 304 and 305, picks up an item from a showcase 309, the item is identified using information obtained by a TOF sensor 308 installed in that showcase 309, and the item is associated with that customer and made available for use at the time of payment.

[0016] Furthermore, when a customer being tracked who is holding a product (i.e., a customer who has not yet paid) passes through passage gate 315 installed at in-store entrance / exit 302, the image captured by security camera 317 installed on the ceiling or the like of store 300 is associated with the time when the customer passed through passage gate 315. This makes it possible to identify a customer who has taken a product out of the store without paying. Note that there may be multiple in-store entrances 302, each of which may be provided with a passage gate 315.

[0017] FIG. 3 is a diagram showing an example of the configuration of an information processing system according to this embodiment, in which the same components as those in FIG. 1 and FIG. 2 are given the same reference numerals. In the information processing system of this embodiment, the tracking device 100 is an information processing device that tracks customers in the store 300 using video of the store 300 captured by the TOF camera 301. The tracking device 100 and the TOF camera 301 may be separate entities, but the tracking device 100 may be incorporated inside the TOF camera 301, and an edge processing type TOF camera 301 that determines the start and end of tracking inside the TOF camera 301 may be used.

[0018] Each showcase 309 (for example, 309A, 309B, ... in Fig. 1 and Fig. 2) is a showcase with a door, and by opening the door, it is possible to take out the products displayed in the showcase 309. The door of each showcase 309 is provided with a sensor, and it is possible to detect that the door of the showcase 309 has been opened by a customer. When the door of the showcase 309 is opened, the information is notified to the tracking device 100 via a LAN or the like at that time (in real time). The door of the showcase 309 is not limited to a door that allows the inside of the showcase to be seen through, such as glass, and may be a door that does not allow the inside of the showcase to be seen through. In addition, in this embodiment, the "showcase" is not limited to a so-called commercial showcase (display shelf), and may be a simple shelf (a shelf with a door in the example of this embodiment) as long as it is capable of storing products for sale.

[0019] Further, each showcase 309 is provided with a TOF camera 308 (for example, 308A, 308B, ... in Fig. 1 and Fig. 2). The TOF camera 308 includes a TOF camera body (corresponding to a TOF camera device described later) and a control unit capable of identifying the type and number of products taken out of the showcase 309 by a customer (or picked up by a customer) from information acquired by the TOF camera body. The TOF camera body is attached to the showcase 309 at an angle capable of capturing an image of a product being taken out of the showcase 309 in which the TOF camera body is installed. The control unit may be realized by a control board or the like connected to the TOF camera body, or may be realized by a personal computer or the like connected to the TOF camera body. Information on the product identified by the TOF camera 308 is notified to the tracking device 100 from the TOF camera 308 via a LAN or the like at the timing of identification (in real time). That is, the TOF camera 308 functions as an identification device that identifies the type and number of products taken out of the showcase 309 from a video image capturing the state in which the products are being taken out of the showcase 309. Note that the type and number of products taken out of the showcase 309 are identified using, for example, a trained image recognition model, but other methods may also be used.

[0020] The TOF cameras 301 and 308 each include a TOF (Time of Flight) camera device (also called a TOF sensor). The TOF camera device is capable of measuring three-dimensional information by utilizing the time of flight of light, and can simultaneously capture a distance image based on the three-dimensional information and, for example, a color visible image (not limited to a color visible image). Using the video captured by such a TOF camera device, it becomes possible to track a customer and identify the product that the customer has picked up.

[0021] The product payment terminal 312 is a payment device for a customer to pay for a product by himself / herself, and there may be a plurality of terminals. A two-dimensional code reader 311 is connected to the product payment terminal 312, and information of the customer who makes a product payment can be obtained by reading a two-dimensional code (e.g., QR code (registered trademark)) displayed on the display of the customer terminal 314 with the two-dimensional code reader 311. In this embodiment, a QR code (registered trademark) is used as an example of a two-dimensional code, but the present invention is not limited to this. When making a product payment, the product payment terminal 312 obtains and uses information of the product that the customer is holding from the tracking device 100. Note that, when the customer takes out the product from the showcase 309, the information of the product that the customer is holding is identified by the TOF camera 308 provided in the showcase 309 and provided to the tracking device 100.

[0022] When the passing gate 315 detects that a customer has notified the passing gate 315 (in real time), the tracking device 100 notifies the tracking device 100 of the information via a LAN or the like. When a customer with an unpaid commodity is passing through the passing gate 315, the tracking device 100 notifies the store manager device (wearable device) 316. At this time, the tracking device 100 associates the time with the image captured by the surveillance camera 317 (surveillance image including at least the vicinity of the store entrance / exit 302) using a time stamp. In other words, the tracking device 100 associates the customer who is about to take the unpaid commodity out of the store with the image in which the customer is captured. The surveillance image of the inside of the store 300 captured by the surveillance camera 317 is transmitted to a data server (not shown) (hereinafter referred to as a "surveillance camera image recording server") via a LAN or the like and stored therein.

[0023] It should be noted that the communication between each device shown in FIG. 3 and the tracking device 100 is not limited to a LAN, and various other communication methods may be used.

[0024] FIG. 4 is a block diagram showing an example of the configuration of the tracking device 100. As shown in FIG. 4, a CPU 201 executes various controls by loading a program stored in a ROM 203 or an external memory 212, etc., into a RAM 202 as necessary and executing the program. The RAM 202 is used as a working area for the CPU 201. The ROM 203 stores various programs and various data.

[0025] The input controller 205 controls input from a keyboard (KB) 209 or a pointing device (mouse, touch panel, etc.) not shown. The video controller 206 controls display on a display 211. Note that in the tracking device 100, the camera device 210 does not have to be connected to the input controller 205. Also, the keyboard 209 and the display 211 do not have to be provided in the tracking device 100.

[0026] The memory controller 207 controls access to an external memory 212 such as a hard disk (HD), a solid state drive (SSD), or a flash memory. A communication I / F controller 208 controls communication with a wired or wireless network. The system bus 204 connects the devices 201 to 208.

[0027] The hardware configuration of the product payment terminal 312 is basically the same as that of the tracking device 100 , but a two-dimensional code reader 311 is connected to the input controller 205 . The hardware configuration of the customer terminal 313 is basically the same as that of the tracking device 100, but the communication I / F controller 208 is realized by, for example, a mobile phone line of a mobile communication system or the like and a communication module of a wireless LAN or the like. The customer terminal 313 can be realized by a so-called smartphone or the like.

[0028] Furthermore, the TOF cameras 301 and 308 can also be basically realized with the same hardware configuration as the tracking device 100, but instead of the keyboard 209, the display 211, etc., they are equipped with the above-mentioned TOF camera device as the camera device 210. Furthermore, the TOF cameras 301 and 308 do not necessarily have to be equipped with the video controller 206 or the display 211. Note that in the TOF camera 308, the above-mentioned process of identifying the product type and the number of products that the customer has taken out of the showcase 309 is realized by the CPU 201 of a control board or the like connected to the camera device 210 executing a program stored in the ROM 203, etc.

[0029] A surveillance camera 317 can also be realized with a similar configuration, but has a camera device 210 that captures visible images instead of the keyboard 209 and the display 211. The surveillance camera 317 does not necessarily have to have the video controller 206 or the display 211.

[0030] Pass gate 315 can also be realized with a similar configuration, but includes a gate mechanism and the like. The store manager device (wearable device) 316 can also be realized with a similar configuration.

[0031] The operation of the information processing system of this embodiment will be described below with reference to a flowchart. 5 and 6 are flowcharts showing an example of the operation of the information processing system of this embodiment. In these figures, the operation of the tracking device 100 is realized by the CPU 201 of the tracking device 100 loading a program stored in the external memory 212 into the RAM 202 as necessary and executing it. The operation of the TOF camera 308 is executed by the control unit provided in the TOF camera 308 described above. The operation of the payment terminal 312 is realized by the CPU of the payment terminal 312 loading a program stored in the external memory into the RAM as necessary and executing it. Note that the "behavior of the customer in the store" written on the left side of FIG. 5 is written for ease of explanation and is not included in the operation of this system.

[0032] 7 is a diagram illustrating various tables used in the information processing system of this embodiment. Each table shown in FIG.

[0033] 5, when a customer (a shopper) opens the door of the showcase 309, this action is detected by a sensor (not shown) provided in the showcase 309. In response, the showcase number and information indicating that the door has been opened are notified to the tracking device 100 via a LAN from a communication interface (not shown) provided in the showcase 309 (S101).

[0034] When the tracking device 100 receives the showcase number from the showcase 309 and information indicating that the door has been opened, it judges whether or not the person in front of the showcase 309 corresponding to the showcase number is the person being tracked (S102). Note that while this system is operating, the TOF camera 301 constantly transmits the captured images (distance image and color image) to the tracking device 100. The tracking device 100 uses this image to track customers, and manages the route of the person being tracked (hereinafter referred to as "tracked person") and the number of people being tracked, using a tracked person table in the store as shown in FIG. 7(a), for example.

[0035] When the tracking device 100 determines that the person in front of the showcase 309 corresponding to the showcase number is the person being tracked (Yes in S102), the process proceeds to S104. The person being tracked corresponds to a customer who has previously opened the doors of showcases A and B and whose tracking has been started, such as customer 305 shown in Fig. 2. The process of S104 will be described later. On the other hand, when it is determined that the person in front of the showcase 309 corresponding to the above showcase number is not the person being tracked (No in S102), the tracking device 100 advances the process to S103.

[0036] In S103, the tracking device 100 starts tracking an untracked person in front of the showcase 309 corresponding to the above showcase number (determines the person as the tracking target), and proceeds to the process of S104. In S104, the tracking device 100 updates the route management of the tracked person and the number of people being tracked. For example, the tracking device 100 updates the tracked person and position information in the in-store tracked person table as shown in FIG.

[0037] Furthermore, when a customer takes an item from the showcase 309, the TOF camera 308 provided in the showcase 309 identifies the type and number of items taken from the showcase, and notifies the tracking device 100 of this together with the showcase number (S105).

[0038] In response to this notification, the tracking device 100 links the tracked person in front of the showcase 309 corresponding to the showcase number included in the notification with the product and quantity included in the notification (S106). Furthermore, in S107, the tracking device 100 updates the product information (information identifying the product, quantity) held in the purchase list of all tracked people (for example, the on-hand product table shown in FIG. 7(b)) based on the information linked in S106.

[0039] Furthermore, a predetermined application (e.g., a store-specific application) is started on the customer terminal 313, and the QR code 314 with the customer information embedded therein is displayed (S108), and when the QR code 314 is read by the two-dimensional code reader 311 connected to the product payment terminal 312, the product payment terminal 312 transmits the customer information (e.g., customer ID) read from the QR code 314 and information indicating that the QR code has been accepted to the tracking device 100 (S109). Note that, if there are multiple payment terminals 312, the payment terminal number for identifying the payment terminal 312, etc., is also included in the transmitted information.

[0040] When the tracking device 100 receives information indicating that the customer information and QR code have been accepted from the product payment terminal 312, it identifies the tracker who preceded the product payment terminal 312 from among all trackers, and links the tracker with the information on the QR code. For example, it registers the customer ID received from the product payment terminal 312 as the customer ID of the record corresponding to the identified tracker in the payment method table as shown in Fig. 7(c). In addition, the tracking device 100 transmits product information (information identifying the product, quantity) held in the purchase list (e.g., Fig. 7(b)) of the identified tracker to the product payment terminal 312 (S110).

[0041] When the product payment terminal 312 receives the product information (information identifying the product, quantity) from the tracking device 100, it displays a product purchase list, for example as shown in 1001 in FIG. 8, based on the product information (S111).

[0042] FIG. 8 is a diagram showing an example of a product payment screen 1000 displayed on the display of the product payment terminal 312 by the product payment terminal 312. The product payment screen 1000 displays a product purchase list 1001, a purchase total amount 1005, payment method selection buttons 1002 to 1004, etc. By selecting one of the payment method selection buttons 1002 to 1004, the customer can make payment using the selected payment method.

[0043] When the product payment terminal 312 accepts the customer's selection of the product payment method (S112), it performs the product payment process using the selected payment method (S113) and transmits payment completion information (including customer information and the payment method) to the tracking device 100.

[0044] When the tracking device 100 receives payment completion information from the product payment terminal 312, it registers the payment method included in the payment information as the payment method of the tracked person who has completed the product payment in, for example, a payment method table (e.g., FIG. 7(c)) based on the payment completion information (S114). Furthermore, the tracking device 100 ends tracking of the tracked person who has completed the product payment (removes them from the tracking target), and updates the route management of the latest tracked person and the number of people being tracked (S104).

[0045] As described above, in the system of this embodiment, tracking of a customer does not start when the customer simply enters the store, but starts when the customer opens the showcase door, and ends when the customer completes payment. This makes it possible to appropriately reduce the number of customers who need to be tracked. As a result, even if the number of customers increases and the store becomes crowded with many customers, it is possible to accurately track the movement of each customer who is moving around in the store.

[0046] In this flowchart, a configuration has been described in which tracking of a tracked person who has completed payment is terminated. However, tracking of the tracked person may be terminated on the condition that the tracked person stands in line at product payment terminal 312 (or passes in front of payment terminal 311). For example, a tracked person who stands in line at payment terminal 311 may be determined to be a customer who intends to make payment, and tracking of the tracked person may be terminated. By doing this, the number of pursuers can be reduced more quickly.

[0047] In addition, if a customer whose tracking has begun by opening the door of a showcase closes the door without removing any products from the showcase, tracking of the customer whose tracking has begun may be terminated. Also, tracking may not start just by opening the showcase door, but may start tracking of a customer standing in front of the showcase when the showcase door is opened and an item is taken from the showcase.

[0048] In addition, this system is configured to read the product that the tracker has taken out of the showcase using a TOF sensor installed in the showcase and use it at the time of payment. This eliminates the need for customers to perform the cumbersome task of having the product payment terminal read the barcode of the purchased product themselves. This makes it possible for customers to easily make payments in unmanned stores, etc.

[0049] As shown in FIG. 6, when the passing gate 315 detects that a customer is passing through the passing gate 315, it notifies the tracking device 100 of the passing information (e.g., the passing gate number and information indicating that the customer is passing through) (S201).

[0050] When the tracking device 100 receives passage information from the passage gate 315, it determines whether or not a tracked person carrying a product has passed through the passage gate 315 corresponding to the passage information. If the tracking device 100 determines that a person who is not a tracked person or a tracked person not carrying a product has passed through the passage gate 315 (No in S202), it determines that the person has left the store without purchasing a product, and ends the processing due to the passage information in S201 above. In this case, if the tracked person passing through the passage gate 315 is a tracked person not carrying a product, the tracking device 100 removes that tracked person from the tracking targets. On the other hand, if the tracking device 100 determines that the tracked person carrying the product has passed through the passing gate 315 (Yes in S202), the process proceeds to S203.

[0051] In S203, the tracking device 100 judges whether the payment method stored in the payment method table (e.g., FIG. 7(c)) of the tracked person who is passing through the gate with the above-mentioned product is "Null". Note that "Null" is stored in the payment method in the payment method table until the product payment is completed. In other words, a tracked person who has "Null" stored in the payment method in the payment method table corresponds to a tracked person who has not completed the product payment.

[0052] If the payment method stored in the payment method table (e.g., FIG. 7(c)) of the tracked person passing through the gate with the above-mentioned product is not "Null" (No in S203), the tracking device 100 determines that the tracked person has left the store after payment, and ends the processing due to the passage information in S201 above. In this case, too, the tracking device 100 removes the tracked person passing through the passage gate 315 from the tracking target. On the other hand, if the payment method stored in the payment method table (e.g., Figure 7(c)) of the tracked person passing through the gate with the above-mentioned product is "Null" (Yes in S203), the tracking device 100 proceeds to S204.

[0053] In S204, the tracking device 100 notifies the store manager device 316 of the detection of an unpaid person (a tracked person passing through the gate with the above-mentioned product). The store manager device 316, which has received the notification of the detection of a non-payment customer from the tracking device 100, notifies the manager of the detection of the non-payment customer by vibrating, sounding an alarm, displaying a warning, or the like (S205).

[0054] Also, in S206, the tracking device 100 transmits the detection time of the unsettled person (the tracked person passing through the gate with the above-mentioned product) to the security camera video recording server (not shown). The security camera video recording server, which receives the notification of the detection time of the unpaid customer from the tracking device 100, records the detection time as a timestamp linked to the shooting time (S207). This makes it possible to record image information of customers who pass through the gate with unpaid goods.

[0055] As described above, the system of this embodiment can associate a customer who is being tracked and is holding a product (i.e., an unpaid customer) with an image captured by a security camera installed on the ceiling or the like of the store when the customer passes through a passage gate installed at the store entrance / exit. This makes it possible to identify the customer who has taken an unpaid product out of the store. In addition, by notifying the store manager when an unpaid customer passes through the passage gate, it becomes possible to deal with the customer attempting to take unpaid merchandise out of the store.

[0056] In the above embodiment, a configuration has been described in which a sensor provided in a showcase detects that the showcase door has been opened, and the opening of the showcase door is used as a trigger to start tracking of a customer in front of the showcase. However, the trigger for starting tracking of a customer is not limited to the detection of the showcase door being opened. Any configuration may be used as long as it detects some customer behavior (a specific action performed by the customer on the showcase or a specific change in the showcase that occurs in response to the action) for the customer to take out the product from the showcase, and the detection is used as a trigger to start tracking of the customer.

[0057] For example, a button for opening the door of the showcase and a sensor for detecting the operation of the button may be provided on the showcase, and the detection of the operation of the button may be used as a trigger for starting tracking of the customer. A sensor for detecting a hand touching the door of the showcase may be provided, and the detection of the hand touching the door may be used as a trigger for starting tracking of the customer. Furthermore, an infrared sensor may be provided on the showcase, and the detection of a customer's hand inserted (entered) into the showcase or a commodity taken out of the showcase may be used as a trigger for starting tracking of the customer. A weight sensor for detecting (measuring) the weight of the commodity contained in the showcase may be provided, and the detection of a commodity removed from the showcase and the weight of the commodity reduced may be used as a trigger for starting tracking of the customer. Furthermore, a TOF camera or the like may be provided in the showcase, and the detection of a commodity being taken out of the showcase may be used as a trigger to start tracking of the customer. Furthermore, when a door sensor for each showcase detects that the door of the showcase has been opened, and then a TOF camera for the showcase with the same identification number detects an item taken out of the showcase, the trigger for starting tracking of the customer may be satisfied when multiple conditions for starting tracking are satisfied. In this way, by setting multiple conditions, it is possible to further narrow down the customers in the store to be tracked.

[0058] Also, when a store employee refills products in a showcase, tracking of the employee is not necessary. For this reason, the employee may wear a certain hat, uniform, or other clothing, or an identifier such as a sash or badge, and a person whose identifier is recognized from the image captured by the TOF camera 301 may not be tracked. The TOF camera 301 outputs, for example, a color image along with a distance image. For this reason, the tracking device 100 that performs tracking processing using the output from the TOF camera can easily determine the identifier of the employee and exclude them from the tracking target. This makes it possible to easily exclude people who do not need to be tracked, such as employees, from the tracking target, thereby reducing the tracking load.

[0059] In this embodiment, the store 300 is provided with an entrance / exit 302, but the entrance and exit may be separated. That is, one or more entrances that only allow entry into the store 300 and one or more exits that only allow exit from the store 300 may be provided, and no entrances that allow entry and exit from the store 300 may be provided. In this configuration, the passage information is transmitted in S201 of Fig. 6 from the passage gate 315 provided at the exit. Furthermore, a configuration may be adopted in which, without providing the passage gate 315, the exit of a customer from the image captured by the TOF camera 301 is detected.

[0060] In addition, the functions of the tracking device 100 may be realized by one computer or by multiple computers. For example, the functions of the tracking device 100 may be realized by using a cloud service or the like.

[0061] The functions of the tracking device 100 may be realized by a control unit included in the TOF camera 301. That is, the control unit included in the TOF camera 301 may execute the above-mentioned functions of the tracking device 100, such as tracking people in the store using video of the inside of the store captured by a camera device included in the TOF camera 301. In this configuration, the TOF camera 301 is configured to be able to communicate with each device of this information processing system, such as the TOF camera 308, the showcase 309, the store payment terminal 312, the passing gate 315, and the store manager device 316, via a LAN or the like.

[0062] As described above, it is possible to appropriately reduce the number of customers who need to be tracked among customers who visit the store. As a result, even if the number of customers increases and the store is crowded with many customers, it is possible to accurately track the movement of each customer who is moving around in the store. Therefore, it is possible to prevent the occurrence of erroneous tracking due to switching of trackers. In addition, by controlling the start of customer tracking using a sensor that can detect the opening and closing of a door on a showcase, it is possible to prevent the cost of the system from increasing and realize a system with an inexpensive configuration that appropriately reduces the number of people that need to be tracked.

[0063] It goes without saying that the configurations and contents of the various data described above are not limited to those described above, and the data may have various configurations and contents depending on the application and purpose. Although one embodiment has been described above, the present invention can be embodied, for example, as a system, an apparatus, a method, a program, a storage medium, etc. Specifically, the present invention may be applied to a system composed of multiple devices, or may be applied to an apparatus composed of a single device. Furthermore, any combination of the above-described embodiments is also included in the present invention.

[0064] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. Furthermore, the present invention may be applied to a system made up of a plurality of devices, or to an apparatus made up of a single device. The present invention is not limited to the above-mentioned embodiment, and various modifications (including organic combinations of the embodiments) are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. In other words, the present invention includes all configurations that combine the above-mentioned embodiments and their modifications. [Explanation of symbols]

[0065] 100 Tracking device 300 stores 301 TOF camera (installed on the ceiling) 308 TOF camera (installed in the showcase) 309 Showcase 312 Product payment terminal 315 Passing Gate 317 Surveillance Camera

Claims

1. A tracking means for performing a person tracking process based on an image captured by an image capturing means for capturing an image of the inside of a store, and for controlling the person to end the tracking process when the person who is the subject of the tracking process makes a product payment; A control means for managing the number of people who are the targets of the tracking process, and for controlling to reduce the number of people when a product payment is made by a person who is the target of the tracking process; An information processing system comprising:

2. The information processing system according to claim 1, characterized in that the tracking means does not target a person as a tracking target if a predetermined characteristic is recognized in the person.

3. 3. The information processing system according to claim 2, wherein the case where a person is recognized to have a predetermined characteristic is a case where the person is recognized to be wearing a predetermined identifier.

4. An information processing system as described in any one of claims 1 to 3, characterized in that it has a notification means for sending a notification to a specified notification destination when the person to be tracked has not made a product payment and it is recognized that the person to be tracked is about to pass through an exit of the store.

5. The information processing system according to claim 4 , wherein the notification destination is a wearable terminal worn by a manager of the store.

6. An information processing system as described in any one of claims 1 to 5, characterized in that when the person to be tracked has not paid for a product and it is recognized that the person to be tracked is about to pass through an exit of the store, the information processing system has a timestamp means which makes it possible to associate the time of the person to be tracked attempting to pass through with an image taken inside the store.

7. An information processing system as described in any one of claims 1 to 6, characterized in that the tracking means controls the tracking process of the person to be terminated when information indicating the completion of product payment is obtained.

8. Further comprising an acquisition means for acquiring information indicating completion of the product payment, The information processing system according to claim 7, characterized in that the acquisition means acquires information indicating that the product payment has been completed on condition that the person is recognized as being in front of the payment terminal performing the product payment.

9. An information processing system as described in any one of claims 1 to 8, characterized in that the tracking process is a process including route management.

10. An information processing system as described in any one of claims 1 to 9, characterized in that the tracking means controls the imaging means to track a person within the store.

11. The information processing system described in any one of claims 1 to 10, characterized in that the control means stores information relating to a person who is the subject of the tracking process in a memory unit, and when a product payment is made by the person who is the subject of the tracking process, deletes the information relating to that person from the memory unit.

12. A display device comprising: a detection means for detecting an action of removing a product from a display area; 12. The information processing system according to claim 1, wherein the tracking process is started in response to the detection of the behavior by the detection means.

13. The information processing system described in Claim 12, characterized in that the tracking process is not initiated if no action of attempting to remove a product from the display area is detected, even if the person has been recognized as entering the store.

14. The information processing system according to claim 12 or 13, characterized in that the detection means detects the behavior based on an image captured by an imaging means provided in the display area.

15. The display area has a door, and the door can be opened to allow removal of merchandise displayed in the display area; The information processing system according to claim 12 or 13, characterized in that the detection means detects an action of attempting to remove a product from the display area by detecting at least one of the following: the door being opened, a person touching the door, or a button for opening the door being operated.

16. A tracking step of performing a person tracking process based on an image captured by an image capturing means for capturing an image of the inside of a store, and controlling to end the tracking process of the person when the person who is the subject of the tracking process makes a payment for a product; a control step of managing the number of people who are the targets of the tracking process, and controlling to reduce the number of people when a product payment is made by a person who is the target of the tracking process; A method for controlling an information processing system comprising:

17. A program for causing at least one computer to function as each of the means of an information processing system described in any one of claims 1 to 15.

Images