Information processing device, control method and program for information processing device

Abstract

To appropriately reduce the number of persons who need to be tracked inside a shop to reduce a tracking load, and also to make it possible to accurately track a movement line of each customer in the shop even when the number of customers increases and the shop becomes crowded with many customers.SOLUTION: A tracking device 100 does not immediately start tracking a person (customer 303) who enters a shop 300, and on condition that sensors provided on showcases 309A detect that doors of the showcases are opened, starts tracking persons (customer 304) who are recognized to be in front of the showcases from a video from a TOF camera 301 as tracking targets. A person (customer 306) who have paid for commodities is excluded from the tracking targets.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device that tracks a person in a store or the like, a control method for the information processing device, and a program. [Background technology]

[0002] In the past, there was technology that tracked the movement of customers in a store as seen on a surveillance camera installed on the ceiling to record customer behavior in the store.Recently, this technology is being considered for use in walk-through unmanned stores where customers can pick up the products they want and pay automatically, as a technology that associates customers with the products they purchase.

[0003] Patent Document 1 discloses a technology that, based on images taken inside a store, associates information about customers entering the store at the entrance / exit with information about the customer's movement within the store, and also associates information about the customer's movement from the payment area within the store to the entrance / exit with information about purchases made in the store. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-162256 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology of Patent Document 1 tracks the in-store behavior of all customers who enter the store from the entrance and exit. Therefore, when a large number of customers enter the store and the store becomes crowded with many customers, the tracking load increases, making it difficult to accurately track the movement of each customer moving around in the store. For example, there is a problem that the person being tracked may change, making it easy for incorrect tracking to occur.

[0006] The present invention has been made to solve the above-mentioned problems, and aims to provide a system that can appropriately reduce the number of people who need to be tracked in a store and reduce the tracking load. [Means for solving the problem]

[0007] The present invention provides a tracking means for tracking a person based on an image captured by a photographing means for photographing the inside of a store, and for controlling the tracking of the person to be tracked so as to terminate the tracking of the person when the person has made a purchase. The present invention also provides a tracking means for managing the number of people who are the subject of the tracking process, and for controlling the tracking of the person when the person has made a purchase. Depending on and a control means for controlling the number of people to be reduced. [Effects of the Invention]

[0008] According to the present invention, it is possible to appropriately reduce the number of people who need to be tracked in a store and reduce the tracking load. As a result, even when the number of customers increases and the store is crowded with many customers, it is possible to accurately track the movement of each customer who is moving around in the store. For example, it is possible to prevent erroneous tracking from occurring when the person being tracked changes. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an overview of an information processing system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a diagram illustrating an overview of an information processing system according to an embodiment of the present invention. [Figure 3] FIG. 1 is a diagram showing an example of the configuration of an information processing system according to an embodiment of the present invention. [Figure 4] FIG. 1 is a block diagram showing an example of the configuration of a tracking device according to an embodiment of the present invention. [Figure 5] 4 is a flowchart showing an example of the operation of the information processing system according to the present embodiment. [Figure 6] 4 is a flowchart showing an example of the operation of the information processing system according to the present embodiment. [Figure 7] 3A to 3C are diagrams illustrating various tables used in the information processing system of the present embodiment. [Figure 8] FIG. 10 is a diagram showing an example of a product payment screen displayed on the product payment terminal. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. 1 and 2 are diagrams for explaining an overview of an information processing system according to an embodiment of the present invention. The information processing system of this embodiment is a system that tracks (also called "pursuit" or "tracking") customers within the store 300 using images captured by a TOF camera 308 installed on the ceiling or the like of the store 300.

[0011] In this system, even if a customer, such as customer 303, enters the store from store entrance / exit 302 through passage gate 315, tracking of that customer does not begin at that point (the customer is not yet considered a tracking target). Since many customers enter the store but leave without purchasing any products, not tracking such customers reduces the number of customers to be tracked and the tracking load.

[0012] When a customer, like customer 304, opens the door of showcase 309 (309A, 309B, ...), the system starts tracking that customer (the customer is the tracking target). This is because a customer who opens the door of showcase 309 is considered to have a high possibility of purchasing product 310 displayed in showcase 309. It should be noted that customer 305 corresponds to a customer that is already being tracked by this system.

[0013] Furthermore, when a customer, like customer 306, makes a payment using product payment terminal 312, the system will stop tracking that customer (remove them from tracking targets). This is because customers who have completed product payments are considered to no longer need to be tracked. In this way, by removing customers who have made payments or otherwise no longer need to be tracked from tracking targets, the number of customers to be tracked can be reduced, and the tracking load can be lightened.

[0014] In this way, the system of this embodiment reduces the number of customers who need to be tracked among those who enter the store, and as a result, makes it possible to accurately track the movement of each customer as they move around the store, even when a large number of customers enter the store and the store becomes crowded with many customers.

[0015] Furthermore, when a customer being tracked, such as customers 304 and 305, picks up an item from showcase 309, the product is identified using information obtained from TOF sensor 308 installed in showcase 309, associated with the customer, and made available for use at the time of payment.

[0016] Furthermore, if a customer being tracked who is holding a product in his / her hand (i.e., a customer who has not yet paid) passes through passing gate 315 installed at store entrance / exit 302, the image captured by surveillance camera 317 installed on the ceiling or the like of store 300 is correlated with the time when the customer passed through passing gate 315. This makes it possible to identify customers who have taken products out of the store without paying. Note that there may be multiple internal entrances / exits 302, each of which may be equipped with a passing gate 315.

[0017] FIG. 3 is a diagram showing an example of the configuration of an information processing system according to this embodiment, and the same components as those in FIGS. 1 and 2 are denoted by the same reference numerals. In the information processing system of this embodiment, tracking device 100 is an information processing device that tracks customers inside store 300 using video of the inside of store 300 captured by TOF camera 301. Note that tracking device 100 and TOF camera 301 may be separate entities, but tracking device 100 may be incorporated inside TOF camera 301, and an edge processing type TOF camera 301 that determines the start and end of tracking inside TOF camera 301 may be used.

[0018] Each showcase 309 (for example, 309A, 309B, etc. in FIGS. 1 and 2) is a showcase with a door, and opening the door allows the removal of products displayed in the showcase 309. A sensor is provided on the door of each showcase 309, and it is possible to detect that the door of the showcase 309 has been opened by a customer. When the door of the showcase 309 is opened, the information is notified to the tracking device 100 via a LAN or the like at that time (in real time). Note that the door of the showcase 309 is not limited to a door that allows the interior of the showcase to be seen through, such as glass, and may be a door that does not allow the interior of the showcase to be seen through. Furthermore, in this embodiment, the "showcase" is not limited to a so-called commercial showcase (display shelf), and may be any type that can store products for sale, such as a simple shelf (a shelf with a door in the example of this embodiment).

[0019] Each showcase 309 is provided with a TOF camera 308 (e.g., 308A, 308B, etc. in FIGS. 1 and 2). The TOF camera 308 includes a TOF camera body (corresponding to a TOF camera device described later) and a control unit capable of identifying the type and number of products removed from the showcase 309 by a customer (or picked up by a customer) from information acquired by the TOF camera body. The TOF camera body is attached to the showcase 309 at an angle that allows it to capture images of products being removed from the showcase 309 in which it is installed. The control unit may be implemented by a control board or the like connected to the TOF camera body, or by a personal computer or the like connected to the TOF camera body. Information about the products identified by the TOF camera 308 is notified to the tracking device 100 via a LAN or the like at the time of identification (in real time). That is, the TOF camera 308 functions as an identification device that identifies the product type and number of products removed from the showcase 309 from a video image capturing the state in which the products are being removed from the showcase 309. Note that the product type and number of products removed from the showcase 309 are identified using, for example, a trained image recognition model, but other methods may also be used.

[0020] The TOF cameras 301 and 308 are equipped with a TOF (Time of Flight) camera device (also called a TOF sensor). The TOF camera device is capable of measuring three-dimensional information using the time of flight of light, and can simultaneously capture a distance image based on the three-dimensional information and, for example, a color visible image (not limited to a color visible image). Images captured by such a TOF camera device can be used to track customers and identify products that customers have picked up.

[0021] The product payment terminal 312 is a payment device that allows a customer to pay for products themselves, and multiple terminals may be provided. A two-dimensional code reader 311 is connected to the product payment terminal 312, and information about the customer making the payment can be obtained by using the two-dimensional code reader 311 to read a two-dimensional code (e.g., a QR code (registered trademark)) displayed on the display of the customer terminal 314. In this embodiment, a QR code (registered trademark) is used as an example of a two-dimensional code, but the present invention is not limited to this. When making the payment, the product payment terminal 312 obtains and uses information about the product being held by the customer from the tracking device 100. Note that when the customer takes a product out of the showcase 309, the information about the product being held by the customer is identified by the TOF camera 308 installed in the showcase 309 and provided to the tracking device 100.

[0022] When the passing gate 315 detects that a customer has notified the passing gate 315 (in real time), it notifies the tracking device 100 of this information via a LAN or the like. When a customer with unpaid merchandise is passing through the passing gate 315, the tracking device 100 notifies the store manager device (wearable device) 316. At this time, the tracking device 100 uses a timestamp to associate the time with the video captured by the surveillance camera 317 (surveillance video that includes at least the area around the store entrance / exit 302). In other words, it associates the customer attempting to take unpaid merchandise out of the store with the video in which the customer appears. The surveillance video of the inside of the store 300 captured by the surveillance camera 317 is transmitted to and stored in a data server (not shown) (hereinafter referred to as the "surveillance camera video recording server") via a LAN or the like.

[0023] It should be noted that communication between the devices shown in FIG. 3 and the tracking device 100 is not limited to LAN, and various other communication methods may be used.

[0024] FIG. 4 is a block diagram showing an example of the configuration of the tracking device 100. As shown in FIG. 4, a CPU 201 performs various controls by loading programs stored in a ROM 203 or an external memory 212, etc., into a RAM 202 as needed and executing the programs. The RAM 202 is used as a work area for the CPU 201. The ROM 203 stores various programs and various data.

[0025] The input controller 205 controls input from a keyboard (KB) 209 and a pointing device (mouse, touch panel, etc.) not shown. The video controller 206 controls display on a display 211. Note that in the tracking device 100, the camera device 210 does not have to be connected to the input controller 205. Furthermore, the keyboard 209 and the display 211 do not have to be provided in the tracking device 100.

[0026] The memory controller 207 controls access to an external memory 212 such as a hard disk (HD), a solid state drive (SSD), or a flash memory. The communication I / F controller 208 controls communication with a wired or wireless network. The system bus 204 connects the devices 201 to 208.

[0027] The hardware configuration of the product payment terminal 312 is basically the same as that of the tracking device 100, but a two-dimensional code reader 311 is connected to the input controller 205. The hardware configuration of customer terminal 313 is also basically the same as that of tracking device 100, but communication I / F controller 208 is realized by, for example, a mobile phone line such as a mobile communication system and a communication module such as a wireless LAN. Customer terminal 313 can be realized by a so-called smartphone or the like.

[0028] Furthermore, TOF cameras 301 and 308 can also be basically realized with the same hardware configuration as tracking device 100, but instead of keyboard 209, display 211, etc., they are equipped with the above-mentioned TOF camera device as camera device 210. Furthermore, TOF cameras 301 and 308 do not necessarily have to be equipped with video controller 206 or display 211. Note that TOF camera 308 realizes processing such as identifying the type and number of products that a customer has taken out of showcase 309 as described above by having CPU 201 of a control board or the like connected to camera device 210 execute a program stored in ROM 203.

[0029] The surveillance camera 317 can also be realized with a similar configuration, but is provided with a camera device 210 that captures visible images instead of the keyboard 209, display 211, etc. The surveillance camera 317 does not necessarily have to be provided with the video controller 206 or the display 211.

[0030] The pass gate 315 can also be realized with a similar configuration, but includes a gate mechanism and the like. The store manager device (wearable device) 316 can also be realized with a similar configuration.

[0031] The operation of the information processing system of this embodiment will be described below using a flowchart. 5 and 6 are flowcharts showing an example of the operation of the information processing system of this embodiment. In these figures, the operation of tracking device 100 is realized by CPU 201 of tracking device 100 loading a program stored in external memory 212 into RAM 202 as needed and executing it. Furthermore, the operation of TOF camera 308 is executed by a control unit provided in TOF camera 308 described above. Furthermore, the operation of payment terminal 312 is realized by the CPU of payment terminal 312 loading a program stored in external memory into RAM as needed and executing it. Note that the "customer's behavior within the store" written on the left side of FIG. 5 is written for ease of explanation and is not included in the operation of this system.

[0032] 7 is a diagram illustrating various tables used in the information processing system of this embodiment. Each table shown in FIG.

[0033] 5, when a customer (a shopper) opens the door of the showcase 309, this action is detected by a sensor (not shown) provided in the showcase 309. In response, the showcase number and information indicating that the door has been opened are notified to the tracking device 100 via a LAN from a communication interface (not shown) provided in the showcase 309 (S101).

[0034] When the tracking device 100 receives the showcase number and information indicating that the door has been opened from the showcase 309, it determines whether the person in front of the showcase 309 corresponding to the showcase number is the person being tracked (S102). Note that while this system is operating, the TOF camera 301 constantly transmits the captured images (distance images and color images) to the tracking device 100. The tracking device 100 uses this image to track customers, and manages the routes of people being tracked (hereinafter referred to as "tracked people") and the number of people being tracked, using a tracked person table in the store such as that shown in FIG. 7(a), for example.

[0035] When the tracking device 100 determines that the person in front of the showcase 309 corresponding to the showcase number is the person being tracked (Yes in S102), the process proceeds to S104. The person being tracked corresponds to a customer who has previously opened the doors of showcases A and B and whose tracking has begun, such as customer 305 shown in Fig. 2. The process of S104 will be described later. On the other hand, if it is determined that the person in front of the showcase 309 corresponding to the showcase number is not the person being tracked (No in S102), the tracking device 100 advances the process to S103.

[0036] In S103, the tracking device 100 starts tracking (tracking) the untracked person in front of the showcase 309 corresponding to the above showcase number, and proceeds to S104. In S104, the tracking device 100 updates the route management of the tracked person and the number of people being tracked. For example, it updates the tracked person and location information in the in-store tracked person table as shown in FIG.

[0037] Furthermore, when a customer takes an item from the showcase 309, the TOF camera 308 provided in the showcase 309 identifies the type and number of items taken from the showcase and notifies the tracking device 100 of this together with the showcase number (S105).

[0038] In response to this notification, the tracking device 100 links the product and quantity included in the notification with the tracked person in front of the showcase 309 corresponding to the showcase number included in the notification (S106). Furthermore, in S107, the tracking device 100 updates the product information (information identifying the product, quantity) held in the purchase list of all tracked people (for example, the on-hand product table shown in FIG. 7(b)) based on the information linked in S106.

[0039] Furthermore, a predetermined application (for example, a store-specific application) is launched on the customer terminal 313, and a QR code 314 with embedded customer information is displayed (S108). When the QR code 314 is read by the two-dimensional code reader 311 connected to the product payment terminal 312, the product payment terminal 312 transmits the customer information (for example, customer ID) read from the QR code 314 and information indicating that the QR code has been accepted to the tracking device 100 (S109). If there are multiple payment terminals 312, the payment terminal number or the like for identifying the payment terminal 312 is also included in the transmitted information.

[0040] When tracking device 100 receives information indicating that it has accepted the customer information and QR code from product payment terminal 312, it identifies the tracker in front of product payment terminal 312 from among all trackers and associates the tracker with the information in the QR code. For example, it registers the customer ID received from product payment terminal 312 as the customer ID of the record corresponding to the identified tracker in a payment method table such as that shown in Figure 7(c). In addition, tracking device 100 transmits product information (information specifying the product, quantity) held in the purchase list (e.g., Figure 7(b)) of the identified tracker to product payment terminal 312 (S110).

[0041] When the product payment terminal 312 receives the product information (information specifying the product, quantity) from the tracking device 100, it displays a product purchase list, for example, as shown in 1001 in FIG. 8, based on the product information (S111).

[0042] FIG. 8 is a diagram showing an example of a product payment screen 1000 displayed on the display of the product payment terminal 312 by the product payment terminal 312. The product payment screen 1000 displays a product purchase list 1001, a purchase total amount 1005, payment method selection buttons 1002 to 1004, etc. By selecting one of the payment method selection buttons 1002 to 1004, the customer can make payment using the selected payment method.

[0043] When the product payment terminal 312 accepts the customer's selection of the product payment method (S112), it performs the product payment process using the selected payment method (S113) and sends payment completion information (including customer information and payment method) to the tracking device 100.

[0044] When the tracking device 100 receives payment completion information from the product payment terminal 312, it registers the payment method included in the payment information as the payment method of the tracked person who has completed the product payment in, for example, a payment method table (e.g., FIG. 7(c)) based on the payment completion information (S114). Furthermore, the tracking device 100 ends tracking of the tracked person who has completed the product payment (removes them from the tracking target), and updates the route management and number of tracked people of the latest tracked person (S104).

[0045] As described above, in the system of this embodiment, tracking of a customer does not begin when the customer simply enters the store. Tracking of the customer begins when the customer opens the showcase door, and tracking of the customer ends when payment is completed. This makes it possible to appropriately reduce the number of customers who need to be tracked. As a result, even when the number of customers increases and the store becomes crowded with many customers, it is possible to accurately track the movement of each customer as they move around the store.

[0046] In this flowchart, a configuration has been described in which tracking of a person being tracked is terminated once payment has been completed. However, tracking of the person being tracked may be terminated on the condition that the person being tracked lines up in line at product payment terminal 312 (or passes in front of payment terminal 311). For example, a person being tracked who lines up in line at payment terminal 311 may be determined to be a customer intending to make payment, and tracking of the person being tracked may be terminated. By doing this, the number of pursuers can be reduced more quickly.

[0047] In addition, if a customer whose tracking has begun after opening the showcase door closes the door without taking any products from the showcase, tracking of the customer whose tracking has begun may be terminated. Furthermore, tracking may not start just by opening the showcase door, but may start when the showcase door is opened and an item is taken from the showcase.

[0048] Furthermore, this system uses a TOF sensor installed in the showcase to read the product the tracker has removed from the showcase and use it during payment. This eliminates the need for customers to manually scan the barcodes of the purchased products into a payment terminal, making it possible for customers to easily make payments in unmanned stores.

[0049] Also, as shown in FIG. 6, when the passing gate 315 detects that a customer is passing through the passing gate 315, it notifies the tracking device 100 of the passing information (e.g., the passing gate number and information indicating that the customer is passing through) (S201).

[0050] When the tracking device 100 receives passage information from the passage gate 315, it determines whether or not a person among the tracked people who is carrying a product has passed through the passage gate 315 corresponding to the passage information. If the tracking device 100 determines that a person who is not a tracked person or a tracked person who is not carrying a product has passed through the passage gate 315 (No in S202), it determines that the person has left the store without purchasing a product, and ends the processing due to the passage information in S201 above. In this case, if the tracked person who is passing through the passage gate 315 is not carrying a product, the tracking device 100 excludes that tracked person from the tracking targets. On the other hand, if the tracking device 100 determines that the tracked person carrying the product has passed through the passage gate 315 (Yes in S202), the process proceeds to S203.

[0051] In S203, the tracking device 100 determines whether the payment method stored in the payment method table (e.g., FIG. 7(c)) of the tracked person passing through the gate with the above-mentioned product is "Null." Note that "Null" is stored in the payment method table until the product payment is completed. In other words, a tracked person whose payment method in the payment method table is set to "Null" corresponds to a tracked person who has not yet completed the product payment.

[0052] If the payment method stored in the payment method table (e.g., FIG. 7(c)) of the tracked person passing through the gate with the above-mentioned product is not "Null" (No in S203), the tracking device 100 determines that the tracked person has left the store after paying, and ends the processing based on the passage information in S201. In this case, the tracking device 100 also excludes the tracked person passing through the passage gate 315 from the tracking targets. On the other hand, if the payment method stored in the payment method table (e.g., Figure 7(c)) of the tracked person passing through the gate with the above-mentioned product is "Null" (Yes in S203), the tracking device 100 proceeds to S204.

[0053] In S204, the tracking device 100 notifies the store manager device 316 of the detection of the unpaid person (the tracked person passing through the gate with the above-mentioned product). The store manager device 316, which has received the notification from the tracking device 100 that a non-payment person has been detected, notifies the manager of the detection of the non-payment person by vibrating, sounding an alarm, displaying a warning, or the like (S205).

[0054] In addition, in S206, the tracking device 100 transmits the detection time of the unsettled person (the tracked person passing through the gate with the above-mentioned product) to the surveillance camera video recording server (not shown). The surveillance camera video recording server receives notification of the detection time of the unpaid customer from the tracking device 100 and records the detection time as a timestamp linked to the shooting time (S207). This makes it possible to record image information of customers who pass through the gate with unpaid products.

[0055] As described above, the system of this embodiment can associate a customer who is being tracked and is holding a product (i.e., a customer who has not yet paid) with the video captured by a surveillance camera installed on the ceiling or other location of the store when the customer passes through a gate installed at the entrance / exit of the store. This makes it possible to identify the customer who has taken an unpaid product outside the store. In addition, by notifying the store manager when an unpaid customer passes through the gate, it becomes possible to deal with customers who are trying to take unpaid merchandise out of the store.

[0056] In the above embodiment, a configuration has been described in which a sensor provided in a showcase detects that the showcase door has been opened, and the opening of the showcase door serves as a trigger to start tracking of a customer standing in front of the showcase. However, the trigger to start tracking of a customer is not limited to the detection of the showcase door being opened. Any configuration may be used as long as it detects some customer behavior (a predetermined action performed by the customer on the showcase or a predetermined change in the showcase that occurs in response to the action) to remove an item from the showcase, and the detection serves as a trigger to start tracking of the customer.

[0057] For example, a showcase may be provided with a button or the like for opening the door and a sensor for detecting the operation of the button or the like, and the sensor's detection of the button or the like may be used as a trigger for starting customer tracking. Alternatively, a sensor may be provided with a sensor for detecting a hand touching the showcase door, and the sensor's detection of the hand touching the door may be used as a trigger for starting customer tracking. Furthermore, the showcase may be provided with an infrared sensor or the like, and the detection of a customer's hand inserting (entering) the showcase or a product being taken out of the showcase may be used as a trigger for starting customer tracking. Alternatively, a weight sensor or the like for detecting (measuring) the weight of products contained in the showcase may be provided, and the detection of a product being removed from the showcase and the weight being reduced may be used as a trigger for starting customer tracking. Furthermore, a TOF camera or the like may be provided in the showcase, and the detection of a commodity being taken out of the showcase may be used as a trigger to start tracking the customer. Furthermore, when a door sensor for each showcase detects that the showcase door has been opened, and then a TOF camera for a showcase with the same identification number detects that an item has been taken out of the showcase, this may be used as a trigger to start tracking a customer when multiple conditions for starting tracking are met. In this way, by setting multiple conditions, it is possible to further narrow down the customers in the store to be tracked.

[0058] Furthermore, when a store employee restocks products in a showcase, tracking of the employee is not necessary. For this reason, the employee may wear a specific hat, uniform, or other clothing, or an identifier such as a sash or badge, and tracking of a person whose identifier is recognized from the video captured by the TOF camera 301 may not be performed. The TOF camera 301 outputs, for example, a color image along with a distance image. Therefore, the tracking device 100, which performs tracking processing using the output from the TOF camera, can easily determine the employee's identifier and exclude them from tracking targets. This makes it possible to easily exclude people who do not need to be tracked, such as employees, from tracking targets, thereby reducing the tracking load.

[0059] Furthermore, in this embodiment, store 300 is provided with entrance / exit 302, but entrances and exits may be separated. That is, one or more entrances that only allow entry into store 300 and one or more exits that only allow exit from store 300 may be provided, and no entrances that allow entry and exit from store 300 may be provided. In this configuration, the passage information is transmitted in S201 of Fig. 6 from passage gate 315 provided at the exit. Furthermore, the store may be configured to detect customers leaving the store from the images captured by the TOF camera 301 without providing the passage gate 315.

[0060] Furthermore, the functions of the tracking device 100 may be realized by one computer or by multiple computers. For example, the functions of the tracking device 100 may be realized by using a cloud service or the like.

[0061] Furthermore, the functions of tracking device 100 may be realized by a control unit included in TOF camera 301. That is, the control unit included in TOF camera 301 may execute the above-described functions of tracking device 100, such as tracking people inside the store using video of the inside of the store captured by a camera device included in TOF camera 301. In this configuration, TOF camera 301 is configured to be able to communicate with each device of this information processing system, such as TOF camera 308, showcase 309, store payment terminal 312, passing gate 315, and store manager device 316, via a LAN or the like.

[0062] As explained above, it is possible to appropriately reduce the number of customers who need to be tracked among customers in the store. As a result, even when the number of customers increases and the store becomes crowded with many customers, it is possible to accurately track the movement of each customer as they move around in the store. This makes it possible to prevent incorrect tracking due to switching of tracked customers. Furthermore, by controlling the start of customer tracking using a sensor that can detect the opening and closing of the door on the showcase, it is possible to prevent the system from increasing in cost and realize a system with an inexpensive configuration that appropriately reduces the number of people who need to be tracked.

[0063] It goes without saying that the configurations and contents of the various data described above are not limited to those described above, and that the data may be configured in various configurations and contents depending on the application and purpose. Although one embodiment has been described above, the present invention can be embodied as, for example, a system, an apparatus, a method, a program, a storage medium, etc. Specifically, the present invention may be applied to a system made up of multiple devices, or may be applied to an apparatus made up of a single device. Furthermore, the present invention also includes any combination of the above embodiments.

[0064] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. Furthermore, the present invention may be applied to a system made up of multiple devices, or to an apparatus made up of a single device. The present invention is not limited to the above-described embodiments, and various modifications (including organic combinations of the embodiments) are possible based on the spirit of the present invention, and these modifications are not excluded from the scope of the present invention. In other words, all configurations that combine the above-described embodiments and their modifications are included in the present invention. [Explanation of symbols]

[0065] 100 Tracking device 300 stores 301 TOF camera (installed on the ceiling) 308 TOF camera (installed in the showcase) 309 Showcase 312 Merchandise payment terminal 315 Passing Gate 317 Surveillance Camera

Claims

1. a tracking means for performing a person tracking process based on an image captured by an image capturing means for capturing images of the inside of the store, and for controlling the tracking process to end when the person being the target of the tracking process makes a payment for a product; A control means for managing the number of people who are the targets of the tracking process and controlling the number of people to be reduced in response to a product payment being made by a person who is the target of the tracking process; An information processing system comprising:

2. 2. The information processing system according to claim 1, wherein said tracking means does not set said person as a tracking target when a predetermined characteristic is recognized in said person.

3. 3. The information processing system according to claim 2, wherein the case where the person is recognized as having a predetermined characteristic is a case where the person is recognized as wearing a predetermined identifier.

4. An information processing system as described in any one of claims 1 to 3, characterized in that it has a notification means for notifying a specified notification destination when the person being tracked has not made a product payment and it is recognized that the person being tracked is about to pass through the exit of the store.

5. 5. The information processing system according to claim 4, wherein the notification destination is a wearable terminal worn by a manager of the store.

6. An information processing system as described in any one of claims 1 to 5, characterized in that it has a timestamp means that, when the person to be tracked has not yet paid for a product and it is recognized that the person to be tracked is about to pass through the exit of the store, it is possible to associate the time of the person's attempt to pass through with an image taken inside the store.

7. 7. The information processing system according to claim 1, wherein the tracking means controls the person tracking process to end when it acquires information indicating completion of payment for a product.

8. The method further includes acquiring means for acquiring information indicating completion of the product payment, The information processing system according to claim 7, characterized in that the acquisition means acquires information indicating that the product payment has been completed on the condition that the person is recognized as being in front of the payment terminal performing the product payment.

9. 9. The information processing system according to claim 1, wherein the tracking process includes route management.

10. 10. The information processing system according to claim 1, wherein the tracking means controls the image capturing means so as to track a person in the store.

11. The control means stores information related to the person who is the target of the tracking process in a storage unit, An information processing system as described in any one of claims 1 to 10, characterized in that when a product payment is made by a person who is the subject of the tracking processing, information related to that person is deleted from the memory unit.

12. A detection means is provided for detecting an action of removing a product from a display area, 12. The information processing system according to claim 1, wherein the tracking process is initiated in response to the detection of the behavior by the detection means.

13. The information processing system according to claim 12, characterized in that the tracking process is not initiated if the person is recognized as entering the store but does not detect any attempt to remove a product from the display area.

14. 14. The information processing system according to claim 12, wherein the detection means detects the behavior based on an image captured by a photographing means provided in the display location.

15. the display area has a door, and by opening the door, the products displayed in the display area can be taken out; The information processing system according to claim 12 or 13, characterized in that the detection means detects an action of attempting to remove a product from the display area by detecting at least one of the following: the door being opened, a person touching the door, or a button for opening the door being operated.

16. The tracking means of the information processing system, a tracking step of performing a person tracking process based on an image captured by an image capturing means for capturing images inside the store, and controlling the tracking process of the person to be terminated when the person being tracked makes a payment for a product; A control step in which a control means of the information processing system manages the number of people who are the targets of the tracking process, and controls to reduce the number of people in response to a product payment being made by the person who is the target of the tracking process; An information processing method comprising:

17. A program for causing at least one computer to function as each means of the information processing system according to any one of claims 1 to 15.

Images