Information processing system, information processing method, and information processing program
A video-based method tracks customer flow lines and product acquisition in retail stores to detect anomalies, addressing high costs and privacy issues, enhancing fraud detection efficiency while protecting customer privacy.
Patent Information
- Application Number
- JP2022136416
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-08-30
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2038-01-31
AI Technical Summary
Existing methods for preventing shoplifting in retail stores, such as RFID tagging and customer monitoring, face issues of high costs and privacy concerns.
An information processing method that tracks customer flow lines and product acquisition status using video data, linking customer movement with product counts without identifying individuals, and storing this information for comparison with POS data to detect anomalies.
Accurately measures product acquisition without infringing on customer privacy, enabling effective fraud detection and reducing the need for continuous human monitoring.
Smart Images

Figure 0007590382000001 
Figure 0007590382000002 
Figure 0007590382000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an information processing method, an information processing device, and an information processing program, and to a technique for processing video data. [Background technology]
[0002] Convenience stores, supermarkets, and other retail stores are equipped with surveillance cameras to record images of the inside of the stores. Generally, store clerks or security guards monitor the images on a monitor to detect crimes and fraud such as shoplifting. However, there are other methods to prevent shoplifting.
[0003] For example, Patent Document 1 discloses a method of placing RFID (Radio Frequency Identification) tags on products. However, the method described in Patent Document 1 has the problem of high costs because it is necessary to attach RFID tags to all products. In addition, it is not possible to obtain information about which customers have acquired the products.
[0004] There is also a method of monitoring customers using technology that tracks customer movements (for example, Patent Document 2). In Patent Document 3, information on the number of items acquired by a customer is linked to an image of the customer and managed. When the customer pays for the items they have acquired, the number of items registered in the POS (Point of Sales) terminal is compared with the number of items acquired by the customer. If these numbers do not match, it is determined that there is a possibility of fraud.
[0005] The method disclosed in Patent Document 3 has the advantage that it does not require store clerks or security guards to continuously monitor the video. [Prior art documents] [Patent documents]
[0006] [Patent Document 1] JP 2007-079615 A [Patent Document 2] JP 2011-170562 A [Patent Document 3] JP 2004-171241 A Summary of the Invention [Problem to be solved by the invention]
[0007] However, the method disclosed in Patent Document 3 has a problem regarding privacy, since images of customers are managed without the customers' permission.
[0008] An object of the present invention is to accurately grasp the product acquisition status of each customer without identifying individuals, while taking into consideration the privacy of the customers. [Means for solving the problem]
[0009] An information processing method according to one aspect of the present invention acquires customer movement line information from video, detects that the customer has acquired a product, links the customer movement line information with information on the number of products acquired by the customer, and stores the information in a memory means.
[0010] An information processing device according to one aspect of the present invention includes an information acquisition means for acquiring customer movement line information from video, a behavior detection means for detecting that the customer has acquired a product, and a recording means for linking the customer movement line information with information on the number of products acquired by the customer and storing the information in a memory means.
[0011] An information processing program according to one aspect of the present invention causes a computer to acquire customer movement information from video, detect when the customer has acquired a product, and link the customer movement information with information on the number of products acquired by the customer and store the information in a memory means. Effect of the Invention
[0012] According to one aspect of the present invention, it is possible to accurately grasp the product acquisition status of each customer without identifying the individual. [Brief description of the drawings]
[0013] [Figure 1] 1 is a block diagram showing a configuration of a store monitoring device according to a first embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of a DB (Data Base) of flow line-acquired score information stored in a storage unit of the in-store monitoring device according to the first embodiment. [Diagram 3] 5 is a flowchart showing a first example of the operation of the in-store monitoring device according to the first embodiment. [Figure 4] 10 is a flowchart showing a second example of the operation of the in-store monitoring device according to the first embodiment. [Diagram 5] FIG. 11 is a block diagram showing the configuration of a store monitoring device according to a second embodiment. [Figure 6] FIG. 11 is a diagram showing a hardware configuration of a store monitoring device according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] [Embodiment 1] The configuration and operation of the in-store monitoring device according to this embodiment will be described with reference to FIGS.
[0015] (Configuration of in-store monitoring device 1) Fig. 1 is a block diagram showing the configuration of a store monitoring device 1 according to this embodiment. As shown in Fig. 1, the store monitoring device 1 includes an imaging unit 11, a movement line analysis unit 12, a behavior in front of the shelf detection unit 13, an abnormality detection unit 14, a notification unit 15, and a memory unit 16.
[0016] The imaging unit 11 captures images of the inside of the store and generates video (video) data. The imaging unit 11 may include one or more cameras for capturing images. The imaging unit 11 transmits video data (pixel data constituting a video frame) obtained by capturing images of the inside of the store to the movement line analysis unit 12, the in-front-shelf behavior detection unit 13, and the anomaly detection unit 14. The memory unit 16 stores movement line-acquired score information.
[0017] The information on customer movement and number of items acquired is information that links customer movement information with information on the number of items acquired by the customer in the store. Note that the number of items acquired by the customer here means the number of items that the customer picked up from the shelves and possesses.
[0018] Fig. 2 is a diagram showing an example of a DB of flow line-acquired point information stored in the memory unit 16. As shown in Fig. 3, in the DB of flow line-acquired point information, for example, an ID (flow line ID) that identifies a customer's flow line, chronological position information of the customer (time / coordinates), and the number of items acquired by the customer may be associated with each other. In Fig. 2, the chronological position of the customer is represented by 3D viewpoint (view) coordinates as seen from the camera of the imaging unit 11. The flow line-acquired point information is not limited to the DB shown in Fig. 2.
[0019] The flow line information includes information on the change in the position of a moving object (here, a customer) over time. The path that a moving object traces is the flow line.
[0020] The operations of the movement line analysis unit 12, the in-shelf behavior detection unit 13, the abnormality detection unit 14, and the notification unit 15 will be described later.
[0021] The store monitoring device 1 does not necessarily have to include the imaging unit 11 and the storage unit 16. In this case, the store monitoring device 1 acquires video data captured by an imaging device (e.g., a camera). The store monitoring device 1 also stores the flow line-acquired score information in a storage device (e.g., a memory).
[0022] (First abnormality detection process) The flow of a first abnormality detection process will be described with reference to Fig. 3 as an example of the abnormality detection process executed by the store monitoring device 1. Fig. 3 is a flow chart showing the flow of the first abnormality detection process.
[0023] As shown in Fig. 3, in the first anomaly detection process, the movement line analysis unit 12 tracks the movement line of customers using the video data received from the imaging unit 11 (S101). More specifically, the movement line analysis unit 12 detects customers from each frame constituting the video at regular intervals, and generates and outputs customer movement line information by analyzing changes in the positions of the detected customers over time. A technology for tracking the movement line of a moving object is disclosed in, for example, Patent Document 2. In this embodiment, a detailed description of the technology for tracking the movement line of customers is omitted.
[0024] The in-shelf behavior detection unit 13 detects the behavior of the customer with respect to the shelf (gondola) by analyzing the video data received from the imaging unit 11, and counts the number of items the customer has acquired from the shelf (S102). A technique for detecting the behavior of a customer acquiring products from a shelf or returning products to the shelf (in-shelf behavior) is disclosed in, for example, Patent Document 3. In this embodiment, a detailed description of the technique for detecting in-shelf behavior will be omitted.
[0025] The in-shelf behavior detection unit 13 links the customer movement line information generated by the movement line analysis unit 12 with information on the number of items acquired by the customer from the shelves, and stores the information in the memory unit 16 as movement line-acquired item number information (S103).
[0026] The abnormality detection unit 14 detects that a customer has entered the area in front of the register. In the area in front of the register, customers are lined up to pay for the products they have acquired. The position and shape of the area where customers are lined up to pay for the products vary depending on the structure of the store. Therefore, the area in front of the register for the abnormality detection unit 14 to detect people may be set for each store.
[0027] When it is detected that a customer has moved into the area in front of the register (Yes in S104), the abnormality detection unit 14 refers to the flow line-acquired point information stored in the memory unit 16, identifies the flow line corresponding to the customer in the front row among the customers in the area in front of the register, and acquires information on the number of items acquired by the customer in the front row (S105). Hereinafter, the customer in the front row among the customers in the area in front of the register will be referred to as the judgment target.
[0028] The abnormality detection unit 14 also acquires information on the number of items registered in the POS terminal after the object of judgment moves into the area in front of the register. Then, the abnormality detection unit 14 compares the number of items registered in the POS terminal with the number of items acquired by the object of judgment (S106).
[0029] In the first abnormality detection process, the abnormality detection unit 14 detects, as an abnormality, the discrepancy between the number of items acquired by the object of judgment and the number of items registered in the POS terminal. If the number of items acquired by the object of judgment does not match the number of items registered in the POS terminal (No in S107), the abnormality detection unit 14 causes the notification unit 15 to notify the abnormality (S108). The notification unit 15 may transmit an abnormality signal (alert), for example, to a mobile terminal carried by the security guard or a terminal in the office.
[0030] In one variant, if the difference between the number of items acquired by the object of judgment and the number of items registered in the POS terminal exceeds a certain number (threshold) greater than 1, the abnormality detection unit 14 may cause the notification unit 15 to notify the abnormality.
[0031] (Second abnormality detection process) With reference to Fig. 4, the flow of a second abnormality detection process will be described as another example of the abnormality detection process executed by the store monitoring device 1. Fig. 4 is a flow chart showing the flow of the second abnormality detection process.
[0032] As shown in FIG. 4, in the second abnormality detection process, the movement line analysis unit 12 tracks the movement line of a customer based on the video image received from the imaging unit 11 (S201).
[0033] The in-shelf behavior detection unit 13 detects the behavior of the customer with respect to the shelves (gondola) from the video received from the imaging unit 11, and counts the number of products that the customer has acquired from the shelves (S202).
[0034] The movement line analysis unit 12 and the shelf front behavior detection unit 13 link the customer movement line information with the information on the number of products acquired by the customer from the shelf, and store the information as movement line-acquired product information in the storage unit 16 (S203). S201 to S203 of the second abnormality detection process are the same as S101 to S103 of the first abnormality detection process.
[0035] The abnormality detection unit 14 detects that a customer has left a specific area. The specific area is, for example, an area where shelves displaying products are arranged, an area in front of the cash register, and other areas where customers are permitted to have unpaid products. Instead of detecting that a customer has left a specific area, the abnormality detection unit 14 may detect that a customer has crossed a specific line. The specific line is the boundary between an area where customers are permitted to have unpaid products and an area where customers are not permitted to have unpaid products.
[0036] When it is detected that a customer has left the specific area (Yes in S204), the abnormality detection unit 14 refers to the movement line-acquired item number information stored in the memory unit 16 to identify the movement line corresponding to the customer who has left the specific area, and acquires information on the number of items acquired by the customer from the memory unit 16 (S205).
[0037] In the second abnormality detection process, the abnormality detection unit 14 detects, as an abnormality, that a customer leaves a specific area while acquiring unpaid merchandise. When a customer leaves a specific area while acquiring unpaid merchandise (Yes in S206), the abnormality detection unit 14 causes the notification unit 15 to notify the abnormality (S207). The method of notification is not particularly limited. The notification unit 15 may send an abnormality signal (alert), for example, to a mobile terminal carried by a security guard or a terminal in the office.
[0038] (Modification) In one modified example, the store monitoring device 1 does not analyze customer movement lines and detect behavior in front of shelves, and an external device (e.g., a network server) may instead perform these functions. In this modified example, the store monitoring device 1 receives the results of the movement line analysis and the detection of behavior in front of shelves from the external device. Alternatively, the external device may generate movement line-obtained score information and store it in the memory unit 16, and the store monitoring device 1 may acquire the movement line-obtained score information stored in the memory unit 16 by the external device.
[0039] According to the configuration of this modified example, the store monitoring device 1 does not need to include the flow line analysis unit 12 and the in-shelf behavior detection unit 13. That is, it is possible to reduce the computer resources required for the processing of customer flow line analysis (S101 in FIG. 3) and in-shelf behavior detection (S102 in FIG. 3).
[0040] (Effects of this embodiment) According to the configuration of this embodiment, customer movement information and information on the number of products acquired by the customer are linked and stored. By using the stored information, it is possible to measure with a simple configuration and accurately how many products each customer acquired.
[0041] This makes it possible to detect suspicious customer behavior, such as customers leaving the store with unpaid items in their hands. Furthermore, since the system does not manage footage of customers, it is possible to appropriately protect customer privacy.
[0042] In addition, when suspicious behavior is detected, the system can alert security guards or store clerks to the abnormality and prompt them to take action. Therefore, security guards and store clerks do not need to continuously monitor the video in order to detect fraud. This reduces the burden on security guards and store clerks and prevents them from overlooking fraud.
[0043] [Embodiment 2] The in-store monitoring device according to this embodiment will be described with reference to FIG.
[0044] (Configuration of in-store monitoring device 2) Fig. 5 is a block diagram showing the configuration of a store monitoring device 2 according to this embodiment. As shown in Fig. 5, the store monitoring device 2 includes an information acquisition unit 21, a behavior detection unit 22, and a recording unit 23. Although not shown, the store monitoring device 2 acquires video data captured by an external imaging device (e.g., a camera). The store monitoring device 2 also stores the flow line-acquired score information in an external storage device (e.g., a memory). Alternatively, the store monitoring device 2 may include an imaging device and a storage device.
[0045] The information acquisition unit 21 acquires customer movement line information generated by analyzing video data captured by an external imaging device. A technology for tracking customer movement lines in a video is disclosed in, for example, Patent Document 2.
[0046] The behavior detection unit 22 detects that a customer has acquired a commodity in the store by using a technique for detecting behavior in front of a shelf (for example, Patent Document 3).
[0047] The recording unit 23 associates the customer movement line information with the information on the number of items acquired by the customer, and stores the information in a storage unit (not shown) as the above-mentioned movement line-acquired item number information. The recording unit 23 in this embodiment corresponds to a part of the movement line analysis unit 12 and the in-shelf behavior detection unit 13 in the first embodiment. The in-store monitoring device 2 according to one modified example does not perform customer movement line analysis and in-shelf behavior detection, and an external device (e.g., a network server) may perform customer movement line analysis and in-shelf behavior detection instead. In this modified example, the recording unit 23 acquires the results of the customer movement line analysis and in-shelf behavior detection performed by the external device, and stores them in a storage unit (not shown).
[0048] (Effects of this embodiment) According to the configuration of this embodiment, customer movement line information and information on the number of items acquired by the customer are linked and stored in a storage means. Therefore, by referring to the information stored in the storage means (movement line-acquired item information), it is possible to accurately measure which customer has acquired how many items. The information on the number of items measured in this way can be used, for example, to prevent shoplifting, because it is possible to know that a customer is trying to take out of the store an item that has not been paid for. Furthermore, unlike an image of a customer, customer movement line information does not include information related to privacy. Therefore, it is possible to appropriately protect the privacy of customers.
[0049] [Embodiment 3] The in-store monitoring device according to this embodiment will be described with reference to FIG.
[0050] (Configuration of in-store monitoring device 3) 6 is a diagram showing the configuration of the in-store monitoring device 3 according to this embodiment. The in-store monitoring device 3 is realized as hardware by a computer device. The in-store monitoring device 3 includes a CPU (Central Processing Unit) 31, a RAM (Random Access Memory) 32, a storage device 33, an input / output device 34, and a communication interface 35.
[0051] The functions of the in-store monitoring device 3 are the same as those of the in-store monitoring device 1 or 2 of the above-mentioned embodiment 1 or 2. In other words, the in-store monitoring device 3 realizes the operation of the functional blocks included in the in-store monitoring device 1 or 2 of the above-mentioned embodiment 1 or 2. The functions of the in-store monitoring device 3 are realized by the CPU 31 executing a program that has been loaded into the RAM 31.
[0052] The storage device 33 includes the storage unit 16 of the embodiment 1. The storage device 33 stores flow line-acquired score information.
[0053] The input / output device 34 includes the notification unit 15 of the embodiment 1. The input / output device 34 may include a user interface such as a display.
[0054] The communication interface 35 is used to acquire video data from an external imaging device.
[0055] (Effects of this embodiment) According to the configuration of this embodiment, the functions of the in-store monitoring device described in the first or second embodiment are realized as hardware using computer resources such as a CPU. Therefore, it is possible to accurately measure how many products each customer has acquired while taking into consideration the privacy of the customers. [Explanation of symbols]
[0056] 1, 2, 3 In-store monitoring equipment 12 Flow line analysis department 13 Shelf front action detection unit 14 Anomaly detection section 15. Notification Department 16 Memory section 21 Information Acquisition Department 22 Behavior detection unit 23 Recording Department
Claims
1. An information acquisition means for acquiring customer movement line information from the video; A behavior detection means for detecting that the customer has acquired a product; A recording means for storing the customer's movement line information and the information on the number of items acquired by the customer in association with each other; a detection means for detecting that a customer has entered an area in front of the register where customers are lined up to pay for the acquired merchandise at the register, and detecting that a first merchandise quantity linked to flow line information corresponding to the customer at the front of the queue does not match a second merchandise quantity paid for by the customer at the register; An information processing system comprising:
2. 2. The information processing system according to claim 1, further comprising a notification means for notifying that a discrepancy between the number of items acquired by the customer and the number of items paid for by the customer at the register has been detected.
3. Acquire customer movement information from the video, Detecting that the customer has acquired the product; The information on the customer's movement line and the information on the number of products acquired by the customer are associated with each other and stored. It is detected that a customer has entered the area in front of the register where customers line up to pay for the acquired products at the register, Detecting whether a first number of items associated with flow line information corresponding to a customer at the front of the queue at the register does not match a second number of items paid for by the customer at the register 23. An information processing method comprising:
4. Acquiring customer movement information from the video; Detecting that the customer has acquired a product; storing the customer's movement line information and the information on the number of products acquired by the customer in association with each other; Detecting that a customer has entered an area in front of the register where customers line up to pay for the acquired products at the register; An information processing program for causing a computer to detect a discrepancy between a first number of items linked to traffic flow information corresponding to a customer at the front of the queue at the register and a second number of items paid for by the customer at the register.
Citation Information
Patent Citations
Illegality monitoring system and program
JP2004171241A
Merchandise management system
JP2007079615A
Security management system and security management method
JP2009009231A
Traffic line association method, device, and program
JP2011170562A
Information processing apparatus, information processing system, and program
JP2015001879A
