Surveillance system

Abstract

To provide a monitoring device capable of efficiently monitoring fraudulent acts of taking out unaccounted commodities from a shop where a system of causing customers themselves to perform accounting is applied, and a program.SOLUTION: The monitoring device includes: imaging means for imaging each of a first area positioned at an entrance, a second area where customers themselves perform accounting of commodities, and a third area positioned at an exit in a checkout area related to registration and accounting of commodities; extraction means for extracting feature information expressing features of a customer from an image captured at each of the areas; tracing means for tracing a moving route through which the same customer reaches the third area from the first area on the basis of a similarity between feature information extracted from the captured image of the first area and feature information extracted from the captured image of each of the second and third areas; and notification means for making a notification when the moving route has reached the third area from the first area, skipping the second area.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] An embodiment of the present invention comprises: Surveillance system Regarding. [Background technology]

[0002] Conventionally, in stores such as supermarkets, product registration and payment are performed by store clerks operating Point Of Sale (POS) terminals. Also, stores use checkout methods called semi-self-service, in which the clerk registers products and the customer pays for them themselves, and checkout methods called self-service, in which the customer registers products and pays for them themselves.

[0003] However, in the above-mentioned semi-self-service and self-service convenience stores, the payment for the products is left to the customer, so there is a possibility of fraudulent behavior called "basket skipping," where the customer leaves the store without paying. A conventional technology is known for semi-self-service convenience stores that provides a first imaging device for capturing an image of the customer who has requested product registration and a second imaging device for capturing an image of the vicinity of the exit, etc., and compares the images of the customer captured by these imaging devices to determine whether or not a customer who has not yet paid is near the exit.

[0004] However, in the conventional technology described above, the determination of whether or not a transaction has been completed is made based on a completion notification sent from the accounting device, rather than using captured images. Therefore, in a surveillance system based on comparing captured images, the conventional technology described above is not efficient, and there is room for further improvement. Summary of the Invention [Problem to be solved by the invention]

[0005] The problem to be solved by the present invention is to provide a system that can efficiently monitor fraudulent acts such as taking unpaid merchandise out of a store in a store where customers pay by themselves. Surveillance system The purpose of this project is to provide [Means for solving the problem]

[0006] The surveillance system of the embodiment includes a first imaging means for imaging customers in a first area where products are registered, a second imaging means for imaging customers in a second area where the customers who have completed their payment are to pass through, and a third imaging means for imaging a third area where the customers who have completed their payment can pass through. The surveillance system identifies the customer who appears most frequently from the images captured for each transaction by the first imaging means, and selects the image in which the area of ​​the face region of the customer is the largest. No. 1 A selection means; a second selection means for identifying the customer who appears most frequently from among the captured images for each transaction captured by the second imaging means, and selecting the captured image in which the area of ​​the face region of the identified customer is the largest; The above No. 1 Selection method and the second selection means Feature information extracted from the captured image selected in step 2; The above Third imaging means of The system includes a tracking means for tracking the movement path of the same customer from the first area to the third area based on the similarity with feature information extracted from the captured image, and an alarm means for issuing an alarm when the movement path indicates that the customer has traveled from the first area to the third area by skipping the second area. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of a store layout according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of the monitoring system according to the first embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of the configuration of the registration device according to the first embodiment. [Figure 4] FIG. 4 is a flowchart illustrating an example of a transmission image selection process executed by the registration device according to the first embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of the configuration of the accounting device according to the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of the configuration of the monitoring server according to the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a data structure of the tracing table. [Figure 8] FIG. 8 is a flowchart illustrating an example of the monitoring process executed by the monitoring server according to the first embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of a store layout according to the second embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of the configuration of a monitoring system according to the second embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of the configuration of the monitoring server according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of a monitoring device and a program according to the present invention will be described in detail with reference to the drawings. In the embodiment shown below, an example in which the present invention is applied to a store such as a supermarket will be described, but the present invention is not limited to this embodiment.

[0009] [First embodiment] Fig. 1 is a diagram showing a schematic example of a store layout according to the first embodiment. As shown in Fig. 1, the store has a checkout area A1 for registering products and making payments. The checkout area A1 includes a registration area A11, a payment area A12, and an exit area A13.

[0010] The registration area A11 is a first area located at the entrance of the checkout area A1, and is provided with a registration device 10. The checkout area A12 is a second area where customers check out for merchandise, and is provided with a checkout device 20. The exit area A13 is a third area located at the exit of the checkout area A1, and is provided with a store exit B11. Although not shown, it is assumed that the store has an area where merchandise for sale is displayed (for example, above the checkout area A1).

[0011] The registration device 10 is a product data processing device that is operated by a store clerk and performs registration processing of products purchased by a customer. The accounting device 20 is a product data processing device that is operated by a customer and performs accounting processing of products registered by the registration device 10. In other words, the registration device 10 and the accounting device 20 realize a checkout method called semi-self-service or the like.

[0012] In a store with the layout shown in FIG. 1, a customer's actions when purchasing an item are as follows. First, the customer places the items to be purchased in a shopping cart or the like and carries it to the registration area A11 (registration device 10). In the registration device 10, a store clerk registers the items by having a reading unit 14 (see FIG. 3), which will be described later, read the barcode attached to the item. Once registration of all items for one transaction has been completed, the store clerk selects (designates) one accounting device 20 to be used for payment, and notifies the customer of that accounting device 20 to guide them to the accounting device. The registration device 10 also transmits accounting information, including information such as the unit price and quantity of each registered item, to the designated accounting device 20.

[0013] Once the customer has completed registering the products, they move to the payment area A12 with the products (shopping basket) and complete the payment at the payment device 20 designated by the store clerk. At that time, the payment device 20 performs the payment process based on the payment information transmitted in advance from the registration device 10. Then, once the payment is completed, the customer places the purchased products in a shopping bag or the like and moves to the exit area A13 (exit B11).

[0014] In this way, in the semi-self-service system, the registration and payment tasks can be shared, which can improve the efficiency of the process by shortening the waiting time at the register, etc. However, in the semi-self-service system, because the payment is left to the customer themselves, there is a possibility of fraudulent behavior, such as basket skipping, in which products are taken out of the store without being paid for.

[0015] Therefore, in the surveillance system of this embodiment, images of the registration area A11, the accounting area A12, and the exit area A13 are captured, and customers who may be committing fraudulent acts are detected based on the captured images. The configuration of the surveillance system according to this embodiment will be described below.

[0016] 2 is a diagram showing a schematic example of the configuration of a monitoring system according to the first embodiment. As shown in the diagram, the monitoring system includes the above-mentioned registration device 10 and accounting device 20, as well as a first camera 31, a second camera 32, a third camera 33, and a monitoring server 40. The registration device 10, accounting device 20, third camera 33, and monitoring server 40 are connected to a network N1 such as a LAN (Local Area network).

[0017] The first camera 31 is an imaging device that captures an image of a customer present in the registration area A11 (registration device 10). The first camera 31 is provided in each of the registration devices 10 (see FIG. 1). More preferably, the first camera 31 is provided in a position in each of the registration devices 10 where it can capture an image of the face of the customer. The registration device 10 and the first camera 31 are connected by a connection line L1. The installation position of the first camera 31 is not particularly important, and it may be provided integrally with the registration device 10, or may be provided on the ceiling of the store in the form of a surveillance camera or the like.

[0018] The registration device 10 cooperates with the first camera 31 to transmit a captured image (hereinafter referred to as a first captured image) captured during the registration process to the surveillance server 40. Note that the concept of a captured image includes still images and moving images.

[0019] Fig. 3 is a diagram showing an example of the configuration of the registration device 10 according to the first embodiment. As shown in Fig. 3, the registration device 10 includes a control unit 11. The control unit 11 has a computer configuration including a CPU (Central Processing Unit), a SoC (System-on-a-Chip), a ROM (Read Only Memory), and a RAM (Random Access Memory).

[0020] The control unit 11 is connected to a display unit 12, an operation unit 13, a reading unit 14, a storage unit 15, a connection interface 16, a communication interface 17, and the like via a bus or the like.

[0021] The display unit 12 has a display device such as a liquid crystal display. Under the control of the control unit 11, the display unit 12 displays various information such as the product names and product codes of registered products. The operation unit 13 has various operation keys and a pointing device, and accepts operations by an operator (store clerk). For example, the operation unit 13 has operation keys for instructing the start and completion of the registration process, and number keys for inputting product codes, the number of products, etc. The operation unit 13 may be a touch panel provided on the display surface of the display unit 12.

[0022] The reading unit 14 is a code reader capable of reading code symbols such as barcodes and two-dimensional codes. For example, the reading unit 14 reads a product code stored in a code symbol attached to a product. Here, the product code is identification information for identifying a product, and unique identification information is assigned to each type of product.

[0023] The storage unit 15 is a storage device such as an HDD (Hard Disk Drive) or a flash memory. The storage unit 15 stores programs executed by the control unit 11 and various data used in executing the programs. For example, the storage unit 15 stores in advance a product master (not shown) that associates the product code of each product with product information including the product name and unit price of the product. The storage unit 15 may also store in advance captured images of the faces of each store clerk and feature information representing facial features extracted from the captured images in order to distinguish between the faces of the store clerk and the faces of customers.

[0024] The connection interface 16 is an interface that can be connected to the first camera 31. The connection interface 16 accepts a first captured image input from the first camera 31. The communication interface 17 is a communication interface that can be connected to the network N1. The communication interface 17 transmits and receives various information to and from external devices (e.g., the accounting device 20 and the monitoring server 40) connected to the network N1.

[0025] In the registration device 10 configured as above, when a store clerk operates to read a code symbol attached to a product to be purchased, the control unit 11 registers the product by storing the product code read by the reading unit 14 in a RAM or the like. When the quantity of the product is input, the control unit 11 registers the quantity in association with the product code. The control unit 11 also acquires a first captured image captured by the first camera 31 while the product registration process is being performed.

[0026] Once registration of products for one transaction is complete, the control unit 11 generates transaction information based on the product codes and points registered up to that point. Here, transaction information includes, for example, the product name and price of each product (product code), the total payment amount, etc. Note that the product name and price are derived based on the product information registered in the product master.

[0027] Furthermore, when a specific accounting device 20 is selected (designated) by a store clerk via the operation unit 13 or the like, the control unit 11 transmits product information to that accounting device 20. The control unit 11 then transmits the first captured images acquired during the registration process to the monitoring server 40. Note that there is no particular restriction on the number of first captured images (still images) or the number of hours (video) to be transmitted to the monitoring server 40. Furthermore, the selection of the accounting device 20 may be configured so that the control unit 11 automatically selects the accounting device 20 based on the availability of the accounting device 20, etc.

[0028] Incidentally, the first captured image serves as a reference image for identifying (recognizing) the same customer in the monitoring process described below. For this reason, it is preferable for the control unit 11 to transmit to the monitoring server 40 a first captured image in which a characteristic part of the customer, such as the face, is captured. Therefore, for example, the control unit 11 may be configured to select a first captured image to be transmitted to the monitoring server 40 from among the first captured images captured during the registration process, based on the face area, the frequency with which the face of the same person appears, and the like. The process of selecting the first captured image will be described below.

[0029] 4 is a flowchart showing an example of a transmission image selection process executed by the registration device 10 of the first embodiment. This process is executed in the background of the registration process. In addition, in this process, known techniques are used for face area detection, face recognition, and the like.

[0030] First, when the control unit 11 receives an instruction to start registration via the operation unit 13 or the like (step S11), the control unit 11 causes the first camera 31 to start capturing an image (step S12).

[0031] Next, the control unit 11 determines whether or not a face area has been detected from the first captured image input from the first camera 31 (step S13). If a face area cannot be detected (step S13; No), the control unit 11 proceeds to step S17. Furthermore, if the control unit 11 detects a face area from the first captured image (step S13; Yes), the control unit 11 compares the features of the face area with the features of the face areas of each store clerk stored in the storage unit 15 to determine whether the captured person is a store clerk (step S14).

[0032] If the captured person is a store clerk (step S14; Yes), the control unit 11 returns the process to step S13. If the captured person is not a store clerk, that is, a customer (step S14; No), the control unit 11 derives the area of ​​the face region (step S15) and temporarily stores the first captured image in descending (or ascending) order of area in a RAM or the like (step S16).

[0033] Next, the control unit 11 judges whether or not an instruction to complete the registration has been given via the operation unit 13 or the like (step S17). If an instruction to complete the registration has not been given (step S17; No), the control unit 11 returns the process to step S13. As a result, while the product is being registered, the first captured images including the customer's facial area captured by the first camera 31 are temporarily stored in order of the area of ​​the facial area.

[0034] When the control unit 11 receives an instruction to complete registration (step S17; Yes), it compares the features of the face area between the temporarily stored first captured images to identify the same person (customer). Next, the control unit 11 identifies the face of the customer that appears most frequently among the identified customers (step S18). For example, the first captured images captured by the first camera 31 may include customers other than the customer involved in the transaction (e.g., customers waiting in line, etc.). Thus, in step S18, the control unit 11 recognizes the same customer included in each of the first captured images, and identifies the customer that appears most frequently as the customer involved in the transaction.

[0035] Next, the control unit 11 selects the first captured image in which the area of ​​the face region of the customer identified in step S18 is the largest from the first captured images including the face region (step S19).Then, the control unit 11 transmits the first captured image selected in step S19 to the monitoring server 40 (step S20), and ends this process.

[0036] Through the above process, the registration device 10 can transmit the first captured image showing the customer's features (face) to the surveillance server 40. This can improve the accuracy of customer recognition in the surveillance process described below.

[0037] Returning to FIG. 2, the second camera 32 is an imaging device that captures images of customers present in the payment area A12. The second camera 32 is provided for each payment device 20. More preferably, the second camera 32 is provided at a position at each payment device 20 where it is possible to capture an image of the face of the customer. The payment device 20 and the second camera 32 are connected by a connection line L2. There is no particular restriction on the installation position of the second camera 32, and it may be provided integrally with the payment device 20, or may be provided on the ceiling of the store in the form of a surveillance camera or the like.

[0038] The accounting device 20 cooperates with the second camera 32 to transmit to the surveillance server 40 an image captured during the accounting process (hereinafter referred to as a second captured image).

[0039] Fig. 5 is a diagram showing an example of the configuration of the transaction device 20 according to the first embodiment. As shown in Fig. 5, the transaction device 20 includes a control unit 21. The control unit 21 has a computer configuration including a CPU, SoC, ROM, RAM, etc.

[0040] The control unit 21 is connected to a display unit 22, an operation unit 23, a printing unit 24, a storage unit 25, a connection interface 26, a communication interface 27, etc. via a bus or the like.

[0041] Display unit 22 has a display device such as a liquid crystal display. Under the control of control unit 21, display unit 22 displays various information such as transaction information. Operation unit 23 has various operation keys and a pointing device, and accepts operations by the operator (customer). For example, operation unit 23 has operation keys for instructing the start and completion of transaction processing. Note that operation unit 23 may be a touch panel provided on the display surface of display unit 22.

[0042] The printing unit 24 is a printing device such as a thermal printer. Under the control of the control unit 21, the printing unit 24 prints details of the transaction (accounting information) on a paper medium such as receipt paper. The memory unit 25 is a storage device such as a HDD or flash memory. The memory unit 25 stores programs executed by the control unit 21 and various data used to execute the programs.

[0043] The connection interface 26 is an interface that can be connected to the second camera 32. The connection interface 26 accepts a second captured image input from the second camera 32. The communication interface 27 is a communication interface that can be connected to the network N1. The communication interface 27 transmits and receives various information to and from external devices (e.g., the registration device 10 and the monitoring server 40) connected to the network N1.

[0044] In addition, the control unit 21 is connected via a bus or the like to a money deposit device that accepts deposits of coins and bills, a money dispensing device that dispenses change, and the like (neither of which is shown).

[0045] In the accounting device 20 configured as described above, the control unit 21 temporarily stores the transaction information sent from the registration device 10 in RAM or the like, and waits for the start of transaction processing related to that transaction information. When a customer moves to the accounting device 20 instructed by the clerk at the registration device 10, they instruct the start of transaction processing via the operation unit 23. When the start of transaction processing is instructed, the control unit 21 accepts payment (deposit) for the product prices based on the temporarily stored transaction information. Then, when the payment is completed, the control unit 21 causes the printing unit 24 to output receipt paper on which the details are printed, and ends the transaction processing.

[0046] Additionally, the control unit 21 acquires second captured images captured by the second camera 32 while the transaction is being processed. Then, when the transaction is completed, the control unit 21 transmits the second captured images acquired during the transaction to the monitoring server 40. Here, the number of second captured images (still images) and the number of hours (video) transmitted to the monitoring server 40 are not particularly important. The control unit 21 may select the second captured images to transmit to the monitoring server 40 based on the area and appearance frequency of the face region, by performing a transmission image selection process similar to that of the registration device 10 described above.

[0047] Returning to Fig. 2, the third camera 33 is an imaging device that captures images of customers present in the exit area A13. The third camera 33 is provided at a position where it is possible to capture images of the faces of customers passing through the exit B11. The third camera 33 transmits the captured image (hereinafter referred to as the third captured image) to the surveillance server 40. The installation position of the third camera 33 is not particularly important, and it may be provided on the ceiling of the store in the form of a surveillance camera or the like.

[0048] The monitoring server 40 is a server device that monitors the behavior of customers in the checkout area A1 based on images captured by the first camera 31, the second camera 32, and the third camera 33. The monitoring server 40 will be described below.

[0049] Fig. 6 is a diagram showing an example of the configuration of the monitoring server 40 according to the first embodiment. As shown in Fig. 6, the monitoring server 40 includes a control unit 41. The control unit 41 has a computer configuration including a CPU, an SoC, a ROM, a RAM, and the like.

[0050] The control unit 41 is connected to the memory unit 42 via a bus or the like. The memory unit 42 is a storage device such as a HDD or a flash memory. The memory unit 42 stores programs executed by the control unit 41 and various data used in executing the programs. The memory unit 42 also stores a tracking table T1 (see FIG. 7) for tracking the movement path of a customer in the checkout area A1.

[0051] In addition, a communication interface (I / F) 43 is connected to the control unit 41 via a bus or the like. The communication interface 43 is connected to the network N1, and transmits and receives various types of information to and from other devices (such as the registration device 10, the accounting device 20, and the third camera 33).

[0052] 6, the control unit 41 includes functional units such as an image acquisition unit 411, a feature extraction unit 412, a registration processing unit 413, a tracking processing unit 414, a fraud detection unit 415, and a notification processing unit 416. These functional units are realized in software by executing a program stored in the storage unit 42, or in hardware by a dedicated processor or the like included in the control unit 41.

[0053] The image acquisition unit 411 acquires captured images captured by each of the first camera 31, the second camera 32, and the third camera 33 via the communication interface 43. More specifically, the image acquisition unit 411 acquires a first captured image transmitted from each of the registration devices 10. The image acquisition unit 411 also acquires a second captured image transmitted from each of the accounting devices 20. The image acquisition unit 411 also acquires a third captured image transmitted from the third camera 33.

[0054] The feature extraction unit 412 extracts feature information representing features of a person (customer) included in each of the captured images (first captured image, second captured image, third captured image) acquired by the image acquisition unit 411. Here, the feature information is, for example, a feature amount representing features of a face area included in the captured image. In addition, there are cases where a face area cannot be detected because a person in a captured image is wearing sunglasses, a mask, or the like. In such a case, the feature extraction unit 412 extracts features of the element from other elements such as the person's clothes, hairstyle, and body type as feature information. Hereinafter, the feature information extracted from each of the first captured image, second captured image, and third captured image are referred to as first feature information, second feature information, and third feature information, respectively. Note that the method of extracting the feature information is not particularly limited, and known techniques such as face recognition and image recognition may be used.

[0055] The registration processing unit 413 is a functional unit that sets a customer to be tracked. More specifically, the registration processing unit 413 registers the first feature information extracted from the first captured image by the feature extraction unit 412 in the tracking table T1 of the storage unit 42.

[0056] Here, Fig. 7 is a diagram showing an example of the data structure of the tracing table T1. As shown in Fig. 7, the tracing table T1 has a tracing target column, a first check column, a second check column, and a third check column as items. Each of these items is prepared for each customer and constitutes one data entry.

[0057] The tracking target column is an item in which the first characteristic information of the customer to be tracked is registered. The registration processing unit 413 registers the first characteristic information extracted from the first captured image in the tracking target column. The first check column is an item for the tracking target customer to check that the product registration has been completed. In this embodiment, since the first captured image is captured by the registration device 10, the registration processing unit 413 registers the first characteristic information in the tracking target column and registers check information indicating that the product registration has been completed in the first check column. The second check column is a column for checking that the tracking target customer has completed the transaction at the accounting device 20. The third check column is a column for checking that the tracking target customer is heading to the exit B11. Hereinafter, registering check information in the first check column, the second check column, and the third check column is also simply referred to as checking.

[0058] 6, the tracking processing unit 414 is a functional unit that tracks the movement path of customers in the accounting area A12 and the exit area A13 by comparing the captured images (characteristic information) of each area. Note that the tracking processing unit 414 uses known technology such as face authentication and image recognition in comparing the captured images (characteristic information).

[0059] Specifically, the tracking processor 414 compares the second feature information extracted from the second captured image with each of the first feature information registered in the tracking table T1, and determines whether or not there is any first feature information whose similarity is equal to or greater than a threshold. If there is any first feature information whose similarity is equal to or greater than a threshold, the tracking processor 414 recognizes that the customer corresponding to the first feature information and the customer corresponding to the second feature information are the same person, and determines that the customer has completed a transaction at the accounting device 20. The tracking processor 414 then registers check information indicating that the transaction has been completed in the second check box of the data entry in which the first feature information is registered. Note that if there are multiple pieces of first feature information whose similarity is equal to or greater than a threshold, the tracking processor 414 selects the first feature information with the highest similarity.

[0060] The tracking processing unit 414 also compares the third feature information extracted from the third captured image with each of the first feature information registered in the tracking table T1, and determines whether or not there is first feature information whose similarity is equal to or greater than a threshold. If there is first feature information whose similarity is equal to or greater than a threshold, the tracking processing unit 414 recognizes that the customer corresponding to the first feature information and the customer corresponding to the third feature information are the same person, and determines that the customer is heading to the exit B11. Then, the tracking processing unit 414 registers check information indicating that the customer is heading to the exit B11 in the third check box of the data entry in which the first feature information is registered. Note that if there are multiple pieces of first feature information whose similarity is equal to or greater than a threshold, the tracking processing unit 414 selects the first feature information whose similarity is the highest, and checks the third check box of the data entry in which the first feature information is registered.

[0061] The fraud detection unit 415 monitors the tracking table T1 and detects customers who may be committing fraud. More specifically, the fraud detection unit 415 determines whether or not there is a possibility of fraud based on the state of the check information registered in the first check box, the second check box, and the third check box, that is, the customer's movement route.

[0062] For example, if the first and third check boxes are checked and the second check box is not checked, there is a possibility that a customer who has registered an item is trying to take the item out of the store without paying for it. Therefore, if the state of the check information registered in the tracking table T1 indicates that the item has left the registration area A11, skipping the payment area A12 and reaching the exit area A13, the fraud detection unit 415 detects the possibility of fraudulent activity.

[0063] Furthermore, if all three check boxes are checked, the fraud detection unit 415 determines that the registration and transaction have been performed normally, and excludes the data entry from tracking. Here, "excluding from tracking" means adding flag information indicating that the data entry is not to be tracked, deleting the data entry itself from tracking table T1, or moving the data entry to another data table. Note that it is preferable for the fraud detection unit 415 to determine whether or not there is a possibility of fraudulent activity when check information is registered in the third check box.

[0064] The notification processor 416 is a functional unit that issues a notification according to the detection result of the fraud detection unit 415. More specifically, the notification processor 416 issues a notification to the store clerk on the condition that the fraud detection unit 415 detects the possibility of fraud. There is no particular restriction on the notification method, and various methods can be adopted. For example, the notification processor 416 may transmit notification information including a message notifying the occurrence of fraud and an image (characteristic information) of the corresponding customer to each of the registration devices 10. In addition, when each store clerk has a portable communication device connected to the network N1, the notification processor 416 may transmit the notification information to the communication device. In addition, when a notification device such as an alarm buzzer or an alarm light is provided in the exit area A13 or the like, the notification may be issued by operating the notification device.

[0065] Next, the operation of the above-mentioned monitoring server 40 will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of a monitoring process executed by the monitoring server 40 of the first embodiment.

[0066] First, the image acquisition unit 411 acquires the captured image transmitted from each device (step S31). Next, the feature extraction unit 412 extracts feature information from the captured image acquired in step S31 (step S32).

[0067] On condition that the first feature information is extracted from the first captured image in step S32 (step S33; Yes), the registration processing unit 413 registers the extracted first feature information in the tracking target column of the tracking table T1 (step S34). Next, the registration processing unit 413 checks the first check column of the data entry in which the first feature information is registered in step S34 (step S35), and returns the process to step S31.

[0068] In addition, the tracking processing unit 414 proceeds to step S37 on the condition that the second feature information is extracted from the second captured image in step S32 (step S33; No → step S36; Yes). The tracking processing unit 414 compares the extracted second feature information with each of the first feature information registered in the tracking table T1, and calculates the similarity between the two feature information (step S37). Next, the tracking processing unit 414 determines whether there is first feature information whose similarity with the second feature information exceeds a predetermined threshold, that is, whether there is a customer corresponding to the customer of the second feature information (step S38).

[0069] If there is no first characteristic information exceeding the threshold, the tracking processing unit 414 determines that there is no match (step S38; No) and returns the process to step S31. If there is first characteristic information exceeding the threshold, the tracking processing unit 414 determines that there is a match (step S38; Yes) and checks the second check box of the data entry in which the first characteristic information is registered (step S39). Then, the tracking processing unit 414 returns the process to step S31.

[0070] In addition, the tracking processing unit 414 proceeds to step S40 on the condition that the third feature information is extracted from the third captured image in step S32 (step S33; No → step S36; No). The tracking processing unit 414 compares the extracted third feature information with each of the first feature information registered in the tracking table T1, and calculates the similarity between the two feature information (step S40). Next, the tracking processing unit 414 determines whether there is first feature information whose similarity with the third feature information exceeds a predetermined threshold, that is, whether there is a customer corresponding to the customer of the third feature information (step S41).

[0071] If there is no first characteristic information exceeding the threshold, the tracking processing unit 414 determines that there is no match (step S41; No) and returns the process to step S31. On the other hand, if there is first characteristic information exceeding the threshold, the tracking processing unit 414 determines that there is a match (step S41; Yes) and checks the third check box of the data entry in which the first characteristic information is registered (step S42).

[0072] Next, the fraud detection unit 415 refers to the state (path of movement) of each check box of the data entry in which the third check box was checked in step S42, and determines whether or not there is a possibility of fraudulent activity (step S43). If the first check box, second check box, and third check box are all checked, the fraud detection unit 415 determines that the checkout procedure for the product has been performed normally (step S43; No), and returns to step S31.

[0073] On the other hand, if the first and third check boxes are checked and the second check box is not checked, the fraud detection unit 415 determines that there is a possibility of fraud (step S43; Yes).Then, the notification processing unit 416 notifies the store clerk by sending notification information or the like (step S44), and returns to step S31.

[0074] As described above, the monitoring server 40 of this embodiment tracks the movement path of customers within the checkout area A1 based on captured images of the registration area A11, the checkout area A12, and the exit area A13. The monitoring server 40 then detects customers who may be committing fraud based on the tracked movement path and notifies a store clerk. This allows the monitoring server 40 to efficiently detect fraudulent behavior such as taking unchecked merchandise out of the store in a monitoring system based on comparing captured images.

[0075] In this embodiment, the registration device 10 is configured to transmit the first captured image to the monitoring server 40 when the registration process is completed, but this is not limited, and the image may be transmitted during the registration process. In addition, the accounting device 20 is also configured to transmit the second captured image to the monitoring server 40 when the transaction process is completed, but this is not limited, and the image may be transmitted during the transaction. In addition, when the accounting device 20 detects a face area of ​​a person (customer) from the second captured image, it may be configured to assume that the customer has made a transaction and transmit the second captured image to the monitoring server 40. In addition, when these configurations are adopted, the criteria for determining whether or not a transaction has been made are looser than those in the above-mentioned embodiment. For this reason, these configurations are preferably used in cases where the customer's movement route is simply tracked.

[0076] Furthermore, in this embodiment, the customer processes the transaction at the accounting device 20 specified by the store clerk, but the configuration is not limited to this and the transaction may be made possible to be made at any accounting device 20. When this configuration is adopted, for example, the transaction can be made at any accounting device 20 by operating the registration device 10, accounting device 20, and monitoring server 40 as follows.

[0077] First, when the control unit 11 of the registration device 10 receives an instruction to end the registration process, it sends a pair of the transaction information generated during the registration process and the first captured image captured during the registration process to the monitoring server 40. In the monitoring server 40, the feature extraction unit 412 extracts first feature information from the first captured image sent from the registration device 10. The registration processing unit 413 registers the extracted first feature information in the tracking table T1 and checks the first check box. The feature extraction unit 412 also associates the registered first feature information with the transaction information paired with the first captured image from which the first feature information was extracted, and registers (stores) it in the tracking table T1, etc.

[0078] On the other hand, a customer who has completed registering the product at the registration device 10 moves to any accounting device 20. When the control unit 21 of the accounting device 20 receives an instruction from the customer to start the accounting process, it transmits the second captured image captured by the second camera 32 to the monitoring server 40. In the monitoring server 40, the feature extraction unit 412 extracts second feature information from the second captured image transmitted from the accounting device 20. Next, the tracking processing unit 414 identifies the first feature information of the customer captured by the accounting device 20 (second camera 32) from the first feature information registered in the tracking table T1 based on the second feature information. Next, the control unit 41 of the monitoring server 40 transmits transaction information associated with the identified first feature information to the accounting device 20 that transmitted the second captured image.

[0079] Next, when the control unit 21 of the accounting device 20 receives the transaction information from the monitoring server 40, it executes the transaction process based on that transaction information. Then, after the transaction process is completed, the control unit 21 of the accounting device 20 sends the second captured image that was captured during the transaction process to the monitoring server 40. The subsequent process is the same as in the embodiment described above.

[0080] [Second embodiment] Next, a second embodiment will be described. In the second embodiment, a store that employs a checkout system in which customers themselves register and pay will be described as an example. Note that elements similar to those in the first embodiment will be given the same reference numerals and descriptions will be omitted.

[0081] Fig. 9 is a diagram showing an example of a store layout according to the second embodiment. As shown in Fig. 9, the store has a checkout area A2 for registering and paying for products. The checkout area A2 includes an entrance area A21, a registration and payment area A22, and an exit area A23.

[0082] The entrance area A21 is a first area located at the entrance to the checkout area A2, and has an entrance B21 to the checkout area A2. The registration and payment area A22 is a second area where customers register products and make payments, and has a registration and payment device 50. The exit area A23 is a third area located at the exit of the checkout area A2, and has a store exit B22. Although not shown, it is assumed that the store has an area where products for sale are displayed (for example, above the checkout area A2).

[0083] The registration and accounting device 50 is a product data processing device that is operated by a customer and performs registration and accounting processes for products purchased by the customer. In other words, the registration and accounting device 50 realizes a checkout method called a self-checkout method.

[0084] In a store with the layout shown in Figure 9, a customer's actions when purchasing an item are as follows. First, the customer places the item to be purchased in a shopping cart or the like and moves from the entrance area A21 (entrance B21) to the registration accounting area A22 (registration accounting device 50). At the registration accounting device 50, the customer registers the item by having a reading unit read the barcode attached to the item. Then, once the registration of the item is complete, the customer performs the transaction at the registration accounting device 50. Then, once the transaction is complete, the customer places the purchased item in a plastic bag or the like and moves to the exit area A23 (exit B22).

[0085] In this way, in the self-service system, the customer registers and pays by himself, which makes it possible to improve the efficiency of the process by shortening the waiting time at the register, etc. However, in the self-service system, because registration and payment are left to the customer, there is a possibility of fraudulent behavior such as basket skipping, just like in the semi-self-service system.

[0086] Therefore, in the surveillance system of this embodiment, images of the entrance area A21, the registration and payment area A22, and the exit area A23 are captured, and customers who may be committing fraudulent acts are detected based on the captured images. The configuration of the surveillance system according to this embodiment will be described below.

[0087] Fig. 10 is a diagram showing an example of the configuration of a monitoring system according to the second embodiment. As shown in Fig. 10, the monitoring system includes a first camera 61, a second camera 62, a third camera 63, and a monitoring server 40a, in addition to the above-mentioned registered accounting device 50. The registered accounting device 50, the first camera 61, the third camera 63, and the monitoring server 40a are connected to a network N2 such as a LAN.

[0088] The first camera 61 is an imaging device that captures an image of a customer present in the entrance area A21. The first camera 61 is provided at a position where it is possible to capture an image of the face of a customer entering the checkout area A2 from the entrance B21. The first camera 61 transmits the captured image (hereinafter referred to as a first captured image) to the monitoring server 40a. The installation position of the first camera 61 is not particularly limited, and it may be provided at the gate of the entrance B21, or may be provided on the ceiling of the store in the form of a surveillance camera or the like.

[0089] The second camera 62 is a camera that captures images of customers present in the registered accounting area A22. The second camera 62 is provided in each registered accounting device 50. More preferably, the second camera 62 is provided in each registered accounting device 50 at a position that allows the face of the customer to be captured. Each pair of the registered accounting device 50 and the second camera 62 is connected by a connection line L3. The installation position of the second camera 62 is not particularly important, and for example, it may be provided integrally with the registered accounting device 50, or may be provided on the ceiling of the store in the form of a surveillance camera or the like.

[0090] The registration accounting device 50 cooperates with the second camera 62 to transmit images captured during the accounting process (hereinafter referred to as second captured images) to the monitoring server 40a. Note that the configuration of the registration accounting device 50 is similar to that of the combination of the registration device 10 and accounting device 20 described above, and therefore a detailed description thereof will be omitted.

[0091] The registration accounting device 50 acquires the second captured image captured by the second camera 62 while the accounting process is being performed. Then, when the accounting process is completed, the registration accounting device 50 transmits the second captured image acquired during the accounting process to the monitoring server 40a. Here, the number of second captured images (still images) and the number of times (video images) to be transmitted to the monitoring server 40a are not particularly important. The registration accounting device 50 may acquire the second captured image while the registration process is being performed, and transmit the acquired second captured image to the monitoring server 40a at the timing when the registration process or the accounting process is completed. In addition, the registration accounting device 50 (control unit) may select the second captured image to be transmitted to the monitoring server 40a based on the area and appearance frequency of the face area by performing a transmission image selection process similar to that of the registration device 10 described above.

[0092] The third camera 63 is an imaging device that captures images of customers present in the exit area A23. The third camera 63 is provided at a position where it is possible to capture images of the faces of customers passing through the exit B22. The third camera 63 transmits the captured image (hereinafter referred to as the third captured image) to the monitoring server 40a. The installation position of the third camera 63 is not particularly important, and it may be provided at the gate of the exit B22, or may be provided on the ceiling of the store in the form of a surveillance camera or the like.

[0093] The monitoring server 40a is a server device that monitors the behavior of customers in the checkout area A2 based on images captured by the first camera 61, the second camera 62, and the third camera 63. The monitoring server 40a will be described below. Note that the same components as those in the first embodiment are given the same reference numerals and will not be described.

[0094] Fig. 11 is a diagram showing an example of the configuration of a monitoring server 40a according to the second embodiment. As shown in Fig. 11, the monitoring server 40a includes a control unit 41a. The control unit 41a has a computer configuration including a CPU, an SoC, a ROM, a RAM, and the like.

[0095] The control unit 41a is connected to a storage unit 42 and a communication interface (I / F) 43 via a bus etc. The storage unit 42 stores a tracing table T1 (see FIG. 7) having the same data configuration as in the first embodiment.

[0096] 11, the control unit 41a includes functional units such as an image acquisition unit 411a, a feature extraction unit 412, a registration processing unit 413, a tracking processing unit 414, a fraud detection unit 415, and a notification processing unit 416. These functional units are realized in software by executing a program stored in the storage unit 42, or in hardware by a dedicated processor or the like included in the control unit 41a.

[0097] The image acquisition unit 411a acquires captured images captured by the first camera 61, the second camera 62, and the third camera 63 via the communication interface 43. More specifically, the image acquisition unit 411a acquires a first captured image transmitted from the first camera 61. The image acquisition unit 411a also acquires a second captured image transmitted from each of the registered accounting devices 50. The image acquisition unit 411a also acquires a third captured image transmitted from the third camera 63.

[0098] The control unit 41a executes the monitoring process described in Fig. 8 by working together with each of the functional units described above. As a result, the monitoring server 40a of this embodiment tracks the movement path of customers within the checkout area A2 based on captured images of the entrance area A21, the registration checkout area A22, and the exit area A23. The monitoring server 40a then detects customers who may be committing fraud based on the tracked movement path and notifies a store clerk. As a result, the monitoring server 40a can efficiently detect fraudulent behavior such as taking unchecked merchandise out of the store in a monitoring system based on comparing captured images.

[0099] Although the embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the invention and its equivalents described in the claims.

[0100] For example, in the above embodiment, the monitoring server 40 (40a) is configured to extract feature information from the captured image, but the present invention is not limited to this, and the feature information may be extracted by the device that transmits the captured image. For example, any or all of the registration device 10, the accounting device 20, and the registration accounting device 50 may be equipped with a feature extraction unit 412 and transmit the feature information extracted from the captured image to the monitoring server 40 (40a). Also, each camera (e.g., the third camera 33, the first camera 61, the third camera 63, etc.) may be equipped with a feature extraction unit 412 and transmit the feature information extracted from the captured image to the monitoring server 40 (40a). In this case, in the monitoring server 40 (40a), the registration processing unit 413 registers the first feature information transmitted from the registration device 10 or the first camera 61 in the tracking table T1. Also, the tracking processing unit 414 tracks the movement route of the same customer based on the similarity between each of the first feature information registered in the tracking table T1 and the second feature information and the third feature information transmitted from the other devices.

[0101] In the above embodiment, the monitoring server 40 (the registration processing unit 413) is configured to register the first feature information extracted from the first captured image in the tracking table T1, but this is not limiting, and the first captured image may be registered in the tracking table T1. In this case, the feature extraction unit 412 cooperates with the tracking processing unit 414 to extract feature information from each captured image when comparing the similarity between the captured images.

[0102] In the above embodiment, the monitoring server 40 (40a) is configured to monitor for fraudulent activities, but this is not limiting and other devices may be configured to monitor. For example, one representative device among the registration devices 10 and accounting devices 20 (or the registration accounting devices 50) may be equipped with the functions of the monitoring server 40 (40a) to monitor for fraudulent activities.

[0103] In the above embodiment, the registration device 10, accounting device 20, and registration accounting device 50 are configured to transmit captured images, but this is not limiting, and each camera may transmit directly. Also, the first camera 31 and the third camera 33, 36 are configured to directly transmit captured images, but they may cooperate with an information processing device such as a PC (Personal Computer) to transmit from the information processing device. When transmitting from an information processing device, the captured image to be transmitted may be selected by performing a process similar to the transmission image selection process. [Explanation of symbols]

[0104] 10 Registration device 20 Accounting Equipment 31, 61 1st Camera 32, 62 2nd Camera 33, 63 3rd Camera 40, 40a Monitoring Server 50 Registered Accounting Device 411, 411a Image acquisition unit 412 Feature Extraction Unit 413 Registration processing unit 414 Tracking Processing Unit 415 Fraud Detection Department T1 Tracking Table [Prior art documents] [Patent documents]

[0105] [Patent Document 1] JP 2013-186495 A

Claims

1. a first imaging means for imaging a customer in a first area where a product is to be registered; a second imaging means for imaging a customer in a second area who is making a payment for the registered product; a third imaging means for imaging a third area through which customers who have completed their transaction can pass; a first selection means for identifying a customer who appears most frequently from among the captured images for each transaction captured by the first imaging means, and selecting an image in which the area of ​​the face region of the identified customer is the largest; a second selection means for identifying the customer who appears most frequently from among the captured images for each transaction captured by the second imaging means, and selecting the captured image in which the area of ​​the face region of the identified customer is the largest; a tracking means for tracking a movement path of a same customer from the first area to the third area based on a similarity between feature information extracted from the captured images selected by the first selection means and the second selection means and feature information extracted from the captured image of the third imaging means; and a notification means for notifying a user when the movement path indicates that the user has traveled from the first area to the third area by skipping the second area; A surveillance system having:

2. 2. The surveillance system according to claim 1, wherein the third imaging means images the third area located at an exit of an area in which the first area and the second area are provided.

3. 3. The surveillance system according to claim 1, wherein the first imaging means images a customer who uses a registration device provided in the first area.

4. 4. The monitoring system according to claim 3, wherein the first selection means, when a process relating to the registration of a commodity for one transaction is completed in the registration device, selects from among the images captured during the process.

5. 5. The surveillance system according to claim 1, wherein the second imaging means images a customer using an accounting device provided in the second area.

6. The surveillance system according to claim 5 , wherein the second imaging device outputs an image captured during a transaction-related process after the transaction has been carried out at the accounting device.

Images