Takeout monitoring system and program

Abstract

To provide a takeout monitoring system which can immediately detect takeout of a portable terminal held by a store to the outside of the store.SOLUTION: A takeout monitoring system comprises: a monitoring camera (detection device) comprising a cart detection unit (detection means) for detecting that a terminal device having a merchandise registration function for a customer to register merchandise for himself / herself, or a shopping cart with the terminal device is taken out to a predetermined area, and a takeout information output unit (output means) for outputting information detected by the cart detection unit to a server device; and the server device comprising a notification instruction unit (notification means) for outputting information for notifying that the terminal or the shopping cart is taken out when the detected information is acquired.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] An embodiment of the present invention relates to a takeout monitoring system and a program. [Background technology]

[0002] In stores such as supermarkets and shopping centers, a style of shopping using a shopping cart that can be moved around the store and can store the products to be purchased is commonly used.Recently, a system has been proposed in which a shopping cart is equipped with a terminal device with product registration and payment functions, allowing customers to register the products they want to purchase and make payments while pushing the shopping cart.

[0003] Such terminal devices are expensive information devices such as tablet terminals, so there is a risk that customers may take them outside the store and steal them. Also, such terminal devices are vulnerable to moisture, so it is necessary to prevent them from being taken outside the store (outdoors).

[0004] Conventionally, a system has been proposed that detects whether a used shopping cart has been left unattended without being returned to a designated location (for example, Patent Document 1).

[0005] In the system of Patent Document 1, if a shopping cart is outside a predetermined range for a predetermined time, the shopping cart is determined to be abandoned. Because this determination takes time, it is not possible to immediately detect that a shopping cart (terminal device) has been taken out of the store. Summary of the Invention [Problem to be solved by the invention]

[0006] The problem to be solved by the present invention is to provide a take-out monitoring system and program that can immediately detect when a mobile terminal owned by a store is taken outside the store. [Means for solving the problem]

[0007] The embodiment of the take-out monitoring system includes a detection device and a server device. The detection device includes a detection means for detecting when a terminal device having a product registration function for allowing a customer to register products or a shopping cart equipped with the terminal device is taken out to a predetermined area, and an output means for outputting information detected by the detection means to the server device. When the server device acquires the detected information, it outputs information for causing a notification that the terminal device or the shopping cart has been taken out. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a system configuration diagram showing an example of a schematic configuration of a takeout monitoring system according to the first and second embodiments. [Figure 2] FIG. 2 is a hardware block diagram showing an example of the hardware configuration of a terminal device included in the takeout monitoring system. [Figure 3] FIG. 3 is a hardware block diagram showing an example of the hardware configuration of the store clerk terminal included in the takeout monitoring system. [Figure 4] FIG. 4 is a hardware block diagram showing an example of the hardware configuration of the server device included in the takeout monitoring system. [Figure 5] FIG. 5 is a diagram showing an example of the contents of a terminal device status management file stored in the server device. [Figure 6] FIG. 6 is a hardware block diagram showing an example of the hardware configuration of a surveillance camera included in the portable surveillance system. [Figure 7] FIG. 7 is a diagram showing an example of a configuration of a main part of a shopping cart according to the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating a method for detecting whether a terminal device has been taken out into a prohibited area. [Figure 9] FIG. 9 is a diagram showing an example of notification information displayed on the screen of a terminal device when the terminal device is taken out into a restricted area. [Figure 10]FIG. 10 is a diagram showing an example of notification information displayed on the screen of the store clerk terminal when the terminal device is taken out into a prohibited area. [Figure 11] FIG. 11 is a functional block diagram showing an example of the functional configuration of the terminal device, the store clerk terminal, the monitoring camera, and the server device included in the takeout monitoring system of the first embodiment. [Figure 12] FIG. 12 is a sequence diagram illustrating an example of the operation of the takeout monitoring system according to the first embodiment. [Figure 13] FIG. 13 is a diagram showing an example of the configuration of the main parts of a shopping cart used in the take-out monitoring system of the second embodiment. [Figure 14] FIG. 14 is a diagram illustrating a method for detecting that a terminal device has been taken out into a take-out prohibited area in the take-out monitoring system of the second embodiment. [Figure 15] FIG. 15 is a system configuration diagram showing an example of a schematic configuration of a takeout monitoring system according to the third embodiment. [Figure 16] FIG. 16 is a diagram illustrating an example of a shopping cart used in the takeout monitoring system of the third embodiment. [Figure 17] FIG. 17 is a diagram illustrating a method for detecting that a terminal device has been taken out into a take-out prohibited area in the take-out monitoring system of the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] (First embodiment) A first embodiment of the present invention will be described below with reference to the accompanying drawings. In the embodiment described below, an example in which the present invention is applied to a take-out monitoring system 100 will be described.

[0010] (Outline of the removal monitoring system) The schematic configuration of the carry-out monitoring system 100 will be described using Figure 1. Figure 1 is a system configuration diagram showing an example of the schematic configuration of the carry-out monitoring system according to the first embodiment. As shown in Figure 1, the carry-out monitoring system 100 includes a terminal device 10, a store clerk terminal 20, a server device 30, a monitoring camera 40, and an alarm device 50. The terminal device 10, the store clerk terminal 20, the server device 30, the monitoring camera 40, and the alarm device 50 are communicatively connected via a network Na. The terminal devices 10, the store clerk terminal 20, the monitoring camera 40, and the alarm device 50 are provided in numbers according to the size of the store.

[0011] The take-out monitoring system 100 detects when a terminal device 10 used by a customer while shopping is taken out of a store and issues a notification.

[0012] The network Na may be the Internet, a virtual private network (VPN), a local area network (LAN), a public communication network, a mobile communication network, or the like, either alone or in appropriate combination. As an example, the network Na may be a combination of the Internet and a mobile communication network.

[0013] In addition, a plurality of wireless access points 2 that can communicate with the server device 30 are connected to the network Na. The wireless access points 2 are connected to the terminal device 10 and the clerk terminal 20 via a network Nb such as a wireless LAN. Therefore, the terminal device 10 and the clerk terminal 20 can communicate with the server device 30.

[0014] The terminal device 10 is a device such as a tablet terminal or a smartphone that is operated by a customer who is shopping. The terminal device 10 is attached to a shopping cart 3, for example.

[0015] The terminal device 10 may be attached to the shopping cart 3 in advance, or a customer entering the store may be loaned the terminal device 10 by the store and then attach the loaned terminal device 10 to the shopping cart 3 by the customer himself.

[0016] The terminal device 10 acquires the product code of the product that the customer wishes to purchase and performs a product registration process to register the product.

[0017] The terminal device 10 also performs cashless payment processing for the registered product using the payment method selected by the customer. Cashless payment methods include, for example, credit card payment, code payment, and electronic money payment.

[0018] Credit card payment is a payment method that uses a customer's credit card. When making a credit card payment, the customer inserts their credit card into the terminal device 10 and has the personal information and debit account information registered on the credit card read to make the payment.

[0019] Code payment is a payment method in which payment is made using a payment app pre-installed on a mobile device such as a smartphone owned by the customer. When making a code payment, the customer displays a code symbol on their mobile device and has the terminal device 10 read the displayed code symbol to make the payment.

[0020] Electronic money payment is a payment method in which payment is made using electronic money loaded onto, for example, an IC card. When making an electronic money payment, a customer performs, for example, short-range wireless communication between the electronic money card they own and the terminal device 10, which allows the terminal device 10 to read the remaining charge on the electronic money card and make a payment within the remaining balance.

[0021] If a customer wishes to pay in cash, the customer pays in cash at an accounting device equipped with a change dispenser (not shown in FIG. 1).

[0022] Terminal device 10 may not have a payment function and may only have the function of executing the product registration process described above. In this case, after the product registration process, terminal device 10 displays a settlement barcode generated based on the registered product data on the screen of terminal device 10. The customer reads the barcode displayed on terminal device 10 by operating a scanner installed in the accounting device, and makes payment at the accounting device.

[0023] The clerk terminal 20 is a small terminal such as a smartphone or tablet terminal carried by a clerk. The clerk terminal 20 is the notification destination when the removal monitoring system 100 detects that the terminal device 10 has been removed. The clerk terminal 20 is also used as a communication device for communication and information sharing between clerks. The clerk terminal 20 is an example of a mobile terminal in the present disclosure.

[0024] The server device 30 monitors the operating status of the terminal device 10 while a customer is shopping using the terminal device 10. The server device 30 also acquires take-out information indicating that the terminal device 10 has been taken out of the store from a monitoring camera 40 (described later). The take-out information is, for example, identification information that identifies the taken-out terminal device 10. If the terminal device 10 and the shopping cart 3 to which the terminal device 10 is attached are associated, the take-out information may be identification information that identifies the shopping cart 3. When the server device 30 acquires the take-out information, the server device 30 causes the taken-out terminal device 10, the store clerk terminal 20, and a notification device 50 (described later) to issue a notification. Specific content of the notification will be described later (see FIGS. 9 and 10). The server device 30 may be a store server installed in the store, or may be provided on, for example, a cloud connected to the network Na. The notification destinations of the server device 30 do not necessarily have to be the terminal device 10 that has been taken out, the store clerk terminal 20, and the notification device 50, and can be changed by appropriate settings.

[0025] The surveillance camera 40 is equipped with an imaging element such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) and captures images of the area around the store's exit, such as a designated area where the terminal device 10 or a shopping cart 3 equipped with the terminal device 10 is prohibited from being taken out. The surveillance camera 40 has an image recognition function for detecting the shopping cart 3 from the captured image. The surveillance camera 40 also reads identification information 65 (see FIG. 7 ) written on the detected shopping cart 3 to identify the shopping cart 3. To improve the detection performance of the shopping cart 3, multiple surveillance cameras 40 may be configured to monitor the same exit 9 (see FIG. 8 ). Since a store typically has multiple exits 9, the server device 30 has a management table (not shown) that indicates which surveillance camera 40 is monitoring which exit 9. The surveillance camera 40 is an example of a detection device in the present disclosure.

[0026] The notification device 50 is installed, for example, at an exit 9 of a store, and issues a notification with light and sound when a customer pushes a shopping cart 3 equipped with a terminal device 10 and exits the store. The notification device 50 is equipped with a light-emitting element such as an LED (not shown), and the lighting, blinking, and extinguishing of the light-emitting element are controlled in response to instructions from the server device 30. The notification device 50 is also equipped with a speaker (not shown), and outputs a notification sound or message in response to instructions from the server device 30. The server device 30 has a management table (not shown) that indicates which notification device 50 is installed at which exit 9 of the store.

[0027] (Hardware configuration of terminal device) The hardware configuration of the terminal device 10 will be described with reference to Fig. 2. Fig. 2 is a hardware block diagram showing an example of the hardware configuration of a terminal device provided in the carry-out monitoring system.

[0028] The terminal device 10 has a configuration in which a control unit 11, a storage unit 12, a peripheral device controller 14, and a communication controller 15 are interconnected via an internal bus 13.

[0029] The control unit 11 controls the overall operation of the terminal device 10. The control unit 11 includes a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, and a RAM (Random Access Memory) 113. The CPU 111 is connected to the ROM 112 and the RAM 113 via internal buses such as an address bus and a data bus. The CPU 111 loads various programs stored in the ROM 112 and the storage unit 12 into the RAM 113. The CPU 111 controls the operation of the terminal device 10 by operating in accordance with the various programs loaded into the RAM 113. In other words, the control unit 11 has the configuration of a typical computer.

[0030] The storage unit 12 is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The storage unit 12 may also be a non-volatile memory such as a flash memory that retains stored information even when the power is turned off. The storage unit 12 stores a control program 121, a product master 122, a product registration file 123, and a payment file 124.

[0031] The control program 121 is a program that controls the overall operation of the terminal device 10. The control program 121 may be provided in a state stored in the storage unit 12, or may be provided by being recorded in an installable or executable file on a computer-readable non-transitory recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD. The control program 121 may also be provided by being stored on a computer connected to a network and downloaded via the network. Furthermore, the control program 121 may be provided or distributed via a network such as the Internet.

[0032] The product master 122 is a master file that stores product information such as product names and prices of products for sale. The product master 122 stores product names, unit prices, additional information, and the like in association with product codes. Note that the contents of the product master 122 are updated as needed, so the terminal device 10 obtains the latest product master 122 via the communication controller 15, for example, from a store server (not shown).

[0033] The product registration file 123 is a file that stores product information corresponding to the product code acquired by the terminal device 10. The information stored in the product registration file 123 is used as the payment target when performing payment processing. The product registration file 123 is stored in association with a transaction code that is issued when product registration is completed.

[0034] The payment file 124 is a file that stores the details of payments made using the cashless payment function of the terminal device 10. In addition to the details stored in the product registration file 123, the payment file 124 stores the payment date and time, the payment method, etc.

[0035] The control unit 11 is connected via a peripheral device controller 14 to a display device 141, an operation device 142, a camera 143, a speaker 144, and the like, which input and output various types of information.

[0036] The display device 141 is, for example, a liquid crystal monitor or an organic EL monitor. The display device 141 displays various screens to the customer, such as a registration details screen showing registered product information and a payment screen showing payment details. The display device 141 also displays notification information (see FIG. 9) when the customer takes the shopping cart 3 into a prohibited area.

[0037] The operation device 142 is, for example, a touch panel that is superimposed on the display device 141. The operation device 142 acquires various pieces of operation information by the customer and outputs the information to the control unit 11.

[0038] The camera 143 photographs the product to be purchased by the customer and reads the code symbol, such as the code symbol in which the product code is registered. Note that the camera 143 may also recognize the product itself.

[0039] The speaker 144 outputs notification information by sound or voice when a customer takes the shopping cart 3 into a prohibited area.

[0040] The peripheral device controller 14 may further be connected to a card reader that reads credit cards or the like, a wireless communication unit that performs short-range wireless communication with electronic money cards or the like, and the like, both of which are not shown in FIG.

[0041] The control unit 11 also communicates with the server device 30 via the communication controller 15. The terminal device 10 outputs its own operating status (shopping, checking out, checkout completed), product registration information, etc. to the server device 30. When the terminal device 10 determines that it has been taken into a prohibited area, it receives an instruction from the server device 30 to issue a notification.

[0042] (Hardware configuration of store clerk terminal) The hardware configuration of the store clerk terminal 20 will be described with reference to Fig. 3. Fig. 3 is a hardware block diagram showing an example of the hardware configuration of the store clerk terminal provided in the takeout monitoring system.

[0043] The store clerk terminal 20 has a configuration in which a control unit 21, a storage unit 22, a peripheral device controller 24, and a communication controller 25 are interconnected by an internal bus 23.

[0044] The control unit 21 controls the overall operation of the clerk terminal 20. The control unit 21 includes a CPU 211, a ROM 212, and a RAM 213. The CPU 211 is connected to the ROM 212 and the RAM 213 via internal buses such as an address bus and a data bus. The CPU 211 loads various programs stored in the ROM 212 and the storage unit 22 into the RAM 213. The CPU 211 controls the operation of the clerk terminal 20 by operating in accordance with the various programs loaded into the RAM 213. In other words, the control unit 21 has the configuration of a typical computer.

[0045] The storage unit 22 is a storage device such as an HDD or SSD. The storage unit 22 may also be a non-volatile memory such as a flash memory that retains stored information even when the power is turned off. The storage unit 22 stores a control program 221.

[0046] The control program 221 is a program that controls the overall operation of the clerk terminal 20. The control program 221 may be provided in a state stored in the storage unit 22, or may be provided by being recorded in an installable or executable file on a computer-readable non-transitory recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD. The control program 221 may also be stored on a computer connected to a network and provided by being downloaded via the network. Furthermore, the control program 221 may be provided or distributed via a network such as the Internet.

[0047] The control unit 21 is connected via a peripheral device controller 24 to a display device 241, an operation device 242, a speaker 243, and the like, which input and output various types of information.

[0048] The display device 241 is, for example, a liquid crystal monitor, an organic EL monitor, etc. The display device 241 displays various screens, etc. to the store clerk. In addition, the display device 241 displays notification information (see FIG. 10) when a customer takes the shopping cart 3 into a take-out prohibited area.

[0049] The operation device 242 is, for example, a touch panel that is superimposed on the display device 241. The operation device 242 acquires various pieces of operation information by the store clerk and outputs the information to the control unit 21.

[0050] The speaker 243 outputs notification information by sound or voice when a customer takes the shopping cart 3 into a prohibited area.

[0051] Furthermore, the control unit 21 communicates with the server device 30 via the communication controller 25. When the clerk terminal 20 detects that the terminal device 10 has been taken out into a take-out prohibited area, the clerk terminal 20 acquires, from the server device 30, identification information for identifying the terminal device 10, information for identifying the exit 9 from which the terminal device 10 has been taken out (for example, identification information of the surveillance camera 40 that detected the take-out), and an instruction to execute an alert. Furthermore, when the clerk terminal 20 has finished dealing with the customer who has taken out the terminal device 10, it outputs to the server device 30 information indicating that the deal with the customer has been completed.

[0052] (Server hardware configuration) The hardware configuration of the server device 30 will be described with reference to Fig. 4. Fig. 4 is a hardware block diagram showing an example of the hardware configuration of the server device included in the takeout monitoring system.

[0053] The server device 30 has a configuration in which a control unit 31, a storage unit 32, and a communication controller 35 are connected to one another via an internal bus 33.

[0054] The control unit 31 controls the overall operation of the server device 30. The control unit 31 includes a CPU 311, a ROM 312, and a RAM 313. The CPU 311 is connected to the ROM 312 and the RAM 313 via internal buses such as an address bus and a data bus. The CPU 311 loads various programs stored in the ROM 312 and the storage unit 32 into the RAM 313. The CPU 311 controls the operation of the server device 30 by operating in accordance with the various programs loaded into the RAM 313. In other words, the control unit 31 has the configuration of a typical computer.

[0055] The storage unit 32 is a storage device such as an HDD or SSD. Alternatively, the storage unit 32 may be a non-volatile memory such as a flash memory that retains stored information even when the power is turned off. The storage unit 32 stores a control program 321 and a terminal device status management file 322.

[0056] The control program 321 is a program that controls the overall operation of the server device 30. The control program 321 may be provided in a state stored in the storage unit 32, or may be provided by being recorded in an installable or executable file on a computer-readable non-transitory recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD. The control program 321 may also be provided by being stored on a computer connected to a network and downloaded via the network. Furthermore, the control program 321 may be provided or distributed via a network such as the Internet.

[0057] The terminal device status management file 322 is a file that manages the operating status of each terminal device 10, and is acquired from the currently used terminal device 10 at any time. The information stored in the terminal device status management file 322 will be described in detail later (see FIG. 5).

[0058] The control unit 31 communicates with the surveillance camera 40, the terminal device 10, the store clerk terminal 20, and the notification device 50 via the communication controller .

[0059] The server device 30 acquires, from the surveillance camera 40, identification information 65 of the shopping cart 3 that has been taken into the prohibited area. The server device 30 also acquires, from the terminal device 10, the status of the terminal device 10 (shopping in progress, checkout in progress, checkout completed) and product registration information, etc. When the server device 30 acquires information indicating that the shopping cart 3 has been taken out, it instructs the terminal device 10, the store clerk terminal 20, and the alarm device 50 associated with the shopping cart 3 to issue an alarm.

[0060] The server device 30 also acquires information from the terminal device 10 that instructed the execution of the alert, indicating that the terminal device 10 has returned from the take-out prohibited area into the store. The server device 30 also acquires information from the store clerk terminal 20, indicating that measures have been taken to return the customer pushing the shopping cart 3 that was the subject of the alert from the take-out prohibited area back into the store. The server device 30 also instructs the terminal device 10, the store clerk terminal 20, and the alert device 50 to cancel the alert.

[0061] (Data structure of terminal device status management file) The data structure of the terminal device status management file 322 stored in the server device 30 will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of the contents of the terminal device status management file stored in the server device.

[0062] The terminal device status management file 322 stores a terminal code 331, cart identification information 332, shopping time 333, last operation time 334, number of registered items 335, registered price 336, status 337, transaction code 338, and take-out flag 339.

[0063] The terminal code 331 is identification information that identifies the terminal device 10 .

[0064] The cart identification information 332 is identification information that identifies the shopping cart 3 to which the terminal device 10 is attached. The cart identification information 332 is also identification information that identifies the shopping cart 3 associated with the terminal device 10 that the store has lent to the customer.

[0065] The shopping time 333 is information indicating the time that has elapsed since the customer checked in. The time of check-in is, for example, the time when the customer issues an instruction to start shopping on the terminal device 10.

[0066] The most recent operation time 334 is information indicating the most recent time when the customer performed some operation on the terminal device 10, such as product registration or payment operation.

[0067] The number of registered products 335 is information indicating the number of products registered at that time.

[0068] The registered price 336 is information indicating the total price of the products registered at that time.

[0069] The status 337 is information indicating the shopping status of a customer using the terminal device 10. The status 337 stores information corresponding to statuses such as "Shopping," "Checkout," and "Checkout Complete." "Shopping" is a status indicating that a customer is selecting a product. Specifically, the status 337 transitions to "Shopping" when the customer instructs the terminal device 10 to start shopping, or, if the customer is a store member, when the customer logs in to the terminal device 10 with their membership code. "Checkout" is a status indicating that registration of all products to be purchased has been completed and payment is in progress. Specifically, the status 337 transitions to "Checkout" when the customer operates the "Subtotal button" on the terminal device 10, indicating completion of registration. "Checkout Complete" is a status indicating that payment has been completed. Specifically, the status 337 transitions to "Checkout Complete" when the terminal device 10 completes payment processing.

[0070] The transaction code 338 is identification information that identifies the transaction. The transaction code 338 is issued by the terminal device 10 or the server device 30, for example, when the customer presses the "subtotal button" indicating completion of registration. In other words, in the "Shopping" status, the transaction code 338 has not yet been issued.

[0071] The take-out flag 339 is a flag that is assigned when the server device 30 acquires from the surveillance camera 40 information indicating that the shopping cart 3 to which the terminal device 10 is attached has been taken into a take-out prohibited area. For example, when the server device 30 acquires from the surveillance camera 40 information indicating that the shopping cart 3 has been taken into a take-out prohibited area, the take-out flag 339 is set to "1," indicating that the shopping cart 3 has been taken out. On the other hand, when the server device 30 does not acquire from the surveillance camera 40 information indicating that the shopping cart 3 has been taken into a take-out prohibited area, the take-out flag 339 is set to "0."

[0072] The information stored in the terminal device status management file 322 is not limited to the information shown in FIG. 5 . The terminal device status management file 322 may also store the current location of the terminal device 10 in the store, etc., acquired from the terminal device 10. The current location of the terminal device 10 in the store is measured, for example, by a positioning function such as a global positioning system (GPS) (not shown) provided in the terminal device 10. The terminal device status management file 322 may also store the connection status between the terminal device 10 and the network Na. By monitoring the connection between the terminal device 10 and the network Na, the takeout monitoring system 100 can at least confirm that the terminal device 10 is present in the store. In this way, by monitoring the location information and communication status of the terminal device 10, it is possible to continuously monitor the terminal device 10 as long as the terminal device 10 is present in the store, even if the terminal device 10 takes a meal or a break in the store while shopping, causing the status of "shopping" or "checkout completed" to continue for a long period of time.

[0073] In addition, the server device 30 can access the terminal device 10 as necessary and obtain the contents of the product registration file 123 and the contents of the payment file 124 stored in the terminal device 10, thereby monitoring the operating status of the terminal device 10 in more detail.

[0074] (Surveillance camera hardware configuration) The hardware configuration of the surveillance camera 40 will be described with reference to Fig. 6. Fig. 6 is a hardware block diagram showing an example of the hardware configuration of a surveillance camera included in a portable surveillance system.

[0075] The surveillance camera 40 has a configuration in which a control unit 41, a storage unit 42, a camera driver 44, and a communication controller 45 are interconnected by an internal bus 43.

[0076] The control unit 41 controls the overall operation of the surveillance camera 40. The control unit 41 includes a CPU 411, a ROM 412, and a RAM 413. The CPU 411 is connected to the ROM 412 and the RAM 413 via internal buses such as an address bus and a data bus. The CPU 411 loads various programs stored in the ROM 412 and the storage unit 42 into the RAM 413. The CPU 411 controls the operation of the surveillance camera 40 by operating in accordance with the various programs loaded into the RAM 413. In other words, the control unit 41 has the configuration of a typical computer.

[0077] The storage unit 42 is a storage device such as an HDD or SSD. Alternatively, the storage unit 42 may be a non-volatile memory such as a flash memory that retains stored information even when the power is turned off. The storage unit 42 stores a control program 421, a cart recognition model 422, and an identification information recognition model 423.

[0078] The control program 421 is a program that controls the overall operation of the surveillance camera 40. The control program 421 may be provided in a state stored in the storage unit 42, or may be provided by being recorded in an installable or executable file on a computer-readable non-transitory recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD. The control program 421 may also be provided by being stored on a computer connected to a network and downloaded via the network. Furthermore, the control program 421 may be provided or distributed via a network such as the Internet.

[0079] The cart recognition model 422 is a model that has learned from a huge number of images of various types of shopping carts photographed from various angles. The cart recognition model 422 is described, for example, by a neural network. The cart recognition model 422 detects an area that appears to be a shopping cart from the input image and outputs it together with an evaluation score that indicates the reliability of the recognition.

[0080] The identification information recognition model 423 is a model that has learned various forms of numbers and alphabets. The identification information recognition model 423 is described, for example, by a neural network. The identification information recognition model 423 detects identification information from the input image and outputs it together with an evaluation score that indicates the reliability of the recognition.

[0081] The control unit 41 is connected to an image pickup element 441 via a camera driver 44 .

[0082] The image sensor 441 is a photoelectric conversion device such as a CCD or CMOS, etc. The image sensor 441 generates a digital image by converting light incident through a lens (not shown) into an electrical signal.

[0083] Furthermore, control unit 41 communicates with server device 30 via communication controller 45. Surveillance camera 40 outputs to server device 30 removal information indicating that shopping cart 3 has been removed into a prohibited area, identification information that identifies surveillance camera 40, and the time the removal was detected.

[0084] (Main components of the shopping cart) The main configuration of the shopping cart 3 will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of the main configuration of the shopping cart in the first embodiment.

[0085] The X-axis shown in FIG. 7 passes through the shopping cart 3 from front to back, and is an axis extending from the rear to the front of the shopping cart 3. The Y-axis passes through the shopping cart 3 from left to right, and is an axis extending from the right to the left in the direction of travel of the shopping cart 3. The Z-axis passes through the shopping cart 3 from top to bottom, and is an axis extending from the bottom to the top of the shopping cart 3.

[0086] The shopping cart 3 comprises a storage section 63, casters 61, and a handle 62. The storage section 63 is basket-shaped with an open top and stores products to be purchased by customers. A plurality of casters 61 are provided on the bottom surface of the shopping cart 3 and comprise a plurality of wheels that can move back and forth and side to side. The handle 62 is connected to the storage section 63 and casters 61 and is held by customers when moving the shopping cart 3.

[0087] The sides and bottom of the storage section 63 are formed with a mesh-like lattice, allowing the stored products to be seen from the outside. The sides and bottom may also be formed with wall surfaces that allow the inside to be seen visually, rather than with a mesh-like lattice.

[0088] The shopping cart 3 also has a terminal device 10 near the handle 62. The terminal device 10 is installed so that the display device 141 (see FIG. 2) faces the customer who is holding the handle 62 and pushing the shopping cart 3. This allows the customer to easily view the information displayed on the display device of the terminal device 10 while pushing the shopping cart 3.

[0089] Identification plates 64 are attached to at least the left and right side surfaces in the Y-axis direction of the storage section 63. Identification information 65 capable of uniquely identifying the shopping cart 3 is written on the identification plates 64. The identification information 65 is composed of, for example, numbers, alphabets, or a combination thereof. The identification information 65 may also include, for example, hiragana, katakana, kanji, and symbols. The identification information 65 may also be uniquely identifiable information embedded in a code such as a barcode or a two-dimensional code. The identification information 65 is written so that when the surveillance camera 40 captures an image of the shopping cart 3, the identification information 65 appears in the captured image at a size that can be recognized by the surveillance camera 40. The identification information 65 is an example of first identification information in the present disclosure.

[0090] (Method for detecting removal of terminal device) A method for detecting that the shopping cart 3 (terminal device 10) has been taken into a prohibited area will be described with reference to Fig. 8. Fig. 8 is a diagram for explaining a method for detecting that the terminal device has been taken into a prohibited area.

[0091] The shopping cart 3 (terminal device 10) is prohibited from being taken outside the exit 9 through which customers pass when leaving the store 7. Therefore, customers return the shopping cart 3 (terminal device 10) to a cart storage area 8 provided near the exit 9 before exiting through the exit 9.

[0092] The surveillance camera 40 is installed to monitor the outside of the exit 9, i.e., the no-take-out area, which is a predetermined area where taking out of the shopping cart 3 (terminal device 10) is prohibited, and monitors the taking out of the shopping cart 3 (terminal device 10). Multiple surveillance cameras 40 may be installed at one exit 9. When multiple surveillance cameras 40 are installed at one exit 9, it is desirable to install them so that they capture the exit 9 from different directions, as shown in FIG. 8. This is because there is a risk that the shopping cart 3 will blend in with the customers passing through the exit 9, so by capturing images from as different directions as possible, the accuracy of detecting the shopping cart 3 can be improved.

[0093] The surveillance camera 40 repeatedly captures images of the exit 9 and repeatedly detects the shopping cart 3 from the captured images. When the shopping cart 3 is detected, the surveillance camera 40 then reads the identification information 65 written on the identification plate 64.

[0094] The surveillance camera 40 outputs to the server device 30 identification information for identifying itself, take-out information indicating that the shopping cart 3 has been detected, and the recognized identification information 65.

[0095] After acquiring from the surveillance camera 40 the identification information that identifies itself, the removal information indicating that the shopping cart 3 has been detected, and the recognized identification information 65, the server device 30 stores a flag "1" indicating that the shopping cart 3 has been removed in the removal flag 339 in the column of the corresponding cart identification information 332 (see Figure 5) in the terminal device status management file 322 stored in itself.

[0096] (Method of notification in terminal device) A method of notification that the server device 30 makes the terminal device 10 perform will be described with reference to Fig. 9. Fig. 9 is a diagram showing an example of notification information that is displayed on the screen of the terminal device when the terminal device is taken out into a prohibited area.

[0097] When the server device 30 is notified that the shopping cart 3 (terminal device 10) has been taken into a prohibited area, it outputs an instruction to the corresponding terminal device 10 to notify that the shopping cart 3 has been taken into a prohibited area, based on the information stored in the terminal device status management file 322.

[0098] When the terminal device 10 receives the notification instruction from the server device 30, the terminal device 10 displays a message such as "Please return the shopping cart to the cart storage area in the store" on its own display device 141. At this time, the terminal device 10 may output a notification sound from the speaker 144 or read out the message displayed on the display device 141.

[0099] A customer who sees or hears the message and realizes that he has left his shopping cart 3 in the cart storage area 8 returns the shopping cart 3 to the cart storage area 8. This prevents the customer from carelessly taking the shopping cart 3 into a restricted area.

[0100] Furthermore, a customer who tries to take the terminal device 10 without permission will see or hear the message, realize that he or she is being monitored, and return the shopping cart 3 to the cart storage area 8. This makes it possible to prevent the unauthorized removal of the terminal device 10.

[0101] Once the terminal device 10 is taken out of the store, it restores its connection to the network Na via the wireless access point 2 when the shopping cart 3 is returned to the store. When the server device 30 detects that the terminal device 10 has restored its connection to the network Na, it instructs the terminal device 10 to cancel the notification. As a result, the terminal device 10 stops the notification.

[0102] (Method of notification at store clerk terminal) A method of notification that the server device 30 makes the store clerk terminal 20 perform will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of notification information that is displayed on the screen of the store clerk terminal when the terminal device is taken out into a prohibited area.

[0103] When the server device 30 is notified that the shopping cart 3 (terminal device 10) has been taken into a prohibited area, it outputs to the store clerk terminal 20 the time when the surveillance camera 40 detected the taking-out, the identification information of the shopping cart 3 or the terminal device 10, and the number of the exit 9, which is information indicating the location where the shopping cart 3 or the terminal device 10 was taken out.

[0104] When the clerk terminal 20 receives a notification instruction from the server device 30, it displays a message on its own display device 241, such as "Terminal device XX (or shopping cart YY) was taken out of entrance / exit AA at 15:42." Here, XX is identification information that identifies the terminal device 10, and YY is identification information that identifies the shopping cart. Also, AA is identification information that identifies the exit 9. At this time, the clerk terminal 20 may output a notification sound from the speaker 243, or may read out the message displayed on the display device 241.

[0105] A store clerk who notices the message goes to the relevant exit 9, finds the relevant terminal device 10 (or shopping cart 3), and urges the customer to return the shopping cart 3 to the cart storage area 8.

[0106] The clerk who has returned the customer to the store presses the "handling complete" button 2411 displayed on the display device 241 of his / her clerk terminal 20. The clerk terminal 20 outputs information indicating that the "handling complete" button 2411 has been pressed to the server device 30.

[0107] When the server device 30 acquires information indicating that the response completion button 2411 has been pressed, it instructs all of the store clerk terminals 20 to cancel the notification. As a result, all of the store clerk terminals 20 stop the notification.

[0108] (Notification method by notification device) When the server device 30 is notified that the shopping cart 3 (terminal device 10) has been taken into a prohibited area, the server device 30 also instructs the notification device 50 to make a notification.

[0109] The notification device 50 lights up or blinks a light-emitting element such as an LED (not shown) upon receiving a notification instruction from the server device 30. Furthermore, the notification device 50 outputs a notification sound or message from a speaker (not shown) upon receiving a notification instruction from the server device 30.

[0110] The customer notices the notification from the notification device 50 and returns the shopping cart 3 to the store. At this time, the terminal device 10 restores its connection with the network Na via the wireless access point 2. When the server device 30 detects that the terminal device 10 has restored its connection with the network Na, it instructs the notification device 50 to cancel the notification. As a result, the notification device 50 stops the notification.

[0111] By making the notification using the notification device 50, not only the customer who has taken out the shopping cart 3 but also customers who are near the exit 9 are effectively deterred from taking out the shopping cart 3.

[0112] (Functional configuration of the removal monitoring system) The functional configuration of the carry-out monitoring system 100 will be described with reference to Fig. 11. Fig. 11 is a functional block diagram showing an example of the functional configuration of the terminal device, the store clerk terminal, the monitoring camera, and the server device included in the carry-out monitoring system of the first embodiment.

[0113] (Functional configuration of terminal device) The control unit 11 of the terminal device 10 deploys the control program 121 in the RAM 113 and operates it, thereby realizing the shopping start instruction unit 161, product code acquisition unit 162, product registration unit 163, payment processing unit 164, status output unit 165, notification unit 166, and take-out flag release instruction unit 167 shown in Figure 11 as functional units.

[0114] The shopping start instruction unit 161 transitions the operating state of the terminal device 10 to "shopping" on the condition that the customer has performed an operation on the operation device 142 to instruct the start of shopping.

[0115] The product code acquisition unit 162 acquires the product code registered in the code symbol by reading the code symbol attached to or printed on the product held over the camera 143. Note that the product code acquisition unit 162 may also acquire the product code corresponding to the product from an image of the product captured by the camera 143.

[0116] The product registration unit 163 registers the product code acquired by the product code acquisition unit 162 in the product registration file 123 .

[0117] The payment processing unit 164 performs cashless payment using the payment method selected by the customer. The payment processing unit 164 also generates a payment file 124 that records the results of the payment.

[0118] The status output unit 165 repeatedly outputs to the server device 30 whether its own status is "shopping," "checkout," or "checkout completed." The status output unit 165 also repeatedly outputs information such as the shopping time, the time of operation, the number of registered items, the registered price, and the transaction code to the server device 30. The repetition time interval is not important, but may be a predetermined time interval, or may be the timing when the status of the terminal device 10 changes, or the timing when some operation is performed on the terminal device 10.

[0119] Upon receiving a notification instruction from the server device 30, the notification unit 166 performs the above-mentioned notification (see FIG. 9) using the display device 141 and the speaker 144. Furthermore, upon receiving a notification cancellation instruction from the server device 30, the notification unit 166 stops the notification.

[0120] When the terminal device 10 is making an announcement and detects that it has become possible to communicate with the server device 30 via the network Nb, the take-out flag release instruction unit 167 instructs the server device 30 to release the take-out flag 339 along with identification information (terminal code) that identifies the server device 30.

[0121] (Functional configuration of store clerk terminal) The control unit 21 of the store clerk terminal 20 loads the control program 221 into the RAM 213 and runs it, thereby realizing a notification unit 261 and a take-out flag release instruction unit 262 shown in FIG. 11 as functional units.

[0122] Upon receiving a notification instruction from the server device 30, the notification unit 261 performs the above-mentioned notification (see FIG. 10) using the display device 241 and the speaker 243. Furthermore, upon receiving notification cancellation from the server device 30, the notification unit 261 stops the notification.

[0123] When the clerk terminal 20 is making an alert and the response completion button 2411 (see Figure 10) is pressed, the take-out flag release instruction unit 262 outputs an instruction to the server device 30 to release the take-out flag 339 along with the terminal code that identifies the terminal device 10.

[0124] (Functional configuration of surveillance cameras) The control unit 41 of the surveillance camera 40 deploys the control program 421 in the RAM 413 and operates it, thereby realizing the imaging control unit 461, cart detection unit 462, identification information recognition unit 463, and carry-out information output unit 464 shown in Figure 11 as functional units.

[0125] The imaging control unit 461 causes the camera driver 44 to cause the imaging element 441 to capture an image.

[0126] The cart detection unit 462 detects the shopping cart 3 from the image captured by the imaging control unit 461. That is, the cart detection unit 462 detects that the shopping cart 3 equipped with the terminal device 10 has been taken out into a predetermined area. Specifically, the cart detection unit 462 detects a pattern similar to the cart recognition model 422 from the captured image. When the cart detection unit 462 detects a pattern that exceeds a preset evaluation score, it determines that the pattern represents a shopping cart. Note that the cart detection unit 462 is an example of a detection means in the present disclosure.

[0127] The identification information recognition unit 463 recognizes the identification information 65 that appears in the image captured by the imaging control unit 461. The identification information recognition unit 463 detects a pattern similar to the identification information recognition model 423 from the captured image. When the identification information recognition unit 463 detects a pattern that exceeds a preset evaluation score, it determines that it has recognized the identification information 65. Note that the processing performed by the identification information recognition unit 463 is widely applied to well-known license plate recognition, handwritten character recognition, etc.

[0128] In this way, the cart detection unit 462 and the identification information recognition unit 463 determine that the shopping cart 3 has been taken out of the exit 9 from a single image captured by the surveillance camera 40. Therefore, the surveillance camera 40 can immediately recognize that the shopping cart 3 (terminal device 10) has been taken out.

[0129] On the condition that it has detected that the terminal device 10 has left the store 7 through the exit 9 and been taken out into a predetermined area, the take-out information output unit 464 outputs identification information 65, which is information for identifying the shopping cart 3 (terminal device 10), identification information for identifying the surveillance camera 40 that detected the removal of the shopping cart 3, and the current time, to the server device 30. The take-out information output unit 464 is an example of an output means in the present disclosure.

[0130] (Functional configuration of server device) The control unit 31 of the server device 30 deploys the control program 321 in the RAM 313 and operates it, thereby realizing the terminal device status management unit 361, the take-out information acquisition unit 362, the notification instruction unit 363, and the take-out flag release unit 364 shown in Figure 11 as functional units.

[0131] The terminal device status management unit 361 manages the status of the terminal device 10 using a terminal device status management file 322 generated based on the operating status, operation information, etc. acquired from the terminal device 10.

[0132] The removal information acquisition unit 362 acquires, from the surveillance camera 40, identification information 65, which is information that identifies the shopping cart 3 (terminal device 10), identification information that identifies the surveillance camera 40 that detected the removal of the shopping cart 3, and the current time.

[0133] When the server device 30 acquires information indicating that the shopping cart 3 (terminal device 10) has exited through the exit 9, the notification instruction unit 363 outputs an instruction to at least the terminal device 10 whose exit was detected and the store clerk terminal 20 carried by the store clerk to notify that the terminal device 10 or the shopping cart 3 has been taken out. Note that in this embodiment, the notification instruction unit 363 also outputs an instruction to the notification device 50 to notify that the terminal device 10 or the shopping cart 3 has been taken out. The notification instruction unit 363 is an example of a notification means in the present disclosure.

[0134] The take-out flag canceling unit 364 acquires an instruction to cancel the take-out flag 339 from the terminal device 10 and the store clerk terminal 20. The take-out flag canceling unit 364 also instructs the terminal device 10, the store clerk terminal 20, and the notification device 50 to cancel the notification that is currently being executed.

[0135] The functional configuration of the takeout monitoring system 100 is not limited to the configuration shown in Fig. 11. For example, the monitoring camera 40 may output captured images to the server device 30, and the server device 30 may detect the shopping cart 3 and recognize the identification information 65.

[0136] (Processing flow of the removal monitoring system) The flow of processing performed by the takeout monitoring system 100 will be described with reference to Fig. 12. Fig. 12 is a sequence diagram showing an example of the operation of the takeout monitoring system of the first embodiment.

[0137] To make it easier to understand how data is received and transmitted between different devices, the processing flow will be explained in chronological order. After arriving at the store, a customer using a terminal device 10 attached to a shopping cart 3 presses, for example, a start shopping button (not shown) displayed on the display device 141. When the start shopping instruction unit 161 of the terminal device 10 detects that the start shopping button has been pressed, it transitions its own operating state to "shopping in progress" and outputs to the server device 30 the current time, its own terminal code, the identification information 65 of the shopping cart 3 associated with it, and a message indicating that the operating state is "shopping in progress" (step S11).

[0138] The terminal device status management unit 361 of the server device 30 acquires information indicating the "shopping" status from the terminal device 10, and starts status management using the terminal device status management file 322 (step S12).

[0139] The product code acquisition unit 162 of the terminal device 10 acquires the product code of the product purchased by the customer. The product registration unit 163 performs a product registration process to register the acquired product code in the product registration file 123. Then, the status output unit 165 outputs the contents of the product registration file 123 to the server device 30 every time a product is registered (step S13). Note that, since a customer generally purchases multiple products, step S13 is repeated multiple times.

[0140] The payment processing unit 164 of the terminal device 10 performs the payment according to the payment method specified by the customer and related to the contents of the product registration file 123. Then, the status output unit 165 outputs to the server device 30 the fact that the terminal device 10 has transitioned to the "checkout" status and the result of the payment processing (step S14).

[0141] The terminal device status management unit 361 of the server device 30 updates the contents of the terminal device status management file 322 every time it acquires information related to product registration and information related to payment processing from the terminal device 10 (step S15).

[0142] The imaging control unit 461 of the monitoring camera 40 causes the camera 40 to capture an image of the exit 9 (step S16).

[0143] The cart detection unit 462 of the surveillance camera 40 detects the shopping cart 3 from the captured image (step S17).

[0144] If a shopping cart 3 is detected in step S17, the identification information recognition unit 463 recognizes the identification information 65 written on the identification plate 64 of the detected shopping cart 3 (step S18).

[0145] The removal information output unit 464 of the surveillance camera 40 outputs to the server device 30 the identification information identifying the surveillance camera 40 that detected the removal of the shopping cart 3, the identification information 65 detected by the surveillance camera 40, and the current time (step S19).

[0146] The monitoring camera 40 repeatedly executes the processes from step S16 to step S19.

[0147] The take-out information acquisition unit 362 of the server device 30 acquires the identification information 65, the identification information specifying the surveillance camera 40 that detected the removal of the shopping cart 3, and the current time from the surveillance camera 40. Then, the terminal device status management unit 361 updates the contents of the terminal device status management file 322 based on the acquired information (step S20).

[0148] The notification instruction unit 363 of the server device 30 instructs the terminal device 10 that has been taken out, the store clerk terminal 20, and the notification device 50 installed at the exit 9 where the removal of the shopping cart 3 has been detected to issue a notification (step S21).

[0149] Upon receiving the instruction to make a notification from the server device 30, the notification unit 166 of the terminal device 10 makes a notification using the display device 141 and the speaker 144 (step S22).

[0150] The notification unit 261 of the store clerk terminal 20 receives the notification instruction from the server device 30 and makes a notification using the display device 241 and the speaker 243 (step S23).

[0151] Upon receiving the instruction to make a notification from the server device 30, the notification device 50 makes a notification using a light-emitting element and a speaker (not shown) (step S24).

[0152] When the take-out flag release instruction unit 167 of the terminal device 10 detects that the terminal device 10 is able to communicate with the server device 30 via the network Nb, it outputs an instruction to the server device 30 to release the take-out flag 339 along with a terminal code that identifies the terminal device 10 (step S25).

[0153] When the clerk terminal 20 is making an alert and the response completion button 2411 (see Figure 10) is pressed, the take-out flag release instruction unit 262 of the clerk terminal 20 outputs to the server device 30 an instruction to release the take-out flag 339 together with the terminal code identifying the terminal device 10 as information indicating that the response to the customer has been completed (step S26).

[0154] When the take-out flag cancellation unit 364 of the server device 30 receives an instruction to cancel the take-out flag 339 from the terminal device 10 or the store clerk terminal 20, it cancels the take-out flag 339 of the corresponding terminal device 10 stored in the terminal device status management file 322 (step S27).

[0155] Next, the take-out flag canceling unit 364 of the server device 30 outputs an instruction to cancel the ongoing notification to the terminal device 10, the store clerk terminal 20, and the notification device 50 (step S28).

[0156] The notification unit 166 of the terminal device 10 receives the notification cancellation instruction from the server device 30 and stops the notification (step S29).

[0157] The notification unit 261 of the store clerk terminal 20 receives an instruction to cancel the notification from the server device 30 and stops the notification (step S30).

[0158] The notification device 50 receives an instruction to cancel notification from the server device 30 and stops the notification (step S31).

[0159] (Modification of the first embodiment) In the first embodiment, the terminal device 10 is described as being attached to the shopping cart 3 in advance, but the terminal device 10 may be a device such as a smartphone that is loaned to the customer by the store. The customer may then attach the terminal device 10 to an empty shopping cart 3 and shop, or may hold the terminal device 10 loaned to them by the store and shop while pushing the shopping cart 3.

[0160] In such a case, the take-out monitoring system 100 needs to know in advance which shopping cart 3 the terminal device 10 loaned by the store is accompanying. Therefore, the customer who has been loaned the terminal device 10 first pairs the terminal device 10 with the shopping cart 3.

[0161] For example, pairing can be performed by reading the information marked on the identification plate 64 of the shopping cart 3 using the camera 143 of the terminal device 10. Alternatively, the customer may input the information marked on the identification plate 64 into the terminal device 10. The terminal device 10 outputs the read or input identification information to the server device 30. The server device 30 associates the acquired identification information with the terminal code 331 of the terminal device 10, thereby associating the terminal device 10 with the shopping cart 3. The server device 30 then stores the identification information 65 of the shopping cart 3 associated with the terminal code 331 of the terminal device 10 in the cart identification information 332 field in the terminal device status management file 322 described above. The subsequent processing is as described in the first embodiment.

[0162] When shopping is completed and the terminal device 10 is returned to a predetermined location, the association is cancelled, for example, by an operation of a store clerk.

[0163] Furthermore, the customer may carry the terminal device 10 provided by the store and proceed with shopping while placing products in a shopping basket or a personal bag, that is, without using the shopping cart 3.

[0164] In such a case, the server device 30 monitors whether the terminal device 10 has established communication with the server device 30 through which wireless access point 2. When a customer exits through an exit 9 while still carrying the terminal device 10, the server device 30 determines that the customer has exited through an exit 9 near the location of the wireless access point 2 with which the customer had communicated before communication with the terminal device 10 was lost, and issues the notification described in the first embodiment. That is, in this configuration, the removal of the terminal device 10 is detected by the communication state between the terminal device 10 and the server device 30, instead of by the surveillance camera 40.

[0165] Furthermore, the terminal device 10 included in the take-out monitoring system 100 may be configured to have only a product registration function, with payment being performed by a cash register. This allows a customer to make a cash payment using a change dispenser included in the cash register.

[0166] (Operation and effect of the first embodiment) As described above, the take-out monitoring system 100 of the first embodiment includes a surveillance camera 40 (detection device) having a cart detection unit 462 (detection means) that detects when a terminal device 10 having a product registration function that allows a customer to register products themselves or a shopping cart 3 equipped with the terminal device 10 is taken out into a predetermined area, and a take-out information output unit 464 (output means) that outputs information detected by the cart detection unit 462 to the server device 30, and a server device 30 having a notification instruction unit 363 (notification means) that, when the detected information is acquired, outputs information to notify the user that the terminal device 10 or the shopping cart 3 has been taken out. Therefore, it is possible to immediately detect and notify the user when a store-owned terminal device 10 is taken out of the store. This makes it possible to prevent the terminal device 10 from being taken out of the store.

[0167] Furthermore, in the removal monitoring system 100 of the first embodiment, the predetermined area is the area exited from the exit 9 of the store 7, and when the monitoring camera 40 (detection device) detects the terminal device 10 or the shopping cart 3 from an image taken around the exit 9, it determines that the terminal device 10 or the shopping cart 3 has been removed from the exit 9. Therefore, by detecting the shopping cart 3 moving together with the terminal device 10 instead of the terminal device 10, it is possible to detect removal of the terminal device 10.

[0168] Furthermore, in the removal monitoring system 100 of the first embodiment, when the monitoring camera 40 (detection device) detects, from the captured image, identification information 65 (first identification information) written on an identification plate 64 attached to the shopping cart 3 and which uniquely identifies the shopping cart 3, the monitoring camera 40 determines that the terminal device 10 has been removed from the exit 9. Therefore, even if any terminal device 10 loaned by the store is attached to the shopping cart 3, the removal of the terminal device 10 can be detected by associating (pairing) the shopping cart 3 with the terminal device 10 in advance.

[0169] (Second embodiment) Next, a description will be given of a second embodiment of a carry-out monitoring system 101. As shown in Fig. 1, the carry-out monitoring system 101 has a system configuration similar to that of the carry-out monitoring system 100, but uses a shopping cart 4 (see Fig. 13) instead of the shopping cart 3.

[0170] (Main components of the shopping cart) The configuration of the main parts of the shopping cart 4 used in the take-out monitoring system 101 of the second embodiment will be described with reference to Fig. 13. Fig. 13 is a diagram showing an example of the configuration of the main parts of the shopping cart used in the take-out monitoring system of the second embodiment.

[0171] The X-axis shown in FIG. 13 is an axis that passes through the shopping cart 4 from front to back and extends from the rear to the front of the shopping cart 4. The Y-axis is an axis that passes through the shopping cart 4 from left to right and extends from the right to the left in the direction of travel of the shopping cart 4. The Z-axis is an axis that passes through the shopping cart 4 from top to bottom and extends from the bottom to the top of the shopping cart 4.

[0172] The shopping cart 4 includes a storage section 63, casters 61, a handle 62, and a terminal device 10. These functions are the same as those of the shopping cart 3 described in the first embodiment, and therefore a description thereof will be omitted.

[0173] Identification plates 66 are attached to the frame of the shopping cart 4 at least at the bottom of the left and right sides in the Y-axis direction of the shopping cart 4. Identification information 67 that can uniquely identify the shopping cart 4 is written on the identification plates 66. The identification information 67 may be, for example, a code symbol such as a barcode, or numbers, letters, or a combination thereof. The identification information 67 may also include, for example, hiragana, katakana, kanji, and symbols. The identification information 67 is written so that when the surveillance camera 40 captures an image of the shopping cart 4, the identification information 67 appears in the captured image at a size that can be recognized by the surveillance camera 40. The identification information 67 is an example of first identification information in the present disclosure.

[0174] (Functional configuration of the removal monitoring system) The carry-out monitoring system 101 of this embodiment has a functional configuration in which the functional configuration of the monitoring camera 40 included in the carry-out monitoring system 100 is partially modified.

[0175] Specifically, the surveillance camera 40 included in the takeout monitoring system 101 does not have the cart detection unit 462 shown in Fig. 11. That is, the surveillance camera 40 included in the takeout monitoring system 101 detects that the shopping cart 4 has been taken out through the exit 9 by recognizing the identification information 67 written on the shopping cart 4 from the captured image. Therefore, the surveillance camera 40 included in the takeout monitoring system 101 does not have the cart recognition model 422 shown in Fig. 6.

[0176] The terminal device 10, the store clerk terminal 20, the server device 30, and the notification device 50 included in the takeout monitoring system 101 have the same functional configuration as the takeout monitoring system 100 (see FIG. 11).

[0177] (Method for detecting removal of terminal device) A method for detecting that the shopping cart 4 (terminal device 10) has been taken out into a take-out prohibited area will be described with reference to Fig. 14. Fig. 14 is a diagram for explaining a method for detecting that the terminal device has been taken out into a take-out prohibited area in the take-out monitoring system of the second embodiment.

[0178] 14 shows the exit 9 of the store as seen from inside the store. Surveillance camera 40 is installed horizontally facing the exit 9, for example, below alarm devices 50 installed on both sides of the exit 9, and monitors identification information 67 written on identification plate 66 when shopping cart 4 is taken out of the exit 9.

[0179] Preferably, the surveillance camera 40 is installed horizontally at a position approximately equal in height to the identification plate 66, facing the exit 9 side.

[0180] By arranging the surveillance cameras 40 in this layout, even when the exit 9 is crowded, the identification information 67 can be monitored without being overlooked.

[0181] On the condition that the monitoring camera 40 detects the identification information 67, the monitoring camera 40 outputs to the server device 30 the identification information that identifies the monitoring camera 40 itself and the detected identification information 67.

[0182] When the server device 30 acquires the identification information that identifies the surveillance camera 40, the detected identification information 67, and the time when the identification information 67 was recognized, it issues an alert to the terminal device 10 that has been taken out, the store clerk terminal 20, and the alert device 50, as described in the first embodiment.

[0183] (Effects of the second embodiment) As described above, in the removal monitoring system 101 of the second embodiment, identification information 67 (first identification information) that uniquely identifies the shopping cart 4 is attached to the bottom of the side of the shopping cart 4, and the monitoring camera 40 (detection device) is installed facing horizontally at a position equal in height to the identification information 67, and when the monitoring camera 40 detects the identification information 67 in a captured image, it determines that the terminal device 10 has been removed to a predetermined area. Therefore, even when the exit 9 is crowded, the identification information 67 can be monitored without being overlooked.

[0184] (Third embodiment) Next, a takeout monitoring system 102 according to a third embodiment will be described.

[0185] (Outline of the removal monitoring system) First, the schematic configuration of the carry-out monitoring system 102 will be described with reference to Fig. 15. Fig. 15 is a system configuration diagram showing an example of the schematic configuration of the carry-out monitoring system according to the third embodiment. As shown in Fig. 15, the carry-out monitoring system 102 includes a terminal device 10, a store clerk terminal 20, a server device 30, a monitoring camera 48, and an alarm device 50.

[0186] Of these, the terminal device 10, the store clerk terminal 20, the server device 30, and the notification device 50 have the same functions as those described in the first embodiment, and therefore a description thereof will be omitted.

[0187] Surveillance camera 48 is a camera provided on shopping cart 5. Surveillance camera 48 is installed facing the floor surface ahead in the direction of travel of shopping cart 5. Surveillance camera 48 captures images of the floor surface in the direction of travel of shopping cart 5, and detects, from the captured images, identification information 68 (see FIG. 17) marked on the floor surface near exit 9 of the store. Surveillance camera 48 is an example of a detection device in the present disclosure. The structure of shopping cart 5 will be described in detail later (see FIG. 16).

[0188] When the surveillance camera 48 detects identification information 68 written on the floor near the exit 9, the removal monitoring system 102 detects that the shopping cart 5 to which it is attached has been removed from the exit 9, and causes the terminal device 10, the store clerk terminal 20, and the alarm device 50 to issue an alarm.

[0189] (Functional configuration of the removal monitoring system) The carry-out monitoring system 102 of this embodiment has a functional configuration in which the functional configuration of the monitoring camera 40 included in the carry-out monitoring system 100 is partially modified.

[0190] Specifically, the surveillance camera 48 included in the takeout monitoring system 102 does not have the cart detection unit 462 shown in Fig. 11. That is, the surveillance camera 48 included in the takeout monitoring system 102 detects that the shopping cart 5 has been taken out of the exit 9 by recognizing the identification information 68 marked on the floor from the captured image. Therefore, the surveillance camera 48 included in the takeout monitoring system 102 does not have the cart recognition model 422 shown in Fig. 6.

[0191] The server device 30 also includes a management table (not shown) that associates identification information that uniquely identifies the surveillance camera 48 attached to the shopping cart 5 with the terminal code of the terminal device 10.

[0192] The terminal device 10, the store clerk terminal 20, the server device 30, and the notification device 50 included in the takeout monitoring system 102 have the same functional configuration as the takeout monitoring system 100 (see FIG. 11).

[0193] (Main components of the shopping cart) The main configuration of the shopping cart 5 used in the third embodiment will be described with reference to Fig. 16. Fig. 16 is a diagram showing an example of a shopping cart used in the take-out monitoring system of the third embodiment.

[0194] The X-axis shown in FIG. 16 is an axis that passes through the shopping cart 5 from front to back and extends from the rear to the front of the shopping cart 5. The Y-axis is an axis that passes through the shopping cart 5 from left to right and extends from the right to the left in the direction of travel of the shopping cart 5. The Z-axis is an axis that passes through the shopping cart 5 from top to bottom and extends from the bottom to the top of the shopping cart 4.

[0195] The shopping cart 5 includes a storage section 63, casters 61, a handle 62, and a terminal device 10. These functions are similar to those of the shopping cart 3 described in the first embodiment, and therefore will not be described further.

[0196] The shopping cart 5 further includes a surveillance camera 48 on the front side in the direction of travel. The surveillance camera 48 is installed facing slightly downward in the direction of travel of the shopping cart 5. The surveillance camera 48 is also connected to the server device 30 via the network Nb (see FIG. 15). The surveillance camera 48 has an image recognition function that detects, from captured images, identification information 68 (see FIG. 17) marked on the floor of the store.

[0197] (Method for detecting removal of terminal device) A method for detecting that the shopping cart 5 (terminal device 10) has been taken out into a take-out prohibited area will be described with reference to Fig. 17. Fig. 17 is a diagram for explaining a method for detecting that the terminal device has been taken out into a take-out prohibited area in the take-out monitoring system of the third embodiment.

[0198] The surveillance camera 48 attached to the shopping cart 5 captures images of the floor surface in the direction of travel of the shopping cart 5 under normal use conditions. Identification information 68 is written on the floor surface near the exit 9 of the store 7, for example, on the floor surface immediately after exiting the exit 9. The identification information 68 may be, for example, a code symbol such as a barcode, or numbers, alphabets, or a combination thereof. The identification information 68 may also include, for example, hiragana, katakana, kanji, and symbols. The identification information 68 may be written directly on the floor surface, or a plate or the like bearing the identification information 68 may be affixed to the floor surface. The identification information 68 includes information associated with the position of the exit 9. The server device 30 includes a management table (not shown) that associates the identification information 68 with the position of the exit 9. The identification information 68 is the second identification information in this disclosure.

[0199] The orientation of the identification information 68 does not matter, but in order to achieve the highest possible recognition performance, it is desirable to place the identification information 68 in a direction perpendicular to the direction in which the shopping cart 5 exits from the exit 9. Furthermore, if the exit width of the exit 9 is wide, multiple pieces of identification information 68 may be provided.

[0200] When the shopping cart 5 (terminal device 10) is taken into a prohibited area, the identification information 68 appears in the image captured by the surveillance camera 48. The surveillance camera 48 recognizes the identification information 68 captured in the image, and outputs the recognized identification information 68, the identification information that identifies the surveillance camera 48, and the time when the identification information 68 was recognized to the server device 30.

[0201] When the server device 30 acquires the recognized identification information 68, the identification information that identifies the surveillance camera 48, and the time when the identification information 68 was recognized, it issues an alert to the terminal device 10 that has been taken out, the store clerk terminal 20, and the alert device 50, as described in the first embodiment.

[0202] In this embodiment, the installation position of the identification information 68 is not limited to the floor of the store. The identification information 68 may be installed on the wall of the alarm device 50 or the ceiling of the store, for example. In this case, however, the installation direction of the surveillance camera 48 is set so that the identification information 68 appears in the image captured by the surveillance camera 48.

[0203] (Operation and effect of the third embodiment) As described above, in the removal monitoring system 102 of the third embodiment, the monitoring camera 48 (detection device) is attached to the shopping cart 5 and determines that the terminal device 10 has been removed to the predetermined area when it detects the identification information 68 (second identification information) installed in the predetermined area. Therefore, even if the exit 9 is crowded, it can reliably detect that the shopping cart 5 has been removed.

[0204] Although the embodiments of the present invention have been described above, these embodiments are merely examples and are not intended to limit the scope of the invention. This novel embodiment can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the inventions and their equivalents as defined in the claims. [Explanation of symbols]

[0205] 10 Terminal Equipment 100, 101, 102 Removal monitoring system 161 Shopping start instruction section 162 Product Code Acquisition Department 163 Product Registration Department 164 Payment processing unit 165 Status output unit 166 Information Department 167 Take-out flag release instruction section 2. Wireless Access Point 20. Clerk terminal (mobile terminal) 261 Information Department 262 Take-out flag release instruction section 3,4,5 Shopping Cart 30 Server device 322 Terminal device status management file 361 Terminal Device Status Management Unit 362 Export Information Acquisition Department 363 Notification instruction unit (notification means) 40, 48 Surveillance cameras (detection devices) 422 Cart Recognition Model 423 Identity Recognition Model 461 Imaging control unit 462 Cart detection unit (detection means) 463 Identification Information Recognition Unit 464 Take-out information output unit (output means) 50 Alarm device 61 Caster 62 Handle 63 Storage unit 64,66 Identification plate 65, 67 Identification information (first identification information) 68 Identification information (secondary identification information) 7 stores 8 Cart Parking 9 Exit Na,Nb network [Prior art documents] [Patent documents]

[0206] [Patent Document 1] Patent Publication No. 2021-114158

Claims

1. a terminal device having a product registration function for allowing a customer to register products by himself / herself, or a detection means for detecting that a shopping cart equipped with the terminal device has been taken into a predetermined area; a detection device having an output means for outputting information detected by the detection means to a server device; and a server device having a notification means for outputting information to notify the user that the terminal device or the shopping cart has been removed when the detected information is acquired. Take-out monitoring system.

2. the predetermined area is an area exiting from an exit of the store, The detection device includes: When the terminal device or the shopping cart is detected from an image taken around the exit, it is determined that the terminal device or the shopping cart has been taken out of the exit. The removal monitoring system according to claim 1 .

3. The detection device includes: determining that the terminal device has been taken out of the exit when first identification information attached to the shopping cart that uniquely identifies the shopping cart is detected from the captured image; The removal monitoring system according to claim 2 .

4. First identification information that uniquely identifies the shopping cart is attached to a lower portion of a side surface of the shopping cart, the detection device is installed facing horizontally at a position equal in height to the first identification information, and determines that the terminal device has been taken into the predetermined area when the detection device detects the first identification information from a captured image; The removal monitoring system according to claim 1 .

5. the detection device determines that the terminal device has been taken out into the predetermined area when detecting second identification information attached to the shopping cart and set in the predetermined area; The removal monitoring system according to claim 1 .

6. A terminal device having a product registration function for customers to register products themselves, or a computer included in a take-out monitoring system that detects and notifies when a shopping cart equipped with the terminal device is taken out into a predetermined area, a detection means for detecting that the terminal device or a shopping cart equipped with the terminal device has been taken out into a predetermined area; an output means for outputting the information detected by the detection means to a server device; a notification means for outputting information to notify the user that the terminal device or the shopping cart has been removed when the detected information is acquired; A program that makes it work.

Images