Shopping system and terminal device

Abstract

We provide technology to prevent unpaid goods from being taken out of stores. [Solution] The shopping system comprises a basket, a camera positioned above the opening of the basket, a first reception unit that accepts registration of items to be purchased, a first detection unit that analyzes an image of the edge of the opening captured by the camera to detect a first object crossing the edge from outside the basket into the basket, and a first notification unit that performs a first notification operation to notify that the items in the basket are unregistered when the first reception unit has not accepted registration of items and the first object is detected.

Description

Technical Field

[0001] The technology disclosed in this specification relates to a shopping system and a terminal device used for shopping for goods.

Background Art

[0002] The merchandise settlement system disclosed in Patent Document 1 captures an image of the merchandise placed on a merchandise transport cart, performs image processing on the image of the merchandise, and recognizes the merchandise. The merchandise settlement system registers the recognized merchandise as merchandise data. The merchandise settlement system settles the merchandise registered as merchandise data.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, it is possible to settle the merchandise to be purchased without waiting for registration by a store clerk at a POS terminal. However, if settlement is performed while the merchandise placed on the merchandise transport cart is not registered as merchandise data, there is a possibility that the unsettled merchandise may be taken out of the store. In this specification, a technology for suppressing the taking out of unsettled merchandise from the store is provided.

Means for Solving the Problems

[0005] The shopping system disclosed herein comprises a basket, a camera positioned above the opening of the basket, a first reception unit for receiving registration of items to be purchased, a first detection unit for analyzing an image of the edge of the opening captured by the camera to detect a first object crossing the edge from outside the basket into the basket, and a first notification unit for performing a first notification operation to notify that the items in the basket are unregistered when the first reception unit has not received registration of items and the first object is detected.

[0006] The presence of a first object crossing the edge of the basket from the outside to the inside suggests that an item has been placed in the basket. According to the above configuration, the user is notified that an unregistered item has been placed in the basket. The user can then be prompted to register the item. By ensuring the item is properly registered, it is possible to prevent unpaid items from being taken out of the store.

[0007] The terminal device itself, the computer program for the terminal device, and the control method for the terminal device are all novel and useful.

[0008] The first detection unit may further detect the number of the first objects and the number of second objects that cross the edge from inside the basket to outside the basket after the first objects have crossed the edge, and the first notification unit may perform the first notification operation if the first reception unit has not accepted product registration, the first objects have been detected, and the number of the first objects is greater than the number of the second objects.

[0009] When an item is placed in a basket, the hand and the item pass over the edge of the basket, and then only the hand passes over the edge. It is also conceivable that the item alone passes over the edge of the basket without the hand passing over it, and the item is placed in the basket. In either case, when an item is placed in the basket, the number of objects that pass from outside to inside is greater than the number of objects that pass from inside to outside. With the above configuration, it is possible to determine that an item has been placed in the basket by comparing the number of first objects with the number of second objects.

[0010] The first detection unit may start the process of detecting the first object when the image of the edge of the opening changes from one line to two or more lines.

[0011] When a single line changes into two or more lines, it means that some object has begun to pass over the edge of the cage. When an object begins to pass over the edge of the cage, detection of that object can be initiated.

[0012] The shopping system may further include a second detection unit that, after the first notification operation is performed, analyzes an image of the edge of the opening captured by the camera to detect a third object crossing the edge from inside the basket to outside the basket, and a first stop unit that stops the first notification operation when the third object is detected.

[0013] The presence of a third object crossing the edge of the basket from the inside to the outside suggests that an unregistered item has been removed from the basket. When it is determined that an unregistered item has been removed from the basket, the first notification operation can be stopped.

[0014] The shopping system further includes a display unit and a display control unit that displays a registration screen on the display unit for manual registration of products when the first reception unit has not accepted product registration by barcode reading and the first object is detected, and the first notification unit may perform the first notification operation when the manual registration of products is not performed on the registration screen.

[0015] If product registration is not done by scanning a barcode, users may manually enter product information after adding items to their cart. With the above configuration, a registration screen for manual entry will be displayed, prompting the user to manually register the products.

[0016] The shopping system may further include a second reception unit for receiving cancellations of product registrations, a third detection unit for analyzing an image of the edge of the opening captured by the camera to detect a fourth object crossing the edge from inside the basket to outside the basket, and a second notification unit for performing a second notification operation to notify that the product registration has not been canceled between the time the second reception unit receives the cancellation of the product registration and the time the fourth object is detected.

[0017] The presence of a fourth object crossing the rim of the basket from inside to outside suggests that a product whose registration has been canceled has been removed from the basket. Conversely, the absence of a fourth object suggests that a product whose registration has been canceled remains in the basket. According to the above configuration, the user is informed that an unregistered product has been placed in the basket, as well as that a product whose registration has not been canceled remains in the basket. By correctly canceling product registrations, it is possible to prevent unpaid products from being taken out of the store.

[0018] Other shopping systems disclosed herein include a basket, a camera positioned above the opening of the basket, a first reception unit for receiving registration of items to be purchased, a second reception unit for receiving cancellation of item registration, a third detection unit for analyzing an image of the edge of the opening captured by the camera to detect a fourth object crossing the edge from inside the basket to outside the basket, and a second notification unit for performing a second notification operation after the second reception unit has received a cancellation of item registration, until the fourth object is detected, to notify that the item registration has not been canceled.

[0019] Even with the above configuration, the correct cancellation of product registrations can prevent unpaid products from being taken out of the store.

[0020] The third detection unit further detects the number of fifth objects that cross the edge from outside the basket to inside the basket, and the number of fourth objects that cross the edge from inside the basket to outside the basket after the fifth objects cross the edge. The shopping system may further include a second stop unit that stops the second notification operation when the fourth object is detected and the number of fifth objects is less than the number of fourth objects.

[0021] When a product is taken out of the basket, the hand passes through the edge of the basket, and then the hand and the product pass through the edge of the basket. That is, when a product is taken out of the basket, the number of objects passing from the outside to the inside is less than the number of objects passing from the inside to the outside. According to the above configuration, it is possible to determine that a product has been taken out of the basket by comparing the number of fifth objects with the number of fourth objects.

[0022] The image of the edge of the opening may include a plurality of frames obtained by continuously imaging the opening.

[0023] By analyzing a plurality of frames, it is possible to analyze the direction in which an object passes through the edge.

Brief Description of Drawings

[0024] [Figure 1] It is a schematic diagram of a shopping system. [Figure 2] It is a flowchart of the process executed by the terminal device. [Figure 3] It is a continuation flowchart of FIG. 2. [Figure 4] It is a flowchart of the analysis process. [Figure 5] It is a schematic diagram of case C1 where a product is put into the basket. [Figure 6] It is a schematic diagram of case C2 where a product is put into the basket. [Figure 7] It is a schematic diagram of case C3 where a product is taken out of the basket. [Figure 8]This is a flowchart illustrating the process performed by the terminal device according to the second embodiment. [Modes for carrying out the invention]

[0025] (Configuration of shopping system 2; Figure 1) Shopping system 2 is used in stores that sell goods 4 such as groceries, daily necessities, general merchandise, and home appliances. Shopping system 2 comprises a terminal device 10, a cart 100, a basket 110, and a server 200. The cart 100 is configured to accommodate the basket 110. When the basket 110 is loaded onto the cart 100, the opening of the basket 110 faces upward.

[0026] The terminal device 10 is a portable terminal device such as a smartphone or tablet. The terminal device 10 is lent to the user by the store. The terminal device 10 is mounted on a mounting base 102 located on top of the cart 100.

[0027] The terminal device 10 comprises a communication unit 12, an input unit 14, a display unit 16, a camera 18, and a control unit 30. The communication unit 12 is an interface used for communication with an external server 200. The server 200 is, for example, located on the Internet. In a modified example, the server 200 may be located on a Local Area Network (LAN).

[0028] The input unit 14 is a user interface capable of receiving user input. The input unit 14 may include, for example, a touchscreen, mechanical keys, or both. The display unit 16 is a display for displaying various information, and may include, for example, a liquid crystal display or an organic EL display. The display unit 16 may or may not be a touchscreen. The camera 18 is exposed from the housing of the terminal device 10 and is capable of capturing images of the area around the terminal device 10.

[0029] The control unit 30 controls parts 12, 14, 16, and 18. The control unit 30 includes a CPU 32 and a memory 34. The memory 34 is composed of volatile and non-volatile memory. The CPU 32 performs various processes according to the program 40 in the memory 34.

[0030] The terminal device 10 is mounted on the mounting base 102 such that the camera 18 faces downward. As a result, the camera 18 of the terminal device 10 is directed towards the opening of the cage 110 and captures an image of the opening of the cage 110. Also, with the terminal device 10 mounted on the mounting base 102, the input unit 14 and display unit 16 of the terminal device 10 face upward.

[0031] In shopping system 2, the user walks through the store pushing a cart 100 while placing items 4 into the basket 110. When the user places items 4 into the basket 110, they input information about the items 4 into the terminal device 10. Methods for inputting information about items 4 include, for example, scanning the barcode displayed on the item 4, and manual input into the input unit 14. In barcode scanning, a barcode is displayed on the item 4, and the barcode captured by the camera 18 is decoded to input information about the item 4 into the terminal device 10. In manual input, for example, the user selects the item they wish to purchase from a list of items 4 displayed on the display unit 16. Manual input is used, for example, for items 4 that do not have a barcode displayed, such as fresh produce and vegetables.

[0032] When information for product 4 is entered into the terminal device 10, it transmits the information for product 4 to the server 200. This registers the product 4 that the user wishes to purchase with the server 200. Once the user has registered all the products 4 they wish to purchase, they enter a payment instruction into the input section 14 of the terminal device 10. The payment instruction is sent from the terminal device 10 to the server 200, and all the products 4 registered with the server 200 are paid for. The payment method is, for example, credit card payment. The shopping system 2 eliminates the need for store clerks to register products 4 on the POS terminal. Users do not have to wait in line in front of store clerks operating the POS terminal. This improves user convenience.

[0033] However, if payment is made without the items 4 placed in the basket 110 being registered with the server 200, there is a possibility that the unpaid items 4 may be taken out of the store. In this embodiment, the process shown in Figure 2 is executed to prevent the unpaid items 4 from being taken out of the store.

[0034] (Processing by terminal device 10; Figures 2 and 3) Referring to Figures 2 and 3, the process executed by the CPU 32 of the terminal device 10 according to the program 40 will be described. The process in Figure 2 is triggered when an instruction to start shopping is input to the input unit 14.

[0035] In S10, the CPU 32 activates the camera 18. The camera 18 captures an image of the opening of the cage 110 and supplies the image of the opening of the cage 110 to the CPU 32. As shown in Figure 1, the camera 18 is positioned above the opening of the cage 110, and the image captured by the camera 18 includes the four edges 110a to 110d of the opening of the cage 110. In the modified example, the image captured by the camera 18 only needs to include at least one of the four edges 110a to 110d.

[0036] In S12, the CPU 32 determines whether or not object 6 has passed through one of the four edges 110a to 110d. As shown in Figure 1, before object 6 passes through edge 110c, edge 110c is imaged as a single line. On the other hand, when object 6 passes through edge 110c, edge 110c is imaged as two lines. If two objects 6 pass through edge 110c, edge 110c may be imaged as three lines. The CPU 32 detects edges 110a to 110d as edges from the image of the opening of the cage 110. The CPU 32 determines that object 6 has passed through one of the four edges 110a to 110d when an edge indicating one of the edges 110a to 110d changes from a single line to two or more lines. Note that the above method is only one example of a method for determining the passage of object 6 through edge 110a, etc.

[0037] If the CPU 32 determines that object 6 has passed through one of the four edges 110a to 110d (YES in S12), the process proceeds to S14. In S14, the CPU 32 starts an analysis process using multiple frames captured by the camera 18 (see Figure 3). The multiple frames are images of the opening of the basket 110 taken in succession. The analysis process analyzes the increase or decrease in the amount of goods inside the basket 110.

[0038] In S16, the CPU 32 determines whether a registration operation has been performed to register information about product 4 to the terminal device 10, i.e., whether the barcode has been read using the camera 18 or manually entered into the input unit 14. If the CPU 32 determines that a registration operation has been performed to the terminal device 10 (YES in S16), the process proceeds to S18.

[0039] In S18, the CPU 32 displays a message on the display unit 16 prompting the user to place item 4 into the basket 110 (for example, "Please put the item in"). While the message in S18 is displayed, operations on the terminal device 10 are restricted.

[0040] In S20, the CPU 32 determines whether the result of the analysis process in S14 is an increase in product 4. If the CPU 32 determines that the result of the analysis process in S14 is an increase in product 4 (YES in S30), it proceeds to S22. In S22, the CPU 32 clears the message in S18. When S22 is finished, the CPU 32 returns to S12.

[0041] Furthermore, if CPU32 determines that the result of the analysis process in S14 is not an increase in product 4 (NO in S30), it returns to S18 and continues displaying the message.

[0042] Furthermore, if CPU32 determines that no registration operation has been performed (NO in S16), it proceeds to S30. S30 is the same as S20. If CPU32 determines that the result of the analysis process in S14 is an increase in product 4 (YES in S30), it proceeds to S32.

[0043] In S32, the CPU 32 displays a message on the display unit 16 prompting the user to remove item 4 from the basket 110 (for example, "Please remove the item"). This is because, despite no registration operation being performed, the number of items 4 has increased, suggesting that an unregistered item 4 has been placed in the basket 110. While the message in S32 is displayed, operations on the terminal device 10 are restricted.

[0044] In the following S34, the CPU 32 resumes the analysis process shown in Figure 3. In S36, the CPU 32 determines whether the result of the analysis process resumed in S34 is a decrease in product 4. If the CPU 32 determines that the result of the analysis process resumed in S34 is a decrease in product 4 (YES in S36), it proceeds to S38. In S38, the CPU 32 clears the message from S32. The reason why the result of the analysis process resumed in S34 is a decrease in product 4 is that it is presumed that an unregistered product 4 was removed from the basket 110. When S38 finishes, the CPU 32 returns to S12.

[0045] Furthermore, if CPU32 determines that the result of the analysis process resumed in S34 is not a decrease in product 4 (NO in S36), it returns to S32 and continues displaying the message.

[0046] Furthermore, if the CPU 32 determines that object 6 has not passed through any of the four edges 110a to 110d (NO in S12), the process proceeds to S50 in Figure 3. In S50, the CPU 32 determines whether or not a cancellation operation has been performed on the terminal device 10 to cancel the registration of product 4 registered with the server 200. In a cancellation operation, the user selects the product 4 they wish to cancel from the list of registered products 4 displayed on the display unit 16. As a result, the terminal device 10 sends an instruction to the server 200 to cancel the product 4 selected by the user.

[0047] If the CPU 32 determines that a cancellation operation has been performed on the terminal device 10 (YES in S50), it proceeds to S52. S52 is the same as S32. In the following S54, the CPU 32 determines, as in S12, whether or not object 6 has passed through one of the four edges 110a to 110d.

[0048] If the CPU 32 determines that object 6 has passed through one of the four edges 110a to 110d (YES in S54), it proceeds to S56. In S56, the CPU 32 performs the analysis process shown in Figure 3. In S58, the CPU 32 determines whether the result of the analysis process in S56 is a decrease in product 4. If the CPU 32 determines that the result of the analysis process in S56 is a decrease in product 4 (YES in S58), it proceeds to S60. In S60, the CPU 32 clears the message from S52. When S60 is finished, the CPU 32 returns to S12.

[0049] Furthermore, if the CPU 32 determines that object 6 does not pass through any of the four edges 110a to 110d (NO in S54), or if it determines that the result of the analysis process in S56 is not a decrease in product 4 (NO in S58), it returns to S52 and continues displaying the message.

[0050] Furthermore, if the CPU 32 determines that no cancellation operation has been performed on the terminal device 10 (NO in S50), the process proceeds to S70. In S70, the CPU 32 determines whether or not a settlement operation has been performed on the terminal device 10 to settle the payment for the product 4 registered with the server 200. A settlement operation is, for example, inputting a settlement instruction into the input unit 14.

[0051] If the CPU 32 determines that a payment operation has been performed on the terminal device 10 (YES in S70), it proceeds to S72. In S72, the CPU 32 sends a payment instruction to the server 200. When S72 is finished, the CPU 32 terminates the process shown in Figure 2. On the other hand, if the CPU 32 determines that a payment operation has not been performed on the terminal device 10 (NO in S70), it returns to S12.

[0052] (Analysis process; Figure 4) In S100, the CPU 32 identifies the number of objects 6 that enter the cage 110 in multiple frames captured by the camera 18. Specifically, the CPU 32 analyzes the features of the moving objects 6 (e.g., color, shape, color combination) in multiple frames and distinguishes the objects 6. The CPU 32 then identifies the number of images of the one or more distinguished objects 6 that cross an edge indicating one of the edges 110a to 110d from outside to inside the cage 110.

[0053] In S102, the CPU 32 identifies the number of objects 6 exiting the cage 110 in multiple frames captured by the camera 18. Specifically, the CPU 32 identifies the number of images of one or more distinguished objects 6 in which an edge indicating one of the edges 110a to 110d crosses from inside to outside the cage 110.

[0054] In S110, the CPU 32 determines whether the number of incoming objects identified in S100 is greater than the number of outgoing objects identified in S102. If the CPU 32 determines that the number of incoming objects is greater than the number of outgoing objects (YES in S110), it proceeds to S112. In S112, the CPU 32 determines an increase in product 4 as a result of the analysis process shown in Figure 4.

[0055] Furthermore, if the CPU 32 determines that the number of objects entering is less than or equal to the number of objects leaving (NO in S110), it proceeds to S120. In S120, the CPU 32 determines whether the number of objects entering is less than the number of objects leaving. If the CPU 32 determines that the number of objects entering is less than the number of objects leaving (YES in S120), it proceeds to S122. In S122, the CPU 32 determines a decrease in product 4 as a result of the analysis process shown in Figure 4.

[0056] Furthermore, if the CPU 32 determines that the number of objects entering is the same as the number of objects leaving (NO in S120), it proceeds to S124. In S124, the CPU 32 determines that there is no increase or decrease in product 4 as a result of the analysis process in Figure 4. When any of S112, S122, or S124 is completed, the CPU 32 terminates the process in Figure 4.

[0057] (Case C1 where product 4 is placed in basket 110; Figure 5) In case C1, the user picks up product 4 with hand 300 and places it into basket 110. In this case, two objects 6, hand 300 and product 4, move. The CPU 32 analyzes the image of hand 300 and the image of product 4 and distinguishes between the two images. As shown in Figure 5, the image of hand 300 and the image of product 4 cross the edge indicating the edge 110c from outside to inside the basket 110. The CPU 32 identifies the direction in which the image of hand 300 is moving, "outside to inside," across multiple frames. Similarly, the CPU 32 identifies the direction in which the image of product 4 is moving, "outside to inside," across multiple frames. Based on this, the CPU 32 identifies the number of objects 6 that go into basket 110 as "2" (S100 in Figure 4).

[0058] In this case, the user leaves product 4 in the basket 110 and moves hand 300 outside the basket 110. The CPU 32 determines that the image of product 4 is not moving from inside the basket 110, while determining that the image of hand 300 is crossing the edge 110c from inside to outside the basket 110. The CPU 32 identifies the direction of movement of the hand 300 image, "from inside to outside," across multiple frames. Based on this, the CPU 32 determines that the number of objects 6 leaving the basket 110 is "1" (S102).

[0059] In this case, the CPU 32 determines that the number of incoming objects 6 ("2") is greater than the number of outgoing objects 6 ("1") (YES in S110), and as a result of the analysis process, it decides to increase the quantity of product 4 (S112).

[0060] (Case C2 where product 4 is placed in basket 110; Figure 6) In case C2, the user throws product 4 into basket 110. In this case, one product 4 moves. As shown in Figure 6, the image of product 4 crosses the edge indicating the edge 110c from outside to inside basket 110. The CPU 32 analyzes the image of product 4 and identifies the direction in which the image of product 4 is moving, "outside to inside," across multiple frames. The CPU 32 identifies the number of objects 6 that enter basket 110 as "1" (S100 in Figure 4).

[0061] Furthermore, in this case, product 4 remains inside the basket 110, and the image of product 4 does not move from inside to outside the basket 110. The CPU 32 determines that the image of product 4 does not cross any of the edges 110a to 110d in multiple frames. The CPU 32 determines that the number of objects 6 leaving the basket 110 is "0" (S102).

[0062] In this case, the CPU 32 determines that the number of incoming objects 6 ("1") is greater than the number of outgoing objects 6 ("0") (YES in S110), and as a result of the analysis process, it decides to increase the quantity of product 4 (S112).

[0063] (Case C3 of removing product 4 from basket 110; Figure 7) In case C3, the user places their hand 300 into the basket 110, picks up the product 4 inside the basket 110, and removes the product 4 from the basket 110. In this case, two objects 6, the hand 300 and the product 4, move. The CPU 32 analyzes the image of the hand 300 and the image of the product 4 and distinguishes between the two images. As shown in Figure 5, the image of the hand 300 crosses the edge indicating the edge 110c from outside to inside the basket 110. The CPU 32 identifies the direction in which the image of the hand 300 is moving, "outside to inside," across multiple frames. The CPU 32 identifies the number of objects 6 entering the basket 110 as "1" (S100 in Figure 4).

[0064] In this case, the user moves the hand 300 holding product 4 out of the basket 110. As shown in Figure 7, the images of the hand 300 and product 4 cross the edge representing the edge 110c from inside to outside the basket 110. The CPU 32 identifies the direction in which the hand 300 image is moving, "from inside to outside," across multiple frames. Similarly, the CPU 32 identifies the direction in which the product 4 image is moving, "from inside to outside," across multiple frames. Based on this, the CPU 32 identifies the number of objects 6 exiting the basket 110 as "2" (S100 in Figure 4).

[0065] In this case, the CPU 32 determines that the number of incoming objects 6 ("1") is less than the number of outgoing objects 6 ("2") (YES in S120), and as a result of the analysis process, it decides to reduce the quantity of product 4 (S122).

[0066] (Effects of this embodiment) Since object 6 crosses one of the edges 110a to 110d of basket 110 from the outside to the inside, it is inferred that product 4 has been placed in basket 110. In this embodiment, if product 4 is detected crossing one of the edges 110a to 110d of basket 110 from the outside to the inside, but the registration operation has not been performed (NO in S16 and YES in S30 in Figure 2), a message prompting the user to remove product 4 from basket 110 is displayed (S32). The user is informed that an unregistered product 4 has been placed in basket 110. The user can be prompted to register product 4. By correctly registering product 4, it is possible to prevent unpaid product 4 from being taken out of the store.

[0067] Furthermore, cases C1 and C2 in Figures 5 and 6 are assumed to be cases in which product 4 is placed into basket 110. In either case, the number of objects 6 that enter is greater than the number of objects 6 that leave (YES in S110 in Figure 4). By comparing the number of objects 6 that enter with the number of objects 6 that leave, it can be determined that product 4 has been placed in basket 110. Note that the analysis process in Figure 4 is merely one example of a method for analyzing the increase or decrease of product 4. Other methods for analyzing the increase or decrease of product 4 using information other than the number may also be employed.

[0068] Furthermore, the CPU 32 monitors changes in the edges 110a to 110d, and if the edge changes from a single line to two or more lines (YES in S12), it starts the analysis process (S14). The analysis process can also be started when some object 6 begins to pass the edge of the cage 110.

[0069] Furthermore, since object 6 crosses one of the edges 110a to 110d of basket 110 from the inside to the outside, it can be inferred that product 4 has been removed from basket 110. In this embodiment, although a cancellation operation has been performed, the message prompting the user to remove product 4 from basket 110 continues to be displayed until product 4 is detected crossing one of the edges 110a to 110d of basket 110 from the inside to the outside (S52 in Figure 3). By removing product 4, whose registration has been canceled, from basket 110, it is possible to prevent unpaid product 4 from being taken out of the store.

[0070] Furthermore, Case C3 in Figure 7 is considered as a case where product 4 is removed from basket 110. In this case, the number of objects 6 that enter is less than the number of objects 6 that exit (YES in S120 in Figure 4). By comparing the number of objects 6 that enter and the number of objects 6 that exit, it can be determined that product 4 has been removed from basket 110.

[0071] (Correspondence) Shopping system 2, basket 110, and camera 18 are examples of "shopping system," "basket," and "camera," respectively. Edge 110a, etc., is an example of "edge." The process in S32 in Figure 2 and the process in S52 in Figure 3 are examples of "first notification operation" and "second notification operation," respectively. Hand 300 and product 4 in case C1 of Figure 5 and hand 300 in case C2 of Figure 6 are examples of "first object." Hand 300 in case C1 of Figure 5 is an example of "second object." Hand 300 and product 4 in case C3 of Figure 7 are examples of "third object" and "fourth object." Hand 300 in case C3 of Figure 7 is an example of "fifth object."

[0072] In Figure 2, steps S16, S14, and S32 are examples of processes implemented by the "first reception unit," "first detection unit," and "first notification unit," respectively. Steps S34 and S38 are examples of processes implemented by the "second detection unit" and "first stop unit," respectively. Steps S50, S56, and S52 in Figure 3 are examples of processes implemented by the "second reception unit," "third detection unit," and "second notification unit," respectively. Step S60 is an example of a process implemented by the "second stop unit."

[0073] (Second example) (Processing by terminal device 10; Figure 8) This embodiment is the same as the first embodiment, except that the processing steps S200 to S204 in Figure 8 have been added.

[0074] If the CPU 32 determines that the result of the analysis process in S14 is an increase in product 4 (YES in S30), it proceeds to S200. In S200, the CPU 32 determines whether or not barcode reading was performed using the camera 18. If the CPU 32 determines that barcode reading was not performed (NO in S200), it proceeds to S202.

[0075] In S202, the CPU 32 displays a registration screen on the display unit 14 for registering products by manual input into the input unit 14. The registration screen includes a selection field for selecting the desired product 4 from a list of products 4, and an input field for entering the quantity of product 4.

[0076] In S204, CPU32 determines whether or not product 4 has been registered using the registration screen in S202. If CPU32 determines that product 4 has not been registered using the registration screen (NO in S204), it proceeds to S32. For example, CPU32 proceeds to S32 if a predetermined time (e.g., 30 seconds) has elapsed since the registration screen was displayed, but product 4 has not been selected and the quantity entered on the registration screen.

[0077] Furthermore, if CPU32 determines that a barcode has been read (YES in S200) or that product 4 has been registered using the registration screen (YES in S204), it skips the processing from S32 onwards and returns to S12.

[0078] If product registration is not performed by barcode scanning, the user places product 4 in the basket 110 and then manually registers product 4. In this embodiment, if an increase in product 4 is detected and barcode scanning has not been performed (YES in S30, NO in S200), the terminal device 10 displays a registration screen for manual input (S202). The registration screen prompts the user to manually register product 4. If manual input is not performed via the registration screen, the terminal device 10 displays a message prompting the user to remove product 4 from the basket 110 (S32). This informs the user that there is an unregistered product 4 in the basket 110.

[0079] The above describes specific examples of the technology disclosed herein, but these are merely illustrative and do not limit the scope of the claims. The technology described in the claims includes various modifications and changes to the specific examples described above. For example, the following modifications may be adopted.

[0080] (Modification 1) The terminal device 10 may be a smartphone owned by the user making the purchase. In this modification, an application program for communicating with the server 200 may be installed on the user's smartphone. The processes shown in Figures 2 to 4 may be executed according to the application program. In this modification, the "camera," "reception unit 1," "first detection unit," and "first notification unit" may be provided on the user's smartphone.

[0081] (Modification 2) The cage 110 may be equipped with a camera capable of communicating with the terminal device 10, rather than the terminal device 10 itself. In this modification, the camera capable of communicating with the terminal device 10 is an example of a "camera".

[0082] (Modification 3) The terminal device 10 may output voice, warning sounds, etc. instead of displaying the messages in S18, S32 in Figure 2 and S52 in Figure 3. In this modification, the output of voice, warning sounds, etc. is an example of the "first notification operation" and the "second notification operation".

[0083] (Modification 4) The processes S34 to S38 in Figure 2 do not need to be executed. In this modification, the "second detection unit" and the "first stopping unit" can be omitted.

[0084] (Modification 5) The processes S52 to S60 in Figure 3 do not need to be executed. In this modification, the "third detection unit" and the "second notification unit" can be omitted.

[0085] (Modification 6) The processes S16 and S30-S38 in Figure 2 may not be executed, and the processes S52-S60 in Figure 3 may be executed instead. In this modification, the "first detection unit" and the "first notification unit" can be omitted.

[0086] (Modification 7) The processes S56 to S60 in Figure 3 do not need to be executed. In this modification, the "second stopping section" can be omitted.

[0087] The technical elements described herein or in the drawings demonstrate technical usefulness individually or in various combinations, and are not limited to the combinations described in the claims at the time of filing. Furthermore, the technologies illustrated herein or in the drawings achieve multiple objectives simultaneously, and achieving even one of these objectives constitutes technical usefulness in itself. [Explanation of symbols]

[0088] 2: Shopping System 4: Product 6:Object 10: Terminal device 12: Communications Department 14: Input section 16: Display section 18: Camera 30: Control Unit 32:CPU 34: Memory 40: Program 100: Cart 102: Installation stand 110: Basket 110a~110d: Edge 200: Server 300: Hand

Claims

1. It is a shopping system, basket, A camera positioned above the opening of the aforementioned cage, The first reception desk accepts registrations for the products to be purchased, A first detection unit analyzes an image of the edge of the opening captured by the camera to detect a first object crossing the edge from outside the basket into the basket, A first notification unit performs a first notification operation to notify that the items in the basket are unregistered when the first reception unit has not accepted the registration of an item and the first object is detected. A shopping system equipped with these features.

2. The first detection unit further detects the number of the first objects and the number of second objects that cross the edge from inside the basket to outside the basket after the first objects have crossed the edge. The shopping system according to claim 1, wherein the first notification unit performs the first notification operation when the first reception unit has not accepted registration of a product, the first object is detected, and the number of the first objects is greater than the number of the second objects.

3. The shopping system according to claim 1, wherein the first detection unit starts processing to detect the first object when the image of the edge of the opening changes from one line to two or more lines.

4. The aforementioned shopping system further, After the first notification operation is performed, a second detection unit analyzes the image of the edge of the opening captured by the camera to detect a third object crossing the edge from inside the cage to outside the cage, A first stop unit that stops the first notification operation when the third object is detected, The shopping system according to claim 1, comprising:

5. The aforementioned shopping system further, Display unit and The first reception unit does not accept product registration by barcode reading, and when the first object is detected, the display control unit displays a registration screen for manual product registration on the display unit. The shopping system according to claim 1, wherein the first notification unit performs the first notification operation when the product is not registered by manual input on the registration screen.

6. The aforementioned shopping system further, The second reception desk accepts cancellations of product registrations, A third detection unit analyzes an image of the edge of the opening captured by the camera to detect a fourth object crossing the edge from inside the cage to outside the cage, A second notification unit performs a second notification operation to notify that the product registration has not been canceled after the second reception unit has received the cancellation of the product registration and before the fourth object is detected, The shopping system according to claim 1, comprising:

7. It is a shopping system, basket, A camera positioned above the opening of the aforementioned cage, The first reception desk accepts registrations for the products to be purchased, The second reception desk accepts cancellations of product registrations, A third detection unit analyzes an image of the edge of the opening captured by the camera to detect a fourth object crossing the edge from inside the cage to outside the cage, After the second reception unit has received the cancellation of the product registration, the second notification unit performs a second notification operation to notify that the product registration has not been canceled until the fourth object is detected, A shopping system equipped with these features.

8. The third detection unit further detects the number of fifth objects that cross the edge from outside the basket to inside the basket, and the number of fourth objects that cross the edge from inside the basket to outside the basket after the fifth objects have crossed the edge. The aforementioned shopping system further, The shopping system according to claim 6 or 7, further comprising a second stop unit that stops the second notification operation when the fourth object is detected and the number of the fifth object is less than the number of the fourth object.

9. The shopping system according to claim 1 or 7, wherein the image of the edge of the aperture includes a plurality of frames obtained by continuously capturing the aperture.

10. terminal device A camera capable of capturing images of the opening of a shopping basket, The first reception desk accepts registrations for the products to be purchased, A first detection unit analyzes an image of the edge of the opening captured by the camera to detect a first object crossing the edge from outside the basket into the basket, A first notification unit performs a first notification operation to notify that the items in the basket are unregistered when the first reception unit has not accepted the registration of an item and the first object is detected. A terminal device equipped with the following features.

Images