Control device, behavioral guidance system, and program

Abstract

To enable shoppers to curb unconscious misbehavior and conscious misbehavior.SOLUTION: A control device includes an action recognition unit, a discrimination unit, a next action reporting unit, a correct action reporting unit, and a warning unit. The action recognition unit recognizes actions performed by a shopper. The discrimination unit discriminates a next action, which is the next action to be taken by the shopper, based on the action recognized by the action recognition unit. Based on the next action determined by the discriminating unit, the next action reporting unit operates a reporting member provided correspondingly to an action area in which the next action is to be performed in a first reporting form. The correct action reporting unit operates the reporting member in a second reporting form when the action recognition unit recognizes that the shopper has performed the next action. When the action recognition unit recognizes that the shopper has performed an action different from the next action, the warning unit allows the reporting member corresponding to the action area in which the shopper has performed the different action to operate in a warning form.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] Embodiments of the present invention relate to a control device, a behavior guidance system, and a program.

Background Art

[0002] In recent years, self-checkout terminals called self-POS terminals for users to register purchased goods and perform accounting processes have been provided, and the number of stores adopting a sales form in which shoppers register and perform accounting processes for goods has been increasing. Generally, in such self-POS terminals, shoppers scan the code symbols attached to goods with a scanner to register the goods.

[0003] In such self-POS terminals, various techniques have been proposed to prevent fraudulent acts. For example, Patent Document 1 proposes a technique of following the face of a robot with the goods held by a customer to give an impression that the terminal user is being monitored and suppressing the occurrence of fraudulent acts such as shoplifting. Further, this Patent Document 1 also discloses notifying a store clerk such as an attendant when some operation error or fraud has occurred.

[0004] Conventionally, there has been no mechanism to visually convey to shoppers, who are users, whether their current actions are correct or incorrect. Therefore, it has not been possible to suppress unconscious incorrect actions or conscious fraudulent actions by users.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] The problem that the embodiments of the present invention aim to solve is to provide a control device, a behavioral guidance system, and a program that can suppress unconscious erroneous behavior and conscious fraudulent behavior by shoppers who are users. [Means for solving the problem]

[0007] In one embodiment, the control device controls notification members provided in each of a plurality of action areas where a plurality of actions included in a series of actions performed by a shopper are performed, and comprises an action recognition unit, a discrimination unit, a next action notification unit, a correct action notification unit, and a warning unit. The action recognition unit recognizes the action performed by the shopper. The discrimination unit determines the next action that the shopper should perform based on the action recognized by the action recognition unit. The next action notification unit operates a notification member provided in the action area where the next action should be performed in a first notification mode, based on the next action determined by the discrimination unit. The correct action notification unit operates the notification member in a second notification mode when the action recognition unit recognizes that the shopper has performed the next action. The warning unit operates a notification member provided in the action area where the different action was performed in a warning mode when the action recognition unit recognizes that the shopper has performed an action different from the next action. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a schematic diagram showing a self-service payment system including a self-service POS terminal to which the behavioral guidance system according to the first embodiment, which includes a control device according to the first embodiment, is applied. [Figure 2] Figure 2 is a perspective view showing an example of the external configuration of a self-service POS terminal. [Figure 3] Figure 3 is a block diagram showing an example of the main circuit configuration of a self-service POS terminal. [Figure 4] Figure 4 is a flowchart showing an example of the main steps in the information processing performed by the processor of a self-service POS terminal. [Figure 5] Figure 5 is a flowchart showing an example of the detailed procedure for the registration action processing shown in Figure 4. [Figure 6] Figure 6 is a schematic diagram showing a self-service payment system including an information terminal to which the behavioral guidance system according to the second embodiment, which includes a control device according to the second embodiment, is applied. [Figure 7] Figure 7 is a perspective view showing an example of the external configuration of an information terminal attached to a shopping cart. [Figure 8] Figure 8 is a schematic diagram illustrating the imaging range of the surrounding area camera and cage area camera installed in the information terminal. [Figure 9] Figure 9 is a block diagram showing an example of the main circuit configuration of an information terminal. [Figure 10] Figure 10 is a flowchart showing an example of the main steps of information processing performed by the processor of an information terminal. [Figure 11] Figure 11 is a flowchart showing an example of the detailed procedure for the registration action processing shown in Figure 10. [Figure 12] Figure 12 is a schematic diagram showing a self-service payment system as a behavioral guidance system according to the third embodiment, including a control device according to the third embodiment. [Figure 13] Figure 13 is a perspective view showing an example of the imaging range of the checkout monitoring camera in Figure 12. [Figure 14] Figure 14 is a sequence diagram illustrating the operation of the payment system. [Figure 15] Figure 15 is a plan view showing an example of the external configuration of a handwashing device to which the behavioral guidance system according to the fourth embodiment, which includes a control device according to the fourth embodiment, is applied. [Modes for carrying out the invention]

[0009] The following describes an embodiment of the control device with reference to the drawings.

[0010] [First Embodiment] In this embodiment, a self-service payment system will be described as an example.

[0011] FIG. 1 is a schematic configuration diagram showing a self-service payment system including a self-service POS terminal to which an action guidance system according to a first embodiment including a control device according to the first embodiment is applied. The payment system includes a plurality of self-service POS terminals 1 and a store server 2. The payment system connects each self-service POS terminal 1 and the store server 2 via a communication network 3 such as a LAN (Local Area Network). The store server 2 provides various services related to this payment system to each self-service POS terminal 1 connected via the communication network 3.

[0012] A self-service POS terminal 1, which is an example of an action guidance system according to the first embodiment, is installed in the checkout area of a store, and a shopper who has finished shopping becomes the user. A customer who has put the purchased goods into a shopping basket or the like in the sales floor where the goods are displayed and moved to the checkout area operates the self-service POS terminal 1 to perform self-checkout for the transaction related to the checkout of the purchased goods, that is, from the registration of the goods to the payment. The self-service POS terminal 1 is an example of a full self-service compatible payment device. The payment device can be paraphrased as a payment terminal, a checkout device, a checkout terminal, a goods registration device, a goods registration terminal, etc.

[0013] FIG. 2 is a perspective view showing the external configuration of the self-service POS terminal 1, and FIG. 3 is a block diagram showing the main circuit configuration of the self-service POS terminal 1.

[0014] As shown in FIG. 2, the self-POS terminal 1 includes a main body 110 installed on the floor surface and a weighing unit 112 installed beside the main body 110. The main body 110 has a display pole 114, a touch panel 116, and two camera poles 118 and 120 attached to its upper part. The main body 110 is provided with a basket stand 122 at the central part of the side surface opposite to the side where the weighing unit 112 is installed. The basket stand 122 is for placing baskets or the like into which a customer coming from the sales floor has put the purchased goods. The customer stands in front of the main body 110 in FIG. 2 to perform operations so that the screen of the touch panel 116 can be seen. Therefore, as seen from the customer, there is a basket stand 122 on the right side and a weighing unit 112 on the left side with the main body 110 in between. In the following description, the side where the customer stands is defined as the front of the main body 110, the side where the weighing unit 112 is installed is defined as the left side of the main body 110, and the side where the basket stand 122 is provided is defined as the right side of the main body 110.

[0015] The weighing unit 112 has a weighing pan 126 provided on the upper part of a housing 124, and a bag holder 128 is attached above the weighing pan 126. The upper surface of the weighing pan 126 serves as a placement surface 130. The bag holder 128 includes a pair of holding arms 132, and the holding arms 132 hold a cash register bag or a shopping bag (so-called my bag) brought by the customer. The weighing unit 112 measures the weight of the goods placed on the placement surface 130 and put into the cash register bag or my bag held by the holding arms 132.

[0016] The display pole 114 has a light-emitting part 134 at its tip end that selectively emits light of, for example, blue and red. The display pole 114 displays the state of the self-POS terminal 1, such as standby, operation, call, error, etc., according to the light-emitting color of the light-emitting part 134.

[0017] The touch panel 116 is composed of a display for displaying various screens to the user who operates the self-POS terminal 1 and a touch sensor for detecting the touch input of the user to the screen. In the self-POS terminal 1, the user is usually a shopper.

[0018] A cage area camera 136 is attached to camera pole 118, and a bagged area camera 138 is attached to camera pole 120.

[0019] The basket area camera 136 is mounted and positioned such that its field of view, or imaging area, falls within the basket stand 122, as shown by the dashed line in Figure 2. More specifically, the basket area camera 136 is configured so that its imaging area captures images from above of the inside of a shopping basket placed on the basket stand 122. Hereinafter, this imaging area of ​​the basket area camera 136 will be referred to as the basket area.

[0020] The bagging area camera 138 is mounted and positioned such that its field of view, or imaging area, includes the mounting surface 130 of the weighing pan 126 of the weighing unit 112, as shown by the dashed line in Figure 2. More specifically, the bagging area camera 138 is positioned so that its imaging area can capture images from above of the opening for loading goods into the shopping bag or reusable bag held by the holding arm 132. Hereinafter, this imaging area of ​​the bagging area camera 138 will be referred to as the bagging area.

[0021] A bagging area LED frame 140 is embedded in the mounting surface 130 of the weighing pan 126 to clearly indicate this bagging area to shoppers. The bagging area LED frame 140 has multiple green LEDs, yellow LEDs, and red LEDs arranged therein, allowing the bagging area to be presented to shoppers as a green, yellow, or red illuminated frame.

[0022] Although not specifically shown in the diagram, a similar LED frame may also be provided on the cage base 122.

[0023] The front of the main unit 110 has a reading window 144 for the scanner 142 (see Figure 3), a card slot 148 for the card reader 146 (see Figure 3), a receipt printing slot 152 for the printer 150 (see Figure 3), and a speaker 154. A scanner area LED frame 156 is positioned around the reading window 144 of the scanner 142. Similar to the bagging area LED frame 140, this scanner area LED frame 156 also has multiple green, yellow, and red LEDs, allowing the reading window 144, or scanner area, to be presented to the shopper as a green, yellow, or red illuminated frame. Furthermore, the front of the main unit 110 also has a coin slot 160, a coin outlet 162, a banknote slot 164, and a banknote outlet 166 for the automatic change dispenser 158 (see Figure 3).

[0024] The bagging area LED frame 140 and the scanner area LED frame 156 are examples of notification members provided in accordance with each of the multiple action areas where multiple actions included in a series of actions performed by a shopper are carried out. The bagging area and the scanner area are examples of multiple action areas. The bagging area LED frame 140 and the scanner area LED frame 156 are examples of visual notification members and light-emitting members.

[0025] Furthermore, a communication cable 168 extends from the right side of the main unit 110 to the outside, and a reader / writer 170 for electronic money media is connected to the end of this communication cable 168. The reader / writer 170 is placed on a stand 172 located on the upper right side of the main unit 110.

[0026] As shown in Figure 3, the self-service POS terminal 1 includes, in addition to the aforementioned weighing unit 112, touch panel 116, light-emitting unit 134, basket area camera 136, bagging area camera 138, bagging area LED frame 140, scanner 142, card reader 146, printer 150, speaker 154, scanner area LED frame 156, automatic change dispenser 158, and reader / writer 170, a processor 174, main memory 176, auxiliary storage device 178, clock 180, communication interface 182, and system transmission path 184. The system transmission path 184 includes an address bus, data bus, control signal lines, etc. The system transmission path 184 connects the processor 174 to the other parts directly or via signal input / output circuits and transmits data signals exchanged between them. The computer of the self-service POS terminal 1 is formed by the connection of the processor 174, main memory 176, and auxiliary storage device 178 via the system transmission path 184. The computer is an example of a control device according to the first embodiment, which controls notification members provided in each of the multiple action areas that perform multiple actions included in a series of actions performed by a shopper.

[0027] The processor 174 corresponds to the central part of the computer described above. The processor 174 controls each part to realize various functions as a self-service POS terminal 1 according to the operating system or application program. The processor 174 is, for example, a CPU (Central Processing Unit). The processor 174 may also be, for example, an MPU (micro processing unit), SoC (system on a chip), DSP (digital signal processor), GPU (graphics processing unit), ASIC (application specific integrated circuit), PLD (programmable logic device), or FPGA (field-programmable gate array). Alternatively, the processor 174 may be a combination of several of these.

[0028] Main memory 176 corresponds to the main memory portion of the computer described above. Main memory 176 includes a non-volatile memory area and a volatile memory area. In the non-volatile memory area of ​​main memory 176, the operating system or application programs are stored. Main memory 176 may also store data necessary for the processor 174 to perform processing to control each part in the non-volatile or volatile memory area. For example, main memory 176 stores the transaction file 186. The transaction file 186 is a data file for storing data related to a single commercial transaction processed by the self-service POS terminal 1. The transaction file 186 stores data such as the transaction number, purchased product data, total number of items, total amount, discount amount, and settlement amount. The transaction number is a series of numbers issued each time a commercial transaction is processed by the self-service POS terminal 1. The purchased product data is record data created for each product bought and sold in the commercial transaction identified by the transaction number. Here, the purchased product data consists of items such as product code, product name, price, number of items, and amount. The points represent the number of items purchased, identified by the product code. The amount is the total price for the number of items purchased. Transaction file 186 can store data for multiple purchased items. The total points are the sum of the points for each purchased item. The total amount is the sum of the amounts for each purchased item. The discount amount is the discount applied to the total amount. The settlement amount is the total amount minus the discount amount. Main memory 176 uses volatile memory areas as work areas where data is rewritten as needed by processor 174. Non-volatile memory areas include, for example, ROM (Read Only Memory). Volatile memory areas include, for example, RAM (Random Access Memory).

[0029] The auxiliary storage device 178 corresponds to the auxiliary storage portion of the computer described above. For example, an EEPROM (Electric Erasable Programmable Read-Only Memory), an HDD (Hard Disc Drive), or an SSD (Solid State Drive) can be the auxiliary storage device 178. The auxiliary storage device 178 stores data used by the processor 174 in performing various processes, data created by the processing performed by the processor 174, etc. For example, the auxiliary storage device 178 has an action sequence storage unit 188 and a product image storage unit 190. The auxiliary storage device 178 may also store the application program described above.

[0030] The action sequence memory unit 188 stores the sequence of actions that a shopper should perform at the self-service POS terminal 1. The sequence of actions includes, for example, the bag setting action of setting a bag such as a shopping bag or reusable bag on the weighing unit 112, the scanning action of scanning the code symbol attached to the product with the scanner 142 in order to register the purchased product, and the bagging action of putting the scanned product into the shopping bag. The action sequence memory unit 188 stores the sequence of actions in this order. The action sequence memory unit 188 is an example of a memory unit that stores the sequence of multiple actions that a shopper performs when registering purchased products.

[0031] The product image storage unit 190 stores images of the appearance of each product sold in the store.

[0032] Clock 180 keeps track of the time. Processor 174 processes the time recorded by clock 180 as the current time.

[0033] The communication interface 182 performs data communication with external devices connected via the network according to a pre-configured communication protocol. The external devices are, for example, the store server 2.

[0034] The scanner 142 reads a code symbol from the product held over the reading window 144. Each product sold in the store is marked with a code symbol that encodes a product ID or other information to identify the product. The code symbol is, for example, a barcode. The code symbol may also be, for example, a two-dimensional data code. The scanner 142 may be a type that reads the code symbol by scanning with a laser beam, or a type that reads the code symbol from an image captured by an imaging device.

[0035] The card reader 146 reads card data recorded on card media such as credit cards and loyalty cards. The card reader 146 pulls the card media inserted into the card slot 148 into the main unit 110, reads the card data, and then ejects it from the card slot 148.

[0036] The printer 150 prints receipt data and other information representing the details of a commercial transaction onto receipt paper. The receipt paper with the printed receipt data is ejected from the output slot 152, cut by a cutter (not shown), and issued as a receipt or invoice.

[0037] Speaker 154 emits voice messages and notification sounds.

[0038] The automatic change dispenser 158 includes a coin unit 192 and a banknote unit 194. The coin unit 192 sorts the coins inserted into the coin slot 160 one by one, identifies the denomination, and stores them in a safe according to their denomination. The coin unit 192 retrieves the coins of the corresponding denomination from the safe, for example based on change data, and dispenses them to the coin outlet 162. The banknote unit 194 sorts the banknotes inserted into the banknote slot 164 one by one, identifies the denomination, and stores them in a safe according to their denomination. The banknote unit 194 retrieves the banknotes of the corresponding denomination from the safe, for example based on change data, and dispenses them to the banknote outlet 166.

[0039] The reader / writer 170 reads and rewrites electronic money recorded on an electronic money medium. The electronic money medium is, for example, a contactless IC card. The electronic money medium may also be an electronic device such as a smartphone or tablet.

[0040] The operation of the self-service POS terminal 1 with this configuration will be explained below. Figure 4 is a flowchart showing an example of the main information processing steps performed by the processor 174 of the self-service POS terminal 1. Unless otherwise specified, the processing operation of the processor 174 shown in Figure 4 is assumed to transition from ACTn (where n is a natural number) to ACT(n+1). Furthermore, the procedure shown in Figure 4 is just one example. The procedure is not particularly limited as long as similar results can be obtained.

[0041] As ACT11, the processor 174 waits for the shopper to initiate an operation. For example, when the processor 174 starts operating after power-on, it displays "Please touch" or similar on the touch panel 116's display and enters a standby state. In ACT11, it determines whether or not the touch panel 116 has been touched. If the shopper has not touched the touch panel 116, the processor 174 determines NO in ACT11 and repeats this ACT11. When the shopper touches the touch panel 116, the processor 174 determines YES in ACT11 and proceeds to ACT12.

[0042] As ACT12, processor 174 accepts the shopper's selection of bag type. For example, processor 174 displays a screen on the touch panel 116 display allowing the shopper to choose whether to purchase a plastic bag or use their own bag, and accepts the shopper's selection. Once a selection is made, processor 174 proceeds to ACT13. Processor 174, which performs the processing in ACT12, is an example of an action recognition unit that recognizes the actions performed by the shopper, and is also an example of a discrimination unit that determines the next action the shopper should take based on the actions recognized by the action recognition unit.

[0043] As ACT13, the processor 174 illuminates the bagging area LED frame 140 in yellow, which is a warning color.

[0044] As ACT14, the processor 174 outputs a bag setting instruction. For example, the processor 174 displays the instruction "Please set the bag" on the touch panel 116 display and outputs the same voice instruction from the speaker 154. The processor 174 that performs the processing of ACT13 is an example of a next action notification unit that operates a notification member provided in the action area where the next action should be performed in a first notification mode, based on the next action determined by the discrimination unit. The bagging area LED frame 140 is an example of a notification member provided in the action area where the next action should be performed. Lighting up in yellow is an example of operation in the first notification mode.

[0045] The processor 174 then performs action detection for the bag-setting action, which is the first action in a series of actions stored in the action sequence memory unit 188, in which the shopper sets a bag, such as a shopping bag or reusable bag, into the weighing unit 112.

[0046] Specifically, first, as ACT15, the processor 174 acquires the bagging area image captured by the bagging area camera 138.

[0047] Then, in ACT16, the processor 174 determines, based on the bagging area image, whether a bag, i.e., a shopping bag or a reusable bag, has been placed in the weighing unit 112. Specifically, the processor 174 recognizes the area of ​​the shopper's hand from the bagging area image and determines whether the hand has entered the bagging area. For example, the processor 174 detects the position of the hand using an object detection technique such as the well-known SSD (Single Shot Multibox Detector) or an attitude estimation technique such as OpenPose. As mentioned above, the bagging area camera 138 is installed so that the mounting surface 130 is included in its field of view, i.e., the imaging area. However, by detecting the bagging area in real time using an object detection technique such as SSD, it is possible to avoid having to strictly set the mounting position and attitude of the bagging area camera 138. If no bag is placed, the processor 174 determines NO in ACT16 and proceeds to ACT15. If a bag is set, the processor 174 determines YES in ACT 16 and proceeds to ACT 17. Note that the determination of whether or not a bag is set may also be made using changes in the values ​​of the weight sensor or other sensors provided by the weighing unit 112. The processor 174 that performs the processing in ACT 16 is an example of an action recognition unit that recognizes actions performed by a shopper.

[0048] As ACT17, processor 174 outputs guidance for the correct action. For example, processor 174 outputs a sound effect indicating the correct action, such as "ding-dong," from speaker 154.

[0049] As ACT18, the processor 174 starts lighting the bagging area LED frame 140, which was lit yellow in ACT13, in green for a certain period of time. This period of time is not strictly defined, but it can be the amount of time required from the moment the shopper sets the bag until they start taking out the items from the shopping basket placed on the basket stand 122. For example, the period of time is about 1 second. The processor 174 that performs this ACT18 process is an example of a correct action notification unit that, when the action recognition unit recognizes that the shopper has performed the next action, operates the notification member that was operated in the first notification mode in a second notification mode that is different from the first notification mode. Lighting up in green for a certain period of time is an example of operation in the second notification mode.

[0050] Next, the processor 174 performs action detection regarding the next action in a series of actions stored in the action sequence memory unit 188: the scanning action, in which the shopper scans the code symbol attached to the product with the scanner 142 in order to register the purchased product, and the bagging action, in which the scanned product is placed in a shopping bag.

[0051] First, as ACT19, the processor 174 acquires the cage area image captured by the cage area camera 136.

[0052] As ACT20, the processor 174 uses known image recognition processing techniques to determine from the basket area image whether the shopping basket placed on the basket stand 122 is empty or not. If the shopping basket is not empty, that is, if there are unscanned items remaining, the processor 174 determines NO in ACT20 and proceeds to ACT21. The processor 174 that performs this ACT20 processing is an example of an action recognition unit that recognizes the actions performed by the shopper, and is also an example of a discrimination unit that determines the next action that the shopper should take based on the actions recognized by the action recognition unit.

[0053] As ACT21, processor 174 executes registration behavior processing, which is the entity for detecting the scanning behavior and the bagging behavior.

[0054] Figure 5 is a flowchart showing an example of the detailed procedure for registering actions in ACT21. First, the processor 174 performs action detection for scanning actions.

[0055] Specifically, as ACT2101, the processor 174 first illuminates the scanner area LED frame 156, which is arranged around the reading window 144 of the scanner 142, in yellow, which is a warning color. The processor 174 that performs this ACT2101 process is an example of a next action notification unit that operates a notification member provided in a first notification mode, corresponding to the action area where the next action should be performed, based on the next action determined by the discrimination unit. The scanner area LED frame 156 is an example of a notification member provided in a corresponding to the action area where the next action should be performed. Illumination in yellow is an example of operation in the first notification mode.

[0056] As ACT2102, processor 174 outputs product scanning instructions. For example, processor 174 displays the message "Please scan the product" on the touch panel 116 display and outputs the same voice instructions from speaker 154.

[0057] As ACT2103, processor 174 acquires the output of scanner 142.

[0058] As ACT2103, the processor 174 determines whether or not the product has been scanned. When the code symbol attached to the product is scanned by the scanner 142, the processor 174 can obtain the product code indicated by the code symbol from the scanner 142. Therefore, the processor 174 can determine whether or not the product has been scanned based on whether or not the product code has been obtained. If the product has been scanned, the processor 174 determines YES in ACT2104 and proceeds to ACT2109. If the product has not been scanned, the processor 174 determines NO in ACT2104 and proceeds to ACT2105. The processor 174 that performs this ACT2104 process is an example of an action recognition unit that recognizes the actions performed by the shopper, and is also an example of a determination unit that determines the next action that the shopper should take based on the actions recognized by the action recognition unit.

[0059] As ACT2105, the processor 174 acquires the bagging area image captured by the bagging area camera 138.

[0060] As ACT2106, processor 174 determines whether or not the items have been bagged based on the bagging area image. Specifically, processor 174 recognizes the area of ​​the shopper's hand from the bagging area image and determines whether or not the hand has entered the bagging area, thereby determining whether or not the items have been bagged. If the items have not been bagged, processor 174 determines NO in ACT2106 and proceeds to ACT2103. Processor 174, which performs the processing in ACT2106, is an example of an action recognition unit that recognizes the actions performed by the shopper.

[0061] This processing loop from ACT2103 to ACT2106 causes the processor 174 to wait for the product to be scanned. It should be noted that, as long as the aforementioned time has elapsed, the green light on the bagging area LED frame 140 may remain on during this waiting period for product scanning.

[0062] If, while waiting for this item to be scanned, an item that has not yet been scanned is bagged, the processor 174 determines YES in ACT2106 and proceeds to ACT2107. In ACT2107, the processor 174 flashes the bagging area LED frame 140 in red, which is a warning color. The processor 174 that performs this ACT2107 process is an example of a warning unit that, when the action recognition unit recognizes that the shopper has performed an action different from the next action, operates a notification member provided in the action area where the different action was performed in a warning mode different from the first and second notification modes. Flashing red is an example of operation in warning mode.

[0063] As ACT2108, processor 174 outputs a warning message for incorrect actions. For example, processor 174 outputs a warning voice message from speaker 154 such as "Please scan the items before bagging them," and displays a similar warning message on the touch panel 116 display for a predetermined time. After that, processor 174 proceeds to the processing of ACT2103.

[0064] While the processing loop from ACT2103 to ACT2106 waits for the product to be scanned, if a product is scanned, processor 430 determines YES in ACT2104 and proceeds to ACT2109. In ACT2109, processor 174 executes the product registration process using the product code of the acquired product. This product registration process is publicly known, so its explanation is omitted. As a result, product sales data for the scanned product is stored in transaction file 186.

[0065] As ACT2110, processor 174 outputs guidance for the correct action. For example, processor 174 outputs a sound effect indicating the correct action, such as "ding-dong," from speaker 154.

[0066] As ACT2111, processor 174 starts lighting the scanner area LED frame 156, which was lit yellow in ACT2101, in green for a certain period of time. This period of time is not strictly defined, but it can be about the amount of time needed for the shopper to start bagging the scanned items. For example, this period of time is about 1 second. Processor 174, which performs the processing of ACT2111, is an example of a correct action notification unit that, when the action recognition unit recognizes that the shopper has performed the next action, operates the notification member that was operated in the first notification mode in a second notification mode that is different from the first notification mode. Lighting in green for a certain period of time is an example of operation in the second notification mode.

[0067] Next, the processor 174 performs action detection regarding the bagging action. Specifically, first, as ACT2112, the processor 174 lights up the bagging area LED frame 140 in yellow, which is a warning color. The processor 174 that performs this ACT2112 process is an example of a next action notification unit that operates a notification member provided in the action area where the next action should be performed in a first notification mode, based on the next action determined by the discrimination unit. The bagging area LED frame 140 is an example of a notification member provided in the action area where the next action should be performed. Lighting up in yellow is an example of operation in the first notification mode.

[0068] As ACT2113, processor 174 outputs bagging instructions. For example, processor 174 displays the instruction "[Product Name] please bag" on the touch panel 116 display and outputs the same voice instruction from speaker 154. Here, [Product Name] is the product name of the product corresponding to the product code registered in ACT2109.

[0069] As ACT2114, processor 174 acquires the bagging area image captured by bagging area camera 138.

[0070] As ACT2115, the processor 174 determines whether or not the product [product name] has been bagged based on the bagging area image. Specifically, the processor 174 recognizes the area of ​​the shopper's hand from the bagging area image and determines whether or not the hand has entered the bagging area. The method for detecting the position of the hand is as described in ACT16. Furthermore, the processor 174 applies known techniques such as template matching to the image around the hand to determine whether the image of the product being held by the hand matches the image of the product [product name]. The product image for each product is stored in the product image storage unit 190 of the auxiliary storage device 178. In this way, the processor 174 can determine whether or not the product [product name] has been bagged. If the product [product name] has been bagged, the processor 174 determines YES in ACT2115 and proceeds to ACT2119. If the product [product name] has not been bagged, the processor 174 determines NO in ACT2115 and proceeds to ACT2116. The processor 174 that performs the ACT2115 processing is an example of an action recognition unit that recognizes the actions performed by the shopper, and is also an example of a determination unit that determines the next action that the shopper should take based on the actions recognized by the action recognition unit.

[0071] As ACT2116, processor 174 determines whether or not items other than [product name] have been bagged. If no items other than [product name] have been bagged, processor 174 determines NO in ACT2116 and proceeds to ACT2114. Processor 174, which performs the processing of ACT2116, is an example of an action recognition unit that recognizes the actions performed by the shopper.

[0072] This processing loop from ACT2114 to ACT2116 puts the processor 174 into a waiting state for the product [product name] to be bagged. Note that, as long as the aforementioned time has elapsed, the green light on the scanner area LED frame 156 may remain on during this waiting period for the product to be bagged.

[0073] If, while the product [product name] is waiting to be bagged, a product other than [product name] is bagged, the processor 174 determines YES in ACT2116 and proceeds to ACT2117. In ACT2117, the processor 174 flashes the bagging area LED frame 140 in red, which is a warning color. The processor 174 that performs this ACT2117 process is an example of a warning unit that, when the action recognition unit recognizes that the shopper has performed an action different from the next action, operates a notification member provided in the action area where the different action was performed in a warning mode different from the first and second notification modes. Flashing red is an example of operation in warning mode.

[0074] As ACT2118, processor 174 outputs a warning message for incorrect actions. For example, processor 174 outputs a warning voice message such as "Please bag the scanned items" from speaker 154, and displays a similar warning message on the touch panel 116 display for a predetermined time. After that, processor 174 proceeds to process ACT2114.

[0075] As ACT2119, processor 174 outputs guidance for the correct action. For example, processor 174 outputs a sound effect indicating the correct action, such as "ding-dong," from speaker 154.

[0076] As ACT2120, processor 174 starts lighting the bagging area LED frame 140, which was lit yellow in ACT2112, in green for a certain period of time. This period of time is not strictly defined, but it can be the amount of time required from the moment the shopper bags the items until they start taking out the next item from the shopping basket placed on the basket stand 122. For example, this period of time is about 1 second. After that, processor 174 returns to the higher-level routine. Processor 174, which performs the processing of ACT2120, is an example of a correct action notification unit that, when the action recognition unit recognizes that the shopper has performed the next action, operates the notification member that was operated in the first notification mode in a second notification mode that is different from the first notification mode. Lighting up in green for a certain period of time is an example of operation in the second notification mode.

[0077] The specified durations for ACT18, ACT2111, and ACT2120 may be the same or different.

[0078] Returning to the explanation of Figure 4, once the scanning and bagging actions for one item in the shopping basket, which constitute the registration action processing of ACT21, are completed, processor 174 proceeds to ACT19. In this way, during the processing loop from ACT19 to ACT21, the shopper registers and bags each item in the shopping basket. Once all items in the shopping basket have been registered and bagged and the shopping basket is empty, processor 174 determines YES in ACT20 and proceeds to ACT22.

[0079] As ACT22, processor 174 performs accounting based on the product sales data stored in transaction file 186. Since this accounting process is publicly known, its explanation is omitted. This allows the shopper to account for their purchases.

[0080] As described above, the computer including the processor 174 of the self-service POS terminal 1 is a control device according to the first embodiment, which controls notification members provided in each of the multiple action areas where multiple actions included in a series of actions performed by a shopper are carried out. The computer recognizes the actions performed by the shopper and, according to the next action to be taken, activates the notification members provided in the action area where the next action will be performed, thereby guiding the shopper to the next action to be taken and where to perform it. If the next action is performed correctly by the shopper, this is notified by the notification member, and if an incorrect action is performed, a warning is issued by the notification member provided in the action area where the incorrect action was performed. Accordingly, the processor 174 comprises: an action recognition unit that recognizes an action performed by a shopper; a determination unit that determines the next action the shopper should perform based on the action recognized by the action recognition unit; a next action notification unit that operates a notification member provided in a corresponding action area where the next action should be performed in a first notification mode based on the next action determined by the determination unit; a correct action notification unit that, when the action recognition unit recognizes that the shopper has performed the next action, operates the notification member that was operated in the first notification mode in a second notification mode different from the first notification mode; and a warning unit that, when the action recognition unit recognizes that the shopper has performed an action different from the next action, operates a notification member provided in a corresponding action area where the different action was performed in a warning mode different from the first and second notification modes. Thus, according to the self-service POS terminal 1 including the control device of the first embodiment, if a shopper performs an incorrect action as their next step, the shopper is warned, thereby preventing unconscious erroneous actions and conscious fraudulent actions by the shopper.

[0081] Furthermore, the series of actions performed by the shopper includes multiple actions performed when the shopper registers the purchased items. The computer of the self-service POS terminal 1 includes an action sequence storage unit 440 in the auxiliary storage device 434 that stores the order of the multiple actions performed when registering the purchased items. The processor 174, which constitutes the discrimination unit, determines the next action based on the order of actions stored in this action sequence storage unit 440. Therefore, according to the self-service POS terminal 1 in the first embodiment, the next action can be easily determined and notified to the shopper.

[0082] Furthermore, multiple actions include at least a scanning action in which the scanner 142 scans a code symbol attached to the purchased item, and the processor 174, which constitutes the action recognition unit, recognizes that the shopper has performed a scanning action based on the output of the scanner 142. Therefore, with the self-service POS terminal 1 in the first embodiment, it is possible to reliably notify shoppers about the essential action of registering purchased items.

[0083] Furthermore, the notification component includes a visual notification component, and the first notification mode, the second notification mode, and the warning mode by the processor 174 are distinguished by the visual notification mode of the visual notification component from each other. Therefore, it becomes possible to ensure that shoppers are reliably aware of the area where they should take their next action, and whether or not they have taken the correct action within that area.

[0084] Furthermore, the visual notification component includes a light-emitting component which is an LED frame, and the first notification mode, the second notification mode, and the warning mode differ from each other in at least one of the light-emitting color and lighting pattern of the light-emitting component. Therefore, it is possible to clearly differentiate the notification methods visually.

[0085] [Second Embodiment] Next, a second embodiment will be described. In this embodiment, a payment system including a pre-registration system will be described as an example, in which shoppers register their purchased items in advance before paying at the checkout machine, using an information terminal that can read product codes attached to a shopping cart, or an information terminal equipped with a scanning function that allows shoppers to scan products themselves while shopping in the store. Note that the same configurations and operations as in the first embodiment will be given the same reference numerals as in the first embodiment, and their explanation will be omitted.

[0086] Figure 6 is a schematic diagram showing a self-service payment system including an information terminal 4 to which the behavioral guidance system according to the second embodiment, which includes a control device according to the second embodiment, is applied. The payment system in the second embodiment includes a communication network 3, a plurality of self-service POS terminals 1 and a store server 2, as well as an information terminal 4, an access point 5, and a plurality of accounting machines 6. Although only one access point 5 is shown in Figure 6, there may be multiple access points 5.

[0087] Information terminal 4 is a terminal used by shoppers to register products while browsing within the store. Access point 5 is a wireless base station for connecting information terminal 4 to the communication network 3. Information terminal 4 does not have a transaction file, but transmits the product code along with its terminal ID to the store server 2 via access point 5, thereby storing the transaction data in a transaction file for each information terminal 4, i.e., each shopper, on the store server 2. Alternatively, a separate management device could be connected to the communication network 3 to manage the transaction files for each information terminal 4. The checkout machine 6 is a payment device used by shoppers to complete transactions based on the transaction data for each information terminal 4 stored on the store server 2. Alternatively, the functions of the checkout machine 6 could be provided by the self-service POS terminal 1 instead of having a separate checkout machine 6.

[0088] The payment system may also include multiple store surveillance cameras 7 connected to the communication network 3. The store surveillance cameras 7 are installed, for example, on the ceiling of each sales area within the store, and capture images from above of the sales area as shoppers scan products placed on display shelves or place products into shopping baskets on shopping carts. The images of the sales area captured by each store surveillance camera 7 are transmitted to the store server 2 and used for monitoring the sales area. It is also possible to transmit the sales area images to information terminals 4 that are within the field of view, i.e., the imaging area, of each store surveillance camera 7. In this case, for example, by installing wireless strength and beacon transmitters and receivers for each access point 5 in the sales area and on the information terminals 4 or shopping carts, the store server 2 can determine the location of each information terminal 4 and transmit the corresponding sales area image from the store surveillance camera 7.

[0089] Figure 7 is a perspective view showing an example of the external configuration of the information terminal 4 attached to the shopping cart C. Figure 8 is a schematic diagram illustrating the imaging range of the surrounding area camera 412 and the basket area camera 414, which are configured separately from the main body 410 of the information terminal 4.

[0090] The main unit 410 of the information terminal 4 includes a main unit 410 equipped with a touch panel 416 and a speaker 418, and to which a card reader 420 is attached, and a scanner 422 separate from the main unit 410.

[0091] The shopping cart C comprises a caster section C1 for movement, a handle frame section C2, and a basket support section C3. The caster section C1 has four wheels for smooth movement on the floor. The handle frame section C2 includes a pair of vertical frames C21, C21 erected on the rear wheel side of the caster section C1, and a handle bar C22 connecting the upper ends of these vertical frames C21, C21. The basket support section C3 is located in front of the middle of the handle frame section C2. The shopping cart C places a shopping basket BA for holding goods on the basket support section C3 and the caster section C1.

[0092] The shopping cart C has a pole C4 attached to one of its vertical frames C21. The tip of the pole C4 is positioned above the handlebar C22. The main unit 410 is attached to the tip of the pole C4 so that the screen of the touch panel 416 faces backward. The card reader 420 is attached to the main unit 410 so that the card slit faces backward. In Figure 7, the card reader 420 is a magnetic card reader. The scanner 422 is located midway along the handlebar C22. The scanner 422 is attached to the handlebar C22 so that the reading window faces forward.

[0093] The surrounding area camera 412 is mounted, for example, on the top surface of the scanner 422. The surrounding area camera 412 is, for example, a 360-degree camera, and is mounted so that the entire circumference of the shopping cart C is within its field of view, or imaging area, as shown by the dashed line in Figure 8. As a result, no matter what position the shopper is in relative to the shopping cart C when taking items from the shelves, the surrounding area camera 412 will capture the image. Hereafter, the imaging area of ​​this surrounding area camera 412 will be referred to as the surrounding area.

[0094] The basket area camera 414 is mounted on the back of the main unit 410, which is in front of the shopping cart C, as shown in Figure 8. The mounting position and orientation of the basket area camera 414 are set so that the basket support section C3 is included in the field of view, or imaging area, as shown by the dashed line in Figure 8. More specifically, the basket area camera 414 is set so that its imaging area can capture images from diagonally above the inside of the shopping basket BA housed in the basket support section C3. Hereinafter, the imaging area of ​​this basket area camera 414 will be referred to as the basket area.

[0095] A basket area LED frame 424 is attached to the upper surface of the basket support section C3 so as to surround the shopping basket BA housed in the basket support section C3. Similar to the bagging area LED frame 140 in the first embodiment, this basket area LED frame 424 has multiple green LEDs, yellow LEDs, and red LEDs arranged therein, allowing the bagging area to be presented to shoppers as a green, yellow, or red illuminated frame.

[0096] Furthermore, a scanner area LED frame 428 is positioned to surround the reading window 426 of the scanner 422. Similar to the scanner area LED frame 156 in the first embodiment, this scanner area LED frame 428 also has multiple green LEDs, yellow LEDs, and red LEDs arranged on it, so that the reading window 426, or scanner area, can be presented to the shopper as a green frame, a yellow frame, or a red illuminated frame.

[0097] The basket area LED frame 424 and the scanner area LED frame 428 are examples of notification members provided in accordance with each of the multiple action areas that perform a series of actions included in a sequence of actions performed by a shopper. The basket support section C3 and the scanner area are examples of multiple action areas. The basket area LED frame 424 and the scanner area LED frame 428 are examples of visual notification members and light-emitting members.

[0098] Figure 9 is a block diagram showing an example of the main circuit configuration of the information terminal 4. As shown in Figure 9, the information terminal 4 includes, in addition to the aforementioned surrounding area camera 412, basket area camera 414, touch panel 416, speaker 418, card reader 420, scanner 422, basket area LED frame 424, and scanner area LED frame 428, a processor 430, main memory 432, auxiliary storage device 434, wireless unit 436, and system transmission line 438. The system transmission line 438 includes an address bus, data bus, control signal lines, etc. The system transmission line 438 connects the processor 430 to the other parts directly or via signal input / output circuits and transmits data signals exchanged between them. The computer of the self-POS terminal 1 is formed by the connection of the processor 430, main memory 432, and auxiliary storage device 434 via the system transmission line 438. The computer is an example of a control device according to the second embodiment, which controls notification members provided corresponding to each of the multiple action areas that perform a series of actions included in a series of actions performed by a shopper.

[0099] The processor 430 corresponds to the central part of the computer described above. The processor 430 controls various parts to realize various functions as an information terminal 4 according to the operating system or application program. The processor 430 may be a CPU, MPU, SoC, DSP, GPU, ASIC, PLD, or FPGA, similar to the processor 174 of the self-service POS terminal 1. Alternatively, the processor 430 may be a combination of several of these.

[0100] Main memory 432 corresponds to the main memory portion of the computer described above. Like the main memory 176 of the self-service POS terminal 1, main memory 432 includes a non-volatile memory area for storing the operating system or application programs, and a volatile memory area used as a work area where data is rewritten as appropriate by the processor 430. Main memory 432 may also store data necessary for the processor 430 to perform processing to control each part in non-volatile or volatile memory areas. The non-volatile memory area is, for example, ROM, and the volatile memory area is, for example, RAM.

[0101] The auxiliary storage device 434 corresponds to the auxiliary storage portion of the computer described above. Similar to the auxiliary storage device 178 of the self-service POS terminal 1, the auxiliary storage device 434 can be composed of, for example, an EEPROM, HDD, or SSD, and stores data used by the processor 430 in various processes, data created by the processing performed by the processor 430, etc. In addition, the auxiliary storage device 434 may also store the application program described above.

[0102] In this second embodiment as well, the auxiliary storage device 434 may have an action sequence storage unit 440 and a product image storage unit 442. The action sequence storage unit 440 stores the sequence of actions that a shopper should perform on the information terminal 4. The product image storage unit 442 stores images of the appearance of each product sold in the store.

[0103] In this embodiment, the sequence of actions stored in the action sequence storage unit 440 includes, for example, the grasping action of a shopper picking up the product they intend to purchase, the scanning action of scanning the code symbol attached to the product with the scanner 422 in order to register the purchased product, and the basket-putting action of placing the scanned product into the shopping basket BA located in the basket receiving section C3 of the shopping cart C. The action sequence storage unit 440 stores the order of these actions in this sequence. The action sequence storage unit 440 is an example of a storage unit that stores the sequence of multiple actions performed by a shopper when registering purchased products.

[0104] The wireless unit 436 communicates wirelessly with the access point 5, and then performs data communication with external devices connected via the network according to a pre-configured communication protocol. The external devices are, for example, the store server 2.

[0105] The scanner 422 reads the code symbol from the product held over the reading window 426. The scanner 422 may be of a type that reads the code symbol by scanning with a laser beam, or it may be of a type that reads the code symbol from an image captured by an imaging device.

[0106] The card reader 420 reads card data recorded on card media such as credit cards and loyalty cards.

[0107] The operation of the self-service POS terminal 1 with this configuration will be explained below. Figure 10 is a flowchart showing an example of the main steps of information processing performed by the processor 430 of the information terminal 4.

[0108] As ACT41, the processor 430 waits for the shopper to initiate an operation. For example, when the processor 430 starts operating after power-on, it displays "Please touch" or similar on the touch panel 416 display and enters a standby state. In ACT41, it determines whether the touch panel 416 has been touched or not. If the shopper has not touched the touch panel 416, the processor 430 determines NO in ACT41 and repeats ACT41. When the shopper touches the touch panel 416, the processor 430 determines YES in ACT41 and proceeds to ACT42.

[0109] As ACT42, processor 430 executes the login process. This login process involves the shopper's member ID, read from the storage medium held by the shopper by the card reader 420, being transmitted to the store server 2 via access point 5 by wireless unit 436 along with the terminal ID of the information terminal 4, and receiving approval. Since this login process is a well-known process, a detailed explanation will be omitted. The store server 2 creates a transaction file for the approved member, updates the transaction data in the corresponding transaction file in response to the transmission of the product code with the terminal ID of the information terminal 4 from the information terminal 4, and transmits the transaction data to the checkout machine 6 when the shopper is ready to pay.

[0110] The processor 430 then performs action detection regarding the first action in a series of actions stored in the action sequence memory unit 440, which is the grasping action in which the shopper picks up the product they intend to purchase.

[0111] Specifically, as ACT43, the processor 430 acquires an ambient image of the surrounding area captured by the ambient area camera 412.

[0112] As ACT44, the processor 430 determines, based on the surrounding area image, whether or not the shopper has grasped an item displayed on the shelf. Specifically, the processor 430 recognizes the area of ​​the shopper's hand from the surrounding area image and determines whether or not an object is being grasped. The method for detecting the position of the hand is as described in the first embodiment. If the shopper is not grasping an item, the processor 430 determines NO in ACT44 and proceeds to ACT45. If the shopper is grasping an item, the processor 430 determines YES in ACT44 and proceeds to ACT46. The processor 430 that performs this ACT44 processing is an example of an action recognition unit that recognizes an action performed by the shopper, and is also an example of a discrimination unit that determines the next action the shopper should take based on the action recognized by the action recognition unit.

[0113] In ACT45, the processor 430 determines whether or not the shopper has declared that they want to pay. For example, the touch panel 416 displays a "Checkout" button that the shopper uses to declare that they want to pay for their purchases, and the shopper touches this "Checkout" button when they want to proceed to checkout. Therefore, the processor 430 can determine whether or not the shopper has declared they want to pay based on whether or not the "Checkout" button has been touched. If the shopper has declared they want to pay, the processor 430 determines YES in ACT45 and proceeds to ACT47. If the shopper has not declared they want to pay, the processor 430 determines NO in ACT45 and proceeds to ACT43.

[0114] Through this processing loop from ACT43 to ACT45, processor 430 awaits the shopper to pick up the goods or declare payment. Once the shopper picks up the goods, processor 430 proceeds to ACT46.

[0115] As ACT46, the processor 430 executes a registration action process, which is the entity of action detection for a scanning action in which the scanner 422 scans a code symbol attached to a product in order to register the purchased product, and a basket-put action in which the scanned product is placed into a shopping basket BA located in the basket receiving section C3 of the shopping cart C.

[0116] Figure 11 is a flowchart showing an example of the detailed procedure for registering actions in ACT46. First, the processor 430 performs action detection for scanning actions.

[0117] Specifically, as ACT4601, the processor 430 first illuminates the scanner area LED frame 428, which is arranged around the reading window 426 of the scanner 422, in yellow, which is a warning color. The processor 430 that performs this ACT4601 process is an example of a next action notification unit that operates a notification member provided in a first notification mode, corresponding to the action area where the next action should be performed, based on the next action determined by the discrimination unit. The scanner area LED frame 428 is an example of a notification member provided in a corresponding to the action area where the next action should be performed. Illumination in yellow is an example of operation in the first notification mode.

[0118] As ACT4602, the processor 430 outputs product scanning instructions. For example, the processor 430 displays the message "Please scan the product" on the touch panel 416's display and also outputs the same voice instructions from the speaker 418.

[0119] As ACT4603, processor 430 acquires the output of scanner 422.

[0120] As ACT4603, the processor 430 determines whether or not the product has been scanned, that is, whether or not it has obtained the product code indicated by the code symbol from the scanner 422. If the product has been scanned, the processor 430 determines YES in ACT4604 and proceeds to ACT4609. If the product has not been scanned, the processor 430 determines NO in ACT4604 and proceeds to ACT4605. The processor 430 that performs this ACT4604 process is an example of an action recognition unit that recognizes the actions taken by the shopper, and is also an example of a determination unit that determines the next action that the shopper should take based on the actions recognized by the action recognition unit.

[0121] As ACT4605, the processor 430 acquires the cage area image captured by the cage area camera 414.

[0122] As ACT4606, the processor 430 determines whether or not an item has been placed in the shopping basket BA based on the basket area image. Specifically, the processor 430 recognizes the area of ​​the shopper's hand from the basket area image and determines whether or not the hand has entered the basket area, thereby determining whether or not an item has been placed in the shopping basket BA. If an item has not been placed in the shopping basket BA, the processor 430 determines NO in ACT4606 and proceeds to ACT4603. The processor 430 that performs this ACT4606 process is an example of an action recognition unit that recognizes the actions performed by the shopper.

[0123] This processing loop from ACT4603 to ACT4606 causes processor 430 to wait for the product to be scanned.

[0124] If, while waiting for this item to be scanned, an unscanned item is placed in the shopping basket BA, the processor 430 determines YES in ACT4606 and proceeds to ACT4607. In ACT4607, the processor 430 causes the basket area LED frame 424 to flash red, which is the warning color. The processor 430 that performs this ACT4607 process is an example of a warning unit that, when the action recognition unit recognizes that the shopper has performed an action different from the next action, operates the notification member provided in the action area where the different action was performed in a warning mode different from the first and second notification modes. Flashing red is an example of operation in the warning mode.

[0125] As ACT4608, processor 430 outputs a warning message for incorrect actions. For example, processor 430 outputs a warning voice from speaker 418 such as "Please scan the item before putting it in the basket," and displays a similar warning message on the touch panel 416 display for a predetermined time. After that, processor 430 proceeds to the processing of ACT4603.

[0126] While the processing loop from ACT4603 to ACT4606 waits for a product to be scanned, if a product is scanned, the processor 430 determines YES in ACT4604 and proceeds to ACT4609. In ACT4609, the processor 430 executes the product registration process using the acquired product code. In this product registration process, as described above, the processor 430 transmits the acquired product code to the store server 2 via the access point 5 using the wireless unit 436.

[0127] As ACT4610, processor 430 outputs guidance for the correct action. For example, processor 430 outputs a sound effect indicating the correct action, such as "ding-dong," from speaker 418.

[0128] As ACT4611, processor 430 starts lighting the scanner area LED frame 428, which was lit yellow in ACT4601, in green for a certain period of time. This period of time is not strictly defined, but it can be the amount of time required for the shopper to start placing the scanned items into the shopping basket BA. For example, this period of time is about 1 second. Processor 430, which performs the processing of ACT4611, is an example of a correct action notification unit that, when the action recognition unit recognizes that the shopper has performed the next action, operates the notification member that was operated in the first notification mode in a second notification mode that is different from the first notification mode. Lighting in green for a certain period of time is an example of operation in the second notification mode.

[0129] Next, the processor 430 performs action detection regarding the basket insertion action. Specifically, first, as ACT4612, the processor 430 lights up the basket area LED frame 424 in yellow, which is a warning color. The processor 430 that performs this ACT4612 process is an example of a next action notification unit that operates a notification member provided in the action area where the next action should be performed in a first notification mode, based on the next action determined by the discrimination unit. The basket area LED frame 424 is an example of a notification member provided in the action area where the next action should be performed. Lighting up in yellow is an example of operation in the first notification mode.

[0130] As ACT4613, processor 430 outputs instructions for placing items in the basket. For example, processor 430 displays the message "[Product Name] Please place it in the basket" on the touch panel 416 display and outputs the same voice guidance from speaker 418. Here, [Product Name] is the product name of the product corresponding to the product code registered in ACT4609.

[0131] As ACT4614, processor 430 acquires the cage area image captured by cage area camera 414.

[0132] As ACT4615, the processor 430 determines, based on the basket area image, whether or not the product [product name] has been placed in the shopping basket BA. Specifically, the processor 430 recognizes the area of ​​the shopper's hand from the basket area image and determines whether or not the hand has entered the basket area. The method for detecting the position of the hand is as described in ACT44. Furthermore, the processor 430 applies known techniques such as template matching to the image of the hand to determine whether the image of the product being held by the hand matches the image of the product [product name]. The product image for each product is stored in the product image storage unit 442 of the auxiliary storage device 434. In this way, the processor 430 can determine whether the product [product name] has been placed in the shopping basket BA. If the product [product name] has been placed in the shopping basket BA, the processor 430 determines YES in ACT4615 and proceeds to ACT4619. If the product [product name] is not bagged, the processor 430 determines NO in ACT4615 and proceeds to ACT4616. The processor 430 that performs the processing of ACT4615 is an example of an action recognition unit that recognizes the actions performed by the shopper, and is also an example of a discrimination unit that determines the next action that the shopper should take based on the actions recognized by the action recognition unit.

[0133] As ACT4616, processor 430 determines whether or not any items other than [product name] have been placed in the shopping basket BA. If no items other than [product name] have been placed in the shopping basket BA, processor 430 determines NO in ACT4616 and proceeds to ACT4614. Processor 430, which performs the processing of ACT4616, is an example of an action recognition unit that recognizes the actions performed by the shopper.

[0134] This processing loop from ACT4614 to ACT4616 causes the processor 430 to wait for the product [product name] to be added to the shopping cart BA. Note that, as long as the aforementioned time has elapsed, the scanner area LED frame 428 may remain lit green while waiting for the product to be added to the cart.

[0135] If, while waiting for product [product name] to be placed in the shopping basket, a product other than [product name] is placed in the shopping basket BA, the processor 430 determines YES in ACT4616 and proceeds to ACT4617. In ACT4617, the processor 430 flashes the basket area LED frame 424 in red, which is a warning color. The processor 430 that performs this ACT4617 process is an example of a warning unit that, when the action recognition unit recognizes that a shopper has performed an action different from the next action, operates a notification member provided in the action area where the different action was performed in a warning mode different from the first and second notification modes. Flashing red is an example of operation in warning mode.

[0136] As ACT4618, processor 430 outputs a warning message for incorrect actions. For example, processor 430 outputs a warning voice message from speaker 418 such as "Please bag the scanned items," and displays a similar warning message on the touch panel 416 display for a predetermined time. After that, processor 430 proceeds to the processing of ACT4614.

[0137] As ACT4619, processor 430 outputs guidance for the correct action. For example, processor 430 outputs a sound effect indicating the correct action, such as "ding-dong," from speaker 418.

[0138] As ACT4620, processor 430 starts lighting the basket area LED frame 424, which was lit yellow in ACT4612, in green for a certain period of time. This period of time is not strictly defined, but like other periods of time, it can be, for example, about 1 second. After that, processor 430 returns to the higher-level routine. Processor 430, which performs the processing of ACT4620, is an example of a correct action notification unit that, when the action recognition unit recognizes that a shopper has performed the next action, operates the notification member that was operated in the first notification mode in a second notification mode that is different from the first notification mode. Lighting in green for a certain period of time is an example of operation in the second notification mode.

[0139] The specified time periods for ACT4611 and ACT4620 may be the same or different.

[0140] Returning to the explanation of Figure 10, once the scanning and basket placement actions for a single item in the shopping basket, which constitute the registration action processing of ACT46, are completed, the processor 430 proceeds to ACT15. Thus, during the processing loop from ACT43 to ACT46, the shopper registers each item and places that item into the shopping basket BA. When the shopper declares payment, the processor 430 determines YES in ACT45 and proceeds to ACT47.

[0141] As ACT47, the processor 430 performs accounting processing. In this accounting processing, for example, the processor 430 sends an accounting command, including the terminal ID of the information terminal 4, to the store server 2 via the access point 5 using the wireless unit 436. Upon receiving this accounting command, the store server 2 issues a unique accounting code and sends a barcode image showing the accounting code back to the information terminal 4. The shopper displays this barcode image on the touch panel 416 display and has the scanner on the checkout machine 6 read it. As a result, the checkout machine 6 obtains the accounting code, retrieves the transaction data corresponding to this accounting code from the store server 2, and can perform the settlement process.

[0142] As described above, the control device can be a computer including the processor 430 of the information terminal 4. Therefore, the same effects as in the first embodiment can be achieved in this second embodiment as well.

[0143] Although the explanation uses an information terminal 4 attached to a shopping cart C as an example, the control device of the information terminal 4 may be a smartphone owned by the shopper. In this case, an application program that performs the functions of the control device according to this embodiment is installed on the smartphone. The shopper uses a shopping cart C equipped with a basket area LED frame 424. Scanning of products is performed using the camera on the smartphone, and a store surveillance camera 7 can be used as the camera used to recognize the shopper's actions. The smartphone acquires a store image from the store surveillance camera 7 corresponding to the location of the smartphone via an access point 5, and extracts an area image from the acquired store image, which is the pixel portion of the area where action detection should be performed as the next action. The area image can be extracted based on the position of the shopper's hand in the store layout or relative to each part of the shopping cart C, which is detected in real time using object detection technology such as an SSD. Since the smartphone does not have a scanner area LED frame 428, it displays a scan message or an image representing the scanning process on its display. When displaying this, the color and / or display pattern can be changed according to whether the action is correct or incorrect.

[0144] [Third Embodiment] Next, a third embodiment will be described. Note that the same configurations and operations as in the first embodiment will be denoted by the same reference numerals as in the first embodiment, and their descriptions will be omitted.

[0145] In the first and second embodiments described above, a behavioral guidance system was described in which a notification member and a control device were provided in the self-POS terminal 1 and the information terminal 4. However, a behavioral guidance system in which the notification member and the control device are distributed across separate devices is also possible.

[0146] Figure 12 is a schematic diagram showing a self-service payment system as a behavioral guidance system according to the third embodiment, which includes a store server 2 as a control device according to the third embodiment. The payment system of this embodiment includes multiple self-service POS terminals 1 and a store server 2, as well as multiple checkout monitoring cameras 8.

[0147] Figure 13 is a perspective view showing an example of the imaging range of the checkout monitoring camera 8. The checkout monitoring camera 8 is provided in a one-to-one correspondence with the self-service POS terminal 1 and is installed on the ceiling or the like above the self-service POS terminal 1. The mounting position and orientation of the checkout monitoring camera 8 are set so that its field of view, or imaging area, includes the basket area and bagging area in the first embodiment, as shown by the dashed line in Figure 13.

[0148] In this embodiment, the self-service POS terminal 1 is not equipped with a basket area camera 136 and a bagging area camera 138, but the notification members, a bagging area LED frame 140 and a scanner area LED frame 156, are provided in the same manner as in the first embodiment.

[0149] Figure 14 is a sequence diagram illustrating the operation of such a payment system. When the self-service POS terminal 1 is operated by a shopper and a bag is selected, as described in ACT11 and ACT12 in the first embodiment, it sends a start notification to the store server 2 (step S1).

[0150] In response to this start notification, the store server 2 determines the next action that the shopper should take and, based on the determined next action, sends a notification and illumination command to the self-service POS terminal 1 (step S2). In this sequence diagram, each operation performed by the store server 2 is actually executed by the processor in the store server 2, but here it is simply described as being performed by the store server 2. The notification and illumination command can be, for example, a command to illuminate the bagging area LED frame 140 of the self-service POS terminal 1 in yellow, which is a warning color, and a command to display the message "Please set the bag" on the touch panel 116 display, as well as a command to output the same voice guidance from the speaker 154. Thus, the store server 2 is an example of an action recognition unit that recognizes the actions performed by the shopper, and an example of a determination unit that determines the next action that the shopper should take based on the actions recognized by the action recognition unit. Furthermore, the store server 2 is an example of a next action notification unit that operates a notification member provided in a first notification mode, corresponding to the action area where the next action should be performed, based on the next action determined by the discrimination unit.

[0151] The processor 174 of the self-service POS terminal 1 receives a notification and lighting command from the store server 2 and, as in ACT13 and ACT14 of the first embodiment, lights the bagging area LED frame 140 in yellow, which is a warning color, displays a guidance message on the touch panel 116 display, and outputs voice guidance from the speaker 154 (step S3). The bagging area LED frame 140 is an example of a notification member provided in a corresponding action area where the next action should be performed. Lighting in yellow is an example of operation in the first notification mode.

[0152] Furthermore, the store server 2 acquires the monitoring images captured by the checkout monitoring camera 8, which is compatible with the self-service POS terminal 1 (step S4).

[0153] The store server 2 then determines the shopper's actions from the monitoring image (step S5). Specifically, the store server 2 first extracts an area image from the acquired monitoring image, which is the pixel portion of the area where action detection should be performed as the next action. For example, if the next action is a bag-setting action, such as setting a shopping bag or reusable bag into the weighing unit 112, the store server 2 extracts the pixel portion corresponding to the bag-packing area as described in the first embodiment from the monitoring image as a bag-packing area image. This corresponds to ACT 15 in the first embodiment. The store server 2 then recognizes the action being performed by the shopper based on the extracted bag-packing area image. The specific method of recognition is as described in the first embodiment. Therefore, the store server 2 is an example of an action recognition unit that recognizes actions performed by a shopper. The store server 2 then determines whether the action of setting the bag has been performed. This corresponds to ACT 16 in the first embodiment.

[0154] When the action of setting a bag is performed, the store server 2 sends a notification and illumination command to the self-service POS terminal 1 (step S6). This notification and illumination command can be, for example, a command to illuminate the bagging area LED frame 140 of the self-service POS terminal 1 in green for a certain period of time, and a command to output a sound effect indicating the correct action, such as "ding-dong," from the speaker 154. Thus, the store server 2 is an example of a correct action notification unit that, when the action recognition unit recognizes that the shopper has performed the next action, operates the notification member that was operated in the first notification form in a second notification form different from the first notification form. Furthermore, the notification and illumination command can continue with a command to illuminate the scanner area LED frame 156, which is arranged around the reading window 144 of the scanner 142 of the self-service POS terminal 1, in yellow, which is a warning color, and a command to display the message "Please scan the product" on the touch panel 116 display, and to output the same voice guidance from the speaker 154. Therefore, the store server 2 is an example of a next action notification unit that operates a notification member provided in a first notification mode, corresponding to the action area where the next action should be performed, based on the next action determined by the discrimination unit.

[0155] The processor 174 of the self-service POS terminal 1 receives a notification and lighting command from the store server 2 and, as in ACT17 and ACT18 of the first embodiment, starts lighting the bagging area LED frame 140 in green for a certain period of time and outputs a sound effect indicating the correct answer from the speaker 154 (step S7). Lighting in green for a certain period of time is an example of operation in the second notification mode. In addition, the processor 174 of the self-service POS terminal 1 lights the scanner area LED frame 156 in yellow, which is a warning color, as in ACT2101 and ACT2102 of the first embodiment, displays guidance on the touch panel 116 display, and outputs voice guidance from the speaker 154. The scanner area LED frame 156 is an example of a notification member provided corresponding to the action area where the next action should be performed. Lighting in yellow is an example of operation in the first notification mode.

[0156] Next, the store server 2 acquires the monitoring image captured by the checkout monitoring camera 8, which is corresponding to the self-service POS terminal 1 (step S8).

[0157] Then, the store server 2 performs a termination determination, that is, it determines whether or not the shopping basket is empty (step S9). Specifically, from the acquired monitoring image, it extracts the pixel portion corresponding to the basket area as described in the first embodiment as the basket area image. This is an operation corresponding to ACT19 in the first embodiment. Next, the store server 2 uses known image recognition processing technology to determine whether or not the shopping basket is empty from the extracted basket area image. This is an operation corresponding to ACT20 in the first embodiment. Thus, the store server 2 is an example of an action recognition unit that recognizes the actions performed by the shopper, and is also an example of a determination unit that determines the next action that the shopper should take based on the actions recognized by the action recognition unit.

[0158] If the shopping basket is not empty, the store server 2 obtains the scan result from the scanner 142 on the self-service POS terminal 1 (step S10). This corresponds to the operation of ACT2103 in the first embodiment. Furthermore, the store server 2 obtains the monitoring image captured by the checkout monitoring camera 8 corresponding to the self-service POS terminal 1.

[0159] The store server 2 then determines the shopper's actions from the scan results and monitoring images (step S11). Specifically, the store server 2 determines from the acquired scan results whether or not a product scan has been performed. This corresponds to ACT2104 in the first embodiment. The store server 2 also extracts the pixel portion corresponding to the bagging area, as described in the first embodiment, from the acquired monitoring images as a bagging area image, and determines whether or not the products have been bagged based on the bagging area image. This corresponds to ACT2104 and ACT2105 in the first embodiment. Therefore, the store server 2 is an example of an action recognition unit that recognizes actions performed by a shopper, and is also an example of a determination unit that determines the next action that the shopper should take based on the actions recognized by the action recognition unit.

[0160] When an action is taken to scan an item or to bag an item, the store server 2 sends a notification / light-up command to the self-service POS terminal 1 (step S12).

[0161] This notification / lighting command can be a command to light up the bagging area LED frame 140 of the self-service POS terminal 1 in green for a certain period of time when a product is scanned, and a command to output a sound effect indicating the correct answer, such as "ding-dong," from the speaker 154. Therefore, the store server 2 is an example of a correct action notification unit that, when the action recognition unit recognizes that a shopper has performed the next action, operates the notification member that was operated in the first notification form in a second notification form different from the first notification form. Furthermore, the notification / lighting command can continue with a command to light up the bagging area LED frame 140 of the self-service POS terminal 1 in yellow, which is a warning color, and to display the message "[Product Name] Please bag it" on the touch panel 116 display, while also outputting the same voice guidance from the speaker 154. Therefore, the store server 2 is an example of a next action notification unit that operates a notification member provided in a first notification mode, corresponding to the action area where the next action should be performed, based on the next action determined by the discrimination unit.

[0162] Furthermore, if an item that has not been scanned is bagged, the notification / lighting command can be a command to cause the bagging area LED frame 140 of the self-POS terminal 1 to flash red, which is a warning color, to output a warning voice such as "Please scan the items before bagging them" from the speaker 154, and to display a similar warning message on the touch panel 116 display for a predetermined time. Thus, the store server 2 is an example of a warning unit that, when the action recognition unit recognizes that a shopper has performed an action different from the next action, operates a notification member provided in the action area where the different action was performed in a warning mode different from the first and second notification modes. Flashing red is an example of operation in the warning mode.

[0163] The processor 174 of the self-service POS terminal 1 receives a notification and lighting command from the store server 2, and outputs sound and lights up according to the command (step S13).

[0164] Specifically, the processor 174 starts lighting the scanner area LED frame 156 in green for a certain period of time, as in ACT2110 and ACT2111 of the first embodiment, and outputs a sound effect indicating the correct answer from the speaker 154. Lighting in green for a certain period of time is an example of operation in the second notification mode. Furthermore, the processor 174 lights the bagging area LED frame 140 in yellow, which is a warning color, as in ACT2112 and ACT2123 of the first embodiment, displays guidance on the touch panel 116 display, and outputs voice guidance from the speaker 154. The bagging area LED frame 140 is an example of a notification member provided in a corresponding action area where the next action should be performed. Lighting in yellow is an example of operation in the first notification mode.

[0165] Alternatively, as in ACT2107 and ACT2108 of the first embodiment, the processor 174 flashes the bagging area LED frame 140 in red, which is a warning color, and outputs a warning sound from the speaker 154, while also displaying a similar warning message on the touch panel 116 display for a predetermined time. Flashing in red is one example of operation in warning mode.

[0166] Furthermore, when a product is scanned, the store server 2 acquires the monitoring image captured by the checkout monitoring camera 8 corresponding to the self-service POS terminal 1 (step S14).

[0167] Then, the store server 2 determines the shopper's actions from this monitoring image (step S15). Specifically, the store server 2 extracts the pixel portion corresponding to the bagging area as described in the first embodiment from the acquired monitoring image as a bagging area image, and determines whether or not the shopper has bagged the product of [product name] or other products based on the bagging area image. This corresponds to the operations ACT2114, ACT2115, and ACT2116 in the first embodiment. Therefore, the store server 2 is an example of an action recognition unit that recognizes the actions performed by the shopper, and is also an example of a determination unit that determines the next action the shopper should take based on the actions recognized by the action recognition unit.

[0168] When an action is taken to bag any of the products, the store server 2 sends a notification and light-up command to the self-service POS terminal 1 (step S16). After that, the store server 2 proceeds to the operation in step S8.

[0169] This notification / lighting command can be a command to light up the bagging area LED frame 140 of the self-service POS terminal 1 in green for a certain period of time when product [product name] is scanned, and a command to output a sound effect indicating the correct answer, such as "ding-dong," from the speaker 154. Therefore, the store server 2 is an example of a correct action notification unit that, when the action recognition unit recognizes that a shopper has performed the next action, operates the notification member that was operated in the first notification form in a second notification form different from the first notification form. Furthermore, the notification / lighting command can continue with a command to light up the scanner area LED frame 156 of the self-service POS terminal 1 in yellow, which is a warning color, and to display the message "Please scan the product" on the touch panel 116 display, while also outputting the same voice guidance from the speaker 154. Therefore, the store server 2 is an example of a next action notification unit that operates a notification member provided in a first notification mode, corresponding to the action area where the next action should be performed, based on the next action determined by the discrimination unit.

[0170] Furthermore, if a product other than [product name] is scanned, the notification / lighting command may be a command to cause the bagging area LED frame 140 of the self-POS terminal 1 to flash red, which is a warning color, to output a warning voice such as "Please bag the scanned product" from the speaker 154, and to display a similar warning message on the touch panel 116 display for a predetermined time. Thus, the store server 2 is an example of a warning unit that, when the action recognition unit recognizes that a shopper has performed an action different from the next action, operates the notification member provided in the action area where the different action was performed in a warning mode different from the first and second notification modes. Flashing red is an example of operation in the warning mode.

[0171] The processor 174 of the self-service POS terminal 1 receives a notification and lighting command from the store server 2, and outputs sound and lights up according to the command (step S17).

[0172] Specifically, the processor 174 starts lighting the bagging area LED frame 140 in green for a certain period of time, as in ACT2119 and ACT2120 of the first embodiment, and outputs a sound effect indicating the correct answer from the speaker 154. Lighting in green for a certain period of time is an example of operation in the second notification mode. Furthermore, the processor 174 lights the scanner area LED frame 156 in yellow, which is a warning color, as in ACT2101 and ACT2102 of the first embodiment, displays guidance on the touch panel 116 display, and outputs voice guidance from the speaker 154. The scanner area LED frame 156 is an example of a notification member provided in a corresponding action area where the next action should be performed. Lighting in yellow is an example of operation in the first notification mode.

[0173] Alternatively, as in ACT2117 and ACT2118 of the first embodiment, the processor 174 flashes the bagging area LED frame 140 in red, which is a warning color, and outputs a warning sound from the speaker 154, while also displaying a similar warning message on the touch panel 116 display for a predetermined time. Flashing in red is one example of operation in warning mode.

[0174] Furthermore, in step S9, if it is determined that the process is complete, i.e., that the shopping basket is empty, the store server 2 sends a payment command to the self-service POS terminal 1 (step S18).

[0175] The processor 174 of the self-service POS terminal 1 receives an accounting command from the store server 2 and performs accounting processing based on the product sales data stored in the transaction file 186 (step S19).

[0176] As described above, the control device can be the store server 2. Therefore, the same effects as in the first embodiment can be achieved in this third embodiment as well.

[0177] Although the above explanation has been given using an example corresponding to the first embodiment, it goes without saying that this embodiment can also be applied to the second embodiment in the same way.

[0178] Furthermore, while we have described an example where store server 2 acts as the control device, it goes without saying that a management server located on a cloud outside the store can also act as the control device.

[0179] [Fourth Embodiment] While the first to third embodiments described an action guidance system corresponding to a series of actions performed by a shopper, the present invention is also applicable to actions other than shopping. Furthermore, although the system recognizes what actions have been performed based on images captured by a camera, it is also possible to perform action detection using sensors other than cameras.

[0180] For example, handwashing has become particularly important in light of the recent spread of infectious diseases. Figure 15 is a plan view showing an example of the external configuration of a handwashing device to which the behavioral guidance system according to the fourth embodiment, which includes a control device according to the fourth embodiment, is applied. The handwashing device includes a washbasin W. The washbasin W is equipped with an automatic faucet W1 from which water is discharged, an automatic soap dispenser W2 from which soapy water is discharged, and an automatic blower dispenser W3 from which warm air is blown out, and an object detection sensor W4 is provided to detect when a hand is in a predetermined position in order to operate each of the dispensers.

[0181] In this embodiment, a water area LED frame 92 is placed at the base of the automatic faucet W1 in the washbasin W, a soap area LED frame 94 is placed at the base of the automatic soap stopper W2, and a blower area LED frame 96 is placed at the base of the automatic blower stopper W3.

[0182] The control device (not shown) defines a series of handwashing actions as rinsing with water, washing with soap, and blowing with a blower. It stores the sequence of these actions, including the number of times and duration of the rinsing and soap washing actions, and illuminates the LED frames 92, 94, and 96 as appropriate in response to the action detected by the object detection sensor W4 on each tap.

[0183] In the first to third embodiments, the detection result of a shopper's hand by a sensor such as the object detection sensor W4 can be used when recognizing any of the actions.

[0184] [Other embodiments] The embodiments of the control device have been described above, but the embodiments are not limited thereto.

[0185] For example, the light-emitting element that serves as the notification element does not have to be configured in a frame shape that surrounds the action area. For example, one green LED, one yellow LED, and one red LED could be placed in any one location within the action area.

[0186] Furthermore, the notification component is not limited to a light-emitting component; it may also be a projection device that projects spotlights, or a message or image displayed on the touch panel displays 116, 416.

[0187] The scanners 142 and 422 are not limited to those that scan code symbols attached to products and output product codes; they may also be cameras that capture images of the product's appearance, and the processors 174 and 430 may identify the product from the product image using pattern matching or the like. In this case, the scanning action described in the embodiment should be read as the imaging action.

[0188] Furthermore, in the first embodiment, the bagging area camera 138 is installed so that the mounting surface 130 is included in its field of view, i.e., imaging area. However, it may be installed so that a wider area is captured. In this case, as described in the second embodiment, a bagging area image can be obtained by extracting the pixel portion corresponding to the bagging area from the image captured by the bagging area camera 138.

[0189] Furthermore, the aforementioned information processing flow performed by processors 174,430 is merely an example and is not limited to this order. For example, the order of ACT13 and ACT14 in the flowchart of Figure 4 may be swapped, or they may be performed in parallel. In this way, the order of processing may be changed or multiple processing may be performed in parallel, as long as no conflict arises with preceding or succeeding processing.

[0190] In addition, several embodiments of the present invention have been described, but these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be carried out in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included within the scope of the invention, as well as within the scope of the invention and its equivalents as described in the claims. [Explanation of Symbols]

[0191] 1…Self-service POS terminal, 2…Store server, 3…Communication network, 4…Information terminal, 5…Access point, 6…Accounting machine, 7…Sales floor surveillance camera, 8…Checkout surveillance camera, 92…Water area LED frame, 94…Soap area LED frame, 96…Blower area LED frame, 110,410…Main unit, 112…Scale unit, 116,416…Touch panel, 118,120…Camera pole, 122…Basket stand, 126…Weighing pan , 128...Bag holder, 130...Placement surface, 132...Holding arm, 136,414...Basket area camera, 138...Bagging area camera, 140...Bagging area LED frame, 142,422...Scanner, 144,426...Reading window, 154,418...Speaker, 156,428...Scanner area LED frame, 174,430...Processor, 176,432...Main memory, 178,434...Auxiliary storage device, 182...Communication interface, 184,438...System transmission path, 186...Transaction file, 188,440...Action sequence storage unit, 190,442...Product image storage unit, 412...Surrounding area camera, 424...Basket area LED frame, 436...Wireless unit, BA...Shopping basket, C...Shopping cart, C3...Basket support section, W...Washbasin, W1...Automatic faucet, W2...Automatic soap dispenser, W3...Automatic blower dispenser, W4...Object detection sensor.

Claims

1. A control device for controlling notification members provided in each of the multiple action areas that perform multiple actions included in a series of actions performed by a shopper, An action recognition unit that recognizes the actions performed by the aforementioned shopper, Based on the behavior recognized by the behavior recognition unit, a determination unit determines the next action that the shopper should take. A next action notification unit operates the notification member provided in the action area where the next action should be performed, in a first notification mode, based on the next action determined by the discrimination unit. When the action recognition unit recognizes that the shopper has performed the subsequent action, the correct action notification unit causes the notification member, which was operated in the first notification mode, to operate in a second notification mode different from the first notification mode. When the action recognition unit recognizes that the shopper has performed an action different from the aforementioned next action, the warning unit activates the notification member provided in the action area where the different action was performed in a warning mode different from the first and second notification modes. A control device having

2. The series of actions performed by the shopper includes a number of actions performed by the shopper when registering the purchased items. The control device further includes a storage unit that stores the sequence of the multiple actions performed when registering the purchased items, The discrimination unit determines the next action based on the order of actions stored in the memory unit. The control device according to claim 1.

3. The aforementioned multiple actions include, at a minimum, a scanning action in which a scanner scans a code symbol attached to the purchased product, or an imaging action in which an imaging unit captures the appearance of the purchased product. The behavior recognition unit recognizes, based on the output of the scanner or the imaging unit, that the shopper has performed the scanning action or the imaging action. The control device according to claim 2.

4. The notification member includes a visual notification member, The first notification mode by the next action notification unit, the second notification mode by the correct action notification unit, and the warning mode by the warning unit are such that the visual notification modes of the visual notification members are different from each other. The control device according to any one of claims 1 to 3.

5. The aforementioned visual notification member includes a light-emitting member, The first notification mode by the next action notification unit, the second notification mode by the correct action notification unit, and the warning mode by the warning unit are such that at least one of the light emission color and lighting pattern of the light-emitting member is different from each other. The control device according to claim 4.

6. Notification members are provided in each of the multiple action areas where multiple actions included in a series of actions performed by a shopper are carried out, A control device for controlling the notification member, Equipped with, The control device is An action recognition unit that recognizes the actions performed by the aforementioned shopper, Based on the behavior recognized by the behavior recognition unit, a determination unit determines the next action that the shopper should take. A next action notification unit operates the notification member provided in the action area where the next action should be performed, in a first notification mode, based on the next action determined by the discrimination unit. When the action recognition unit recognizes that the shopper has performed the subsequent action, the correct action notification unit causes the notification member, which was operated in the first notification mode, to operate in a second notification mode different from the first notification mode. When the action recognition unit recognizes that the shopper has performed an action different from the aforementioned next action, the warning unit activates the notification member provided in the action area where the different action was performed in a warning mode different from the first and second notification modes. A behavioral guidance system that includes [the following features].

7. A computer comprising a processor and memory controls notification members provided in each of multiple action areas that perform multiple actions included in a series of actions performed by a shopper. Action recognition unit that recognizes the actions performed by the aforementioned shopper, A determination unit determines the next action that the shopper should take, based on the action recognized by the action recognition unit and the order of the multiple actions stored in the memory. A next action notification unit operates the notification member provided in the action area where the next action should be performed, in a first notification mode, based on the next action determined by the discrimination unit. When the action recognition unit recognizes that the shopper has performed the subsequent action, the correct action notification unit causes the notification member, which was operated in the first notification mode, to operate in a second notification mode different from the first notification mode. When the action recognition unit recognizes that the shopper has performed an action different from the subsequent action, the warning unit activates the notification member provided in the action area where the different action was performed in a warning mode different from the first and second notification modes. A program designed to function as such.

Images