Customer monitoring and notification device and customer monitoring and notification program

Abstract

To provide a customer monitoring device and a customer monitoring notification program that prevent missing of an ordering opportunity.SOLUTION: A customer monitoring device according to an embodiment includes a communication unit, an image recognition unit, a skeleton estimation unit, a first detection unit, a second detection unit, and a notification output unit. The communication unit receives imaged video data. The image recognition unit extracts and recognizes feature quantities of a specific article on a table and a user from the video data. The skeleton estimation unit estimates a skeleton of the user and generates skeleton data thereof. The first detection unit detects whether a feature quantity indicating the specific article is present on the table recognized by the image recognition unit. The second detection unit detects a preset elbow bending angle from the skeletal data. The notification output unit outputs a notification instruction when the first detection unit or the second detection unit detects the feature quantity and the elbow bending angle.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] Embodiments of the present invention relate to a customer monitoring notification device and a customer monitoring notification program.

Background Art

[0002] Conventionally, in a restaurant such as a restaurant, there is a customer monitoring device that monitors the behavior of customers in order to quickly take orders from customers who are dining inside the store. The customer monitoring device is connected to an imaging unit and acquires the state inside the store. The customer monitoring device recognizes a customer from the acquired image or video, and determines that the customer is trying to call a store clerk when the customer performs an operation above a preset threshold value from the movements of the hands and face, and notifies the store clerk.

[0003] However, when the customer's movement is small enough not to reach the threshold value, the customer monitoring device does not notify the store clerk. In such a case, it may happen that the store clerk misses the opportunity to take an order from the customer.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The problem to be solved by the present invention is to provide a customer monitoring device and a customer monitoring notification program that do not miss the order opportunity.

Means for Solving the Problems

[0006] In order to achieve the above problems, the customer monitoring device of the present embodiment includes a communication unit, an image recognition unit, a skeleton estimation unit, Control unit andIt has a notification output unit. The communication unit receives captured video data. The image recognition unit extracts and recognizes the features of specific items on the table and the user from the video data. The skeleton estimation unit estimates the user's skeleton and generates skeleton data. The control unit controls a first detection that detects whether a feature representing a specific item exists on the table recognized by the image recognition unit, and a second detection that detects a pre-set elbow bending angle from the skeletal data. The notification output unit is the first inspection Come out or second inspection If detection is made by output, and If skeletal data cannot be detected again after a predetermined time has elapsed, a notification instruction will be issued. [Brief explanation of the drawing]

[0007] [Figure 1] A block diagram showing an example of a customer monitoring system according to the first embodiment. [Figure 2] A block diagram showing an example of a customer monitoring device according to the first embodiment. [Figure 3] A functional block diagram showing an example of the functional configuration of a customer monitoring device according to the first embodiment. [Figure 4] A figure showing an example of skeletal data according to the first embodiment. [Figure 5] A diagram showing an example of the data structure of a flag management file according to the first embodiment. [Figure 6] A flowchart showing an example of customer monitoring processing according to the first embodiment. [Figure 7] A flowchart showing an example of behavioral monitoring processing according to the first embodiment. [Figure 8] A flowchart showing an example of table monitoring processing according to the first embodiment. [Figure 9] A figure showing an example of a display image according to the first embodiment. [Figure 10] A flowchart showing an example of behavioral monitoring processing according to the second embodiment. [Modes for carrying out the invention]

[0008] (First embodiment) The following description of this embodiment will be made with reference to the drawings. The customer monitoring device 10 of this embodiment determines whether to activate a flag by performing image recognition on the customer's skeletal data M generated from video data acquired from an imaging unit 20 installed in a store such as a restaurant, and on the video data or an image extracted from the video data.

[0009] This is a block diagram showing an example of a customer monitoring system 1 according to the first embodiment. The customer monitoring system 1 consists of, for example, a customer monitoring device 10, an imaging unit 20, and a notification unit 30, and each unit is connected by a communication network. The communication network is, for example, a LAN (Local Area Network). The LAN may be a wired LAN or a wireless LAN.

[0010] The customer monitoring device 10 monitors the actions of customers and the status of their tables. Customers may also be referred to as patrons, store users, or consumers. The customer monitoring device 10 acquires video data of the store from the connected imaging unit 20 and monitors the actions of customers and the status of their tables. The customer monitoring device 10 estimates the customer's skeleton from the acquired video data to monitor their actions and generates skeleton data M. The customer monitoring device 10 uses image recognition technology to monitor the status of the tables. The customer monitoring device 10 also determines whether it is an opportunity to order based on the skeleton data M and the image of the table, and if it determines that it is an opportunity to order, it sends a notification instruction to the notification unit 30. Such a monitoring device 10 may be an independent device, or it may be part of a personal computer, in-store server, or off-store server installed in the store.

[0011] The imaging unit 20 captures images of the store interior. The imaging unit 20 outputs the light formed on the image sensor as an electrical signal, and generates video data by converting, compressing, and encoding the signal. After generating the video data, the imaging unit 20 transmits the video data to the monitoring device 10. In addition, the imaging unit 20 performs development processing, including gradation conversion, noise reduction, and scratch correction, when generating the video data. The imaging unit 20 may be placed in different locations within the store that can monitor the entire area used by customers, or multiple imaging units 20 may be placed so that each corresponds to multiple tables on a one-to-one basis.

[0012] The notification unit 30 notifies the store staff of the ordering opportunity. The notification unit 30 notifies the store staff of the table where the ordering opportunity has occurred according to the notification instruction received from the customer monitoring device 10. For example, the notification unit 30 includes a lamp corresponding to each table and notifies the store staff by making the lamp emit light. In addition to the notification by lighting the lamp, the notification unit 30 may also perform a notification by sound.

[0013] FIG. 2 is a block diagram showing an example of the customer monitoring device according to the first embodiment. The customer monitoring device 10 includes a control unit 100, a storage unit 101, a clock unit 102, and a communication I / F 103, and each unit is connected via a bus.

[0014] The control unit 100 is composed of a CPU (Central Processing Unit) 1001, a ROM 1002, and a RAM 1003. The CPU 1001 controls the entire customer monitoring device 10. The ROM (Read Only Memory) 1002 stores various programs such as programs used for driving the monitoring device 10 and various data. The RAM (Random Access Memory) 1003 is used as a work area for the CPU 1001 and develops various programs and various data stored in the ROM 1002 and the storage unit 101. The control unit 100 executes various functional processes of the customer monitoring device 10 by operating according to the information processing program stored in the ROM 1002 and the storage unit 101 and developed in the RAM 1003.

[0015] The storage unit 101 is composed of a storage medium such as a HDD (Hard Disc Drive) or a flash memory. The storage unit 101 stores software such as an operating system and other necessary application programs, as well as user information, for the operation of the customer monitoring device 10. In addition, the storage unit 101 stores a flag management file 1011, and may also store video data acquired from the imaging unit 20 and image data extracted from the video data. Further, the flag management file 1011 describes a unique table number corresponding to each table. Note that the flag management file 1011 may be stored in a storage medium or device accessible from the customer monitoring device 10, and may be stored in an external HDD, server, management terminal installed in the store, or the like.

[0016] The clock unit 102 functions as a time information source of the customer monitoring device 10. The control unit 100 acquires the current date and time based on the time information measured by the clock 102. Further, the control unit 100 may be based on the time indicated by the clock unit 102 when performing processing at a preset time.

[0017] The communication I / F 103 is an interface for communicating with the imaging unit 20 and the notification unit 30. The video data generated by the imaging unit 20 is taken into the customer monitoring device 10 via the communication I / F 103. Further, the notification instruction output from the customer monitoring device 10 to the notification unit 30 is output via the communication I / F 103.

[0018] FIG. 3 is a functional block diagram showing an example of the functional configuration of the customer monitoring device 10 according to the first embodiment. The control unit 100 realizes an image recognition unit 201, a skeleton estimation unit 202, a table monitoring unit 203, an action monitoring unit 204, and a notification output unit 205 as functional units by a program stored in the ROM 1002 or the storage unit 101.

[0019] The image recognition unit 201 recognizes tables and customers from video data acquired via the imaging unit 20. Similar to known image recognition technologies, the image recognition unit 201 recognizes objects on the table by extracting features from the acquired video data. In other words, the image recognition unit 201 can recognize whether glasses, mugs, or plates are placed on the table from the features extracted from the video data. Furthermore, when imaging tables in a store with a small number of imaging units 20, that is, when one imaging unit 20 does not monitor one table, the image recognition unit 201 identifies the table being used by a customer based on the features, arrangement, or distance of each table.

[0020] Furthermore, the image recognition unit 201 recognizes people from features extracted from video data, similar to how it recognizes tables.

[0021] Although the image recognition unit 201 was described using an example of recognizing tables and customers from video data, it may also extract images from the video and recognize tables and customers from the image data.

[0022] The skeleton estimation unit 202 estimates the skeleton of the customer recognized by the image recognition unit 201 and generates skeleton data M. The technique for skeleton estimation may be a known AI technique such as deep learning. The skeleton estimation unit 202 understands the customer's movements from the generated skeleton data M. The skeleton estimation unit 202 recognizes order action 1 and order action 2 from the customer's movements. In this embodiment, skeleton estimation is performed based on the acquired video, but it may also be performed from image data extracted from the video.

[0023] Figure 4 shows an example of skeletal data M according to the first embodiment. The skeletal estimation unit 202 estimates the skeleton of the customer recognized by the image recognition unit 201. After the skeletal estimation is complete, the skeletal estimation unit 202 generates skeletal data M. Markers are placed on the skeletal data M generated by the skeletal estimation unit 202 at positions necessary to identify the movement of the body, such as the eyes, shoulders, elbows, and wrists. For example, the image recognition unit 201 recognizes that the right arm is extended to the right and horizontally to the ground when marker M1 placed at the right shoulder, marker M2 placed at the right elbow, marker M3 placed at the right wrist, and marker M4 set at the center of the chest are positioned in a roughly straight line.

[0024] Returning to Figure 3, the table monitoring unit 203 monitors the state of the table being used by the customer. Based on the state of the table recognized by the image recognition unit 201, it determines whether there are any empty glasses, mugs, plates, etc. If it determines that there are any empty glasses, mugs, plates, etc., the image recognition unit 201 turns on flag 3 in the record containing the table number corresponding to the monitored table in the flag management file 1011, which will be described later.

[0025] The behavior monitoring unit 204 monitors customer movements from the skeletal data M generated by the skeletal estimation unit 202. The behavior monitoring unit 204 recognizes the area around a table as a single monitoring area. These monitoring areas are separated. Therefore, each table has one monitoring area set up. If the behavior monitoring unit 204 determines that a customer has taken either order action 1 or order action 2 (described later) within a monitoring area, it turns on flag 1 or flag 2 in the record containing the table number corresponding to the table that is a monitoring area in the flag management file 1011 (described later). By recognizing the area around a table as a monitoring area, the behavior monitoring unit 204 recognizes that actions taken by customers within a monitoring area are actions taken by customers using the corresponding table.

[0026] Although it has been explained that the behavior monitoring unit 204 is configured to recognize the monitoring area in order to associate table numbers with customers, this may be done by the image recognition unit 201 or an independent function, or by a combination of both.

[0027] The notification output unit 205 notifies the notification unit 30 that an order opportunity has arisen. The notification output unit 205 checks the flags that are active in the flag management file 1011, which will be described later. If an active flag is found, it extracts the table number from the record that indicates the flag is active. After extracting the table number, the notification output unit 205 sends a notification instruction consisting of the extracted table number to the notification unit 30.

[0028] Figure 5 shows an example of the data structure of the flag management file 1011 according to the first embodiment. The flag management file 1011 consists of records that contain a unique table number indicating a table installed in the store and a flag corresponding to each table number. The flags described in the records are set to either "0" to indicate invalidity or "1" to indicate validity. Each record contains flags such as flag 1, flag 2, and flag 3. Flag 1 is a flag related to an action (hereinafter referred to as order action 1) in which the angle of the elbow bend is a certain value based on the customer's movements. Therefore, if the customer is recognized as having performed order action 1, flag 1 becomes "1", and if the customer is recognized as not having performed order action 1 or if the flag is reset in the customer monitoring process described later, flag 1 becomes "0". Flag 2 is a flag related to an action (hereinafter referred to as order action 2) in which the customer raises their hand based on their movements. Therefore, if the customer is recognized as having performed order action 2, flag 2 becomes "1", and if the customer is recognized as not having performed order action 2 or if the flag is reset in the customer monitoring process described later, flag 2 becomes "0". Flag 3 is a flag related to the state of glasses, mugs, and plates on the table when they are empty. Therefore, if it is recognized that there are empty glasses or mugs or finished plates on the table, Flag 3 will be "1". If it is recognized that there are glasses or mugs with drinks remaining on the table, or plates with food still on them, or if the flag is reset during the customer monitoring process described later, Flag 1 will be "0".

[0029] Furthermore, in order to determine that an order is placed, the behavior monitoring unit 204 must recognize that the customer's elbow is bent at a pre-set angle. The behavior monitoring unit 204 recognizes the customer's elbow bending angle from the generated skeletal data M. Specifically, the behavior monitoring unit 204 uses a marker M2 placed on the right elbow as the origin of the elbow bending angle, and measures the angle between a marker M1 placed on the shoulder and a marker M3 set on the right wrist to determine if it is an order. The angle that the behavior monitoring unit 204 recognizes as an order is, for example, between 70 and 100 degrees. In this embodiment, the determination was made using the elbow bending angle of the right arm, but it may also be made using the elbow bending angle of the left arm. Also, the setting for the elbow bending angle that determines an order is placed may be changed.

[0030] In order to determine that an order has been placed (action 2), the behavior monitoring unit 204 must determine that the customer has raised their hand. In this embodiment, raising a hand refers to a state in which, for example, marker M3 is positioned higher on the head side than marker M1 or marker M4. In this embodiment, the determination of whether a hand has been raised was made by the movement of the right arm, but it may also be made by the movement of the left arm, in which case the condition may be set that the marker placed on the left wrist is positioned higher than marker 4 or the marker placed on the left shoulder. Alternatively, the determination may be made by whether the marker placed on the wrist is positioned higher than the marker placed on the elbow.

[0031] Next, we will explain an example of customer monitoring processing when a customer enters the store and sits down at the table assigned to table number 1. First, before the customer enters the store, the imaging unit 20 takes images of the store and generates video data. After generating the video data, the imaging unit 20 transmits the video data to the customer monitoring device 10. After receiving the video data, the customer monitoring device 10's image recognition unit 201 recognizes the table and identifies which table number the imaged table is based on the table's features, arrangement, or distance. After the customer enters the store and sits down at table number 1, the imaging unit 20 transmits the video data showing the customer to the customer monitoring device 10, and the image recognition unit 201 recognizes the customer. After customer recognition, the skeleton estimation unit 202 estimates the customer's skeleton and generates skeleton data M.

[0032] The behavior monitoring unit 204 recognizes the area around the table the customer is using as a monitoring area and monitors the customer's behavior. When the customer makes an action within the monitoring area, if the action is recognized as either Order Action 1, where the elbow bends at an angle of 70 to 100 degrees, or Order Action 2, where the wrist is positioned in the center of the chest or towards the head from the elbow, the unit accesses the flag management file 1011 and activates Flag 1 or Flag 2, which is recorded in the record of Table Number 1, the table corresponding to the monitoring area.

[0033] The table monitoring unit 203 checks the status of the table after a predetermined time has elapsed, and if it confirms that there are empty glasses, empty mugs, or empty plates, it activates flag 3 for table number 1.

[0034] The control unit 100 within the customer monitoring device 10 accesses the flag management file 1011 to check if there are any records with the flag enabled. If confirmed, it sends a notification instruction to reset the flag in the record and then returns to behavioral monitoring.

[0035] In this embodiment, the process is described as monitoring customer behavior followed by monitoring tables, but this is not limited to this; the process of monitoring tables may be performed first, or in parallel.

[0036] Figures 5 to 7 are flowcharts showing an example of customer monitoring processing according to the embodiment. Figure 5 is a flowchart showing an example of customer monitoring processing according to the first embodiment. In this embodiment, an example is described in which a small number of imaging units 20 are used to monitor the entire store.

[0037] The control unit 100 acquires video data (ACT101). The control unit 100 acquires video data from the imaging unit 20 via the communication interface 103. The control unit 100 loads the acquired video data into the RAM 1003.

[0038] The control unit 100 recognizes the table (ACT102). The control unit 100 recognizes the table by extracting table features from the video data via the image recognition unit 201. It also identifies the table number based on the table features, arrangement, or distance.

[0039] The control unit 100 recognizes the customer (ACT103). The control unit 100 recognizes the customer from the customer's features via the image recognition unit 201.

[0040] The control unit 100 generates skeletal data M (ACT104). The control unit 100 estimates the skeleton from the customer's features recognized via the skeleton estimation unit 202 and generates skeletal data M. Markers M1 to Mn are also placed in the skeletal data M at the eyes, center of the chest, both shoulders, both elbows, both wrists, etc.

[0041] The control unit 100 performs behavior monitoring (ACT 105). The control unit 100 monitors the customer's behavior via the behavior monitoring unit 204. If the control unit 100 determines from the customer's actions that order action 1 or order action 2 has been performed, it activates the corresponding flag.

[0042] The control unit 100 performs table monitoring (ACT106). If the control unit 100 detects that there are empty plates or other items on the table via the table monitoring unit 203, it activates flag 3.

[0043] The control unit 100 checks if the flag is enabled (ACT107). The control unit 100 accesses the flag management file 1011 to check if there are any enabled flags. If the check reveals that there are no enabled flags (NO in ACT107), it returns to ACT105.

[0044] If the control unit 100 accesses the flag management file 1011 and confirms that there are any active flags (ACT 107 YES), the control unit 100 issues a notification instruction (ACT 108). The control unit 100 extracts the table number from the flag management file 1011 via the notification output unit 205 and sends the notification instruction to the notification unit 30 via the communication interface 103.

[0045] After sending the notification instruction, the control unit 100 resets the flag (ACT109). The control unit 100 changes the flag in the record containing the table number extracted by the notification output unit 205 from enabled to disabled. After resetting the flag, the control unit 100 returns to processing ACT105.

[0046] Next, we will describe the details of the behavior monitoring and table monitoring processes. Figure 6 is a flowchart showing an example of the behavior monitoring process according to the first embodiment. The behavior monitoring process is performed by the control unit 100 via the behavior monitoring unit 204.

[0047] The control unit 100 monitors the area (ACT201). The control unit 100 recognizes the area around the table being used by a customer as a single monitoring area, and distinguishes whether an action was taken by the customer using that table.

[0048] The control unit 100 checks if the customer is taking action (ACT202). The control unit 100 monitors whether the customer is moving, and if not (NO in ACT202), it returns to ACT201 and monitors the monitoring area again.

[0049] The control unit 100 checks if the customer is performing order action 1 (ACT203). The control unit 100 determines the angle of the forearm and upper arm, with the customer's elbow as the origin, based on markers M1 to M3. If the elbow bending angle is within a preset range (NO in ACT203), flag 1 is activated.

[0050] If the control unit 100 determines, based on the judgment of ACT203, that the elbow bending angle is not a preset angle (YES in ACT203), then it checks whether the order action is 2 (ACT204). If the check reveals that the marker M3 located on the wrist is located closer to the head than the marker M4 located in the center of the chest or the marker M2 located on the elbow (YES in ACT204), then the control unit 100 activates flag 2 (ACT205).

[0051] If the confirmation reveals that the conditions for determining order action 2 are not met (NO in ACT204), the control unit 100 returns to ACT201.

[0052] Figure 7 is a flowchart showing an example of table monitoring processing according to the first embodiment. The processing related to table monitoring is performed by the control unit 100 via the table monitoring unit 203.

[0053] The control unit 100 checks if the set time has elapsed (ACT301). The control unit 100 accesses the clock unit 102 and checks if the preset time has elapsed. After checking, if the preset time has not elapsed (NO in ACT301), it returns to ACT301.

[0054] If a pre-set time has elapsed (ACT301 YES), the control unit 100 checks the table (ACT302).

[0055] The control unit 100 checks for the presence of empty plates, glasses, mugs, etc., based on the features on the table (ACT303). If no empty plates, glasses, mugs, etc., are found, the process returns to ACT301.

[0056] If the check reveals empty plates, empty glasses, empty mugs, etc., the control unit 100 activates flag 3 (ACT304).

[0057] This configuration allows the customer monitoring device 10 to notify store staff without missing any opportunities to place an order.

[0058] (Second embodiment) In the second embodiment, when the customer monitoring device 10 determines whether the customer is performing an ordering action 1, it also checks whether the customer is performing a gripping action, such as holding a cup or other container in their hand. In the second embodiment, parts that are the same as those in the embodiments of Figures 1 to 9 are indicated by the same reference numerals.

[0059] Figure 10 is a flowchart showing an example of the behavior monitoring process according to the second embodiment. After confirming the order action 1, the control unit 100 checks whether a grasping action is being performed (ACT401). The control unit 100 checks whether a container such as a glass can be recognized near marker M3 via the image recognition unit 201. In other words, the image recognition unit 201 is used to check whether the customer is holding a glass or a mug. If the check confirms that it is a grasping action (YES in ACT401), flag 1 is enabled.

[0060] If the confirmation does not recognize it as a grasping action (NO in ACT402), the control unit 100 proceeds to the process of confirming order action 2.

[0061] Even in such cases, the monitoring device 10 can notify customers of potential orders without missing any opportunities, thereby reducing false alarms.

[0062] Furthermore, this embodiment is not limited to the above example, and various modifications may be made. The notification unit 30 has a display unit that displays the situation inside the store and may display the image S shown in Figure 9. In this case, the notification output unit 205, which is one of the functions implemented by the control unit 100 via a program, may display a message in addition to the table number as a notification instruction. A file related to this message is stored in the storage unit 101, and multiple patterns of messages may be stored.

[0063] Furthermore, if a customer who has generated skeletal data M leaves their seat, the control unit 100 may access the clock unit 102 and, if the customer does not return after a predetermined time has elapsed, send a message to the notification unit 30 stating that the customer has not returned to their seat as part of a notification instruction.

[0064] Furthermore, if the control unit 100 detects the characteristics of items other than plates and glasses on the table after a customer has finished using it, it may send a message to the notification unit informing them that an item has been left behind.

[0065] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0066] 10...Customer monitoring device 20…IMG Department 30…Notification section 100... Control Unit 101...Storage section 1011... Flag management file 201...Image Recognition Unit 202... Skeletal Estimation Unit 203... Table Monitoring Unit 204... Behavioral Monitoring Unit 205...Notification output section

Claims

1. A communication unit that receives captured video data, An image recognition unit extracts and recognizes the characteristics of specific items on a table and the user from the aforementioned video data. A skeleton estimation unit that estimates the user's skeleton and generates skeleton data, A control unit controls a first detection that detects whether a feature quantity indicating a specific item exists on the table recognized by the image recognition unit, and a second detection that detects a preset elbow bending angle from the skeletal data. A notification output unit outputs a notification instruction when detection is performed by the first detection or the second detection, and when the control unit fails to detect the skeletal data again after a preset time has elapsed. A customer monitoring device having the following features.

2. In the customer monitoring device, A receiving procedure for receiving captured video data, An image recognition procedure for extracting and recognizing the characteristics of specific items on a table and users from the aforementioned video data, A skeletal estimation procedure for estimating the user's skeleton and generating skeletal data, A control procedure that controls a first detection, which detects whether a feature representing a specific item exists on the table recognized by the image recognition procedure, and a second detection, which detects a pre-set elbow bending angle from the skeletal data. A program that implements a notification output procedure that outputs a notification instruction when detection is performed by the first detection or the second detection, and when the skeletal data cannot be detected again after a predetermined time has elapsed by the control procedure.

Images