User management system and user management program

Abstract

This user management system for managing the location of a user of a facility comprises: a camera for capturing an image of a preset place in the facility; a display device 20; and a control device. The control device is configured to identify the user in an image captured by the camera, identify the location of the user in the facility, generate a map image indicating the location of the user, and display the map image on the display device 20.

Description

User Management System and User Management Program

[0001] The present disclosure relates to a user management system and a user management program.

[0002] For example, in the user management system disclosed in Patent Document 1, when a customer visits a store, is called, and leaves the store, the customer is photographed, and customer information is generated based on the photographed image. Based on the generated information, the information of the customers in the call waiting state is displayed in a list on a monitor as a customer status display screen. On the customer status display screen, the customers staying in the store (waiting area) and the customers who have left the store and are outside the store are grouped and displayed in a list.

[0003] Japanese Patent Application Laid-Open No. 2015-232791

[0004] In the system of Patent Document 1, it is possible to grasp whether a customer is staying in the store or has left the store, but it is not possible to grasp the location of the customer in the store. In this regard, there is room for improvement in the technology of Patent Document 1.

[0005]

[0005] An object of the present disclosure is to provide a user management system and a user management program that can easily grasp the location of users of a facility.

[0006] The user management system of the present disclosure is a user management system that manages the location of users of a facility, and includes a camera that images a location preset in the facility, a display device, and a control device. The control device identifies the user shown in the captured image captured by the camera, identifies the location of the user in the facility, generates a map image indicating the location of the user, and is configured to display the map image on the display device.

[0007] In the user management system of the present disclosure, it is preferable that the control device is configured to identify the attribute of the user shown in the captured image and display the attribute on the display device.

[0008] In the user management system of the present disclosure, it is preferable that the control device is configured to display the captured image on the display device.

[0009] In the user management system of this disclosure, it is preferable that the control device is configured to determine the time of the user's visit to the facility based on the time at which the user is identified as being in the captured image, and to display the time of the visit on the display device.

[0010] Preferably, the user management system of this disclosure includes a reception device installed in the facility and operated by the visiting user, and a reception camera which is the camera that captures the user operating the reception device, and the control device is configured to identify the user who is captured in the image captured by the reception camera, identify the time of reception of the user by the reception device, and display the reception time on the display device.

[0011] In the user management system of this disclosure, it is preferable that an exit camera is provided at the exit of the facility and captures images of exiters leaving the facility from the exit, and that the control device is configured to identify the exiters captured in the images captured by the exit camera and to display the exiters on the display device.

[0012] In the user management system of this disclosure, it is preferable that the control device is configured to determine the time of departure of the exiting person from the facility based on the time of departure of the exiting person shown in the image captured by the exit camera, and to display at least one of the departure time and the elapsed time since the departure time on the display device.

[0013] In the user management system of this disclosure, it is preferable that the control device is configured to display on the display device the location of the user identified based on the most recent captured image, or the last location of the user identified based on past captured images.

[0014] In the user management system of this disclosure, it is preferable that the control device is configured to display information on the display device indicating that it failed to identify the user based on the most recent captured image when displaying the last known location of the user on the display device.

[0015] Preferably, the user management system of this disclosure comprises a signage device that provides information to the user, and a signage camera that takes an image of the user standing in front of the signage device, and the control device is configured to identify the user who is in the image captured by the signage camera, to identify the user's location within the facility, and to reflect the user's location in the map image.

[0016] The user management system of this disclosure can also be applied as a user management program. The user management program of this disclosure is a user management program used in a system for managing the location of facility users, and causes a computer to execute the following: a user identification function that identifies the user in an image captured by a camera that captures a predetermined location in the facility; a location identification function that identifies the location of the user in the facility; a map generation function that generates a map image showing the location of the user; and a display function that displays the map image on a display device.

[0017] This disclosure enables the realization of a user management system and user management program that allows for easy tracking of the location of facility users.

[0018] This is a floor plan of the facility. This is a block diagram showing the configuration of the user management system. This is a flowchart showing the processing of the user management system. This is another flowchart showing the processing of the user management system. This is a diagram of a display device that shows a map image. This is a diagram of a database showing the visit history of users. This is a flowchart showing the function of supporting staff in recording customer service information.

[0019] [Facility Overview] The facility illustrated in Figure 1 is, for example, a bank branch. This facility has an entrance / exit A, an automated reception area B, an automated transaction booth C, a waiting lobby D, a customer service counter E, a consultation booth F, a business meeting room G, a back room H, and a restroom I. Visitors (corresponding to 'users') can walk from entrance / exit A to either the automated reception area B or the automated transaction booth C.

[0020] A visitor camera 11 (corresponding to "camera") is placed in the automated reception area B, and the camera's field of view includes entrance A. Therefore, visitors entering the facility are captured by the visitor camera 11.

[0021] An exit camera 12 (corresponding to "camera" and "exit camera") is positioned at entrance / exit A, and the imaging range of the exit camera 12 includes the boundary between entrance / exit A and automated reception space B. People leaving the facility are imaged by the exit camera 12.

[0022] In automated reception space B, a customer registration device 1 and an electronic registration desk 2 are located. Both customer registration device 1 and electronic registration desk 2 are considered "reception devices." Both customer registration device 1 and electronic registration desk 2 are operated by the visitor.

[0023] The customer reception device 1 has a touch panel display (not shown) and accepts input from visitors regarding their purpose of visit and other matters. The customer reception device 1 is equipped with a facial recognition camera 1A. The facial recognition camera 1A captures images of visitors standing in front of the customer reception device 1. In other words, the facial recognition camera 1A captures images of visitors operating the customer reception device 1.

[0024] The electronic registration desk 2 has a touch panel display (not shown) and a stylus (not shown), and accepts entries from visitors for transactions such as cash deposits, cash withdrawals, bank transfers, tax payments, and utility bill payments. The electronic registration desk 2 is equipped with a facial recognition camera 2A. The facial recognition camera 2A captures images of visitors standing in front of the electronic registration desk 2. In other words, the facial recognition camera 2A captures images of visitors operating the electronic registration desk 2. At least one of the facial recognition cameras 1A and 2A corresponds to the 'camera' and the 'reception camera'.

[0025] Multiple automated teller machines (ATMs) 3 are located in automated transaction booth C. The ATMs 3 are used by individual customers to deposit or withdraw cash from their deposit accounts. The ATMs 3 are connected to the accounting server via an internal network (not shown). When a cash deposit or withdrawal is made using an ATM 3, the account update process is also performed automatically. In addition to the ATMs 3, automated transaction booth C may also contain other devices such as currency exchange machines and tax / public fund collection machines.

[0026] An in-store camera 13 (corresponding to "camera") is installed in the automated trading booth C. Visitors operating the automated currency deposit machine 3 are photographed by the in-store camera 13. The in-store camera 13 photographs the visitor's back, but it is also possible for the in-store camera 13 to be positioned above or adjacent to the left or right side of the automated currency deposit machine 3, so that the in-store camera 13 photographs the visitor's front.

[0027] The automated teller machine 3 has a built-in facial recognition camera 3A. The facial recognition camera 3A captures images of visitors standing in front of the automated teller machine 3. Note that the facial recognition camera 3A may also be included in the category of 'camera'.

[0028] Waiting lobby D is furnished with several chairs, a digital signage 4 (corresponding to the "signage device"), and a call device 5. Visitors wait in the chairs until they are called by a teller. The digital signage 4 has a display for advertising (e.g., an LCD monitor, an organic LED monitor, etc.).

[0029] The digital signage 4 is equipped with a facial recognition camera 4A (corresponding to "camera" and "signage camera"). The facial recognition camera 4A captures images of visitors standing in front of the digital signage 4. Based on the images captured by the facial recognition camera 4A, the characteristics of the visitors standing in front of the digital signage 4 are analyzed, and advertisements that match those characteristics are displayed on the digital signage 4's screen.

[0030] The call device 5 includes a display that shows the call number and a speaker for voice guidance. The display may be an LCD monitor, an organic LED monitor, a 7-segment monitor, etc. When the call device 5 is operated by a teller, the call number is displayed on the display of the call device 5, and a message prompting the customer corresponding to that call number to move to counter E is emitted from the speaker of the call device 5.

[0031] Cameras 14 and 15 (both equivalent to "cameras") are installed in the automated trading booth C. Visitors waiting or walking around in the waiting lobby D are captured by cameras 14 and 15.

[0032] A teller is stationed at window E to handle accounting transactions. When a visitor is called by the teller, they move to window E to perform various procedures (cash deposits, cash withdrawals, bank transfers, tax payments, utility bill payments, etc.). Although not shown in Figure 1, the teller carries a display device 20 (see Figure 2).

[0033] Consultation booth F is a space for providing consultations, for example, regarding customer loans. For this reason, consultation booth F is equipped with a door and is isolated from the waiting lobby D.

[0034] Meeting room G is a space for conducting business with a larger number of people than consultation booth F. For this reason, meeting room G has a door and is isolated from waiting lobby D.

[0035] Backyard H is staffed by tellers and other personnel. Backyard H is a restricted area where visitors are prohibited from entering. There are no cameras or other devices installed in toilet I. Although not shown in Figure 1, a control device 30 (see Figure 2) is located in backyard H.

[0036] [Configuration of the User Management System] The user management system in this disclosure will be described based on Figure 2. The user management system in this disclosure includes a facial recognition camera 1A of the customer reception device 1, a facial recognition camera 2A of the electronic register desk 2, a facial recognition camera 3A of the automated teller machine 3, a facial recognition camera 4A of the digital signage 4, a call device 5, an arrival camera 11, an departure camera 12, in-store cameras 13, 14, 15, a display device 20, and a control device 30. The cameras in this disclosure described above (facial recognition cameras 1A, 2A, 3A, 4A, arrival camera 11, departure camera 12, in-store cameras 13, 14, 15, etc.) are, for example, color cameras that capture visible light and have a CMOS (complementary metal-oxide-semiconductor) sensor.

[0037] The display device 20 is a tablet computer with a so-called touch panel display, carried by tellers at counter E or facility staff (e.g., lobby concierges). The display device 20 has a camera 21, a selection reception unit 22, and a display unit 23. The camera 21 captures images of, for example, the faces of tellers or visitors. The selection reception unit 22 and the display unit 23 are, for example, selection buttons displayed on the touch panel display. The selection reception unit 22 receives operations from the teller, etc. Operations from the teller include, for example, operations to call a visitor to counter E. Various information related to the teller's duties and the visitor's purpose of visit is displayed on the display unit 23.

[0038] The control device 30 includes a visitor identification unit 31, a location identification unit 32, a map generation unit 33, an attribute identification unit 34, and a storage unit 35. The control device 30 may also be composed of multiple devices or processors.

[0039] The visitor identification unit 31 identifies visitors based on their appearance information captured in images taken by facial recognition cameras 1A, 2A, 3A, 4A, arrival camera 11, exit camera 12, in-store cameras 13, 14, 15, etc. The appearance information includes at least one of the following: facial information, information about clothing, information about bone structure, and information about whether or not the visitor is accompanied by children. This appearance information is detected using a neural network (so-called artificial intelligence) identified by mechanical deep learning.

[0040] The location identification unit 32 identifies the location of visitors within the facility based on the positions of the in-store cameras 13, 14, 15, etc., and the field of view of the captured images (it identifies whether or not the visitor is present within the facility, and if so, its exact location. However, if the location cannot be confirmed, or if the unit does not have the function to identify the location, it may only identify that the visitor is present). "Identifying the visitor's location" means identifying where the visitor is within the facility. For example, it may identify which area of ​​the facility the visitor is in (e.g., by room, by individual sofas in the waiting lobby D), or it may identify the visitor's exact coordinates within the facility. If it fails to determine which area or coordinates the visitor is in, it may only identify that the visitor is present within the facility. The location identification unit 32 identifies visitors by analyzing the captured images and performing image recognition. Location identification may be based on which camera captures the visitor, or on where the visitor is located in the captured image. In this case, the location identification unit 32 may track the visitor identified by the visitor identification unit 31 while capturing images with in-store cameras 13, 14, 15, etc., using, for example, optical flow technology.

[0041] The map generation unit 33 generates a map image showing the location of visitors within the facility. If a visitor leaves the facility midway, that visitor will be displayed outside the facility (in the list of visitors who left midway) in the map image. Figure 5 shows an example of a map image generated by the map generation unit 33. The map image may be a plan view or a perspective view, as in the example in Figure 5. The map image may be in the form of a simplified illustration or in the form of actual images taken inside the store. The map image may also be generated based on images captured by a camera inside the store. The map image is displayed on the display unit 23 of the display device 20. That is, the control device 30 is configured to display the map image on the display device 20.

[0042] The attribute specifying unit 34 specifies the attribute information of a visitor based on the appearance information of the visitor shown in the captured image by each camera installed in the facility, the input information of at least one of the store reception device 1 and the electronic cash register 2, and the like. The attribute information of the visitor includes, for example, the age group and gender determined based on the appearance information, and further includes the purpose of visit and the like. The age group and gender are determined using the above-mentioned artificial intelligence. The purpose of visit is obtained from the information input to at least one of the store reception device 1 and the electronic cash register 2. And the control device 30 is configured to display the attribute on the display device 20, for example, as shown in FIG. 5. In the example of FIG. 5, it is pop-up displayed on the display device 20 that the attribute of the visitor with the visitor ID "106" is "a woman in her 30s".

[0043] The storage unit 35 is a storage medium including at least one of a volatile memory, a non-volatile memory, a hard disk drive, and the like. In the storage unit 35, as information about visitors in the facility, visitor status information, time information such as the visitor's arrival time, visitor attribute information, a list of中途退出者 (dropouts), the visitor's call number, and the like are stored.

[0044] Note that the visitor status information is information indicating whether the procedure of the visitor has been completed or not. In the present embodiment, the status information includes information of "unknown person", "uncompleted", "ATM completed", and "counter completed".

[0045] 〔Regarding the processing of the user management system〕Based on the flowcharts of FIGS. 3 and 4, the processing of the user management system will be described. Note that the flowcharts of FIGS. 3 and 4 are flowcharts assuming the case of tracking one visitor, the start of the processing is step #01, and the end of the processing is step #13. When a plurality of visitors exist simultaneously, the flowcharts of FIGS. 3 and 4 are processed in parallel for the plurality of visitors at the same time.

[0046] When the visitor camera 11 captures an image of a visitor entering the facility, the visitor identification unit 31 of the control device 30 identifies the visitor's appearance information (step #01). The visitor identification unit 31 determines whether the visitor's appearance information matches the appearance information of a person who leaves the facility midway, as described later (step #02). Step #03, which is processed when the determination in step #02 is Yes, will be described later. If the visitor's appearance information does not match the appearance information of a person who leaves the facility midway, as described later (step #02: No), the control device 30 records the visitor's arrival time in the storage unit 35 and registers a new visitor ID along with the appearance information (step #04). At this time, the initial value of the visitor ID status information is registered as "unknown person". The control device 30 also determines the visitor's arrival time at the facility based on the time when the visitor was identified as being in the captured image. The identified arrival time is displayed on the display device 20. In the example shown in Figure 5, a pop-up notification on the map image indicates that the visitor with visitor ID "109" arrived at "13:50".

[0047] Once the registration of a new visitor ID is complete, the facial recognition camera 3A of the ATM 3 determines whether or not facial information (part of the appearance information) associated with the visitor ID has been detected (step #05). If facial information is detected by the facial recognition camera 3A (step #05: Yes), it is determined whether or not a transaction has been executed at the ATM 3 based on the visitor's actions (step #06). The determination of whether or not a transaction has been executed may be made by the visitor identification unit 31 of the control device 30 performing a determination process based on the image from the facial recognition camera 3A, or it may be made by the transmission of information regarding the transaction from the ATM 3 to the control device 30.

[0048] When transaction processing based on the operation of the visitor is executed in the currency automated teller machine 3 (step #06: Yes), the status information of the visitor ID corresponding to the visitor is updated to "ATM completed" (step #07). At this time, the attribute information associated with the visitor ID is updated, and "ATM" is added to the data in the visit purpose column of the attribute information. Thereby, for example, a facility staff member or the like can grasp that the purpose of the visit of the visitor is a transaction using the currency automated teller machine 3 (ATM). If the determination in step #05 and the determination in step #06 are both No, the processing of the user management system proceeds to step #08.

[0049] If face information is not detected in the face recognition camera 3A (step #05: No), the visitor is likely to go through procedures at the service counter E. Also, even if face information is detected in the face recognition camera 3A (step #05: Yes) and transaction processing based on the operation of the visitor is executed in the currency automated teller machine 3 (step #06: Yes), the visitor may go through other procedures at the service counter E. For this reason, it is determined whether or not face information (a part of the appearance information) associated with the visitor ID is detected in at least one of the face recognition camera 1A of the store reception device 1 and the face recognition camera 2A of the electronic ledger stand 2 (step #08).

[0050] When face information associated with the visitor ID is detected in at least one of the face recognition cameras 1A and 2A (step #08: Yes), it is determined whether or not a new call number has been issued (step #09). When a new call number has been issued (step #09: Yes), the call number is associated with the visitor ID corresponding to the face information (step #10). At this time, the status information of the visitor ID of the visitor is updated to "incomplete". If the status information of the visitor ID corresponding to the visitor is "ATM completed", the status information is rewritten from "ATM completed" to "incomplete".

[0051] In step #10, the attribute information associated with the visitor ID is updated, and the purpose of the visit, for example, received by the visitor reception device 1, is added to the data in the purpose of visit field of the attribute information. The purpose of the visit received by the visitor reception device 1 may include, for example, "deposit," "withdrawal," "inheritance," "bank transfer," etc., as shown in Figure 6.

[0052] Furthermore, in step #10, the control device 30 identifies the visitor captured in the image taken by at least one of the facial recognition cameras 1A and 2A, and identifies the visitor's registration time by at least one of the customer registration device 1 and the electronic registration desk 2. The identified registration time is displayed on the display device 20. In the example in Figure 5, the map image shows a pop-up indicating that the registration time for visitor ID "104" is "13:30".

[0053] Furthermore, it is conceivable that after a visitor operates at least one of the customer registration device 1 and the electronic register desk 2, and the status information of the visitor ID for that visitor is updated to "incomplete," the visitor may then operate the automated teller machine 3 to conduct a separate transaction while waiting. In this case, the status information of the visitor ID for that visitor will remain "incomplete."

[0054] In the example shown in Figure 5, visitors with visitor IDs "107", "108", and "109" are lined up at the customer reception device 1. Each time a new call number is assigned to these visitors, that call number is associated with each visitor ID. At this time, the control device 30 may be configured to display on the display device 20 an image showing the appearance of the visitors lined up at the customer reception device 1 (for example, an image captured by an in-store camera 13, a facial recognition camera 1A, or an exit camera 12). The image showing the appearance of the visitors lined up at the customer reception device 1 may be displayed on the display unit 23 of the display device 20 as a thumbnail, or as a pop-up display in response to touch operation.

[0055] If no new call number is issued (Step #09: No), the user management system proceeds to Step #11, which is described below. Situations in which a new call number is not issued include, for example, if the visitor stops operating the visitor registration device 1 or the electronic registration desk 2, or if the visitor misoperates the visitor registration device 1 or the electronic registration desk 2.

[0056] If no facial information associated with the visitor ID is detected in at least one of the facial recognition cameras 1A and 2A (Step #08: No), the user management system proceeds to Step #11, which is described below. In Step #11, it is determined whether or not appearance information is detected in the exit camera 12. The control device 30 then identifies the exiter who is captured in the image taken by the exit camera 12.

[0057] When external information is detected based on the image captured by the exit camera 12 (Step #11: Yes), it is determined whether the status information of the visitor ID corresponding to the external information is "counter completed" or "ATM completed" (Step #12). If the status information of the visitor ID corresponding to the external information is "counter completed" or "ATM completed" (Step #12: Yes), the control device 30 records the exit time of the visitor ID, assuming that the visitor has completed their business (Step #13). At this time, in consideration of the visitor's privacy, the control device 30 erases the external information of the visitor ID. Visit history other than the external information of the visitor ID is stored as history information in the storage unit 35.

[0058] If the status information of the visitor ID corresponding to the external information is neither "counter completed" nor "ATM completed" (Step #12: No), the control device 30 considers the visitor to have left before completing the procedure, adds the visitor ID to the list of visitors who left prematurely, and records the time of departure (Step #14). As a result, the visitor captured by the exit camera 12 is added to the list of visitors who left prematurely, and the visitor ID is displayed in the list of visitors who left prematurely on the display unit 23 of the display device 20. At this time, the control device 30 determines the time of departure of the visitor from the facility based on the identified time of the visitor captured in the image.

[0059] For example, in the example shown in Figure 5, a message is displayed indicating that visitor ID "102" has left the facility at 13:35, and this message is displayed outside the facility (on the list of visitors who have left prematurely). In the example shown in Figure 5, the time of departure from the facility is displayed, but the elapsed time since departure may also be displayed. That is, the control device 30 is configured to display visitors who have left the facility on the display device 20. In this case, the control device 30 may be configured to display at least one of the departure time and the elapsed time since departure on the display device 20.

[0060] Once the processing in step #14 is complete, the process returns to step #01. When a visitor who has left the facility returns, the visitor's appearance information is identified based on the image captured by the visitor camera 11 (step #01). If the appearance information identified based on the image captured by the visitor camera 11 matches the appearance information of the person who left the facility (step #02: Yes), the control device 30 removes the visitor ID corresponding to that appearance information from the list of people who have left the facility and updates the location and time of the visitor ID (step #03). As a result, in the map image, the visitor who has left the facility is removed from the list of people who have left the facility and is displayed within the facility.

[0061] If no external information is detected by the exit camera 12 (step #11: No), the user management system proceeds to step #15 shown in Figure 4. Figure 4 shows the processes from step #15 to step #26. After completing the processes shown in Figure 4, the user management system returns to step #08 shown in Figure 3.

[0062] In step #15, it is determined whether or not the appearance information of the visitor ID has been detected based on the images captured by the visitor camera 11, the in-store cameras 13, 14, and 15, and the facial recognition cameras 1A, 2A, 3A, and 4A. If the appearance information of the visitor ID is detected (step #15: Yes), the location identification unit 32 of the control device 30 updates the location and time of the visitor ID corresponding to the appearance information (step #16). As a result, the location of the visitor on the map display within the facility is updated on the map image displayed on the display unit 23 of the display device 20 (see Figure 5). In addition, the visitor's activity time information in the "activity area (lobby)", "activity area (ATM)", "activity area (digital signage)", etc. is updated in the history information shown in Figure 6.

[0063] If the appearance information of the visitor ID is not detected based on the captured image (Step #15: No), it is possible that the visitor has gone to the restroom I, or that the visitor's figure is hidden behind a newspaper that the visitor has spread out in the waiting lobby D, or that the visitor's figure is not captured by the in-store cameras 13, 14, 15, etc., because the visitor is sitting hunched over in a chair in the waiting lobby D and looking downwards at the screen of a mobile device. In such cases, the control device 30 is configured to display on the display device 20 that the visitor's whereabouts are unknown and the visitor's last known location on the map display (Step #17).

[0064] For example, in the example shown in Figure 5, a message is displayed indicating that visitor ID "103" has been missing since 13:40, and the last known location is shown on the map image. Toilet I is located near the last known location of visitor ID "103". Therefore, the facility staff can infer that this visitor went to toilet I.

[0065] In other words, the control device 30 is configured to display the last known location of a visitor identified based on past captured images on the display device 20 when it fails to identify the visitor based on the most recent captured image. Furthermore, when the control device 30 displays the last known location of a visitor identified based on past captured images on the display device 20, it is configured to display information on the display device 20 indicating that it failed to identify the user based on the most recent captured image.

[0066] The "latest captured image" refers to the most recent captured image taken at a time and interval predetermined by each camera. For example, the latest captured image may be the image taken when the visitor identification unit 31 outputs a visitor identification request signal in response to a visitor ID call operation by the teller in the calling device 5, and the visitor identification unit 31 performs visitor identification processing based on the identification request signal.

[0067] If appearance information is detected in step #15 (step #15: Yes), after processing in step #16, the control device 30 determines whether or not the appearance information was detected based on the image captured by the facial recognition camera 4A of the digital signage 4 (step #18). If the appearance information was detected based on the image captured by the facial recognition camera 4A (step #18: Yes), the control device 30 displays an advertisement on the digital signage 4 that corresponds to the attribute information of the visitor ID that is associated with the appearance information (step #19). If the appearance information was not detected based on the image captured by the facial recognition camera 4A (step #18: No), the user management system proceeds to step #20.

[0068] For example, in the example shown in Figure 5, a visitor with visitor ID "101" is standing in front of the digital signage 4. Therefore, based on the attribute information of visitor ID "101", advertisements that are likely to match the visitor's hobbies, preferences, and interests are displayed on the digital signage 4's screen. At this time, the control device 30 identifies the visitor in the image captured by the facial recognition camera 4A (corresponding to "camera" and "signage camera"), identifies the visitor's location within the facility, and reflects the visitor's location in the map image. The control device 30 may also display the captured image showing the visitor standing in front of the digital signage 4 on the display device 20.

[0069] As mentioned above, the visitor ID attribute information includes information about the visitor's age group, gender, purpose of visit, etc. For example, if the visitor is elderly, advertisements for reverse mortgages or pension services may be displayed on the digital signage 4 screen. For example, if the visitor is young, advertisements for investment savings or matchmaking services may be displayed on the digital signage 4 screen. Furthermore, if the visitor is a young male, advertisements for car loans may be displayed on the screen, and if the visitor is a young female, advertisements for overseas travel insurance may be displayed on the screen. Furthermore, if the visitor is accompanied by children, advertisements for personal liability insurance, home loans, or education loans may be displayed on the digital signage 4 screen. By identifying multiple visitors who are paying attention to the digital signage 4, it is possible to sequentially provide advertisements tailored to each attribute and to collect a wide range of statistical results on the effectiveness of the advertisements.

[0070] When a visitor is called to teller at counter E (Step #20: Yes), and the visitor appears at counter E (Step #21: Yes), the procedure at counter E is carried out (Step #22). Once the procedure at counter E is completed (Step #23: Yes), the status information of the visitor ID corresponding to that visitor is updated to "Counter Completed" (Step #24). If the procedure at counter E is not completed (Step #23: No), the visitor will return to a waiting state such as waiting again in waiting lobby D. Therefore, if the determination in Step #23 is No, the user management system returns to Step #08 shown in Figure 3.

[0071] If a visitor is not called to the teller at window E (Step #20: No), or if a visitor is called to the teller at window E but does not appear at window E (Step #21: No), it is possible that the visitor has been kept waiting for a long time. For this reason, the control device 30 determines whether the elapsed time since the time of arrival recorded in the visitor ID has exceeded a predetermined time (for example, 30 minutes) (Step #25).

[0072] If a visitor is waiting for longer than the predetermined time (Step #25: Yes), the control device 30 is configured to display an alert with the visitor's ID on the display device 20 (Step #26). The user management system then returns to Step #08 shown in Figure 3. If the determination in Step #25 is No, the user management system returns to Step #08 shown in Figure 3.

[0073] For example, in the example shown in Figure 5, visitor ID "105" has been sitting in a chair in waiting lobby D for more than the pre-set time. This visitor may be waiting to be called by the teller at window E without entering anything at the visitor registration device 1 or the electronic registration desk 2, or they may be waiting, mistakenly believing they will be called by the teller again, even though they have completed the procedures at window E and are ready to leave. To address such situations, step #26 processes the display device 20 to show that visitor ID "105" has switched to an alert state at 13:45. This allows facility staff to approach visitor ID "105" and provide appropriate guidance.

[0074] Furthermore, if a visitor is called by the teller at window E but does not appear at window E (Step #21: No), it is possible that the visitor is concentrating on the screen of the digital signage 4 or the screen of their mobile device and has not noticed the teller at window E calling them. In such cases, the teller can also appropriately approach the visitor while confirming the location of the called visitor on the display unit 23 of the display device 20.

[0075] [Visitor History Information] As shown in Figure 6, visitor history information of visitors to the facility is recorded in the database as a list of visitor IDs. This list can be displayed on the display unit 23 of the display device 20. In the list shown in Figure 6, multiple visitor IDs are recorded. Each visitor ID is associated with the following: call number, time of arrival (I), time of use of the customer reception device 1 (II), time of entry on the electronic register 2 (III), time (IV) to (VI) when the visitor's appearance information was detected based on images captured by in-store cameras 13, 14, 15, etc., time when an alert was displayed based on the processing of step #26 shown in Figure 4, time (VII) when the visitor was called by the teller at counter E, time of departure (VIII), time of early departure, time of return visit, time of re-call, attribute information including age, gender, and purpose of visit, last updated status information, and record of the staff member who handled the visit.

[0076] The time spent in the store is calculated and recorded as the difference between the time of arrival (I) and the time of departure (VIII). The waiting time is calculated and recorded as the difference between the time of use of the customer reception device 1 (II) and the time of calling (VII). The actual processing time is calculated as the difference between the time of use of the customer reception device 1 (II) and the time of departure (VIII). Note that the time of early departure (the difference between the time of early departure and the time of return visit) may be subtracted from both the time spent in the store and the waiting time.

[0077] By generating visitor history information for each visitor ID, as shown in Figure 6, the records can be used to verify visitor waiting times, the quality of staff service, and problems in guiding visitors within the facility. This makes it easier for facility managers, for example, to identify areas for improvement in order to enhance customer satisfaction, and to plan for restructuring visitor flow and reallocating personnel.

[0078] [Support for Staff Record-Keeping of Customer Service Activities Using a Display Device] As shown in Figure 7, facility staff may perform customer service activities such as providing consultations and information about products and services for visitors. The display device 20 is carried by facility staff and has a function to support the recording of customer service activities by staff.

[0079] When recording customer service using the display device 20, it is conceivable that the staff member would operate the display device 20 to start recording before the customer service begins and stop recording after the customer service ends. However, staff members are often focused on a variety of customer service tasks and may forget to operate the recording device. Therefore, the challenge lies in how to reliably record the staff member's customer service tasks.

[0080] The customer service support function of the display device 20 in this disclosure is configured to allow the recording of the staff member's customer service while eliminating the need for the staff member to perform complicated operations on the display device 20 during customer service. To this end, facial recognition of the staff member using the camera 21 of the display device 20 is used.

[0081] As explained using the flowchart in Figure 7, first, the staff member takes an image of their face with the camera 21 of the display device 20 they are carrying, and registers the staff member's face information in the display device 20 (step #31). If the staff member's face information has already been registered in the display device 20, the process in step #31 may be omitted.

[0082] When an employee provides customer service to a visitor, the employee ensures that the person being served is visible in the camera 21 of the display device 20 (step #32). At this time, the control module of the display device 20 determines whether or not the person being served (visitor) is visible in the camera 21 (step #33).

[0083] At this time, the display device 20 can be configured so that the image captured by the camera 21 is not displayed on the display unit 23. This makes it less likely for visitors to feel uncomfortable when they see their own face being captured on the screen of the display unit 23 of the display device 20. For example, when an attendant is guiding a visitor to a product or service, if the display unit 23 is shown to the visitor with the product or service inquiry screen displayed on it, the determination in step #33 will easily be Yes.

[0084] Instead of making a determination in step #33, the system may be configured to proceed to step #34 if, for example, it is determined that the distance between the visitor and the staff member is less than or equal to a predetermined distance based on the visual information of the images captured by the in-store cameras 13, 14, and 15.

[0085] If the determination in step #33 is "Yes", the time when the staff member started serving the customer is recorded (step #34). The control module of the display device 20 then determines whether or not the end of the customer service has been detected (step #35). The determination of whether or not the customer service has ended may be configured to determine, for example, whether or not a predetermined amount of time has elapsed continuously since the person being served (visitor) was no longer visible in the camera 21. Alternatively, the determination of whether or not the customer service has ended may be configured to determine, for example, whether or not the distance between the visitor and the staff member has moved beyond a predetermined distance based on the appearance information of the images captured by the in-store cameras 13, 14, and 15. Or, the determination of whether or not the customer service has ended may be configured to be made based on, for example, the position detection function of the display device 20 (e.g., positioning technology using a global navigation satellite system, indoor positioning technology using short-range wireless communication, etc.).

[0086] If the result in step #35 is "Yes," the time when the staff member finished serving customers is recorded (step #36). Furthermore, staff members are imaged by, for example, in-store cameras 13, 14, and 15, and their location within the facility is identified by the location identification unit 32. Additionally, the staff member's location information within the facility, customer service time, etc., are automatically compiled. This prevents input errors and fraudulent data entry by staff members, ensuring that the staff member's sales activities with customers are captured as accurately as possible. This compiled record is used for analyzing staff members' lobby sales and promotional activities, as well as for staff member evaluation.

[0087] [Effects of the Embodiment] The user management system of this disclosure includes cameras that capture images of pre-set locations in the facility (at least one of the following: arrival camera 11, exit camera 12, in-store cameras 13, 14, 15, and face recognition cameras 1A, 2A, 3A, 4A), a display device 20, and a control device 30. The control device 30 is configured to identify users (visitors) captured in the images taken by the cameras, to identify the location of the users in the facility, to generate a map image showing the location of the users, and to display the map image on the display device 20.

[0088] In this configuration, the location of users is displayed on a map image. Therefore, even if users are concentrated in areas such as the lobby, or if the lobby is larger than other areas, facility staff can easily locate where users are on the display device 20. This realizes a user management system that makes it easy to grasp the location of facility users.

[0089] In this disclosure, the control device 30 is configured to identify the user's attributes captured in the image and to display the attributes on the display device 20.

[0090] With the above configuration, facility staff can easily check the attributes of users on the display device 20. As a result, facility staff can more easily find users by using attributes shown on the map image (e.g., age and gender). Furthermore, when user attributes are displayed in a list, facility managers can easily extract areas for improvement to enhance customer satisfaction based on user attributes, making it easier to develop improvement plans.

[0091] In this disclosure, the control device 30 is configured to display the captured image on the display device 20.

[0092] With the above configuration, facility staff can easily check the condition of users by viewing the captured images displayed on the display device 20.

[0093] In this disclosure, the control device 30 is configured to determine the time of the user's arrival at the facility based on the time at which the user is identified as being captured in the image, and to display the arrival time on the display device 20.

[0094] With the above configuration, facility staff can easily check the time of a user's arrival on the display device 20. As a result, facility staff can, for example, understand how long a user is waiting.

[0095] This disclosure includes a reception device (at least one of a customer reception device 1 and an electronic registration desk 2) installed in the facility and operated by visitors, and a reception camera (at least one of face recognition cameras 1A and 2A) which is used to capture images of the user operating the reception device. The control device 30 is configured to identify the user captured in the image taken by the reception camera, identify the time of the user's reception by the reception device, and display the reception time on the display device 20.

[0096] With the above configuration, users operating the reception device are captured by the reception camera, and facility staff can easily check on the display device 20 whether or not a visitor has completed the procedure at the reception device. As a result, facility staff can take appropriate action, such as speaking to visitors who have not completed the procedure at the reception device. In addition, accurate statistical information such as the time spent in the store and waiting time can be collected based on attribute information such as the purpose of the visitor.

[0097] This disclosure includes an exit camera (exit camera 12) installed at the exit (entrance / exit A) of the facility, which captures images of people exiting the facility from the exit. The control device 30 is configured to identify the people exiting who are captured in the images taken by the exit camera and to display the people on the display device 20.

[0098] With the above configuration, facility staff can easily check the display device 20 to identify users (exiters) who have left the facility midway through their stay.

[0099] In this disclosure, the control device 30 is configured to determine the time of departure of an exiter from the facility based on the time of departure of the exiter captured in the image taken by the exit camera (exit camera 12), and to display at least one of the departure time and the elapsed time since departure on the display device 20.

[0100] With the above configuration, facility staff can easily check on the display device 20 how much time has passed since a user (departureee) left the facility.

[0101] In this disclosure, the control device 30 is configured to display on the display device 20 the location of a user identified based on the most recent captured image, or the last known location of a user identified based on past captured images.

[0102] With the above configuration, facility staff can ascertain the location of the last identified user and infer their subsequent whereabouts.

[0103] In this disclosure, the control device 30 is configured to display information on the display device 20 indicating that it failed to identify the user based on the most recent captured image when displaying the last known location of a user identified based on past captured images on the display device 20.

[0104] With the above configuration, facility staff can quickly notice when a user goes missing, and it becomes easier to track down the missing user.

[0105] The system includes a signage device (digital signage 4) that provides information to users, and a signage camera (face recognition camera 4A) that captures images of users standing in front of the signage device. The control device 30 is configured to identify users in the images captured by the signage camera, determine the location of users within the facility, and reflect the location of users in the map image.

[0106] With the above configuration, the user's location can be detected by the image captured by the signage camera, and facility staff can easily confirm the user's location on the display device 20.

[0107] [Other Embodiments] This disclosure is not limited to the configurations exemplified in the embodiments described above, and representative other embodiments of this disclosure are described below.

[0108] (1) This disclosure may also be a user management program used in a system for managing the location of facility users (visitors). The user management program causes a computer to execute the following functions: a user identification function that identifies users in images captured by cameras (at least one of the following: visitor camera 11, exit camera 12, in-store cameras 13, 14, 15, and face recognition cameras 1A, 2A, 3A, 4A) that capture images of predetermined locations in the facility; a location identification function that identifies the location of users in the facility; a map generation function that generates a map image showing the location of users; and a display function that displays the map image on a display device 20. At least one of steps #01, #05, #08, #15, and #18 shown in Figures 3 to 4 corresponds to the "user identification function". At least one of steps #01 and #16 shown in Figures 3 to 4 corresponds to the "location identification function". The visitor identification unit 31 shown in Figure 2 implements the "user identification function". The location identification unit 32 shown in Figure 2 has a "location identification function" implemented in it. The map generation unit 33 shown in Figure 2 has a "map generation function" implemented in it. The control device 30 shown in Figure 2 has a "display function" implemented in it. The user management program may consist of, for example, an executable program generated by a compiler and a linker, or it may consist of code executed sequentially by an interpreter.

[0109] (2) In the above-described embodiment, a bank branch was used as an example of a facility. The user management system of this disclosure is not limited to this embodiment and can be used in a variety of facilities such as financial institutions other than banks (trust banks, securities companies, etc.), government offices, hospitals, welfare facilities, post offices, restaurants, retail stores, travel agencies, waiting lobbies of train and bus stations, airport waiting lobbies, event venues, entertainment facilities, sports facilities, etc. For example, a travel agency may have users who come to obtain catalogs, but with the user management system of this disclosure, it is possible to manage the number of users who do not interact with the counter, and it is possible to analyze demand based on attribute information such as gender, age, and group. For example, if the facility is a hospital, there are multiple consultation counters, and if a user is far away from the call device 5, they may not notice the call from the consultation counter. Nurses, assistants, etc. can check the floor for users who do not notice the call. For example, if the facility is a food court in a shopping mall, there are multiple tenant stores, and it is possible that the person who was called may not come. Staff can confirm that the user is waiting in line at another tenant's location, and can respond by calling the user again or changing the method of notification.

[0110] (3) The facility map image may be a multi-layered map image. There may be cases where multiple applications, procedures, and business departments are located on different floors. Even if a user cannot be found in such a case, staff members can be aware that the user is on another floor for other applications, etc. This allows staff members to take flexible measures such as waiting or changing the queue, improving user convenience.

[0111] (4) In the above-described embodiment, an entrance / exit A is provided in the facility. The facility is not limited to this embodiment, and entrances and exits may be located separately or in multiple locations.

[0112] (5) The "reception device" may be at least one of the customer reception device 1 and the electronic registration desk 2. Also, the "reception camera" may be at least one of the facial recognition cameras 1A and 2A.

[0113] (6) In the embodiments described above, the cameras were exemplified as an arrival camera 11, an exit camera 12, in-store cameras 13, 14, 15, and face recognition cameras 1A, 2A, 3A, 4A. There may be only one camera.

[0114] (7) In the above-described embodiment, the display device 20 is a tablet computer having a so-called touch panel display, which is carried by the teller at counter E or the facility staff. The display device 20 may also be the display of a management server operated by a remote administrator.

[0115] (8) The camera of this disclosure is not limited to a color camera that captures visible light, but may be, for example, in combination with an infrared sensor or in combination with a LiDAR (LiDAR) distance measuring reflectivity photoelectric sensor.

[0116] (9) The camera used to capture images of visitors viewing the digital signage 4 may be, in addition to the facial recognition camera 4A, a ceiling camera that captures images of the area where the digital signage 4 is installed.

[0117] (10) Users of the facility are not limited to visitors to the facility, but may also include, for example, employees or workers at the facility.

[0118] Furthermore, the configurations disclosed in the above-described embodiments (including other embodiments, the same applies hereinafter) can be applied in combination with configurations disclosed in other embodiments, as long as no inconsistencies arise. In addition, the embodiments disclosed herein are illustrative, and the embodiments of this disclosure are not limited thereto and can be modified as appropriate without departing from the purpose of this disclosure.

[0119] This disclosure is applicable to user management systems and user management programs.

[0120] 1: Customer registration device (reception device) 1A: Facial recognition camera (camera, reception camera) 2: Electronic registration desk (reception device) 2A: Facial recognition camera (camera, reception camera) 3A: Facial recognition camera (camera) 4: Digital signage (signage device) 4A: Facial recognition camera (camera, signage camera) 11: Visitor camera (camera) 12: Exit camera (camera, exit camera) 13: In-store camera (camera) 14: In-store camera (camera) 15: In-store camera (camera) 20: Display device 30: Control device A: Entrance / exit (exit)

Claims

1. A user management system for managing the location of facility users, comprising: a camera that takes images of predetermined locations in the facility; a display device; and a control device, wherein the control device is configured to identify the user in the image captured by the camera, identify the location of the user in the facility, generate a map image showing the location of the user, and display the map image on the display device.

2. The user management system according to claim 1, wherein the control device is configured to identify the attributes of the user captured in the image and to display the attributes on the display device.

3. The user management system according to claim 1 or 2, wherein the control device is configured to display the captured image on the display device.

4. The user management system according to any one of claims 1 to 3, wherein the control device is configured to determine the time of the user's visit to the facility based on the time at which the user is identified as being captured in the image, and to display the time of the visit on the display device.

5. A user management system according to any one of claims 1 to 4, comprising: a reception device installed in the facility and operated by a visiting user; and a reception camera as the camera for capturing images of the user operating the reception device, wherein the control device is configured to identify the user captured in the image captured by the reception camera, identify the time of reception of the user by the reception device, and display the time of reception on the display device.

6. The user management system according to any one of claims 1 to 5, comprising an exit camera provided at the exit of the facility and capturing images of exiters leaving the facility from the exit, wherein the control device is configured to identify the exiter captured in the image captured by the exit camera and to display the exiter on the display device.

7. The user management system according to claim 6, wherein the control device is configured to determine the time of departure of the exiting person from the facility based on the time of departure of the exiting person shown in the image captured by the exit camera, and to display at least one of the departure time and the elapsed time since the departure time on the display device.

8. The user management system according to any one of claims 1 to 7, wherein the control device is configured to display on the display device the location of the user identified based on the most recent captured image, or the last location of the user identified based on past captured images.

9. The user management system according to claim 8, wherein the control device is configured to display information on the display device indicating that it failed to identify the user based on the latest captured image when displaying the last known location of the user on the display device.

10. A user management system according to any one of claims 1 to 9, comprising: a signage device that provides information to the user; and a signage camera that takes images of the user standing in front of the signage device, wherein the control device is configured to identify the user who is captured in the image captured by the signage camera, identify the location of the user in the facility, and reflect the location of the user in the map image.

11. A user management program used in a system for managing the location of facility users, the program causing a computer to execute the following functions: a user identification function that identifies the user in an image captured by a camera that captures a predetermined location in the facility; a location identification function that identifies the location of the user in the facility; a map generation function that generates a map image showing the location of the user; and a display function that displays the map image on a display device.

Images