Information processing device, information processing method, and program

Abstract

To provide a facility management system for automating or saving labor of facility operation.SOLUTION: In an information processing device, person specification means specifies a person who enters a facility. Group estimation means estimates a group composed of a plurality of persons. Group presentation means presents an estimation result of the group to a person. Group setting means sets a group on the basis of a response to the estimation result of the group.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] This disclosure relates to the technical field of facility management systems that automate or reduce the manpower required for facility operation.

Background Art

[0002] In recent years, unmanned and manpower-reduced AI (Artificial Intelligence) stores have been introduced in supermarkets, convenience stores, etc. Due to the impact of the recent spread of infectious diseases, it is expected that the demand for unmanned and manpower-reduced AI stores will further increase in the future.

[0003] As technologies for automating or reducing the manpower required for store operation, Patent Documents 1 to 3 can be cited. Patent Document 1 describes an article estimation device that automatically discriminates products taken out from shelves by a person and associates the person with the products. Patent Document 2 describes a group estimation device that estimates a group including at least some of a plurality of target persons based on gaze information, etc. Further, Patent Document 3 describes a shopping management device that performs shopping in a group composed of a plurality of users in online shopping and a designated single user makes a lump-sum payment.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0005] This disclosure aims to improve the technologies disclosed in the prior art documents. [Means for solving the problem]

[0006] From one perspective of this disclosure, the information processing device is A means of identifying individuals who enter the facility, A group configuration means for configuring a group consisting of multiple users, Estimation means for estimating the decision-maker and companions of the group based on the user's state or behavior recognized from images captured by cameras installed within the facility, A transmission means for transmitting information about the payer and accompanying persons of the group, and information about the products acquired by the payer and accompanying persons, to the terminal of a user belonging to the group. It is equipped with.

[0007] From another perspective of this disclosure, the information processing methods performed by computers are: Identify users who have entered the facility, Set up a group consisting of multiple users, Based on the user's state or behavior recognized from images captured by cameras installed within the facility, the decision-maker and accompanying persons of the group are estimated. The system transmits information about the payer and accompanying person of the group, as well as information about the products acquired by the payer and accompanying person, to the terminals of users belonging to the group.

[0008] In yet another aspect of this disclosure, the program is Identify users who have entered the facility, Set up a group consisting of multiple users, Based on the user's state or behavior recognized from images captured by cameras installed within the facility, the decision-maker and accompanying persons of the group are estimated. The computer is instructed to perform a process that transmits information about the payer and accompanying persons of the group, as well as information about the products acquired by the payer and accompanying persons, to the terminals of users belonging to the group. [Brief explanation of the drawing]

[0009] [Figure 1] The configuration of the facility management system according to the first embodiment is shown. [Figure 2] The general configuration of the management server is shown below. [Figure 3]This is an example of the data structure of the registered user information DB. [Figure 4] This is an example of the data structure of the product information DB. [Figure 5] This is an example of the data structure of the customer information DB. [Figure 6] This is an example of the data structure of the purchased product list DB. [Figure 7] This is a schematic diagram showing the state of Store 5. [Figure 8] This is an example of the display of the group confirmation screen. [Figure 9] This is an example of the display of the purchased product list screen. [Figure 10] Shows the schematic configuration of the mobile terminal. [Figure 11] This is a flowchart of the group settlement process according to the first embodiment. [Figure 12] This is a flowchart of the product association process. [Figure 13] This is a flowchart of the group setting process. [Figure 14] Shows the configuration of the facility management system according to the second embodiment. [Figure 15] This is a flowchart of the group settlement process according to the second embodiment. [Figure 16] Shows the configuration of the information processing apparatus according to the third embodiment. [Figure 17] This is a flowchart of the process according to the third embodiment. [Modes for Carrying Out the Invention]

[0010] <First Embodiment> Hereinafter, the first embodiment of this disclosure will be described with reference to the drawings. [Configuration of Facility Management System] FIG. 1 shows the configuration of the facility management system 100 according to this embodiment. The facility management system 100 is a system for managing facilities, and examples of facilities include stores, airports, stations, amusement parks, event venues, companies, etc. In this embodiment, as an example of a facility, a specific example when the present application is applied to a store will be described.

[0011] The facility management system 100 mainly comprises a management server 1, a mobile terminal 2 used by users, multiple cameras 10 installed inside or outside the store 5, product shelves 14, an entrance 16, and an exit 18. The presence or absence of gates at the entrance 16 and exit 18 is optional.

[0012] The management server 1 and the camera 10 are connected via any network 3, such as a local area network or the internet, by at least one of either a wired or wireless connection. Alternatively, the management server 1 and the camera 10 may be directly connected. Furthermore, the management server 1 and the mobile terminal 2 are connected via any network 3, such as the internet, in a manner that allows for communication.

[0013] The facility management system 100 first estimates the group and payer for users who enter store 5 and presents the estimation results to the user. Then, based on the user's response to the estimation results, the facility management system 100 sets the group and payer. Furthermore, based on the payer's payment information, the facility management system 100 settles the total amount of all items purchased by the users belonging to the group in one go. Here, a group is a collection of users who shop together, such as family or friends. The payer is the person who pays for all the items purchased by the users belonging to the group.

[0014] A user enters the store alone or with other users, retrieves the items they intend to purchase (hereinafter also referred to as "purchased items") from the products displayed on the shelves 14, and places them in a cart. Specifically, User 20, shown in Figure 1, is a man in his 30s who enters the store with his wife, User 22, and his daughter, User 24. A cart is a container used by users to move their purchased items around in the store 5, and multiple users belonging to the same group share one cart. In this disclosure, Users 20, 22, and 24 are a family and belong to the same group. The user who pays for the purchased items is called the payer, and other users belonging to the group are called companions. In the example in Figure 1, User 20 is the payer, and Users 22 and 24 are companions.

[0015] Camera 10 is an imaging device. Camera 10 includes an image sensor such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxid Semiconductor) sensor.

[0016] Cameras 10 are installed in multiple locations inside or outside the store, such as near the entrance 16 of store 5, in the area where goods are displayed in store 5 (hereinafter also referred to as the "sales floor"), near the exit 18 of store 5, and in the parking lot of store 5. Cameras 10 installed near the entrance 16 capture images of users entering the store. Cameras 10 installed in the sales floor capture images of users and goods present in the sales floor. Cameras 10 installed near the exit 18 capture images of users leaving the store. Cameras 10 installed in the parking lot capture images of the car a user is in, any passengers, and the user's position in the vehicle. Store 5 is equipped with a number and arrangement of cameras 10 that can capture images of users and goods present in any location within store 5. Cameras 10 transmit image data, including images acquired through imaging, to the management server 1.

[0017] Based on the image data acquired from camera 10, management server 1 identifies the user who entered the store, identifies the purchased items acquired by each user, estimates the group, and estimates the payer. Then, management server 1 presents the estimation results to the user and sets the group and payer based on the user's response to the estimation results. Furthermore, based on the payer's payment information, management server 1 settles the total amount of all purchased items acquired by users belonging to the group in one go.

[0018] Management Server 1 can be implemented on one or more computers. Management Server 1 may be installed at Store 5. Alternatively, Management Server 1 may be implemented on a cloud, which is a collection of computing resources.

[0019] Mobile terminal 2 is a portable terminal device, such as a smartphone or tablet, used by a user shopping at store 5. Mobile terminal 2 has functions such as receiving estimated group and payment person results from management server 1 and displaying them as a group confirmation screen. Mobile terminal 2 also has functions such as sending user responses to management server 1, such as adding or removing companions or changing the payment person, as a response to the estimated results, on the group confirmation screen. Note that in Figure 1, for the sake of explanation, only the mobile terminal 2 used by user 20 is shown, but other users at store 5 may also possess mobile terminal 2.

[0020] [Device configuration] Next, the configurations of the management server 1 and the mobile terminal 2 will be explained with reference to Figures 2 to 10.

[0021] (Management Server) Figure 2 shows the schematic configuration of the management server 1. The management server 1 mainly comprises a storage unit 31, a communication unit 32 for data communication, and a control unit 33. These elements are interconnected by a bus line 30.

[0022] The storage unit 31 is composed of memory such as a hard disk or flash memory. The storage unit 31 stores the program executed by the control unit 33, and the information necessary for the control unit 33 to execute predetermined processes by executing the program. In this embodiment, the storage unit 31 includes a registrant information DB (Database) 35, a product information DB 36, a customer information DB 37, and a purchased product list DB 38. The storage unit 31 may also store various store management-related information other than the registrant information DB 35, product information DB 36, customer information DB 37, and purchased product list DB 38.

[0023] The Registered User Information DB35 is a database containing information about users who have registered as members of Store 5 (hereinafter also referred to as "Registered Users"). Figure 3 shows an example of the data structure of the Registered User Information DB35. As shown in the figure, the Registered User Information DB35 stores Registered User information such as registration ID, name, contact information, payment information, personal characteristics information, and attribute information. Membership registration for Store 5 can be performed by any method.

[0024] The registration ID is identification information used to identify the registrant. The name is the registrant's name. Contact information is information about contacts necessary to communicate with the registrant, such as the email address of the mobile device 2 used by the registrant. Payment information is information used for processing payments for purchased goods, such as the registrant's credit card number or bank account number. Personal characteristic information is information used to determine whether a user is a registrant or not, using image data of the user entering the store, and to identify who the registrant is if they are. Personal characteristic information may be the image data itself including the user's face, or it may be a feature quantity calculated from the image data including the user's face. Attribute information is information indicating the relationship between registrants, such as "family" or "friends." For example, if the registrant with registration ID "M001" and the registrant with registration ID "M002" are a married couple, information such as "M002 and family" will be stored as attribute information for registration ID "M001."

[0025] Product Information DB36 is a database containing information about products handled by store 5. Figure 4 shows an example of the data structure of Product Information DB36. As shown in the figure, Product Information DB36 stores product information such as product ID, product name, price, and product feature information. The product ID is identification information for identifying a product, and may be, for example, a JAN (Japanese Article Number) code. The product name is the name of the product, and the price is the price of the product. Product feature information is information used to identify what product a user has acquired by using image data of the area containing the product. Product feature information may be the image data of the area containing the product itself, or it may be feature quantities calculated from the image data of the area containing the product.

[0026] The Customer Information DB37 is a database of information about users present within Store 5. Figure 5 shows an example of the data structure of the Customer Information DB37. As shown in the figure, the Customer Information DB37 stores customer information such as registration ID, temporary ID, name, type, person image information, exit flag, and group ID, which are all related to users present within Store 5. As will be explained in detail later, customer information is generated when a user enters the store and is updated as needed in response to changes in group structure, etc., as long as the user and the group to which that user belongs exist within Store 5.

[0027] The registration ID is the registration ID issued when a user enters the store and registers as a member, and is stored if the user is registered. The temporary ID is identification information temporarily set to identify a user who enters the store, and is issued and stored if the user is not registered. The name is the name of the user who enters the store, and is stored by referring to the registrant information DB35 if the user is registered. The type is information indicating whether the user who enters the store is the payer, a companion, or undecided (it is unclear whether they are the payer or a companion). The person image information is image data of the area including the face of the user who enters the store (hereinafter also referred to as "face image data"). The exit flag is a flag that indicates whether the user has left store 5 (exited (y)) or not (not exited (n)). The exit flag is set to "not exited (n)" when the entrant information is generated. The exit flag is set to "exited (y)" when the user exits store 5 through exit 18, and the user is considered to have left the store. The group ID is an identifier that identifies the group to which a user belongs.

[0028] If a user visits the store alone and does not belong to a group, the group ID may be left blank, or it may be treated as a group of one user and a group ID may be set. Also, if there are multiple possible groups to which the user belongs and it is not possible to identify a single group, the group ID may store all of the candidate group IDs. In this case, it is desirable to store the group IDs in order of the likelihood of the user belonging to the group.

[0029] The Purchased Item List DB38 is a database containing information about items purchased by users at store 5. Figure 6 shows an example of the data structure of the Purchased Item List DB38. As shown in the figure, the Purchased Item List DB38 stores purchased item list information such as buyer information, purchased item ID, product name, price, quantity, subtotal, group ID, and total amount. As will be explained in detail later, the purchased item list information is generated when a user acquires a purchased item and is updated as long as the user and the group to which that user belongs exist within store 5.

[0030] Purchaser information is information that identifies the user who retrieved the purchased product from the product shelf 14, and consists of a registered user ID and a temporary user ID. If the user is a registered user, the registered user ID is stored in the purchaser information; if the user is not registered, the temporary user ID is stored. The purchased product ID is the product ID of the purchased product. The product name is the product name of the purchased product, and the price is the price of the purchased product. The product name and price are stored by referring to the product information DB 36 based on the product ID of the purchased product. The quantity is the number of purchased products. The subtotal is the total price of the purchased products retrieved by each user. The group ID is the group ID of the group to which the user belongs. The total amount is the total price of all purchased products retrieved by users belonging to the group.

[0031] If a user visits the store alone and does not belong to a group, the group ID and total amount may be left blank, or the user may be treated as part of a single-user group, and the group ID and total amount may be stored accordingly.

[0032] The communication unit 32 communicates with the mobile terminal 2 and the camera 10 by wired or wireless means.

[0033] The control unit 33 includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc. (not shown), and performs various controls on each component within the management server 1. In this embodiment, the control unit 33 includes a person identification unit 40, a tracking unit 41, a relevance calculation unit 42, a linking unit 43, a group estimation unit 44, a presentation unit 45, a reception unit 46, a group setting unit 47, a payer setting unit 48, and a payment processing unit 49.

[0034] The person identification unit 40 identifies a user who has entered the store using image data of the user acquired from the camera 10. Specifically, the person identification unit 40 acquires image data of the user who has entered the store from the camera 10 installed near the entrance 16 of the store 5 and acquires the user's characteristic information. At this time, the person identification unit 40 may also acquire face image data from the image data of the user who has entered the store. The person identification unit 40 then compares the person characteristic information of the user who has entered the store with the person characteristic information of each registered person read from the registered person information DB 35 to determine whether the user who has entered the store is a registered person, and if so, who the registered person is. If the user who has entered the store is a registered person, the person identification unit 40 generates new store entrant information including the registration ID and name read from the registered person information DB 35, the user's face image data, and an exit flag (n), and stores it in the store entrant information DB 37. On the other hand, if the user who enters the store is not a registered user, the person identification unit 40 generates new store entry information including the issued temporary ID, the user's facial image data, and an exit flag (n), and stores it in the store entrant information DB 37.

[0035] The tracking unit 41 uses user image data acquired from the camera 10 to track the movements of each user within the store and acquires it as tracking data. Specifically, the movements of users 20, 22, and 24 tracked by the tracking unit 41 are shown by dotted lines in Figure 1. When a user leaves the store 5 through exit 18, the tracking unit 41 considers that the user has left the store and sets the exit flag in the customer information DB 37 to "Exited (y)".

[0036] The relevance calculation unit 42 performs interaction detection using user image data acquired from the camera 10 and tracking data from the tracking unit 41, and calculates the degree of relevance between multiple users based on arbitrary determination factors. Examples of determination factors for calculating relevance include the distance between users, the direction of their faces, opening and closing of their mouths, physical movements, and whether or not they share a container for carrying products. The relevance calculation unit 42 calculates relevance not only based on information such as image data and tracking data within the store, but also based on information such as image data and tracking data from the parking lot and other stores, for example, whether they arrived in the same car or shopped together at other stores. The relevance calculation by the relevance calculation unit 42 is performed as needed.

[0037] Here, we will explain in detail the criteria used to calculate relevance. The distance between users, which is a criterion, is measured by recognizing the coordinates of each user in the real world based on the user's image data. Specifically, the relevance calculation unit 42 acquires the physical distance between users, measured by machine learning-based human detection and camera calibration, as data related to the distance between users.

[0038] The determination element, face orientation, is detected by analyzing the user's gaze based on the user's image data. Specifically, the relevance calculation unit 42 recognizes the user's face orientation and what they are looking at through machine learning-based face orientation detection and gaze detection, and acquires the registered ID or temporary ID of the user with whom the user is making eye contact as data related to face orientation. The relevance calculation unit 42 also acquires the registered IDs or temporary IDs of multiple users who are simultaneously looking at the same thing as data related to face orientation.

[0039] Furthermore, the correlation calculation unit 42 acquires data related to face orientation, not only when users make eye contact with each other at the same time, but also the number of times each user turns their face and gazes towards a predetermined user. Specifically, the number of times each user turns their face and gazes towards a predetermined user is, for example, the number of times user A looks at user B. The group estimation unit 44, described later, may estimate that users belong to the same group if the number of times they turn their face and gazes towards each other exceeds a predetermined number. The conditions for the group estimation unit 44 to estimate that users belong to the same group can be arbitrarily set, for example, "when user A looks at user B more than 5 times" or "when user A looks at user B more than 3 times AND user B looks at user A more than 3 times".

[0040] The determination element, mouth opening and closing, is detected by analyzing the user's mouth based on the user's image data. Specifically, the relevance calculation unit 42 obtains the registered ID or temporary ID of the speaking user as data related to mouth opening and closing through machine learning-based speech detection.

[0041] The physical contact element, which is the determination factor, estimates a person's skeleton, posture, body shape, appearance, or silhouette based on the user's image data, and detects physical contact between users, such as linking arms or holding hands. Specifically, the relevance calculation unit 42 obtains the registered ID or temporary ID of the users who are in physical contact as data related to physical contact through pose estimation using machine learning.

[0042] The determination factor, whether or not a container is shared, detects whether or not a container such as a shopping cart, basket, or eco-bag used to hold purchased items is shared among multiple users, based on the user's image data. Specifically, the relevance calculation unit 42 obtains the registered ID or temporary ID of the users who are sharing the container as data.

[0043] The relevance calculation unit 42 may calculate the relevance from a single determination element, or it may calculate the relevance from a combination of multiple determination elements. For example, the relevance calculation unit 42 may detect users conversing with each other and calculate the relevance based on two determination elements such as "mouth opening and closing" and "face orientation." The relevance calculation unit 42 may also calculate the relevance by taking attribute information into account. Furthermore, the relevance calculation unit 42 may calculate the relevance using tracking data, not just image data.

[0044] The linking unit 43 uses user image data acquired from the camera 10 to identify the user who acquired the purchased product and the acquired purchased product. The linking unit 43 also links the identified user with the purchased product and stores it in the purchased product list DB 38.

[0045] Figure 7 is a schematic diagram showing the store 5 according to the first embodiment. The store 5 is equipped with product shelves 14 on which products are displayed. A camera 10 is also provided near the product shelves 14 at a position that allows it to capture an area including the face of the user 20 and the purchased products 51.

[0046] Specifically, the linking unit 43 acquires image data of the area containing the product from the camera 10 and obtains product feature information of the purchased product 51 from the image data. The linking unit 43 uses the product feature information of the purchased product 51 to detect when the purchased product 51 is taken out of the product shelf 14. Detection of the purchase of the purchased product 51 being taken out of the product shelf 14 is performed, for example, by detecting that the product feature information of the purchased product 51 moves out of the area corresponding to the product shelf 14 (or from a position where it touches to a position where it does not touch) between a series of consecutive image data in a time series. The area corresponding to the product shelf 14 in the image data may be recorded in advance as coordinates in the image data, or it may be determined by image recognition of the product shelf 14 from the image data. The linking unit 43 may also detect when the user 20 has acquired the purchased product 51 by detecting from the image data that the purchased product 51 has been placed in the cart. The linking unit 43 identifies the purchased product 51 by comparing the product feature information of the purchased product 51 with the product feature information read from the product information DB 36, and obtains the product ID, product name, and price of the purchased product 51.

[0047] Furthermore, the linking unit 43 acquires facial image data of the user 20 who retrieved the purchased item 51 from the camera 10. The user 20 who retrieved the purchased item 51 is, for example, the user closest to the purchased item 51. The linking unit 43 identifies the user 20 who retrieved the purchased item 51 by comparing the person characteristic information of the facial image data of the user 20 who retrieved the purchased item 51 with the person characteristic information of the facial image data read from the customer information DB 37, and obtains the registered ID or temporary ID of that user 20.

[0048] The linking unit 43 then stores the registered ID or temporary ID of the user 20 who retrieved the purchased item 51, the item ID of the purchased item 51, the item name, price, and quantity in the purchased item list DB 38. At this time, the linking unit 43 calculates a subtotal and stores it in the purchased item list DB 38 as well. If the user belongs to a group, the linking unit 43 also calculates the total amount and updates the purchased item list DB 38.

[0049] Furthermore, the method for identifying the user 20 who took out the purchased item 51 may involve acquiring image data of the area including the user's hand from the camera 10 and applying methods based on the position and movement of the hand, or methods based on tracking data.

[0050] The group estimation unit 44 estimates groups of multiple users based on image data, attribute information, and relevance scores acquired from the camera 10. For example, the group estimation unit 44 takes attribute information into account and estimates that users with a relevance score above a threshold belong to the same group. If there are no users with a relevance score above a threshold, the group estimation unit 44 estimates that the user is shopping alone. The group estimation by the group estimation unit 44 is performed continuously while the user is inside the store 5.

[0051] In this way, by estimating groups based on the degree of relevance calculated by combining various judgment factors, the management server 1 can improve the accuracy of group settings. Furthermore, by taking into account attribute information stored in advance before entering the store, the management server 1 can estimate and set groups more accurately.

[0052] Furthermore, the group estimation unit 44 estimates the payer from among the users that make up the group. If the group estimation unit 44 estimates that a user is shopping alone, it estimates that user to be the payer.

[0053] Further details regarding the group estimation method and the decision-maker estimation method will be described later.

[0054] The presentation unit 45 presents the user with the group estimation result from the group estimation unit 44. The presentation unit 45 also presents the user with the group estimation result from the group estimation unit 44. Specifically, the presentation unit 45 presents the estimation result to the user who is estimated or set as the setter for the group. Since the user who is estimated or set as the setter is a registered user who stores payment information, the presentation unit 45 transmits the estimation result to the mobile terminal 2 used by the user based on the contact information read from the registered user information 35. For example, the presentation unit 45 transmits a group confirmation screen as the estimation result. The mobile terminal 2 displays the received group confirmation screen, and the user performs a predetermined operation on the group confirmation screen, which will be described later.

[0055] In this way, by presenting the estimation results to the user who is estimated or designated as the payer, the user who pays for the purchased goods can add or remove other users belonging to the group.

[0056] Here, we will explain the group confirmation screen. Figure 8 is an example of a group confirmation screen. As shown in Figure 8(a), the group confirmation screen has instructions for the user 60, face image data of the user estimated to be the payer (hereinafter also referred to as "payer face image data") 61, face image data of users estimated to be companions (hereinafter also referred to as "companion face image data") 62 and 63, a modify button 64, add buttons 65 and 67, and exclude buttons 66 and 68.

[0057] The payer face image data 61 is face image data of the user who is presumed to be the payer, and is read from the customer information DB 37. The correct button 64 is a button to be pressed when the payer is different, as described in instruction 60, to specify the correct payer. Specifically, after pressing the correct button 64, the user may specify the payer by entering the name or registration ID of the correct payer, or they may specify the payer by taking a picture of the correct payer with the camera built into the mobile terminal 2 and sending the face image data of the correct payer.

[0058] The companion face image data 62 and 63 are face image data of users who are presumed to be companions, and are read from the customer information DB 37. The add buttons 65 and 67 are pressed when the users indicated by companion face image data 62 and 63, respectively, are companions, as described in instruction 60, and confirm that the users belong to the same group. On the other hand, the exclude buttons 66 and 68 are pressed when the users indicated by companion face image data 62 and 63, respectively, are not companions, as described in instruction 60, and do not acknowledge that the users belong to the same group, thus excluding them. Presses of various buttons and predetermined operations by the user are transmitted from the mobile terminal 2 to the management server 1 as responses to the estimation results.

[0059] In this way, by presenting a group confirmation screen that includes the user's facial image data as the estimation result, users can easily recognize the users who make up the group they belong to. Therefore, users can respond accurately to the estimation result, and the management server 1 can improve the accuracy of group settings.

[0060] If the group confirmation screen does not include users shopping together, the user may specify users belonging to the same group by taking a picture of the users shopping together with the camera built into the mobile terminal 2 and sending the facial image data to the management server 1. Specifically, if user 20, as shown in Figure 1, does not include his daughter, user 24, on the group confirmation screen which is the group estimation result, he takes a picture of user 24 with the camera built into the mobile terminal 2 and sends the facial image data of user 24 to the management server 1. Based on the facial image data of user 24, the management server 1 checks whether the user indicated by the facial image data exists in the store 5, and if it does, sets user 24 to belong to the same group as user 20.

[0061] Furthermore, in this embodiment, a group confirmation screen as shown in Figure 8(a) is presented as the estimation result, but the screen configuration is arbitrary as long as it allows for the presentation of estimation results and user response. For example, as shown in Figure 8(b), a purchase item list 70 based on purchase item list information may also be displayed so that the purchase items acquired by each user indicated by the payer's facial image data and the accompanying person's facial image data can be seen. This allows the user to check what each user has purchased and then decide whether to add them to the same group or exclude them.

[0062] Furthermore, the display unit 45 displays the estimation results not only when a user enters the store, but also at any time when the group composition changes. In addition, the display unit 45 may display the estimation results to all users belonging to the group, not just the user who is estimated or set as the group's payer. Specifically, if the store 5 is equipped with a shared terminal that users can use freely, the display unit 45 can display the estimation results to all users by transmitting the estimation results to the shared terminal. Here, the shared terminal is a tablet terminal similar to the mobile terminal 2 and is connected to the management server 1 so as to be able to communicate with it.

[0063] The reception unit 46 receives the user's response to the estimation results presented by the presentation unit 45 from the user's mobile terminal 2 via the communication unit 32. Specifically, the reception unit 46 obtains information regarding modifications to the estimation results, user designation, additions, and exclusions as responses.

[0064] The group setting unit 47 sets up groups based on the estimation result and the response to the estimation result. The group setting unit 47 then issues a group ID to identify the group and stores the group ID of each user belonging to the group in the customer information DB 37. For users who do not belong to a group, for example, a hyphen is stored as their group ID. In addition, the group setting unit 47 stores the group ID of each user belonging to the group in the purchased product list DB 38, and also calculates and stores the total amount of all purchased products acquired by users belonging to the same group.

[0065] The payer setting unit 48 sets the payer based on the estimation result and the response to the estimation result. The payer setting unit 48 then stores in the customer information DB 37 the type of user who is the payer as the payer and the type of user who is an accompanying person as the accompanying person.

[0066] The payment processing unit 49, at the moment all users belonging to the group exit from exit 18 of store 5, refers to the registrant information DB 35 and performs payment processing for all purchased items acquired by each user belonging to the group based on the payment information of the payer. Payment processing is the process of paying the price based on the payment information. Specifically, the payment processing unit 49 authenticates the payer using biometric authentication and performs payment processing based on the payment information of the authenticated payer. Here, biometric authentication can be, for example, authentication using facial images, iris scans, fingerprints, gait patterns, or voice.

[0067] The display unit 45 may also transmit the purchase item list information to the mobile terminal 2 used by the payer via the communication unit 32 before the payment processing is performed by the payment processing unit 49. Figure 9 is an example of the purchase item list screen. Specifically, as shown in Figure 9(a), the purchase item list screen includes the payer's face image data and the accompanying person's face image data, and displays the product name, quantity, price, and subtotal of the purchase items acquired by each user indicated by the face image data, the total amount of all purchase items acquired by each user, and an approval button 72. In this case, when the user presses the approval button 72, the payment processing unit 49 obtains information from the mobile terminal 2 via the communication unit 32 that the purchase item list has been approved, and then processes the payment for all purchased items. In other words, the payment processing unit 49 may perform the payment processing after the user has confirmed the purchase items and approved the payment.

[0068] Furthermore, the purchased items list screen is not limited to including the payer's facial image data and the accompanying person's facial image data. As shown in Figure 9(b), the screen configuration is arbitrary as long as the correspondence between the users constituting the group and the purchased items acquired by each user is known.

[0069] In the above configuration, the memory unit 31 is an example of a memory means, the person identification unit 40 is an example of a person identification means, and the relevance calculation unit 42 is an example of a relevance calculation means. Furthermore, the group estimation unit 44 is an example of a group estimation means and a decision-maker estimation means, the presentation unit 45 is an example of a group presentation means and a decision-maker presentation means, the group setting unit 47 is an example of a group setting means, and the decision-maker setting unit 48 is an example of a decision-maker setting means. In addition, the settlement processing unit 49 is an example of a settlement processing means, and the alert processing unit 50 is an example of an alert processing means.

[0070] (Mobile device) Figure 10 shows the schematic configuration of the mobile terminal 2. The mobile terminal 2 mainly comprises a display unit 91, an input unit 92, a storage unit 93, a communication unit 94, and a control unit 95. These elements are interconnected via a bus line 90.

[0071] The display unit 91 displays various information, such as a group confirmation screen and a purchased product list screen, based on the control of the control unit 95.

[0072] The input unit 92 is an interface that accepts user input through predetermined operations on the group confirmation screen and the purchased product list screen, and includes, for example, a touch panel, buttons, or a voice input device.

[0073] The storage unit 93 is composed of memory such as a hard disk or flash memory. The storage unit 93 stores programs executed by the control unit 95, and information necessary for the control unit 95 to perform predetermined processes by executing the programs. For example, the storage unit 93 may store a dedicated application program that is activated when shopping at store 5 and controls the display of various screens such as a group confirmation screen and a purchase list screen.

[0074] The communication unit 94 communicates with the management server 1 via wireless communication. The control unit 95 includes a CPU, ROM, RAM, etc. (not shown) and performs various controls on each component within the mobile terminal 2.

[0075] [Method for estimating groups and methods for estimating decision-makers] Next, we will explain the methods for estimating the group and the decision-makers.

[0076] (Group estimation method based on user behavior before entering the store) Users who are family members, such as spouses or parents and children, can register this fact as attribute information in the registrant information DB35 before entering the store. In this case, the group estimation unit 44 estimates the group of users present in the store 5 based on the attribute information read from the registrant information DB35. Alternatively, instead of using attribute information, family members may link their accounts at the time of member registration. Furthermore, even if they are family members, if a user is a minor or otherwise unable to register as a member and does not have an account, their facial photograph may be registered in advance and stored in the registrant information DB35.

[0077] The group estimation unit 44 may calculate the degree of relevance based on image data acquired from a camera 10 installed in a parking lot outside the store, and estimate the group of users present inside the store 5. Specifically, the group estimation unit 44 estimates that users who were in the same car belong to the same group based on the image data.

[0078] If store 5 is a facility in a large shopping mall, the group estimation unit 44 may estimate that a group set up in another store in the shopping mall is the group of users present in store 5. In this case, the stores located in the shopping mall share the registered user information DB 35 and the customer information DB 37.

[0079] The system may store in the storage unit 31 historical information such as customer information and purchase list information from when a user previously visited store 5, and the group estimation unit 44 may calculate the degree of relevance based on this historical information to estimate the group of users present in store 5. Specifically, since users who entered the store together last time are likely to belong to the same group, the group estimation unit 44 estimates that they belong to the same group this time as well.

[0080] If a user's mobile device 2 is using a product purchase app (for example, the Amazon app) to shop at store 5, and that app is linked to other SNS (Social Networking Service) apps (for example, Line, Facebook, etc.), the group estimation unit 44 may identify users who have relationships such as friends on the SNS app, and if the identified users are present in store 5, it may estimate that those users belong to the same group.

[0081] Based on information from the SNS app used by the user on the mobile device 2, if the time and location information of multiple users are the same, the group estimation unit 44 may estimate that the multiple users belong to the same group.

[0082] (Method for estimating the payer based on user behavior before entering the store) Users who are family members, such as a married couple or parent and child, specify in advance who the payer will be within the family before entering the store, and store this as a type in the registered user information DB35. In this case, the group estimation unit 44 estimates the payer of the group based on the type read from the registered user information DB35.

[0083] Among friends, whether they split the bill, pay in one lump sum, and who pays varies each time, making it difficult to remember their attributes or categories in advance like with family. Therefore, the group estimation unit 44 may estimate the user who entered the store first within the group as the payer based on image data acquired from the camera 10 located near the entrance 16.

[0084] The group estimation unit 44 may estimate a user driving a car to be the payer based on image data acquired from a camera 10 installed in a parking lot outside the store. Alternatively, the group estimation unit 44 may identify the user's means of arrival at the store based on image data acquired from a camera 10 installed outside the store, and estimate a user who arrived by car to be the payer.

[0085] (Group estimation method based on user behavior while inside the store) The relevance calculation unit 42 calculates the degree of relevance by detecting interactions between users in store 5, and the group estimation unit 44 estimates groups based on the degree of relevance. At this time, the relevance calculation unit 42 may calculate the degree of relevance by taking into account attribute information and tracking data, and the group estimation unit 44 may estimate groups by taking into account attribute information and tracking data.

[0086] (Method for estimating the payer based on the behavior of users while inside the store) The group estimation unit 44 estimates the user pushing a shopping cart to be the payer based on image data acquired from the camera 10 installed in the store 5. The group estimation unit 44 may also estimate the payer to be a user carrying a container for purchased goods, such as a basket or eco-bag, rather than just a shopping cart.

[0087] The group estimation unit 44 may acquire user appearance information based on image data obtained from a camera 10 installed in the store 5, and estimate the payer based on said appearance information. Specifically, the group estimation unit 44 estimates the payer to be a user who matches pre-set conditions such as "male in his 40s" or "wearing branded goods," based on estimated age, gender, height, body shape, clothing, accessories, etc. that can be determined from the appearance information.

[0088] If audio data can be obtained from the camera 10 installed inside the store 5, the group estimation unit 44 may identify the user who made a statement indicating they would make a payment, such as "I'll pay today," based on the image data and audio data, and estimate that user to be the payer.

[0089] If physical data such as the user's heart rate and body temperature can be obtained from cameras 10 and sensors installed in the store 5, the payment person is likely to be in a state of tension, so the group estimation unit 44 may estimate the user with a high heart rate and body temperature to be the payment person based on the physical data.

[0090] The group estimation unit 44 estimates that if there is only one user in the group whose payment information is stored in the registrant information DB 35, i.e., a user who is a registrant, then that user is the payer.

[0091] The system may store in the storage unit 31 historical information such as customer information and purchase list information from when a user previously used store 5, and the group estimation unit 44 may estimate a user who has previously made a payment based on the historical information. In addition, the group estimation unit 44 may determine, for example, that if a user has acquired a purchase item that is cheaper than previously purchased items based on the historical information, there is a high probability that the user is the payer, and estimate that user to be the payer.

[0092] Thus, even if the payer is not pre-stored in the memory unit 31 before entering the store, the group estimation unit 44 can estimate the payer based on the user's behavior while inside the store.

[0093] (Group estimation method based on user behavior immediately before payment) If a camera 10 or shared terminal is installed near exit 18, all users shopping together will be captured by the camera 10 or shared terminal. In this case, the group estimation unit 44 may estimate that the users included in the image data belong to the same group based on the image data acquired from the camera 10 or shared terminal.

[0094] Based on image data acquired from camera 10 located near exit 18 of store 5, the group estimation unit 44 may estimate, through interaction detection, that users who are seen with their arms linked or shaking hands near exit 18 belong to the same group.

[0095] In this way, by using image data that includes all members of the group acquired near the exit, the group estimation unit 44 can accurately estimate and set the group to which the user belongs and the users that make up that group.

[0096] (Method for estimating the payer based on the user's behavior immediately before payment) Based on image data acquired from the camera 10 located near exit 18 or from a shared terminal, the group estimation unit 44 may estimate that a user raising their hand near the exit is the person making the payment.

[0097] The group estimation unit 44 combines one or more of the above-mentioned elements to estimate the groups of users present in the store 5.

[0098] [Group payment processing] Next, we will explain the overview of group payment processing by the facility management system 100. Group payment processing is the process of setting up a group and a payer for users who enter store 5. Furthermore, group payment processing is the process of settling the total amount of all purchases made by users belonging to the group, based on the payer's payment information.

[0099] Figure 11 is a flowchart illustrating the overview of the group settlement process according to the first embodiment. The group settlement process includes a product linking process S200 and a group setting process S300. The group settlement process is primarily implemented by the management server 1 executing a pre-prepared program.

[0100] When a user enters store 5, the management server 1 identifies the user who entered the store based on image data acquired from camera 10 located near entrance 16 (step S100). Specifically, the management server 1 acquires the user's face image data and personal feature information based on the acquired image data. The management server 1 then compares the personal feature information of the entering user with the personal feature information of each registered user read from the registered user information DB 35 to determine whether the entering user is a registered user, and if so, who that user is.

[0101] The management server 1 determines whether the identified user is a registered user (step S101). If the user is a registered user (step S101; Yes), it generates new customer information including the registration ID and name read from the registered user information DB 35, the user's facial image data, and the exit flag (n), and stores it in the customer information DB 37 (step S102). On the other hand, if the user is not a registered user (step S101; No), the management server 1 issues a temporary ID, generates new customer information including the temporary ID, the user's facial image data, and the exit flag (n), and stores it in the customer information DB 37 (step S103).

[0102] Then, the management server 1 uses the user image data acquired from the camera 10 to track each user's movements within the store and acquires tracking data (step S104). Furthermore, the management server 1 performs product linking processing (step S200).

[0103] Next, we will explain the product linking process. The product linking process links the purchased products acquired by a user with that user. Figure 12 is a flowchart of the product linking process. This process is executed by the management server 1.

[0104] The management server 1 determines whether or not a product has been acquired by the user based on the image data acquired from the camera 10 (step S201). Specifically, the management server 1 acquires image data of the area containing the product from the camera 10 and obtains product feature information from the image data. Then, the management server 1 uses the product feature information to detect when a product is taken out of the product shelf 14. If the product has not been acquired (step S201; No), the management server 1 terminates the product linking process and proceeds to step S300 of the group settlement process shown in Figure 11.

[0105] On the other hand, if an item is acquired (step S201; Yes), the management server 1 uses the image data acquired from the camera 10 to identify the acquired item and the user who acquired the item (step S202). Specifically, the management server 1 identifies the purchased item by comparing the item's characteristic information with the item's characteristic information read from the item information DB 36, and acquires the item ID, item name, and price of the purchased item. The management server 1 also identifies the user who retrieved the purchased item by comparing the person's characteristic information from the user's face image data with the person's characteristic information from the customer information DB 37, and acquires the user's registration ID or temporary ID. The management server 1 then links the identified purchased item with the user and stores it in the purchased item list DB 38 (step S203). With this, the management server 1 finishes the item linking process and proceeds to step S300 of the group settlement process shown in Figure 11.

[0106] Next, we will explain the group configuration process. The group configuration process is the process of setting the group to which a user belongs and the decision-maker for that group. Figure 13 is a flowchart of the group configuration process. This process is executed by the management server 1.

[0107] The management server 1 performs interaction detection using user image data acquired from the camera 10 and tracking data from the tracking unit 41, and calculates the degree of relevance between multiple users present in the store 5 based on arbitrary determination factors (step S301). Determination factors for calculating the degree of relevance include, for example, the distance between users, the direction of their faces, opening and closing of their mouths, physical movements, and whether or not they are sharing a container for carrying goods. The management server 1 calculates the degree of relevance as needed.

[0108] The management server 1 then estimates groups composed of multiple users based on pre-stored attribute information and calculated relevance. For example, the group estimation unit 44 takes attribute information into account and estimates that users whose relevance is above a threshold belong to the same group (step S302). The management server 1 also estimates that a user does not belong to a group and is shopping alone if there are no users whose relevance is above a threshold.

[0109] Furthermore, the management server 1 estimates the payer from among the users who make up the group (step S303). If it estimates that a user does not belong to a group and is shopping alone, the management server 1 estimates that user to be the payer.

[0110] Next, the management server 1 presents the estimated results for the group and the payer to the user (step S304). Specifically, the management server 1 presents the estimated results for the group and the payer to the user who is estimated or set as the payer for that group. Since the user who is estimated or set as the payer is basically a registered user who stores payment information, the management server 1 sends the estimated results for the group and the payer to the mobile terminal 2 used by that user, based on the contact information read from the registered user information 35. Specifically, the management server 1 sends a group confirmation screen as the estimated result to the mobile terminal 2.

[0111] Mobile terminal 2 displays the received group confirmation screen, and the user performs the prescribed operations on the group confirmation screen according to the instructions. Management server 1 receives information regarding the prescribed operations performed by the user from mobile terminal 2 via communication unit 32 as a response to the estimation result (step S305). Specifically, management server 1 obtains information regarding corrections to the estimation result as a response.

[0112] Management server 1 determines whether the response to the estimation result includes modifications (step S306). Here, modifications include information regarding the exclusion of companions from the group estimation result and information regarding modifications to the decision-maker estimation result.

[0113] If the response to the estimation result includes modifications (step S306; Yes), the management server 1 modifies the estimated group and decision-maker (step S307). Specifically, if the estimated group result includes information about excluding companions, the management server 1 excludes companions for whom the exclude button was pressed and sets only companions for whom the add button was pressed in the group. Also, if the estimated decision-maker result includes information about modifications, the management server 1 sets the decision-maker according to the modifications (step S308). On the other hand, if the response to the estimation result does not include modifications (step S306; No), the management server 1 sets the estimated group and decision-maker as they are (step S308).

[0114] Once the group and payer settings are complete, the management server 1 updates the customer information DB 37 and the purchased item list DB 38 (step S309). Specifically, when the management server 1 sets up a group, it issues a group ID to identify the group and stores the group ID of each user belonging to the group in the customer information DB 37. For users who do not belong to a group, for example, a hyphen is stored as their group ID. Furthermore, the management server 1 stores the group ID of each user belonging to the group in the purchased item list DB 38 and calculates and stores the total amount of all purchased items acquired by users belonging to the same group. Also, when the management server 1 sets up a payer, it stores the type of the user who is the payer as the payer and the type of the accompanying user as the accompanying user in the customer information DB 37.

[0115] Then, the management server 1 finishes the group configuration process and proceeds to step S105 of the group settlement process shown in Figure 11.

[0116] The management server 1 refers to the exit flag in the customer information DB 37 and determines whether all users belonging to the group have left the store (step S105). If not all users belonging to the group have left the store, that is, if at least one user belonging to the group is still in store 5 (step S105; No), the management server 1 returns to the process in step S104 and, depending on the user's actions within store 5, tracks users, links users to products, and estimates and sets up groups as needed.

[0117] On the other hand, if all users belonging to a group leave the store (step S105; Yes), the management server 1 refers to the registered user information DB 35 and, using the payment information of the payer, settles the total amount of all purchased items acquired by each user belonging to the group (step S106). Specifically, the payment processing unit 49 authenticates the payer using biometric authentication and settles the payment based on the payment information of the authenticated payer. If a user enters the store alone and does not belong to a group, the management server 1 considers that all members of the group have left the store when that user leaves. This completes the group settlement process according to the first embodiment.

[0118] As described above, the facility management system 100 of this embodiment estimates the group and the payer for a user who enters store 5 and presents the estimation results to the user. The facility management system 100 then sets the group and the payer based on the user's response to the estimation results. Furthermore, based on the payer's payment information, the facility management system 100 performs a single payment process for the total amount of all items acquired by users belonging to the group.

[0119] According to this, even users who have not registered as members in advance can shop at Store 5 by belonging to a group with a payer. Therefore, minors who do not have payment information and cannot register as members can also enjoy shopping at Store 5 with their families.

[0120] Furthermore, the facility management system 100 automatically estimates groups through interaction detection and other means, but it also presents the estimation results to the user for confirmation, thereby improving the accuracy of group settings. This prevents situations such as setting strangers who are not shopping together in the same group, or paying for other people's purchases together.

[0121] Furthermore, the facility management system 100 automatically estimates the approver based on image data acquired from the camera 10, and presents the estimation result to the user for confirmation, thereby improving the accuracy of the approver setting.

[0122] In current unmanned or labor-saving AI stores, users are often required to register as members in advance, and unregistered individuals are usually not allowed to enter. Furthermore, unmanned or labor-saving AI stores are generally based on the premise that registered users will shop individually, and are not designed for groups of family or friends to shop together. However, this facility management system 100 can provide a system that allows users to enter automated or unmanned facilities without prior registration and enables group shopping.

[0123] [Differentiation] (First variation) In the above embodiment, the linking unit 43 identifies the purchased product acquired by the user using the user's image data acquired from the camera 10 and product feature information. However, the linking unit 43 is not limited to this and can also use location information to identify the purchased product. Here, the location information is, for example, information such as "the first row from the top of the designated product shelf 14 is rice balls" and "the rice balls from left to right are plum rice ball, salmon rice ball, and tuna rice ball," and the specifications and format of the data are arbitrary, such as image data of the product display and coordinates within the image data.

[0124] Furthermore, the tying unit 43 may use not only images but also weight sensors built into the product shelves 14 to identify purchased items. In this case, weight sensors are installed on each product shelf, and if, for example, the weight of one rice ball is removed from the display area of ​​salmon rice balls, the tying unit 43 detects that a salmon rice ball has been taken and identifies the purchased item. By combining image analysis and weight sensors, the tying unit 43 can improve the accuracy of purchasing item identification and reduce errors.

[0125] (Second variation) The group estimation unit 44 estimates the groups based on a pre-set priority when it is estimated that a single user belongs to multiple groups. The pre-set priority is stored in the registrant information DB 35.

[0126] (Third variation) In the above embodiment, users belonging to the same group share containers such as carts, baskets, and eco-bags, but the system is not limited to this; each user may place their purchased items in various containers. In this case, the group estimation unit 44 does not estimate the payer based on whether or not containers are shared, but estimates the payer based on other factors.

[0127] (Fourth variation) In the above embodiment, one payer is assigned to a group, and based on the payer's payment information, the payment processing for the total amount of all products acquired by multiple users belonging to the group is performed all at once. However, the system is not limited to this, and multiple payers may be assigned to a group, and payment processing may be performed based on the payment information of each payer. In this case, the total amount of all products acquired by multiple users belonging to the group is divided among the number of payers, and payment processing is performed for the divided amount based on the payment information of each payer. It should be noted that, in addition to dividing the amount, an arbitrary payment amount may be set separately for each payer. For example, if the total amount is 8,000 yen and users A and B are assigned as payers, the payment processing unit 49 may perform payment processing for 3,000 yen based on user A's payment information and payment processing for 5,000 yen based on user B's payment information.

[0128] (Fifth variation) In the above embodiment, the payer is a pre-registered member and the registrant information DB35 stores the payment information. However, the embodiment is not limited to this, and a user whose payment information is not stored can also be the payer. In this case, a self-checkout register that can communicate with the management server 1 is installed in the store 5, and the payer uses the self-checkout register to pay for all purchased items acquired by users belonging to the group in cash or other means. Alternatively, the user may transmit payment information such as a credit card number to the management server 1 by performing a predetermined operation on the mobile terminal 2 while shopping. In this case, the payment processing unit 49 performs payment for all purchased items acquired by users belonging to the group based on the received payment information.

[0129] (Sixth variation) In the above embodiment, when all users belonging to the group leave the store, the total amount of all purchased items acquired by the users belonging to the group is settled all at once. However, the timing of settlement is not limited to this, and the settlement processing unit 49 may settle the price of the purchased items acquired by companion A when companion A leaves the store, and the price of the purchased items acquired by companion B when companion B leaves the store. In other words, the settlement processing unit 49 may settle the price of the purchased items acquired by a user each time they leave the store, based on the settlement information of the payer.

[0130] (Seventh variation) In the above embodiment, payment processing is performed when all users belonging to the group have left the store. In this case, regardless of whether the user who left was the payer or a companion, if even one user belonging to the group remains in store 5, payment processing will not be performed and shopping can continue. However, the management server 1 is not limited to this, and may also perform payment processing based on the payer's payment information when the payer leaves the store. In this case, companions will not be able to shop in store 5 once the payer leaves.

[0131] Furthermore, if a flapper gate capable of communicating with the management server 1 is installed at exit 18, the user's departure may be restricted until the payment process is complete.

[0132] (Variation 8) In the above embodiment, the presentation unit 45 presents the estimated group results and the estimated decision-makers together to the user as a group confirmation screen. However, it is not limited to this, and the estimated group results and the estimated decision-makers may be presented to the user separately. For example, the presentation unit 45 may present the estimated decision-makers to the decision-makers and the estimated group results to some or all of the users belonging to the group.

[0133] (9th variation) A list of purchased items, showing the items acquired by each user, can be displayed at any time requested by the user. Specifically, the display unit 45 transmits the purchased item list information to the mobile terminal 2 when it receives a request for the purchased item list via the communication unit 32 from the mobile terminal 2 used by the user through a predetermined operation. By displaying the purchased item list based on the purchased item list information received by the mobile terminal 2, the user can easily confirm their purchased items.

[0134] Furthermore, in response to a request for a list of purchased items, the purchased item list information may be sent to the shared terminal. This allows users who do not own the mobile terminal 2 to easily check their purchased items by displaying the purchased item list based on the information received by the shared terminal.

[0135] (10th variation) Electronic shelf labels that can communicate with the management server 1 may be provided on the product shelf 14. In this case, the display unit 45 may refer to the purchased product list DB 38 when a user takes a product from the product shelf 14 and display the product names and quantities of products already purchased by the user on the electronic shelf label of the purchased product. The display unit 45 may also display the product names and quantities of products already purchased by other users who make up the group to which the user belongs, not just the user themselves. Furthermore, the display unit 45 may also display the total amount for all members of the group, including the product taken out, for example, "The current total amount is 695 yen." This makes it convenient for the user to check the products already purchased by other users in the group and the total amount for all members of the group at the time they take out a product.

[0136] <Second Embodiment> A second embodiment of this disclosure will be described below with reference to the drawings. As with the first embodiment, this embodiment will describe a specific example of applying the present invention to a store as an example of a facility.

[0137] [Facility Management System Configuration] The configuration of the facility management system 100x according to this embodiment is the same as that of the first embodiment, so for convenience, a detailed explanation will be omitted. However, in this embodiment, the facility management system 100x is equipped with shared terminals that can be used by users near the exit 18 and on the carts. In addition, electronic shelf labels are provided on the product shelves 14. The shared terminals and electronic shelf labels are connected to the server 1x in a way that allows them to communicate with it.

[0138] [Device configuration] Next, the configuration of the management server 1x will be described with reference to Figure 14. Figure 14 shows the schematic configuration of the management server 1x. The management server 1x mainly comprises a storage unit 31, a communication unit 32 for data communication, and a control unit 33x. These elements are interconnected by a bus line 30. Note that the storage unit 31 and the communication unit 32 are the same as in the first embodiment, so their description will be omitted.

[0139] The control unit 33x includes a CPU, ROM, RAM, etc. (not shown), and performs various controls on each component in the management server 1. In this embodiment, the control unit 33x includes a person identification unit 40, a tracking unit 41, a relevance calculation unit 42, a linking unit 43, a group estimation unit 44, a presentation unit 45, a reception unit 46, a group setting unit 47, a payer setting unit 48, a payment processing unit 49, and a warning processing unit 50.

[0140] Here, we will describe the security processing unit 50, which differs from that of the first embodiment. The security processing unit 50 sets users whose payment information is not associated with them as targets for security and executes security processing on these targets. Specifically, whether a user is shopping alone or in a group, if their payment information is stored in the storage unit 31, that user will not be designated as a target for security. On the other hand, if a user enters the store and shops alone, and their payment information is not stored in the storage unit 31, that user will be designated as a target for security.

[0141] Furthermore, if multiple users enter the store and shop as a group, even if a user's payment information is not stored in the storage unit 31, if a payer is set for the group to which the user belongs, that user will not be designated as a person under surveillance. On the other hand, even if multiple users enter the store, if a user does not belong to a group, or if a payer is not set for the group to which the user belongs, that user will be designated as a person under surveillance. In other words, if the payment information for the price of the purchased goods acquired by a user at store 5 is unknown, the surveillance processing unit 50 will designate that user as a person under surveillance and execute surveillance processing.

[0142] The specific security procedures will now be explained. First, the security processing unit 50 notifies the person under security of a message prompting them to register their payment information or join a group with a payer. Since the person under security is an unregistered person whose payment information is not stored, their contact information is unknown. Therefore, the security processing unit 50 notifies, for example, a shared terminal or electronic shelf label located near the person under security of a message to the effect of, "Your payment information is unknown, and you do not appear to belong to any group. Please register your payment information or join a group with a payer."

[0143] The security processing unit 50 may also use a spotlight to display a message on a screen on the floor. Alternatively, the security processing unit 50 may use a security robot or drone to display the message via screen or audio.

[0144] At this time, a user who is registered but has been designated as a person under surveillance will perform an action such as clearly capturing their own face on a shared terminal. As a result, the management server 1 can identify the user as a registered user based on the high-resolution facial image data of that user.

[0145] Furthermore, once the security processing unit 50 has set a person to be monitored, it presents the person's facial image data to the store employee. Specifically, the security processing unit 50 transmits the person's facial image data and tracking data to a mobile terminal used by the store employee or a shared terminal in the back room that is only accessible to the store employee. Based on the person's facial image data and tracking data, the store employee monitors the person's movements and takes appropriate action, such as speaking to them, if necessary.

[0146] Furthermore, the security processing unit 50 may, if necessary, provide information about the person under surveillance to users present within the store 5. For example, if suspicious behavior is detected in the person under surveillance, such as intentionally trying to belong to the same group as a trustworthy user, the security processing unit 50 will provide the user with a message urging them to be cautious.

[0147] Furthermore, if a flapper gate capable of communicating with the management server 1 is installed at exit 18, the security processing unit 50 will restrict the departure of a person under surveillance who has acquired purchased goods. If a person under surveillance who has acquired purchased goods leaves store 5, the security processing unit 50 will notify the security room and the police.

[0148] Furthermore, if the person under surveillance does not possess payment information such as a credit card or bank account, the security processing unit 50 may allow them to pay for their purchases at a self-checkout counter, thereby removing them from the security setting and enabling them to leave the store. In addition, if a person under surveillance who does not possess payment information approaches the vicinity of Exit 18 or a flapper gate, the security processing unit 50 may guide them to pay at a self-checkout counter using signage or other means.

[0149] [Group payment processing] Next, we will explain the overview of group payment processing by the facility management system 100x. Group payment processing is the process of setting up a group and a payer for users who enter store 5. Furthermore, group payment processing is the process of settling all purchases made by users belonging to the group based on the payer's payment information. In addition, in group payment processing by the facility management system 100x, users who are not linked to payment information are set as persons to be monitored, and a monitoring process is executed.

[0150] Figure 15 is a flowchart illustrating the overview of the group settlement process according to the second embodiment. The group settlement process includes a product linking process S200 and a group setting process S300. The product linking process S200 and the group setting process S300 are the same as in the first embodiment, so their explanation is omitted for convenience. The group settlement process is mainly implemented by the management server 1 executing a pre-prepared program.

[0151] When a user enters store 5, the management server 1 identifies the user who entered based on image data acquired from camera 10 located near entrance 16 (step S400). The management server 1 then determines whether the identified user is a registered user or not (step S401). If the user is a registered user (step S401; Yes), the management server 1 generates new customer information including the registration ID and name read from the registered user information DB 35, the user's facial image data, and an exit flag (n), and stores it in the customer information DB 37 (step S402). On the other hand, if the user is not a registered user (step S402; No), the management server 1 issues a temporary ID, generates new customer information including the temporary ID, the user's facial image data, and an exit flag (n), and stores it in the customer information DB 37 (step S403).

[0152] Then, the management server 1 uses the user image data acquired from the camera 10 to track each user's movements within the store and acquires it as tracking data (step S404). Furthermore, the management server 1 performs product linking processing (step S200). Once the product linking processing is complete, the management server 1 performs group setting processing (step S300).

[0153] Once the group setting process is complete, the management server 1 determines whether the user is attempting to leave the store based on image data acquired from the camera 10 located near the exit 18 (step S405). If the user is not attempting to leave the store (step S405; No), the management server 1 returns to the process in step S404 and, depending on the user's actions within the store 5, performs user tracking, user-to-product linking, group estimation, and setting as needed.

[0154] On the other hand, if the user is about to leave the store (step S405; Yes), the management server 1 refers to the purchased product list DB38 and determines whether the user has retrieved the purchased products (step S406). If the user has not retrieved the purchased products (step S406; No), the management server 1 proceeds to step S408. On the other hand, if the user has retrieved the purchased products (step S406; Yes), the management server 1 refers to the customer information DB37 and determines whether the user is a person under suspicion (step S407).

[0155] If the user is a person under suspicion (step S407; Yes), the management server 1 performs a warning process (step S410). For example, as a warning process, the management server 1 may notify the person under suspicion of a message prompting them to register payment information or join a group with other payers. The management server 1 may also present the face image data of the person under suspicion to the store staff. Furthermore, if a flapper gate is installed at exit 18, the management server 1 may restrict the departure of the person under suspicion who has acquired purchased goods. After performing the warning process, the management server 1 returns to the process in step S405.

[0156] On the other hand, if the user is not a person under suspicion (step S407; No), the management server 1 refers to the exit flag in the customer information DB 37 and determines whether all users constituting the group have left the store (step S408). If not all users constituting the group have left the store, that is, if at least one user belonging to the group is still in store 5 (step S408; No), the management server 1 returns to the process in step S404 and, depending on the user's actions within store 5, tracks the user, links the user to products, and estimates and sets up groups as needed.

[0157] On the other hand, if all users constituting a group leave the store (step S408; Yes), the management server 1 refers to the registered user information DB 35 and uses the payment information of the payer to settle all purchased items acquired by each user constituting the group (step S409). Specifically, the payment processing unit 49 authenticates the payer using biometric authentication and settles the payment based on the payment information of the authenticated payer. If a user enters the store alone and does not belong to a group, the management server 1 considers that all members of the group have left the store when that user leaves. With this, the group settlement process according to the second embodiment is completed.

[0158] In the group payment process described above, the management server 1 performs a security check when the person under security who has acquired the purchased items attempts to leave the store. However, it is not limited to this, and can perform any security check at any time while the person under security is inside store 5.

[0159] As described above, the facility management system 100x of this embodiment sets users who have entered the facility but are not linked to payment information as persons to be monitored and executes a warning process as needed. The facility management system 100x is intended for unmanned and labor-saving AI facilities, and does not require prior membership registration, allowing groups of users who do not have payment information to shop. However, on the other hand, there is a high need to prevent fraudulent activities by malicious users. Therefore, executing a warning process on persons to be monitored acts as a deterrent against fraudulent activities such as shoplifting or having other users make payments, thereby strengthening the security function.

[0160] <Third Embodiment> Figure 16 shows the configuration of an information processing device according to the third embodiment of this disclosure. The information processing device 80 includes a person identification means 81 for identifying a person who has entered a facility, a group estimation means 82 for estimating a group composed of multiple people, a group presentation means 83 for presenting the group estimation result to the person, and a group setting means 84 for setting the group based on the response to the group estimation result.

[0161] Figure 17 is a flowchart of the processing performed by the information processing device 80. Person identification means 81 identifies a person who has entered the facility (step S81). Group estimation means 82 estimates a group consisting of multiple people (step S82). Group presentation means 83 presents the group estimation result to the person (step S83). Group setting means 84 sets a group based on the response to the group estimation result (step S84).

[0162] According to the third embodiment, multiple individuals who enter the facility can be grouped together, enabling management and payment on a group basis.

[0163] In addition, some or all of the above embodiments (including modifications, the same applies hereinafter) may also be described as follows, but are not limited to the following.

[0164] (Note 1) A means of identifying individuals who enter the facility, A group estimation method for estimating a group composed of multiple individuals, A group presentation means for presenting the estimation results of the group to the person, A group setting means for setting the group based on the response to the estimation result of the group, An information processing device equipped with the following features.

[0165] (Note 2) The person identification means acquires a facial image from an image captured by a camera installed inside or outside the facility, and identifies the person based on the facial image. The group presentation means is an information processing device according to Appendix 1 that presents estimation results including facial images of individuals presumed to belong to the group.

[0166] (Note 3) The facility further comprises a correlation calculation means for calculating the degree of correlation between multiple persons within the facility, The group estimation means estimates the group based on the degree of relevance, The information processing device described in Appendix 2 calculates the degree of relevance based on one or more of the following data: the distance between people in the image, the orientation of their faces, the opening and closing of their mouths, their physical movements, and whether or not they share a container for carrying goods.

[0167] (Note 4) The system includes a storage means for storing attribute information of the aforementioned person, The correlation calculation means is an information processing device according to Appendix 3 that calculates the correlation using the attribute information.

[0168] (Note 5) A means for estimating the decision-maker from among the individuals constituting the aforementioned group, A means for presenting the estimated result of the decision-maker to the person, A setter setting means for setting the setter based on the response to the estimated result of the setter, A payment processing means that, based on the payment information of the aforementioned payer, collectively processes the payments for all products acquired by each person constituting the group, An information processing device according to any one of the appendices 1 to 4, comprising:

[0169] (Note 6) The payment person estimation means is an information processing device according to Appendix 5, which acquires a facial image from an image captured by a camera installed inside or outside the facility, and estimates the payment person based on one or more of the following data: the estimated age of the person included in the image, the order in which they entered the store, their gender, and whether or not they have a container to carry the acquired goods.

[0170] (Note 7) The group estimation means is an information processing device according to any one of the appendices 1 to 6, which estimates that persons included in images captured by a camera installed near the exit of the facility belong to the same group.

[0171] (Note 8) The group presentation means is an information processing device according to any one of the appendices 5 to 7, which presents the estimated results of the group to the decision-maker.

[0172] (Note 9) The group presentation means presents, as an estimated result of the group, a list associating the persons constituting the group with the goods acquired by each person and the total amount of the goods acquired by each person.

[0173] (Note 10) The information processing device according to any one of the appendices 5 to 9, further comprising a warning processing means for issuing a warning to the person if the person does not possess payment information and does not belong to a group that includes a payer.

[0174] (Note 11) Identify the person who entered the facility, We estimate that the group consists of multiple individuals. The estimation results of the aforementioned group were presented to the aforementioned person, An information processing method for setting up the group based on the response to the estimation result of the group.

[0175] (Note 12) Identify the person who entered the facility, We estimate that the group consists of multiple individuals. The estimation results of the aforementioned group were presented to the aforementioned person, A recording medium containing a program that causes a computer to perform a process of setting up the group based on the response to the estimation result of the group.

[0176] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the structure and details of the present invention can be made that are understandable to those skilled in the art within the scope of the present invention. That is, the present invention naturally includes the full disclosure, including the claims, and various modifications and alterations that those skilled in the art could make in accordance with the technical idea. Furthermore, each disclosure of the above-mentioned patent documents and other references is incorporated herein by reference. [Explanation of Symbols]

[0177] 1. Management Server 2 Mobile devices 3 Network 5 stores 10 Cameras 14 Product shelf 16 Entrance 18 Exit 20, 22, 24 users 100 Facility Management Systems

Claims

1. A means of identifying individuals who enter the facility, A group configuration means for configuring a group consisting of multiple users, Estimation means for estimating the decision-maker and companions of the group based on the user's state or behavior recognized from images captured by cameras installed within the facility, A transmission means for transmitting information about the payer and accompanying persons of the group, and information about the products acquired by the payer and accompanying persons, to the terminal of a user belonging to the group. An information processing device equipped with the following features.

2. The information processing apparatus according to claim 1, further comprising a payment processing means for settling the payment for the product based on approval information from the terminal.

3. The information processing device according to claim 1 or claim 2, further comprising a setter setting means for changing the setter based on instructions entered by a user.

4. The information processing device according to claim 1, wherein the group setting means changes the group based on instructions entered by the user.

5. A group estimation means for estimating a group composed of multiple users, A group presentation means for presenting the estimated results of the group to the user, Equipped with, The information processing apparatus according to any one of claims 1 to 4, wherein the group setting means sets the group based on the response to the estimation result of the group.

6. The person identification means acquires a facial image from images captured by a camera installed inside or outside the facility, and identifies the user based on the facial image. The information processing device according to claim 5, wherein the group presentation means presents estimation results including facial images of persons presumed to belong to the group.

7. The facility further comprises a correlation calculation means for calculating the degree of correlation between multiple users within the facility, The group estimation means estimates the group based on the degree of relevance, The information processing apparatus according to claim 6, wherein the correlation calculation means calculates the correlation based on one or more data points included in the image, such as the distance between users, the orientation of their faces, the opening and closing of their mouths, physical movements, and whether or not they share a container for carrying goods.

8. The system includes a storage means for storing the attribute information of the user, The information processing apparatus according to claim 7, wherein the relatedness calculation means calculates the relatedness using the attribute information.

9. A method of information processing performed by a computer, Identify users who have entered the facility, Set up a group consisting of multiple users, Based on the user's state or behavior recognized from images captured by cameras installed within the facility, the decision-maker and accompanying persons of the group are estimated. An information processing method for transmitting information about the payer and accompanying persons of the group, and information about the products acquired by the payer and accompanying persons, to the terminal of a user belonging to the group.

10. Identify users who have entered the facility, Set up a group consisting of multiple users, Based on the user's state or behavior recognized from images captured by cameras installed within the facility, the decision-maker and accompanying persons of the group are estimated. A program that causes a computer to execute a process to transmit to the terminals of users belonging to the aforementioned group information about the payer and their companions, and information about the products acquired by the payer and their companions.

Images