Information processing device, information processing method, and program

Abstract

To provide a facility management system for automating or saving labor of facility operation.SOLUTION: In an information processing device, person specification means specifies a person who enters a facility. Group estimation means estimates a group composed of a plurality of persons. Group presentation means presents an estimation result of the group to a person. Group setting means sets a group on the basis of a response to the estimation result of the group.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present disclosure relates to the technical field of facility management systems that automate or reduce the labor required for facility operations. [Background technology]

[0002] In recent years, unmanned and labor-saving AI (Artificial Intelligence) stores have been introduced in supermarkets, convenience stores, etc. Due to the recent spread of infectious diseases, the demand for unmanned and labor-saving AI stores is expected to increase further in the future.

[0003] Patent Documents 1 to 3 are cited as examples of technologies for automating or labor-saving store operations. Patent Document 1 describes an item estimation device that automatically identifies an item that a person has taken from a shelf and associates the person with the item. Patent Document 2 describes a group estimation device that estimates a group including at least some of multiple targets based on gaze information or the like. Patent Document 3 further describes a shopping management device for online shopping in which a group of multiple users shop and one designated user pays in a lump sum. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. WO2020 / 179480 [Patent Document 2] JP 2020-091622 A [Patent Document 3] JP 2015-099504 A Summary of the Invention [Problem to be solved by the invention]

[0005] This disclosure is intended to improve upon the techniques disclosed in the prior art documents. [Means for solving the problem]

[0006] In one aspect of the present disclosure, an information processing device includes: A person identification means for identifying a user who has entered the facility; A group setting means for setting a group composed of a plurality of users; a transmission means for transmitting information on the payer and accompanying person of the group and information on products acquired by the payer and accompanying person to terminals of users belonging to the group; Equipped with.

[0007] In another aspect of this disclosure, a computer-implemented information processing method includes: Identifying users who have entered the facility; Set up a group consisting of multiple users, Information on the payer and accompanying person of the group and information on the products acquired by the payer and accompanying person are transmitted to the terminals of the users belonging to the group.

[0008] In another aspect of this disclosure, the program includes: identifying a user who has entered the facility; Set up a group consisting of multiple users, The computer executes a process of transmitting information on the payer and accompanying person of the group, and information on products acquired by the payer and accompanying person, to terminals of users belonging to the group. [Brief description of the drawings]

[0009] [Figure 1] 1 shows the configuration of a facility management system according to a first embodiment. [Diagram 2] 2 shows a schematic configuration of a management server. [Diagram 3] 13 is an example of a data configuration of a registrant information DB. [Figure 4] 1 is an example of a data configuration of a product information DB. [Diagram 5] 13 is an example of a data configuration of a store visitor information DB. [Figure 6] 13 is an example of a data configuration of a purchase product list DB. [Figure 7] FIG. 2 is a schematic diagram showing the state of a store 5. [Figure 8] 13 is a display example of a group confirmation screen. [Figure 9] 13 is a display example of a purchase product list screen. [Figure 10] 2 shows a schematic configuration of a mobile terminal. [Figure 11] 13 is a flowchart of a group payment process according to the first embodiment. [Figure 12] 13 is a flowchart of a product linking process. [Figure 13] 13 is a flowchart of a group setting process. [Figure 14] 1 shows the configuration of a facility management system according to a second embodiment. [Figure 15] 13 is a flowchart of a group payment process according to the second embodiment. [Figure 16] 13 shows a configuration of an information processing device according to a third embodiment. [Figure 17] 13 is a flowchart of a process according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] First Embodiment Hereinafter, a first embodiment of the present disclosure will be described with reference to the drawings. [Facility management system configuration] 1 shows the configuration of a facility management system 100 according to this embodiment. The facility management system 100 is a system that manages facilities, such as stores, airports, stations, amusement parks, event venues, companies, etc. In this embodiment, a specific example of the application of the present application to a store will be described as an example of a facility.

[0011] The facility management system 100 mainly comprises a management server 1, a mobile terminal 2 used by a user, a number of cameras 10 installed inside or outside a store 5, product shelves 14, an entrance 16, and an exit 18. The entrance 16 and exit 18 may or may not have a gate.

[0012] The management server 1 and the camera 10 are connected by at least one of a wired connection and a wireless connection via any network 3 such as a local area network or the Internet. The management server 1 and the camera 10 may be directly connected. The management server 1 and the mobile terminal 2 are connected to be able to communicate with each other via any network 3 such as the Internet.

[0013] The facility management system 100 first estimates the group and payer for a user who enters the store 5, and presents the estimation result to the user. The facility management system 100 then sets the group and payer based on the user's response to the estimation result. Furthermore, the facility management system 100 makes a single payment for the total amount of all products acquired by users belonging to the group, based on the payment information of the payer. Here, a group is a collection of users who shop together, such as family or friends. Also, a payer is a person who pays for all products acquired by users belonging to the group.

[0014] A user enters the store alone or with other users, acquires a product to be purchased (hereinafter also referred to as a "purchased product") from among the products displayed on the product shelf 14, and places it in a cart. Specifically, the user 20 shown in FIG. 1 is a man in his 30s who enters the store with his wife, user 22, and daughter, user 24. A cart is a container used by a user who is shopping in the store 5 when moving around with purchased products, and multiple users belonging to the same group share one cart. In this disclosure, the user 20, user 22, and user 24 are a family and belong to one group. In addition, a user who pays for the purchased product is called a payer, and users other than the payer who belong to the group are called companions. In the example of FIG. 1, the user 20 is the payer, and the users 22 and 24 are companions.

[0015] The camera 10 is an imaging device and includes an image sensor such as a Charge Coupled Device (CCD) sensor or a Complementary Metal Oxid Semiconductor (CMOS) sensor.

[0016] The cameras 10 are provided at multiple locations inside or outside the store, such as near the entrance 16 of the store 5, a place where products are displayed in the store 5 (hereinafter also referred to as the "sales floor"), near the exit 18 of the store 5, and the parking lot of the store 5. The camera 10 provided near the entrance 16 captures an image of a user entering the store. The camera 10 provided in the sales floor captures an image of users and products present in the sales floor. The camera 10 provided near the exit 18 captures an image of a user leaving the store. The camera 10 provided in the parking lot captures an image of the car in which the user is riding, passengers, and the user's riding position. The store 5 is provided with a number and arrangement of cameras 10 capable of capturing an image of users and products present anywhere in the store 5. The camera 10 transmits image data including the image acquired by capturing the image to the management server 1.

[0017] Based on image data acquired from camera 10, management server 1 identifies users who have entered the store, identifies the purchased items acquired by each user, estimates the group, and estimates the payer. Then, management server 1 presents the estimation results to the users, and sets the group and the payer based on the user's response to the estimation results. Furthermore, management server 1 performs payment all at once for the total amount of all purchased items acquired by users belonging to the group based on the payment information of the payer.

[0018] The management server 1 is realized on a single computer or on multiple computers. The management server 1 may be installed in a store 5. The management server 1 may also be realized on a cloud, which is a collection of computer resources.

[0019] The mobile terminal 2 is a portable terminal device such as a smartphone or tablet used by a user who is shopping at the store 5. The mobile terminal 2 has a function of receiving the estimation results of the group and the payer from the management server 1 and displaying them as a group confirmation screen. The mobile terminal 2 also has a function of transmitting to the management server 1, as the user's response to the estimation results, operations such as adding or removing a companion or changing the payer operated by the user on the group confirmation screen. Note that, for convenience of explanation, only the mobile terminal 2 used by the user 20 is shown in FIG. 1, but other users may have mobile terminals 2 in the store 5.

[0020] [Device configuration] Next, the configurations of the management server 1 and the mobile terminal 2 will be described with reference to FIGS.

[0021] (Management Server) 2 shows a schematic configuration of the management server 1. The management server 1 mainly comprises a storage unit 31, a communication unit 32 that performs data communication, and a control unit 33. These elements are connected to each other by a bus line 30.

[0022] The storage unit 31 is configured with a memory such as a hard disk or a flash memory. The storage unit 31 stores a program executed by the control unit 33, and information necessary for the control unit 33 to execute a predetermined process by executing the program. In this embodiment, the storage unit 31 includes a registrant information DB (Database) 35, a product information DB 36, a store visitor information DB 37, and a purchased product list DB 38. The storage unit 31 may store various types of information related to store management other than the registrant information DB 35, the product information DB 36, the store visitor information DB 37, and the purchased product list DB 38.

[0023] The registrant information DB35 is a database of information on users (hereinafter also referred to as "registrants") who have registered as members of the store 5. FIG. 3 shows an example of the data configuration of the registrant information DB35. As shown in the figure, the registrant information DB35 stores registrant information such as registration ID, name, contact information, payment information, personal characteristic information, attribute information, etc. Registration as a member of the store 5 can be performed by any method.

[0024] The registration ID is identification information for identifying the registrant. The name is the name of the registrant. The contact information is information about contacts required for communicating with the registrant, such as the email address of the mobile terminal 2 used by the registrant. The payment information is information used for payment processing of purchased items, such as the credit card number or bank account number of the registrant. The personal characteristic information is information for determining whether the user who entered the store is a registrant or not, and for identifying who the user is if the user is a registrant, using image data of the user who entered the store. The personal characteristic information may be the image data itself of the area including the user's face, or may be a feature calculated from the image data of the area including the user's face. The attribute information is information indicating the relationship between the registrants, such as "family" or "friend". For example, if the registrant of the registration ID "M001" and the registrant of the registration ID "M002" are a married couple, information such as "M002 and family" is stored as the attribute information of the registration ID "M001".

[0025] The product information DB36 is a database of information related to products handled in the store 5. FIG. 4 is an example of the data configuration of the product information DB36. As shown in the figure, the product information DB36 stores product information such as a product ID, a product name, a price, and product feature information. The product ID is identification information for identifying a product, and may be, for example, a JAN (Japanese Article Number) code. The product name is the name of the product, and the price is the price of the product. The product feature information is information for identifying what product the user has purchased by using image data of an area including the product. The product feature information may be the image data itself of the area including the product, or may be a feature calculated from the image data of the area including the product.

[0026] The store visitor information DB37 is a database of information about users who are present in the store 5. FIG. 5 is an example of the data configuration of the store visitor information DB37. As shown in the figure, the store visitor information DB37 stores information about users who are present in the store 5, such as a registration ID, a temporary ID, a name, a type, a person's image information, a store exit flag, and a group ID, as store visitor information. The store visitor information, which will be described in detail later, is generated when a user enters the store, and is updated as needed in response to changes in the group composition, etc., as long as the user and the group to which the user belongs are present in the store 5.

[0027] The registration ID is a registration ID issued when the user who entered the store registered as a member, and is stored when the user is a registered user. The temporary ID is identification information temporarily set to identify the user who entered the store, and is issued and stored when the user is an unregistered user. The name is the name of the user who entered the store, and is stored by referring to the registered user information DB 35 when the user is a registered user. The type is information indicating whether the user who entered the store is a payer, an escort, or an unknown undecided payer or escort. The person image information is image data of an area including the face of the user who entered the store (hereinafter, also referred to as "face image data"). The exit flag is a flag indicating whether the user has left the store 5 (left the store (y)) or not (not left the store (n)). The exit flag is set to "not left the store (n)" at the time when the store visitor information is generated. In addition, when the user leaves the exit 18 of the store 5, the user is considered to have left the store, and the exit flag is set to "left the store (y)". The group ID is identification information for identifying the group to which the user belongs.

[0028] If the user comes to the store alone and does not belong to a group, the group ID may be left blank, or the user may be considered to be in a group of one. If there are multiple candidate groups to which the user belongs and it is not possible to identify one group, the group ID may store multiple candidate group IDs. In this case, it is desirable to store the group IDs in order of the probability that the user belongs to them.

[0029] Purchased item list DB38 is a database of information about purchased items acquired by the user at store 5. Fig. 6 shows an example of the data configuration of purchased item list DB38. As shown in the figure, purchased item list DB38 stores purchased item list information such as purchaser information, purchased item ID, item name, price, quantity, subtotal, group ID, and total amount. Note that, as will be described in detail later, the purchased item list information is generated when the user acquires a purchased item, and is updated as needed as long as the user and the group to which the user belongs are present in store 5.

[0030] The purchaser information is information indicating the user who removed and acquired the purchased item from the product shelf 14, and is composed of a registrant ID and a temporary ID. If the user is a registrant, the registrant ID is stored as the purchaser information, and if the user is unregistered, the temporary ID is stored. The purchased item ID is the item ID of the purchased item. The item name is the item name of the purchased item, and the price is the price of the purchased item. The item name and price are stored by referring to the item information DB 36 based on the item ID of the purchased item. The quantity is the number of items purchased. The subtotal is the total amount of the prices of the purchased items acquired by each user. The group ID is the group ID of the group to which the user belongs. The total amount is the total amount of the prices of all purchased items acquired by users belonging to the group.

[0031] In addition, if a user visits the store alone and does not belong to a group, the group ID and total amount may be left blank and nothing may be stored, or the user may be considered to be in a group of one person, and the group ID and total amount may be stored.

[0032] The communication unit 32 communicates with the mobile terminal 2 and the camera 10 via wired or wireless communication.

[0033] The control unit 33 includes a central processing unit (CPU), a read only memory (ROM), a random access memory (RAM), etc. (not shown), and performs various controls on each component in the management server 1. In this embodiment, the control unit 33 includes a person identification unit 40, a tracking unit 41, an association degree calculation unit 42, a linking unit 43, a group estimation unit 44, a presentation unit 45, a reception unit 46, a group setting unit 47, a settling party setting unit 48, and a payment processing unit 49.

[0034] The person identification unit 40 uses the image data of the user acquired from the camera 10 to identify the user who has entered the store. Specifically, the person identification unit 40 acquires image data of the user who has entered the store from the camera 10 installed near the entrance 16 of the store 5, and acquires the personal characteristic information of the user. At this time, the person identification unit 40 may acquire face image data from the image data of the user who has entered the store. Then, the person identification unit 40 compares the personal characteristic information of the user who has entered the store with the personal characteristic information of each registered person read from the registered person information DB 35 to identify whether the user who has entered the store is a registered person, and if so, who the user is. If the user who has entered the store is a registered person, the person identification unit 40 generates new store visitor information including the registration ID and name read from the registered person information DB 35, the user's face image data, and a store exit flag (n), and stores the information in the store visitor information DB 37. On the other hand, if the user who entered the store is not a registered user, the person identification unit 40 generates new store entry information including the issued temporary ID, the user's face image data, and a store exit flag (n), and stores it in the store visitor information DB37.

[0035] The tracking unit 41 tracks the movement of each user within the store using image data of the user acquired from the camera 10, and acquires the tracking data. Specifically, the movements of user 20, user 22, and user 24 tracked by the tracking unit 41 are shown by dotted lines in Fig. 1. When a user leaves through exit 18 of store 5, the tracking unit 41 considers the user to have left the store, and sets the store exit flag in the store visitor information DB 37 to "left the store (y)."

[0036] The relevance calculation unit 42 detects interactions using image data of users acquired from the camera 10 and tracking data by the tracking unit 41, and calculates relevance between multiple users based on arbitrary determination factors. Determination factors for calculating relevance include, for example, the distance between users, the direction of their faces, opening and closing of their mouths, physical movements, and whether or not they share a container for carrying products. The relevance calculation unit 42 calculates relevance based not only on information such as image data and tracking data in the store, but also on information such as image data and tracking data in the parking lot and in other stores, such as when users came in the same car or shopped together at other stores. The relevance calculation unit 42 calculates relevance at any time.

[0037] Here, the determination factor for calculating the relevance will be described in detail. The distance between users, which is the determination factor, is measured by recognizing the coordinates of each user in the real world based on the image data of the users. Specifically, the relevance calculation unit 42 acquires the physical distance between users measured by human body detection using machine learning and camera calibration as data regarding the distance between users.

[0038] The face direction, which is a determination factor, is detected by analyzing the user's gaze based on image data of the user. Specifically, the relevance calculation unit 42 recognizes the face direction of the user and what the user is looking at by detecting the face direction and gaze using machine learning, and obtains the registered ID or temporary ID of the user who is making eye contact as data related to the face direction. In addition, the relevance calculation unit 42 obtains the registered IDs or temporary IDs of multiple users who are looking at the same thing at the same time as data related to the face direction.

[0039] Furthermore, the relevance calculation unit 42 not only obtains that the users make eye contact at the same time, but also obtains the face direction and the number of times that the users turn their eyes to a specific user as data on the face direction. Specifically, the face direction and the number of times that the users turn their eyes to a specific user are, for example, the number of times that user A looks at user B, and the group estimation unit 44 described later may estimate that the users are in the same group when the face direction and the number of times that the users turn their eyes are equal to or greater than a predetermined number of times. The condition under which the group estimation unit 44 estimates that the users are in the same group can be set arbitrarily, for example, "when user A looks at user B more than five times," "when user A looks at user B more than three times, and when user B looks at user A more than three times," and so on.

[0040] The mouth opening and closing, which is a determination element, is detected by analyzing the user's mouth based on the image data of the user. Specifically, the relevance calculation unit 42 obtains the registered ID or provisional ID of the speaking user as data regarding the opening and closing of the mouth by detecting speech using machine learning.

[0041] The physical contact, which is a determination element, is detected by estimating a person's skeleton, posture, body shape, appearance, silhouette, etc. based on image data of the user, and detecting physical contact between users, such as folding arms, holding hands, etc. Specifically, the relevance calculation unit 42 acquires the registered ID or temporary ID of the user who is in physical contact as data related to the physical contact by pose estimation using machine learning.

[0042] The determination element of whether or not a container is shared is to detect whether or not a container such as a cart, basket, or eco bag for putting purchased items is shared by multiple users based on the image data of the users. Specifically, the relevance calculation unit 42 acquires the registered ID or temporary ID of the users who share the container as data.

[0043] The relevance calculation unit 42 may calculate the relevance from one determination element, or may calculate the relevance by combining a plurality of determination elements. For example, the relevance calculation unit 42 may detect users who are talking to each other based on two determination elements, such as "mouth opening / closing" and "face direction", and calculate the relevance. The relevance calculation unit 42 may also calculate the relevance by taking attribute information into account. The relevance calculation unit 42 may also calculate the relevance by using tracking data, not limited to image data.

[0044] The linking unit 43 identifies the user who has purchased a purchased item and the purchased item, using the image data of the user acquired from the camera 10. The linking unit 43 also links the identified user with the purchased item, and stores the linked item in the purchased item list DB38.

[0045] 7 is a schematic diagram showing the state of a store 5 according to the first embodiment. Product shelves 14 are installed in the store 5, and products are arranged on the product shelves 14. In addition, a camera 10 is provided near the product shelves 14 at a position capable of capturing an image of an area including the face of a user 20 and purchased products 51.

[0046] Specifically, the linking unit 43 acquires image data of an area including a product from the camera 10, and acquires product feature information of the purchased product 51 from the image data. The linking unit 43 detects that the purchased product 51 is taken out of the product shelf 14 using the product feature information of the purchased product 51. The detection of the purchase product 51 being taken out of the product shelf 14 is performed, for example, by detecting that the product feature information of the purchased product 51 moves from within the area corresponding to the product shelf 14 to outside (or from a contacting position to a non-contacting position) between a plurality of image data that are consecutive in time series. The area corresponding to the product shelf 14 in the image data may be recorded in advance as coordinates in the image data, or may be determined by image recognition of the product shelf 14 from the image data. The linking unit 43 may also detect the acquisition of the purchased product 51 by the user 20 by detecting that the purchased product 51 is put into a cart from the image data. The linking unit 43 compares the product characteristic information of the purchased product 51 with the product characteristic information read from the product information DB 36 to identify the purchased product 51 and acquire the product ID, product name, and price of the purchased product 51.

[0047] Furthermore, the linking unit 43 acquires facial image data of the user 20 who has taken out the purchased product 51 from the camera 10. The user 20 who has taken out the purchased product 51 is, for example, the user who is closest to the purchased product 51. The linking unit 43 compares personal characteristic information of the facial image data of the user 20 who has taken out the purchased product 51 with personal characteristic information of the facial image data read from the store visitor information DB 37 to identify the user 20 who has taken out the purchased product 51, and acquires the registered ID or temporary ID of the user 20.

[0048] Then, the linking unit 43 stores the registration ID or temporary ID of the user 20 who retrieved the purchased product 51, and the product ID, product name, price, and quantity of the purchased product 51 in the purchased product list DB 38. At this time, the linking unit 43 calculates a subtotal and stores it in the purchased product list DB 38. Furthermore, if the user belongs to a group, the linking unit 43 calculates the total amount and updates the purchased product list DB 38.

[0049] The method of identifying the user 20 who has taken out the purchased item 51 may be a method based on the position and movement of the hand obtained by acquiring image data of the area including the user's hand from the camera 10, or a method based on tracking data.

[0050] The group estimation unit 44 estimates a group consisting of multiple users based on image data acquired from the camera 10, attribute information, relevance, etc. For example, the group estimation unit 44 estimates that users whose relevance is equal to or greater than a threshold belong to the same group, taking into account attribute information. If there is no user whose relevance is equal to or greater than the threshold, the group estimation unit 44 estimates that the user is shopping alone. The group estimation unit 44 estimates the group at any time while the user is in the store 5.

[0051] In this way, by estimating a group based on the degree of association calculated by combining various determination elements, the management server 1 can improve the accuracy of group setting. Also, by taking into account attribute information stored before entering the store, the management server 1 can estimate and set a group more accurately.

[0052] Furthermore, the group estimation unit 44 estimates a payer from among the users who make up the group. When the group estimation unit 44 estimates that a user is shopping alone, it estimates that the user is the payer.

[0053] The group estimation method and the payer estimation method will be described in detail later.

[0054] The presentation unit 45 presents the result of the group estimation by the group estimation unit 44 to the user. The presentation unit 45 also presents the result of the payment person estimation by the group estimation unit 44 to the user. Specifically, the presentation unit 45 presents the estimation result to the user who has been estimated or set as the payment person of the group. Since the user who has been estimated or set as the payment person is a registrant who stores payment information, the presentation unit 45 transmits the estimation result to the mobile terminal 2 used by the user based on the contact information read from the registrant information 35. The presentation unit 45 transmits, for example, a group confirmation screen as the estimation result. The mobile terminal 2 displays the received group confirmation screen, and the user performs a predetermined operation, which will be described later, on the group confirmation screen.

[0055] In this way, by presenting the estimation result to the user who is estimated or set as the payer, the user who pays for the purchased item can add or remove other users who belong to the group.

[0056] Here, the group confirmation screen will be described. Fig. 8 is an example of the group confirmation screen. As shown in Fig. 8(a), the group confirmation screen has instructions 60 for the user, face image data of a user presumed to be the payer (hereinafter also referred to as "payer face image data") 61, face image data of a user presumed to be a companion (hereinafter also referred to as "companion face image data") 62 and 63, a correction button 64, add buttons 65 and 67, and remove buttons 66 and 68.

[0057] The payer face image data 61 is face image data of the user presumed to be the payer, and is read from the store visitor information DB 37. The correction button 64 is a button that is pressed when the payer is incorrect, as described in the instructions 60, to specify the correct payer. Specifically, after pressing the correction button 64, the user may specify the payer by inputting the name or registered ID of the correct payer, or may specify the payer by capturing an image of the correct payer with a camera built into the mobile terminal 2 and transmitting the face image data of the correct payer.

[0058] The companion face image data 62 and 63 are face image data of users estimated to be companions, and are read from the store visitor information DB 37. The add buttons 65 and 67 are buttons that are pressed when the users indicated by the companion face image data 62 and 63, respectively, are companions, as described in the instruction 60, to acknowledge that the users belong to the same group. On the other hand, the remove buttons 66 and 68 are buttons that are pressed when the users indicated by the companion face image data 62 and 63, respectively, are not companions, as described in the instruction 60, to reject the fact that the users belong to the same group and to remove the users. The presses of various buttons and predetermined operations by the user are transmitted from the mobile terminal 2 to the management server 1 as responses to the estimation results.

[0059] In this way, by presenting a group confirmation screen including the user's face image data as an inference result, the user can easily recognize the users who make up the group to which the user belongs. Therefore, the user can accurately respond to the inference result, and the management server 1 can improve the accuracy of group setting.

[0060] If the group confirmation screen does not include users shopping together, the user may specify users who belong to the same group by capturing images of users shopping together with a camera built into the mobile terminal 2 and transmitting the facial image data to the management server 1. Specifically, if user 24, who is the user's daughter, is not included in the group confirmation screen, which is the group estimation result, user 20 shown in FIG. 1 captures user 24 with a camera built into the mobile terminal 2 and transmits the facial image data of user 24 to the management server 1. Based on the facial image data of user 24, the management server 1 checks whether or not the user indicated by the facial image data exists in the store 5, and if so, sets user 24 to belong to the same group as user 20.

[0061] In this embodiment, a group confirmation screen as shown in Fig. 8(a) is presented as an inference result, but the screen configuration is not limited to this and is arbitrary as long as it allows the inference result to be presented and the user to respond. For example, as shown in Fig. 8(b), a purchased item list 70 based on the purchased item list information may be displayed so that the purchased items acquired by each user indicated by the payer face image data and the companion face image data can be seen. In this way, the user can check what each user has purchased and then decide whether to add them to the same group or exclude them.

[0062] Furthermore, the presentation of the estimation result by the presentation unit 45 is performed when the user enters the store, or at any timing when the composition of the group changes. Furthermore, the presentation unit 45 may present the estimation result not only to the user estimated or set as the payer of the group, but also to all users belonging to the group. Specifically, when a shared terminal that users can freely use is provided in the store 5, the presentation unit 45 can present the estimation result to all users by transmitting the estimation result to the shared terminal. Here, the shared terminal is a tablet terminal similar to the mobile terminal 2, and is connected to the management server 1 so as to be able to communicate with it.

[0063] The reception unit 46 receives a user's response to the inference result presented by the presentation unit 45 from the mobile terminal 2 used by the user via the communication unit 32. Specifically, the reception unit 46 obtains, as the response, information regarding correction of the inference result, user designation, addition, removal, and the like.

[0064] The group setting unit 47 sets a group based on the estimation result and the response to the estimation result. Then, the group setting unit 47 issues a group ID for identifying the group, and stores the group ID of each user belonging to the group in the store visitor information DB 37. For example, a hyphen is stored as the group ID of a user not belonging to a group. Furthermore, the group setting unit 47 stores the group ID of each user belonging to the group in the purchased item list DB 38, and calculates and stores the total price of all purchased items acquired by users belonging to the same group.

[0065] The settler setting unit 48 sets the settler based on the estimation result and the response to the estimation result. Then, the settler setting unit 48 stores the type of the user who is the settler as the settler and the type of the accompanying user as the accompanying person in the store visitor information DB 37.

[0066] When all users belonging to the group leave the exit 18 of the store 5, the payment processing unit 49 refers to the registrant information DB 35 and collectively performs payment processing for all purchased items acquired by each user belonging to the group using the payment information of the payer. The payment processing is a process of paying the price using the payment information. Specifically, the payment processing unit 49 authenticates the payer using biometric authentication and performs payment processing based on the payment information of the authenticated payer. Here, examples of biometric authentication include authentication using a face image, iris, fingerprint, gait, voice, etc.

[0067] The presenting unit 45 may transmit the purchased item list information to the mobile terminal 2 used by the payer via the communication unit 32 before the payment processing by the payment processing unit 49 is performed. FIG. 9 is an example of a purchased item list screen. Specifically, as shown in FIG. 9(a), the purchased item list screen includes the payer's face image data and the companion's face image data, and displays the product name, quantity, price, and subtotal of the purchased items acquired by each user indicated by the face image data, the total amount of all the purchased items acquired by each user, and an approval button 72. In this case, when the user presses the approval button 72, the payment processing unit 49 obtains information from the mobile terminal 2 via the communication unit 32 that the purchased item list has been approved, and performs payment for all the purchased items. In other words, the payment processing unit 49 may perform payment processing after the user confirms the purchased items and approves the payment.

[0068] Furthermore, the purchased item list screen is not limited to one that includes the payer's facial image data and companion's facial image data, and the screen configuration can be arbitrary as long as it shows the correspondence between the users who make up the group and the purchased items acquired by each user, as shown in Figure 9(b).

[0069] In the above configuration, the memory unit 31 is an example of a memory means, the person identification unit 40 is an example of a person identification means, and the relevance calculation unit 42 is an example of a relevance calculation means. Furthermore, the group estimation unit 44 is an example of a group estimation means and a settler estimation means, the presentation unit 45 is an example of a group presentation means and a settler presentation means, the group setting unit 47 is an example of a group setting means, and the settler setting unit 48 is an example of a settler setting means. Furthermore, the payment processing unit 49 is an example of a payment processing means, and the alert processing unit 50 is an example of an alert processing means.

[0070] (Mobile devices) 10 shows a schematic configuration of the mobile terminal 2. The mobile terminal 2 mainly includes a display unit 91, an input unit 92, a storage unit 93, a communication unit 94, and a control unit 95. These elements are connected to each other via a bus line 90.

[0071] The display unit 91, under the control of the control unit 95, displays various types of information such as a group confirmation screen and a purchased product list screen.

[0072] The input unit 92 is an interface that accepts user input through a predetermined operation on the group confirmation screen or the purchased product list screen, and corresponds to, for example, a touch panel, a button, a voice input device, or the like.

[0073] The storage unit 93 is configured with a memory such as a hard disk or a flash memory. The storage unit 93 stores programs executed by the control unit 95 and information necessary for the control unit 95 to execute predetermined processes by executing the programs. For example, the storage unit 93 may store a dedicated application program that is activated when shopping at the store 5 and that controls the display of various screens such as a group confirmation screen and a purchased item list screen.

[0074] The communication unit 94 communicates with the management server 1 by wireless communication. The control unit 95 includes a CPU, a ROM, a RAM, etc. (not shown), and performs various controls on the components in the mobile terminal 2.

[0075] [Group estimation method and settlement person estimation method] Next, the group estimation method and the payer estimation method will be described.

[0076] (Method of group estimation based on user behavior before entering a store) Users who are family members such as a husband and wife or a parent and child store this fact as attribute information in the registrant information DB 35 before entering the store. In this case, the group estimation unit 44 estimates a group of users present in the store 5 based on the attribute information read from the registrant information DB 35. It is also possible to link accounts at the time of member registration within a family without using attribute information. Also, if a user is a family member but is unable to register as a member because he or she is a minor and does not have an account, a facial photo of the user may be registered in advance and stored in the registrant information DB 35.

[0077] The group estimation unit 44 may calculate the degree of association based on image data acquired from a camera 10 installed in a parking lot outside the store, and estimate a group of users present in the store 5. Specifically, the group estimation unit 44 estimates, based on the image data, that users who are in the same car are in the same group.

[0078] When the store 5 is a facility of a large shopping mall, the group estimation unit 44 may estimate a group set in another store in the shopping mall as a group of users present in the store 5. In this case, the registered user information DB35 and the store visitor information DB37 are shared between the stores installed in the shopping mall.

[0079] Store visitor information and purchased item list information when a user has previously visited the store 5 may be stored as history information in the storage unit 31, and the group estimation unit 44 may calculate the degree of association based on the history information and estimate the group of users present in the store 5. Specifically, since users who entered the store together last time are highly likely to belong to the same group, the group estimation unit 44 may estimate that they will be in the same group this time as well.

[0080] If a product purchasing app (e.g., the Amazon app) that a user uses to shop at store 5 using mobile terminal 2 is linked to other SNS (Social Networking Service) apps (e.g., Line, Facebook, etc.), the group estimation unit 44 may identify users who have relationships such as friends in the SNS app, and if the identified users are present within store 5, may estimate that the users are in the same group.

[0081] If the time and location information of multiple users is the same based on information of an SNS app that the users use on their mobile terminals 2, the group estimation unit 44 may estimate that the multiple users are in the same group.

[0082] (Method of predicting who made a payment based on user behavior before entering a store) Users who are family members such as a husband and wife or a parent and child specify in advance who in the family is the payer before entering the store, and store this as a type in the registered user information DB 35. In this case, the group estimation unit 44 estimates the payer of the group based on the type read out from the registered user information DB 35.

[0083] Since friends may split the bill, pay in one lump sum, or pay differently, it is difficult to memorize attribute information or types in advance, as is the case with family members. Therefore, the group estimation unit 44 may estimate the first user in the group to enter the store as the payer, based on image data acquired from the camera 10 installed near the entrance 16.

[0084] The group estimation unit 44 may estimate that a user driving a car is the payer based on image data acquired from a camera 10 installed in a parking lot outside the store. The group estimation unit 44 may also identify the means by which users came to the store based on image data acquired from a camera 10 installed outside the store, and estimate that a user who came to the store by car is the payer.

[0085] (Method of predicting groups based on user behavior while in a store) The association degree calculation unit 42 calculates the association degree by detecting an interaction between users in the store 5, and the group estimation unit 44 estimates a group based on the association degree. At this time, the association degree calculation unit 42 may calculate the association degree taking into account attribute information and tracking data, and the group estimation unit 44 may estimate a group taking into account attribute information and tracking data.

[0086] (Method of estimating who made a payment based on the user's behavior while in a store) The group estimation unit 44 estimates the user pushing the cart as the payer based on image data acquired from the camera 10 installed in the store 5. Note that the estimation is not limited to the cart, and the group estimation unit 44 may estimate a user carrying a container for holding purchased items, such as a basket or an eco bag, as the payer.

[0087] The group estimation unit 44 may acquire appearance information of the user based on image data acquired from the camera 10 installed in the store 5, and estimate the payer based on the appearance information. Specifically, the group estimation unit 44 estimates a user who matches a preset condition, such as "male in his 40s" or "wearing a branded product," as the payer based on estimated age, sex, height, body type, clothing, accessories, and the like that can be determined from the appearance information.

[0088] When voice data can be acquired from a camera 10 installed in the store 5, the group estimation unit 44 may identify a user who has made a statement indicating that they will make a payment, such as "I'll pay today," based on the image data and voice data, and estimate that user to be the payer.

[0089] When physical data regarding the user's heart rate and body temperature can be acquired from the camera 10 or sensor installed in the store 5, the payment person is likely to be in a tense state, so the group estimation unit 44 may estimate that a user with a high heart rate or body temperature is the payment person based on the physical data.

[0090] If there is one user in the group whose payment information is stored in the registrant information DB 35, that is, one who is a registrant, the group estimation unit 44 estimates that the user is the payer.

[0091] Store visitor information and purchased item list information from when a user has previously used the store 5 may be stored as history information in the storage unit 31, and the group estimation unit 44 may estimate a user who was a payer in the past as the payer based on the history information. Also, the group estimation unit 44 may determine that, for example, if a user has acquired a purchased item that is cheaper than a previously purchased item based on the history information, the user himself / herself is likely to be the payer, and estimate the user as the payer.

[0092] In this way, even if the payer is not stored in the storage unit 31 before entering the store, the group estimation unit 44 can estimate the payer based on the behavior of the user while in the store.

[0093] (Method of group estimation based on user behavior immediately before payment) If a camera 10 or a shared terminal is installed near the exit 18, it is assumed that a plurality of users shopping together are all captured by the camera 10 or the shared terminal. In this case, the group estimation unit 44 may estimate, based on the image data acquired from the camera 10 or the shared terminal, that the users included in the image data belong to the same group.

[0094] Based on image data acquired from a camera 10 installed near an exit 18 of the store 5, the group estimation unit 44 may detect interactions and estimate that users who are holding their arms together or shaking hands near the exit 18 belong to the same group.

[0095] In this way, by using image data including all members of the group acquired near the exit, the group estimation unit 44 can accurately estimate and set the group to which the user belongs and the users who make up that group.

[0096] (Method of estimating the person who made a payment based on the user's behavior immediately before the payment) Based on image data acquired from a camera 10 or a shared terminal installed near the exit 18, the group estimation unit 44 may estimate that a user who is raising his or her hand near the exit is the person making the payment.

[0097] The group estimation unit 44 estimates a group of users present in the store 5 by combining one or more of the above-mentioned elements.

[0098] [Group payment processing] Next, an overview of group payment processing by the facility management system 100 will be described. The group payment processing is processing for setting a group and a payer for users who enter the store 5. The group payment processing is processing for making a collective payment for the total amount of all purchased products acquired by users belonging to the group, based on the payment information of the payer.

[0099] 11 is a flowchart showing an overview of the group payment process according to the first embodiment. The group payment process includes a product linking process S200 and a group setting process S300. The group payment process is mainly realized by the management server 1 executing a program prepared in advance.

[0100] When a user enters the store 5, the management server 1 identifies the user based on image data acquired from the camera 10 installed near the entrance 16 (step S100). Specifically, the management server 1 acquires facial image data and personal characteristic information of the user based on the acquired image data. The management server 1 then compares the personal characteristic information of the user who entered the store with the personal characteristic information of each registered person read from the registered person information DB 35 to identify whether the user who entered the store is a registered person, and if so, who the user is.

[0101] The management server 1 determines whether the identified user is a registered user (step S101), and if the user is a registered user (step S101; Yes), it generates new store visitor information including the registration ID and name read from the registered user information DB 35, the user's face image data, and a store exit flag (n), and stores the information in the store visitor information DB 37 (step S102).On the other hand, if the user is not a registered user (step S101; No), the management server 1 issues a temporary ID, generates new store visitor information including the temporary ID, the user's face image data, and a store exit flag (n), and stores the information in the store visitor information DB 37 (step S103).

[0102] Then, the management server 1 tracks the movement of each user in the store using the image data of the user acquired from the camera 10, and acquires tracking data (step S104). Furthermore, the management server 1 performs a product linking process (step S200).

[0103] Next, the product linking process will be described. The product linking process is a process for linking a purchased product acquired by a user with the user. Fig. 12 is a flowchart of the product linking process. This process is executed by the management server 1.

[0104] The management server 1 determines whether or not a product has been acquired by a user based on image data acquired from the camera 10 (step S201). Specifically, the management server 1 acquires image data of an area including the product from the camera 10, and acquires product characteristic information from the image data. The management server 1 then uses the product characteristic information to detect that a product has been removed from the product shelf 14. If a product has not been acquired (step S201; No), the management server 1 ends the product linking process, and proceeds to step S300 of the group payment process shown in FIG. 11.

[0105] On the other hand, if a product is acquired (step S201; Yes), the management server 1 uses the image data acquired from the camera 10 to identify the acquired product and the user who acquired the product (step S202). Specifically, the management server 1 identifies the purchased product by comparing the product characteristic information of the purchased product with the product characteristic information read from the product information DB 36, and acquires the product ID, product name, and price of the purchased product. The management server 1 also identifies the user who retrieved the purchased product by comparing the personal characteristic information of the face image data of the user who retrieved the purchased product with the personal characteristic information of the face image data read from the store visitor information DB 37, and acquires the registered ID or temporary ID of the user. Then, the management server 1 links the identified purchased product with the user, and stores it in the purchased product list DB 38 (step S203). As a result, the management server 1 ends the product linking process, and proceeds to the process of step S300 of the group payment process shown in FIG. 11.

[0106] Next, the group setting process will be described. The group setting process is a process for setting a group to which a user belongs and a settlor for the group. Fig. 13 is a flowchart of the group setting process. This process is executed by the management server 1.

[0107] The management server 1 detects interactions using image data of users acquired from the camera 10 and tracking data from the tracking unit 41, and calculates the degree of association between multiple users present in the store 5 using any determination factor (step S301). Determination factors for calculating the degree of association include, for example, the distance between users, the direction of their faces, opening and closing of their mouths, physical movements, and whether or not they share containers for carrying products. The management server 1 calculates the degree of association at any time.

[0108] Then, the management server 1 estimates groups consisting of multiple users based on pre-stored attribute information and the calculated relevance. For example, the group estimation unit 44 estimates that users whose relevance is equal to or greater than a threshold belong to the same group, taking into account the attribute information (step S302). Note that, when there is no user whose relevance is equal to or greater than the threshold, the management server 1 estimates that the user does not belong to a group and is shopping alone.

[0109] Furthermore, the management server 1 estimates a payer from among the users who make up the group (step S303). If it is estimated that the user does not belong to a group and is shopping alone, the management server 1 estimates that the user is the payer.

[0110] Next, the management server 1 presents the estimated results of the group and the payer to the user (step S304). Specifically, the management server 1 presents the estimated results of the group and the payer to the user who has been estimated or set as the payer of the group. Since the user who has been estimated or set as the payer is basically a registrant who has stored payment information, the management server 1 transmits the estimated results of the group and the payer to the mobile terminal 2 used by the user based on the contact information read from the registrant information 35. Specifically, the management server 1 transmits a group confirmation screen to the mobile terminal 2 as the estimated result.

[0111] The mobile terminal 2 displays the received group confirmation screen, and the user performs a predetermined operation on the group confirmation screen according to the instructions. The management server 1 receives information on the predetermined operation by the user from the mobile terminal 2 via the communication unit 32 as a response to the estimation result (step S305). Specifically, the management server 1 obtains information on corrections to the estimation result as the response.

[0112] The management server 1 determines whether the response to the estimation result includes any corrections (step S306). Here, the corrections include information on the exclusion of the accompanying person from the estimation result of the group and information on the correction of the estimation result of the payer.

[0113] If the response to the estimation result includes a correction (step S306; Yes), the management server 1 corrects the estimated group and the payer (step S307). Specifically, if the response to the estimation result of the group includes information regarding the exclusion of a companion, the management server 1 excludes the companion for whom the exclude button was pressed from the group, and sets only the companion for whom the add button was pressed to the group. If the response to the estimation result includes information regarding a correction to the payment person's estimation result, the management server 1 sets the payment person according to the correction (step S308). On the other hand, if the response to the estimation result does not include a correction (step S306; No), the management server 1 sets the estimated group and the payer as they are (step S308).

[0114] When the group and the payer are set, the management server 1 updates the store visitor information DB 37 and the purchased item list DB 38 (step S309). Specifically, when the management server 1 sets a group, it issues a group ID to identify the group, and stores the group ID of each user belonging to the group in the store visitor information DB 37. For example, a hyphen is stored as the group ID of a user who does not belong to a group. Furthermore, the management server 1 stores the group ID of each user belonging to the group in the purchased item list DB 38, and calculates and stores the total amount of all purchased items acquired by users belonging to the same group. Furthermore, when the management server 1 sets the payer, it stores the type of the user who is the payer as the payer and the type of the accompanying user as the accompanying person in the store visitor information DB 37.

[0115] Then, the management server 1 ends the group setting process, and proceeds to the process of step S105 of the group payment process shown in FIG.

[0116] The management server 1 refers to the store exit flag in the store visitor information DB 37 and determines whether all users belonging to the group have left the store (step S105). If all users belonging to the group have not left the store, that is, if one or more users belonging to the group are present in the store 5 (step S105; No), the management server 1 returns to the process of step S104 and performs user tracking, user-product association, and group estimation and setting as needed according to the user's behavior in the store 5.

[0117] On the other hand, if all users belonging to the group have left the store (step S105; Yes), the management server 1 refers to the registered user information DB 35 and makes a payment for the total amount of all purchased items acquired by each user belonging to the group, using the payment information of the payer (step S106). Specifically, the payment processing unit 49 authenticates the payer by biometric authentication and makes a payment based on the payment information of the authenticated payer. Note that if a user enters the store alone and does not belong to a group, the management server 1 considers that all members of the group have left the store when the user leaves the store. This ends the group payment processing according to the first embodiment.

[0118] In this way, the facility management system 100 of this embodiment estimates the group and payer for a user who enters the store 5 and presents the estimation result to the user. The facility management system 100 then sets the group and payer based on the response to the estimation result. Furthermore, the facility management system 100 performs payment processing all at once for the total amount of all products acquired by users belonging to the group, based on the payment information of the payer.

[0119] According to this, even a user who has not registered as a member in advance can shop at the store 5 by belonging to a group that has a payer. Therefore, children who are underage and do not have payment information and cannot register as a member can also enjoy shopping at the store 5 with their families.

[0120] Furthermore, the facility management system 100 automatically estimates groups by detecting interactions, etc., but the estimation results are presented to the user for confirmation, improving the accuracy of group setting. This makes it possible to prevent setting people who are not shopping together as the same group, and to prevent people from paying for items purchased by others together.

[0121] Furthermore, the facility management system 100 automatically estimates the payer based on image data acquired from the camera 10, and presents the estimated result to the user for confirmation, thereby improving the accuracy of the payer setting.

[0122] In current unmanned and labor-saving AI stores, users must register as members in advance, and unregistered people are often not allowed to enter the store. Furthermore, unmanned and labor-saving AI stores are basically premised on registered users shopping alone, and are not intended for shopping in groups such as with family or friends. However, such a facility management system 100 can provide a system that allows users to enter automated or unmanned facilities without registering as members in advance, and enables group shopping.

[0123] [Variations] (First Modification) In the above embodiment, the linking unit 43 identifies the purchased product acquired by the user using the image data of the user acquired from the camera 10 and the product feature information. However, this is not limited to this, and the linking unit 43 can also use location information to identify the purchased product. Here, the location information is, for example, information such as "the first row from the top of a specific product shelf 14 is rice balls" and "the rice balls are, from the left, plum rice balls, salmon rice balls, and tuna rice balls", and the specifications and format of the data, such as image data at the time of product display and coordinates within the image data, are arbitrary.

[0124] Furthermore, the linking unit 43 may use not only images but also weight sensors built into the product shelves 14 to identify purchased products. In this case, a weight sensor is provided on each product shelf, and when it detects that the weight of one rice ball has been removed from the display area of ​​salmon rice balls, for example, the linking unit 43 detects that a salmon rice ball has been taken and identifies the purchased product. By combining image analysis and a weight sensor, the linking unit 43 can improve the accuracy of identifying purchased products and reduce mistakes.

[0125] (Second Modification) When it is estimated that one user belongs to multiple groups, the group estimation unit 44 estimates the group based on a preset priority. Note that the preset priority is stored in the registrant information DB 35.

[0126] (Third Modification) In the above embodiment, users belonging to the same group share containers such as carts, baskets, and eco-bags, but this is not limiting, and each user may put purchased items in various containers. In this case, the group estimation unit 44 does not estimate the payer based on whether or not the container is shared, but estimates the payer based on other factors.

[0127] (Fourth Modification) In the above embodiment, one payer is set for each group, and payment processing is performed for the total amount of all products acquired by multiple users belonging to the group at once based on the payment information of the payer. However, this is not limited to this, and multiple payers may be set for each group, and payment processing may be performed based on the payment information of each payer. In this case, the total amount of all products acquired by multiple users belonging to the group is divided among the number of payers, and payment processing is performed for the divided amount based on the payment information of each payer. It is not limited to dividing the amount, and an arbitrary payment amount may be set separately for each payer. For example, if the total amount is 8000 yen and user A and user B are set as payers, the payment processing unit 49 may perform payment processing for 3000 yen based on the payment information of user A, and payment processing for 5000 yen based on the payment information of user B.

[0128] (Fifth Modification) In the above embodiment, the payer is registered as a member in advance, and the registered user information DB 35 stores the payment information, but the payer may be a user whose payment information is not stored. In this case, a self-checkout register capable of communicating with the management server 1 is provided in the store 5, and the payer pays for all purchased items acquired by users belonging to the group using the self-checkout register, for example, in cash. In addition, the user may transmit payment information, such as a credit card number, to the management server 1 by performing a predetermined operation on the mobile terminal 2 while shopping. In this case, the payment processing unit 49 makes payment for all purchased items acquired by users belonging to the group based on the received payment information.

[0129] (Sixth Modification) In the above embodiment, when all users belonging to a group leave the store, the total amount of all purchased items acquired by users belonging to the group is paid together. However, the timing of payment is not limited to this, and the payment processing unit 49 may pay for the purchased items acquired by companion A when companion A leaves the store, and pay for the purchased items acquired by companion B when companion B leaves the store. In other words, the payment processing unit 49 may pay for the purchased items acquired by a user who has left the store, based on the payment information of the payer, every time a user leaves the store.

[0130] (Seventh Modification) In the above embodiment, payment processing is performed when all users belonging to a group have left the store. In this case, regardless of whether the user who left the store is the payer or a companion, as long as there is at least one user belonging to the group remaining in the store 5, payment processing is not performed and shopping can be continued. However, this is not limited to the above, and the management server 1 may perform payment processing based on the payment information of the payer when the payer leaves the store. In this case, the companion will not be able to shop at the store 5 when the payer leaves the store.

[0131] If a flapper gate capable of communicating with the management server 1 is installed at the exit 18, the user may be restricted from leaving the store until the payment process is completed.

[0132] (Eighth Modification) In the above embodiment, the presentation unit 45 presents the estimation result of the group and the estimation result of the setter together as a group confirmation screen to the user, but this is not limiting, and the estimation result of the group and the estimation result of the setter may be presented to the user separately. For example, the presentation unit 45 may present the estimation result of the setter to the setter, and present the estimation result of the group to some or all of the users belonging to the group.

[0133] (Ninth Modification) The purchased item list showing the purchased items acquired by each user can be presented at any timing requested by the user. Specifically, the presentation unit 45 transmits purchased item list information to the mobile terminal 2 at the timing when a purchased item list request made by a predetermined operation from the mobile terminal 2 used by the user is received via the communication unit 32. By displaying the purchased item list based on the purchased item list information received by the mobile terminal 2, the user can easily check the purchased items.

[0134] In response to the purchase item list request, the purchase item list information may be transmitted to the shared terminal. In this way, the shared terminal displays the purchase item list based on the received purchase item list information, so that even a user who does not own the mobile terminal 2 can easily check the purchased items.

[0135] (Tenth Modification) The product shelf 14 may be provided with an electronic shelf label capable of communicating with the management server 1. In this case, the presentation unit 45 may refer to the purchased product list DB 38 at the timing when the user takes out a product from the product shelf 14, and present the product name and the number of purchased products already acquired by the user on the electronic shelf label of the purchased product. The presentation unit 45 may present not only the product name and the number of purchased products already acquired by the user, but also the product name and the number of purchased products already acquired by other users constituting the group to which the user belongs. Furthermore, the presentation unit 45 may present the total amount of all the members of the group, including the taken out product, for example, "The current total amount is 695 yen." This is convenient because the user can check the purchased products already acquired by other users belonging to the group and the total amount of all the members of the group at the timing when the user takes out the product.

[0136] <Second embodiment> Hereinafter, a second embodiment of the present disclosure will be described with reference to the drawings. As in the first embodiment, in this embodiment, a specific example in which the present application is applied to a store as an example of a facility will be described.

[0137] [Facility management system configuration] The configuration of the facility management system 100x according to this embodiment is similar to that of the first embodiment, and therefore a description thereof will be omitted for the sake of brevity. However, the facility management system 100x according to this embodiment is provided with shared terminals that can be used by users near the exit 18 or on carts. Also, electronic shelf labels are provided on the product shelves 14. The shared terminals and electronic shelf labels are connected to the server 1x so as to be able to communicate with each other.

[0138] [Device configuration] Next, the configuration of the management server 1x will be described with reference to Fig. 14. Fig. 14 shows a schematic configuration of the management server 1x. The management server 1x mainly includes a storage unit 31, a communication unit 32 that performs data communication, and a control unit 33x. These elements are connected to each other by a bus line 30. Note that the storage unit 31 and the communication unit 32 are the same as those in the first embodiment, and therefore the description thereof will be omitted.

[0139] The control unit 33x includes a CPU, a ROM, a RAM, etc. (not shown), and performs various controls on each component in the management server 1. In this embodiment, the control unit 33x includes a person identification unit 40, a tracking unit 41, an association degree calculation unit 42, a linking unit 43, a group estimation unit 44, a presentation unit 45, a reception unit 46, a group setting unit 47, a settling party setting unit 48, a payment processing unit 49, and a vigilance processing unit 50.

[0140] Here, the alert processing unit 50 different from the first embodiment will be described. The alert processing unit 50 sets a user with no linked payment information as a person to be alerted, and executes alert processing on the person to be alerted. Specifically, whether the user is shopping alone or in a group, if the user's payment information is stored in the storage unit 31, the user is not set as a person to be alerted. On the other hand, if the user is shopping alone in a store, and if the user's payment information is not stored in the storage unit 31, the user is set as a person to be alerted.

[0141] Furthermore, when multiple users enter a store and shop as a group, even if the user's payment information is not stored in the memory unit 31, the user will not be set as a person to be alerted if a payer is set for the group to which the user belongs. On the other hand, even if multiple users enter a store, if the user does not belong to a group or if a payer is not set for the group to which the user belongs, the user will be set as a person to be alerted. In other words, if the payment information for the price of the product purchased by the user at the store 5 is unknown, the alert processing unit 50 sets the user as a person to be alerted and executes alert processing.

[0142] A specific example of the alert processing will now be described. First, the alert processing unit 50 notifies the alert target person of a message encouraging them to register payment information or join a group with a payment person. The alert target person is an unregistered person with no payment information stored, and so contact information is unknown. For this reason, the alert processing unit 50 notifies, for example, a shared terminal or electronic shelf label installed near the alert target person of a message such as "Payment information is unknown, and it appears that you do not belong to a group. Please register payment information or join a group with a payment person."

[0143] The security processing unit 50 may use a spotlight to display a display on the floor and announce a message on the display. The security processing unit 50 may also use a security robot or drone to announce a message on a screen or by voice.

[0144] In this case, a user who is a registered user but has been set as a person under surveillance performs an operation such as taking a clear image of his / her face on the shared terminal, which allows the management server 1 to identify the user as a registered user based on the highly accurate face image data of the user.

[0145] Furthermore, when the alert processing unit 50 sets a person to be alerted, it presents facial image data of the person to the store clerk. Specifically, the alert processing unit 50 transmits facial image data and tracking data of the person to a mobile terminal used by the store clerk or a shared terminal installed in the back yard that can be used only by the store clerk. The store clerk pays close attention to the movements of the person to be alerted based on the facial image data and tracking data of the person to be alerted, and calls out to the person if necessary.

[0146] Furthermore, the alert processing unit 50 may, as necessary, present information on the alert target to a user present in the store 5. For example, when a suspicious behavior of the alert target is confirmed, such as intentionally attempting to belong to the same group as honest users, the alert processing unit 50 presents a message to the user to call attention to the situation.

[0147] Furthermore, if a flapper gate capable of communicating with the management server 1 is installed at the exit 18, the security processing unit 50 restricts the person to be alerted who has acquired purchased goods from leaving the store 5. If the person to be alerted who has acquired purchased goods leaves the store 5, the security processing unit 50 notifies the security office or the police.

[0148] If the person to be watched does not have payment information such as a credit card or bank account, the alert processing unit 50 may remove the setting of the person as a person to be watched and allow the person to leave the store by paying for the purchased items at a self-checkout register. Also, if a person to be watched who does not have payment information approaches the exit 18 or the flapper gate, the alert processing unit 50 may guide the person to pay at a self-checkout register using signage or the like.

[0149] [Group payment processing] Next, an overview of group payment processing by the facility management system 100x will be described. The group payment processing is processing for setting a group and a payer for users who enter the store 5. The group payment processing is processing for making a collective payment for all purchased items acquired by users belonging to the group based on the payment information of the payer. Note that in the group payment processing by the facility management system 100x, users who are not linked to payment information are set as persons to be monitored, and monitoring processing is executed.

[0150] Fig. 15 is a flow chart showing an overview of the group payment process according to the second embodiment. The group payment process includes a product linking process S200 and a group setting process S300. The product linking process S200 and the group setting process S300 are similar to those in the first embodiment, and therefore descriptions thereof will be omitted for the sake of brevity. The group payment process is mainly realized by the management server 1 executing a program prepared in advance.

[0151] When the user enters the store 5, the management server 1 identifies the user based on image data acquired from the camera 10 installed near the entrance 16 (step S400). Then, the management server 1 judges whether the identified user is a registered user or not (step S401). If the user is a registered user (step S401; Yes), the management server 1 generates new store visitor information including the registration ID and name read from the registered user information DB 35, the user's face image data, and a store exit flag (n), and stores the information in the store visitor information DB 37 (step S402). On the other hand, if the user is not a registered user (step S402; No), the management server 1 issues a temporary ID, generates new store visitor information including the temporary ID, the user's face image data, and a store exit flag (n), and stores the information in the store visitor information DB 37 (step S403).

[0152] Then, the management server 1 tracks the movement of each user in the store using the image data of the user acquired from the camera 10, and acquires the tracking data (step S404). Furthermore, the management server 1 performs a product linking process (step S200). After the product linking process is completed, the management server 1 performs a group setting process (step S300).

[0153] When the group setting process is completed, the management server 1 judges whether or not the user is about to leave the store based on the image data acquired from the camera 10 installed near the exit 18 (step S405). If the user is not about to leave the store (step S405; No), the management server 1 returns to the process of step S404, and tracks the user, associates the user with products, and estimates and sets groups as needed according to the user's behavior within the store 5.

[0154] On the other hand, if the user is about to leave the store (step S405; Yes), the management server 1 refers to the purchased item list DB 38 and determines whether or not the user has acquired a purchased item (step S406). If the user has not acquired a purchased item (step S406; No), the management server 1 proceeds to the processing of step S408. On the other hand, if the user has acquired a purchased item (step S406; Yes), the management server 1 refers to the store visitor information DB 37 and determines whether or not the user is a person to be monitored (step S407).

[0155] If the user is a person under surveillance (step S407; Yes), the management server 1 executes surveillance processing (step S410). For example, as surveillance processing, the management server 1 may send the person under surveillance a message encouraging them to register payment information or to join a group that includes the person who made the payment. The management server 1 may also present facial image data of the person under surveillance to a store clerk. Furthermore, if a flapper gate is installed at the exit 18, the management server 1 may restrict the person under surveillance who has acquired a purchased item from leaving the store. After executing surveillance processing, the management server 1 returns to the processing of step S405.

[0156] On the other hand, if the user is not a person under surveillance (step S407; No), the management server 1 refers to the store exit flag in the store visitor information DB 37 and determines whether all users constituting the group have left the store (step S408). If all users constituting the group have not left the store, that is, if one or more users belonging to the group are present in the store 5 (step S408; No), the management server 1 returns to the process of step S404 and performs user tracking, linking users with products, and group estimation and setting as needed according to the user's behavior in the store 5.

[0157] On the other hand, if all users in the group have left the store (step S408; Yes), the management server 1 refers to the registered user information DB 35 and makes a payment for all purchased items acquired by each user in the group collectively using the payment information of the payer (step S409). Specifically, the payment processing unit 49 authenticates the payer by biometric authentication and makes a payment based on the payment information of the authenticated payer. Note that if a user enters the store alone and does not belong to a group, the management server 1 considers that all members of the group have left the store when the user leaves the store. This ends the group payment processing according to the second embodiment.

[0158] In the above group payment processing, the management server 1 executes the alert processing when the person to be watched who has acquired the purchased items attempts to leave the store, but this is not limited to the above and any alert processing can be executed at any time while the person to be watched is within the store 5.

[0159] In this way, the facility management system 100x of this embodiment sets users who have entered the facility and are not linked to payment information as people to be monitored, and executes monitoring processing as necessary. The facility management system 100x assumes unmanned and labor-saving AI facilities, does not require membership registration in advance, and allows users who do not have payment information to shop in groups. However, on the other hand, there is a high need to prevent fraudulent acts by malicious users. Therefore, executing monitoring processing on users who have become people to be monitored serves as a deterrent to fraudulent acts such as shoplifting and having other users make payments, thereby enhancing crime prevention functions.

[0160] <Third embodiment> 16 shows the configuration of an information processing device according to a third embodiment of this disclosure. The information processing device 80 includes a person identification means 81 for identifying a person who has entered a facility, a group estimation means 82 for estimating a group composed of a plurality of people, a group presentation means 83 for presenting the group estimation result to the person, and a group setting means 84 for setting the group based on a response to the group estimation result.

[0161] 17 is a flowchart of processing by the information processing device 80. The person identification means 81 identifies a person who has entered the facility (step S81). The group estimation means 82 estimates a group consisting of a plurality of people (step S82). The group presentation means 83 presents the group estimation result to the person (step S83). The group setting means 84 sets a group based on a response to the group estimation result (step S84).

[0162] According to the third embodiment, a plurality of people who have entered a facility can be set as a group, and management and settlement can be made on a group basis.

[0163] In addition, some or all of the above-described embodiments (including modified examples, the same applies below) can be described as, but are not limited to, the following notes.

[0164] (Appendix 1) A person identification means for identifying a person who has entered the facility; A group estimation means for estimating a group composed of a plurality of persons; a group presenting means for presenting the group estimation result to the person; a group setting means for setting the group based on a response to the group estimation result; An information processing device comprising:

[0165] (Appendix 2) The person identification means acquires a face image from an image captured by a camera installed inside or outside the facility, and identifies the person based on the face image; The information processing device according to claim 1, wherein the group presentation means presents an estimation result including face images of people estimated to be part of the group.

[0166] (Appendix 3) further comprising a correlation calculation means for calculating a correlation between a plurality of people present in the facility; The group estimation means estimates the group based on the relevance degree, The information processing device described in Appendix 2, wherein the relevance calculation means calculates the relevance based on one or more of the following data: the distance between the people included in the image, the direction of their faces, the opening and closing of their mouths, their physical movements, and whether or not they share a container for carrying a product.

[0167] (Appendix 4) A storage means for storing attribute information of the person, The information processing device according to claim 3, wherein the relevance calculation means calculates the relevance using the attribute information.

[0168] (Appendix 5) a payer estimation means for estimating a payer from among the people constituting the group; a payment processor that presents the payment processor's estimation result to the person; a settler setting means for setting the settler based on a response to the settler estimation result; a payment processing means for collectively paying for all the products acquired by the persons constituting the group based on the payment information of the payer; 5. The information processing device according to claim 1, further comprising:

[0169] (Appendix 6) The information processing device described in Appendix 5, wherein the payment person estimation means acquires a facial image from an image captured by a camera installed inside or outside the facility, and estimates the payment person based on one or more of data of the people included in the image, such as their estimated age, the order of entry into the store, their gender, and whether or not they have a container to carry the acquired product.

[0170] (Appendix 7) The information processing device according to any one of claims 1 to 6, wherein the group estimation means estimates that people included in an image captured by a camera installed near an exit of the facility belong to the same group.

[0171] (Appendix 8) The information processing device according to any one of Supplementary claims 5 to 7, wherein the group presentation means presents an estimation result of the group to an approver.

[0172] (Appendix 9) The information processing device according to any one of appendices 1 to 8, wherein the group presentation means presents a list correlating the people constituting the group, the products acquired by each person, and the total price of the products acquired by each person, as an estimation result of the group.

[0173] (Appendix 10) An information processing device as described in any one of Appendices 5 to 9, further comprising an alert processing means for performing alert processing on a person if the person does not have payment information and does not belong to a group that includes the payment person.

[0174] (Appendix 11) Identifying people who enter the facility A group consisting of a plurality of people is estimated, presenting the group estimation result to the person; An information processing method for setting the group based on a response to an estimation result of the group.

[0175] (Appendix 12) Identifying people who enter the facility A group consisting of a plurality of people is estimated, presenting the group estimation result to the person; A recording medium having a program recorded thereon for causing a computer to execute a process of setting the group based on a response to the group estimation result.

[0176] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In other words, the present invention naturally includes various modifications and corrections that a person skilled in the art could make in accordance with the entire disclosure, including the claims, and the technical ideas. In addition, the disclosures of the above cited patent documents and the like are incorporated herein by reference. [Explanation of symbols]

[0177] 1 Management Server 2. Mobile devices 3. Network 5 stores 10. Camera 14 Product shelf 16 Entrance 18 Exit 20, 22, 24 users 100 Facility Management System

Claims

1. A person identification means for identifying a user who has entered the facility; A group setting means for setting a group composed of a plurality of users; a transmission means for transmitting information on the payer and accompanying person of the group and information on products acquired by the payer and accompanying person to terminals of users belonging to the group; An information processing device comprising:

2. The information processing apparatus according to claim 1 , further comprising a payment processing means for performing payment for the product based on approval information from the terminal.

3. 3. The information processing apparatus according to claim 1, further comprising a settlor setting unit for changing the settlor based on an instruction input by a user.

4. 2. The information processing apparatus according to claim 1, wherein the group setting means changes the group based on an instruction input by a user.

5. A group estimation means for estimating a group composed of a plurality of users; a group presenting means for presenting the group estimation result to the user; Equipped with The information processing apparatus according to claim 1 , wherein the group setting means sets the groups based on a response to the estimation result of the groups.

6. The person identification means acquires a face image from an image captured by a camera installed inside or outside the facility, and identifies the user based on the face image; The information processing apparatus according to claim 5 , wherein the group presentation means presents an estimation result including face images of people who are estimated to be part of the group.

7. a correlation calculation unit that calculates a correlation between a plurality of users in the facility, The group estimation means estimates the group based on the relevance degree, The information processing device according to claim 6, wherein the relevance calculation means calculates the relevance based on one or more of the following data contained in the image: the distance between users, the direction of their faces, the opening and closing of their mouths, physical movements, and whether or not they share a container for carrying products.

8. A storage means for storing attribute information of the user is provided, The information processing apparatus according to claim 7 , wherein the relevance calculation means calculates the relevance by using the attribute information.

9. 1. A computer-implemented information processing method, comprising: Identifying users who have entered the facility; Set up a group consisting of multiple users, An information processing method for transmitting information on a payer and accompanying person of the group and information on products acquired by the payer and accompanying person to terminals of users belonging to the group.

10. Identifying users who have entered the facility; Set up a group consisting of multiple users, A program that causes a computer to execute a process of transmitting information about the payer and accompanying person of the group, and information about products acquired by the payer and accompanying person, to terminals of users belonging to the group.

Images