Information processing system, information processing method and program

Abstract

An information processing system, an information processing method, and a program are provided that grasp the behavior of users of facilities such as sports gyms and further enhance support for facility management. [Solution] According to one aspect of the present invention, an information processing system includes at least one processor, and the processor executes a program to perform the following steps: In a first acquisition step (activity A101), identification information for identifying a user is associated with a captured image showing at least a portion of the user, and the captured image is defined as a specific image related to the user; in a second acquisition step (activity A105), an internal image of the interior of a facility where the user is staying is acquired; and in a management step (activity A108), behavioral information and identification information are associated and managed. The behavioral information is information including the user's behavior history, which is identified based on the specific image and the internal image.

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] For example, Patent Document 1 discloses a "system for supporting the operation of a sports gym." In the sports gym operation support system, a door unit 3 includes an unlocking control unit that unlocks the door based on an unlocking instruction from a server device, a user's terminal device 2 includes a first receiving unit that receives screen data, a display unit that displays a door unlocking instruction screen based on the screen data, an operation unit that issues an unlocking instruction on the unlocking instruction screen, and a first transmitting unit that sends an unlocking request to the server device when the unlocking instruction is issued, and a server device 1 includes a second receiving unit that accepts the unlocking request, an unlocking control unit that controls the door unit to unlock based on the request, a second transmitting unit that sends the unlocking instruction to the door unit and also sends a discount coupon to the user's terminal device, and a charging unit that charges the user a gym usage fee when the door is unlocked. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-041673 Summary of the Invention [Problem to be solved by the invention]

[0004] However, there is still room for improvement in the functionality of the above-mentioned known technologies. In particular, there is a need for technology that can grasp the behavior of users at facilities such as sports gyms and provide more comprehensive support for facility management.

[0005] In view of the above circumstances, the present invention provides an information processing system and the like that can further enhance support for facility management. [Means for solving the problem]

[0006] According to one aspect of the present invention, there is provided an information processing system comprising at least one processor, the processor being configured to execute a program to perform the following steps: in a first acquisition step, identification information for identifying a user is associated with a captured image showing at least a portion of the user and the captured image is acquired, and the captured image is defined as a specific image relating to the user; in a second acquisition step, an internal image of the interior of a facility where the user is staying is acquired; and in a management step, behavioral information and identification information are associated with each other and managed, the behavioral information being information including the user's behavior history identified based on the specific image and the internal image.

[0007] According to this embodiment, it is possible to further enhance support for facility management. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a configuration diagram illustrating an information processing system 1. FIG. [Figure 2] FIG. 2 is a block diagram showing a hardware configuration of an information processing device 2. [Figure 3] FIG. 2 is a block diagram showing the hardware configuration of a user terminal 3. [Figure 4] FIG. 2 is a block diagram showing the hardware configuration of the imaging device 5. [Figure 5] FIG. 2 is a functional block diagram showing functions of the information processing device 2. [Figure 6] 1 is an activity diagram showing an example of the flow of information processing executed in the information processing system 1. FIG. [Figure 7] FIG. 1 is a schematic diagram of the entrance 6 of facility Z1. [Figure 8] FIG. 10 is a schematic diagram of a user information table T1. [Figure 9] This is a plan view of the interior of facility Z1. [Figure 10] 10 is a diagram showing an example of an internal image IM2 captured by the imaging device 5. FIG. [Figure 11] FIG. 10 is a schematic diagram of an action history table T2. [Figure 12] 1 shows an example of a proposal screen 9 displayed on the display unit 44 of the employee terminal 4. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to the accompanying drawings. Various features shown in the following embodiments can be combined with each other.

[0010] Incidentally, the program for realizing the software appearing in one embodiment may be provided as a non-transitory computer-readable medium, or may be provided so that it can be downloaded from an external server, or may be provided so that the program is started on an external computer and its functions are realized on a client terminal (so-called cloud computing).

[0011] Furthermore, various information processing according to an embodiment may realize input and output corresponding to the input. Here, the form of information referenced in such information processing (hereinafter referred to as reference information) is not limited as long as an output is obtained as a result of the input. The reference information may be, for example, rule-based information such as a database, a lookup table, or a predetermined function (including a decision formula such as a regression formula constructed using a statistical method), a trained model that has previously trained the correlation between input and output, or a generative AI such as a large-scale language model or a visual language model that can output a desired result in response to a prompt input.

[0012] In one embodiment, a "unit" may include, for example, a combination of hardware resources implemented by a circuit in the broad sense and software information processing that can be specifically realized by these hardware resources. In one embodiment, various information is handled, and this information is represented, for example, by physical values ​​of signal values ​​representing voltage and current, high and low signal values ​​as a binary bit set consisting of 0 or 1, or quantum superposition (so-called quantum bits), and communication and calculations can be performed on a circuit in the broad sense.

[0013] Furthermore, a circuit in the broad sense is a circuit realized by at least an appropriate combination of a circuit, circuitry, processor, memory, etc. The processor may be a general-purpose processor or a dedicated circuit. That is, it includes an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)), etc.

[0014] 1. Hardware Configuration In this section, the hardware configuration of one embodiment will be described.

[0015] 1.1 Information Processing System 1 FIG. 1 is a configuration diagram showing an information processing system 1. The information processing system 1 includes an information processing device 2, a user terminal 3, an employee terminal 4, and an imaging device 5. The information processing device 2, the user terminal 3 which is a user terminal, and the employee terminal 4 are configured to be able to communicate with each other via a telecommunications line. The information processing device 2 and the imaging device 5 are also configured to be able to communicate with each other via a telecommunications line. In one embodiment, the information processing system 1 is made up of one or more devices or components. For example, if the information processing system 1 is made up of only the information processing device 2, the information processing system 1 can be the information processing device 2. These components will be described below.

[0016] 1.2 Information processing device 2 2 is a block diagram showing the hardware configuration of the information processing device 2. The information processing device 2 includes a communication bus 20, a communication unit 21, a storage unit 22, and a processor 23. The communication unit 21, the storage unit 22, and the processor 23 are electrically connected via the communication bus 20 inside the information processing device 2.

[0017] <Communications Department 21> The communication unit 21 is preferably a wired communication means such as USB, IEEE1394, Thunderbolt (registered trademark), wired LAN network communication, etc., but may also include wireless LAN network communication, mobile communication such as 3G / LTE / 5G, BLUETOOTH (registered trademark) communication, etc. as needed. In other words, it is more preferable to implement it as a collection of multiple communication means. In other words, the information processing device 2 may communicate various information from the outside via the communication unit 21 and the network.

[0018] <Storage section 22> The storage unit 22 stores various pieces of information defined above. This can be implemented, for example, as a storage device such as a solid state drive (SSD) that stores various programs and the like related to the information processing device 2 executed by the processor 23, or as a memory such as a random access memory (RAM) that stores temporarily required information (arguments, arrays, etc.) related to the program operations. The storage unit 22 stores various programs, variables, etc. related to the information processing device 2 executed by the processor 23.

[0019] <Processor 23> The processor 23 processes and controls the overall operations related to the information processing device 2. The processor 23 is, for example, a central processing unit (CPU) not shown. The processor 23 realizes various functions related to the information processing device 2 by reading out predetermined programs stored in the storage unit 22. In other words, information processing by software stored in the storage unit 22 is specifically realized by the processor 23, which is an example of hardware, and can be executed as each functional unit included in the processor 23. These will be described in more detail in the next section. Note that the processor 23 is not limited to being single, and multiple processors 23 may be provided for each function. A combination of these may also be used.

[0020] 1.3 User terminal A user terminal is a terminal owned by a user. User terminals include a terminal 3 (hereinafter referred to as user terminal 3) operated by a first user who uses the facility, and a terminal 4 (hereinafter referred to as employee terminal 4) operated by a second user who is an employee of the facility. The user terminal may be in any form, such as a smartphone, tablet terminal, computer, or any other device that can access the information processing device 2 via a telecommunications line.

[0021] 3 is a block diagram showing the hardware configuration of the user terminal 3. The following description will be given taking the user terminal 3 as an example. The user terminal 3 includes a communication bus 30, a communication unit 31, a storage unit 32, a processor 33, a display unit 34, and an input unit 35. The communication unit 31, the storage unit 32, the processor 33, the display unit 34, and the input unit 35 are electrically connected via the communication bus 30 inside the user terminal 3. The description of the communication unit 31, the storage unit 32, and the processor 33 will be omitted as they are the same as the description of each unit in the information processing device 2.

[0022] <Display section 34> The display unit 34 displays a screen of a graphical user interface (GUI) that can be operated by the user. The display unit 34 may be included in the housing of the user terminal 3 or may be attached externally. Specifically, the display unit 34 may be implemented as a display device such as a CRT display, a liquid crystal display, an organic EL display, or a plasma display. It is preferable that these display devices are implemented by selectively using them depending on the type of the user terminal 3.

[0023] <Input section 35> The input unit 35 accepts operation inputs made by the user. The operation inputs are transferred as command signals to the processor 33 via the communication bus 30. The processor 33 can execute predetermined control or calculations based on the transferred command signals as necessary. The input unit 35 may be included in the housing of the user terminal 3 or may be externally attached. For example, the input unit 35 may be implemented as a touch panel integrated with the display unit 34. When the input unit 35 is implemented as a touch panel, the user can input tap operations, swipe operations, etc. to the input unit 35. Instead of a touch panel, a switch button, a mouse, a QWERTY keyboard, etc. can be used as the input unit 35.

[0024] 1.4 Imaging device 5

[0025] Fig. 4 is a block diagram showing the hardware configuration of the imaging device 5. As shown in Fig. 4, the imaging device 5 includes a communication bus 50, a communication unit 51, a storage unit 52, a control unit 53, and an imaging unit 54. The communication unit 51, the storage unit 52, the control unit 53, and the imaging unit 54 are electrically connected via the communication bus 50 inside the imaging device 5. The description of the communication unit 51, the storage unit 52, and the control unit 53 is omitted because they are the same as the description of each unit in the information processing device 2.

[0026] The imaging unit 54 is a so-called vision sensor (camera) configured to be able to capture information of the outside world. For example, the imaging unit 54 executes a process of capturing an image of a user entering a facility at the entrance of the facility. The captured image may be stored in the storage unit 52 in association with date and time information indicating the date and time of capture. Furthermore, for example, the imaging unit 54 executes a process of capturing an image of the interior of the facility. The captured image may be recorded in the storage unit 52 as time-series data, and the captured image may include the interior status of the facility. Details will be described later.

[0027] 2. Functional configuration In this section, the functional configuration will be described. Fig. 5 is a functional block diagram showing the functions of the information processing device 2. As described above, information processing by software stored in the storage unit 22 can be specifically realized by hardware (specifically, the processor 23), and executed as each functional unit included in the processor 23. In other words, the information processing system 1 has each functional unit in a program.

[0028] Specifically, the processor 23 has, as its functional units, a reception unit 231, an acquisition unit 232, a management unit 233, an authentication unit 234, an identification unit 235, an evaluation unit 236, an update unit 237, a display control unit 238, and an artificial intelligence unit 239.

[0029] The reception unit 231 is configured to receive various pieces of information as a reception step. Specifically, the reception unit 231 is configured to receive information via the communication unit 21 or the storage unit 22 and to be able to read the information into a working memory. Preferably, the reception unit 231 receives, from the user via the user terminal 3, a request to display identification information IF1 that can identify the user.

[0030] The acquisition unit 232 is configured to acquire various information as an acquisition step. For example, the acquisition unit 232 acquires user identification information IF1 via the user terminal and the reader RD. Also, for example, the acquisition unit 232 acquires internal images IM2 captured by the imaging device 5. The internal images IM2 include images captured of the interior of the facility where the user is staying.

[0031] The management unit 233 is configured to manage various information as a management step. Specifically, the management unit 233 manages various information or data in association with identification information IF1 that identifies a user. The various information or data may be stored in a storage unit of various devices such as the storage unit 22 of the information processing device 2 or the user terminal 3, or may be stored in a predetermined storage area including an external storage device.

[0032] The authentication unit 234 executes authentication processing related to authentication of a user as an authentication step. For example, the storage unit 22 stores identification information IF1 and the like of users registered as users of the facility. Here, when the acquisition unit 232 acquires at least the identification information IF1 from a user via the user terminal 3, if the identification information IF1 is stored in the storage unit 22, the authentication unit 234 authenticates the user. Details of the authentication processing will be described later.

[0033] The identification unit 235 is configured to identify various pieces of information as an identification step. Specifically, the identification unit 235 identifies behavioral information IF2 including the user's behavior history within the facility based on a specific image IM1 that can identify the user and an internal image IM2 that captures the interior of the facility used by the user. Here, the specific image IM1 is an image for identifying the user's behavioral information within the facility and is an image that shows a part of the user.

[0034] The evaluation unit 236 is configured to evaluate various characteristics as the evaluation step. Preferably, the evaluation unit 236 evaluates the working conditions of the employee based on the behavior information IF2 of the employee identified by the identification unit 235.

[0035] The update unit 237 is configured to update various pieces of information as an update step. Preferably, the update unit 237 updates various pieces of information managed by the information processing system 1, or causes an external device to update information managed by the external device. For example, as an update step, the update unit 237 updates an image of a user managed by the information processing system 1 to a new image (specific image IM1) received from the imaging device 5.

[0036] As a display control step, the display control unit 238 displays various pieces of information stored in the storage unit 22 or screens containing such information in a viewable manner on a terminal such as the user terminal 3. This allows various pieces of information to be presented to a user operating the user terminal 3. When the phrase "display" is used, it does not matter whether the display medium is in a local environment or whether processing for displaying is performed via the network 11. The display control unit 238 controls visual information such as screens, images, icons, and messages to be displayed on the display of the terminal. The display control unit 238 may only generate rendering information for displaying the visual information on the terminal.

[0037] The artificial intelligence unit 239 is configured to receive input from each functional unit and return the instructed output. The artificial intelligence used by each functional unit of the information processing device 2 may be a common one, or may be prepared individually for each functional unit.

[0038] The artificial intelligence unit 239 may include a learning model such as a language model including a generative AI, a Transformer, a Recurrent Neural Network (RNN), or the like.

[0039] The language model is an example of a learning model based on a machine learning algorithm. Specific examples of machine learning algorithms include nearest neighbor methods, naive Bayes methods, decision trees, support vector machines, and deep learning using neural networks. The artificial intelligence unit 239 can apply the above algorithms as appropriate.

[0040] The artificial intelligence unit 239 may have a trained model constructed by a learning method such as supervised learning, unsupervised learning, or self-supervised learning. In supervised learning, machine learning is performed using training data (training data). The training data consists of pairs of input data for learning and output data (correct answer data). In addition, the language model may not only be trained for a specific task, but may also be a general-purpose model that can be used for a wide range of tasks. The learning model included in the artificial intelligence unit 239 can undergo additional training as transfer learning or fine tuning. 3. Information Processing The information processing of this embodiment will be described below.

[0041] 3.1 Overview of Information Processing 6 is an activity diagram showing an example of the flow of information processing executed in the information processing system 1. In this embodiment, a case will be described in which a behavior history indicating the behavior of a user X, who is a first user, from the time he enters a facility until he leaves the facility is tracked.

[0042] Here, user X is assumed to be a user who uses a facility. The facility may be, for example, an exercise facility. An exercise facility is a facility that provides an environment for users to exercise, such as a sports club, fitness club, or sports gym. In this embodiment, user X is assumed to be a member of sports gym Z, and to use facility Z1 owned by sports gym Z. Furthermore, facility Z1 is assumed to include, for example, an area where multiple types of exercise equipment (machines) are installed. Furthermore, sports gym Z is assumed to have a lesson studio for yoga, dance, etc. within its premises. The type and configuration of the facility are not limited to this.

[0043] First, as a first acquisition step, the acquisition unit 232 acquires identification information IF1 of user X (activity A101). Next, as a first acquisition step, the acquisition unit 232 acquires a captured image that shows at least a part of user X (activity A102). In this embodiment, this captured image is defined as a specific image IM1 related to the user.

[0044] Preferably, the processing of activity A101 and activity A102 is executed when user X enters facility Z1. Alternatively, such processing may be executed when user X enters a predetermined area included in facility Z1. The predetermined area may be, for example, an area with entry restrictions. That is, the captured image (specific image IM1) of user X acquired in activity A102 may be an image captured when user X enters facility Z1 or a predetermined area included in the facility. According to this aspect, when identifying a user's behavior history using internal image IM2, an image captured when the user enters facility Z1 can be used as specific image IM1. Therefore, since an image captured immediately before the user enters facility Z1 can be used to identify the user's behavior history, it is easier to identify the user and the identification accuracy can be improved compared to, for example, identifying a user based on a pre-registered user's facial image.

[0045] A specific example of the processing of activity A101 and activity A102 will be described with reference to Fig. 7. Fig. 7 is a schematic diagram of the entrance 6 of facility Z1. Preferably, a reader RD and an imaging device 5 are installed near the door through which users enter.

[0046] Specifically, in response to a request from user X to display a two-dimensional code for entering facility Z1 input to user terminal 3, display control unit 238 causes a two-dimensional code such as a QR code (registered trademark) to be displayed on display unit 34 of user terminal 3. Here, the two-dimensional code includes at least user identification information IF1. Preferably, the two-dimensional code also includes an expiration date. In other words, each time a request is received from user X, a different two-dimensional code may be generated and displayed on display unit 34.

[0047] Next, the reader RD reads the two-dimensional code displayed on the display unit 34 of the user terminal 3, and transmits image data of the captured two-dimensional code to the information processing device 2, whereby the acquisition unit 232 acquires the image data via the communication unit 21. Preferably, the image data is transmitted to the information processing device 2 in association with date and time information indicating the date and time when the two-dimensional code was read.

[0048] Furthermore, an image capturing device 5 installed at the entrance of the facility Z1 captures an image capturing at least a portion of the user X. Here, the acquisition unit 232 acquires the captured image of the user X via the communication unit 21.

[0049] In this way, as a first acquisition step, the acquisition unit 232 acquires identification information IF1 that identifies the user X and a captured image in which at least a part of the user appears, in association with each other. At this time, the captured image in which the user appears is defined as a specific image IM1 related to the user.

[0050] The processor 23 compares the specific image IM1 with the internal image IM2. In this case, the identification unit 235 can identify an image of user X from the internal image IM2 using the specific image IM1. For example, the specific image IM1 may be an image including the facial region of user X. Furthermore, for example, the specific image IM1 may be an image including the entire region of user X from head to toe. Furthermore, for example, the specific image IM1 may be an image including the region of user X's upper body, including the face and waist. In other words, the specific image IM1 includes a part that shows the external characteristics of the user, i.e., user X. Here, "at least a part of user X" refers to a part of user X's face and body that can identify the user, preferably part or all of the upper body. According to this aspect, the part of the user included in the image can be identified from the internal image IM2 captured inside the facility based on the specific image of the user.

[0051] Next, the authentication unit 234 authenticates user X based on at least the identification information IF1 acquired in activity A101 (activity A103). Specifically, if the user information corresponding to the identification information IF1 satisfies the entry conditions, the authentication unit 234 permits user X to enter. The entry conditions may include, for example, that the two-dimensional code has not expired. Furthermore, for example, the entry conditions may include that user X identified from the identification information IF1 is a valid member. Furthermore, for example, the entry conditions may include that the time the two-dimensional code is read is within the facility usage hours of the plan subscribed to by user X.

[0052] (User information table T1) 8 is a schematic diagram of the user information table T1. The user information table T1 is an example of a database that is referenced to authenticate the user X in, for example, the activity A103 in FIG.

[0053] In this embodiment, the user information table T1 is a database containing information about users of the sports gym Z. For example, when a user signs a new contract for use of the sports gym Z, the reception unit 231 may receive personal information about the user or the type of usage contract, and a record may be added to the user information table T1 in response to the reception unit 231 receiving the personal information about the user or the type of usage contract. The user information table T1 may be stored in the storage unit 22 of the information processing device 2, or may be stored in another server.

[0054] As shown in FIG. 8, the user information table T1 may include information such as a member ID (T11), a name (T12), a contract plan (T13), a specific image (T14), a group of reference images (T15), and a behavioral characteristic (T16). The member ID (T11) is information that can identify a user who is a member of the sports gym Z. The identification unit 235 can identify a user by matching the identification information IF1 acquired in activity A101 of FIG. 6 with the member ID (T11) in the user information table T1. Furthermore, the name (T12) is information that indicates the name of the user.

[0055] The contract plan (T13) is information indicating the type of usage pattern of the user at the gym Z. For example, the time period or area in which the user can stay at the gym Z may differ depending on the type of contract plan. In activity A103 of FIG. 6, whether the user is permitted to enter the facility Z1 may be determined based on the contract plan of the user corresponding to the member ID (T11).

[0056] The specific image (T14) is image data of the specific image IM1 corresponding to the user. For example, when a user is authenticated in activity A103 of Fig. 6, the specific image (T14) of the user may be stored in association with the user's member ID (T11).

[0057] The reference image group (T15) indicates information that allows identification of a specific image IM1 included in an image group that includes multiple specific images IM1. The behavioral characteristic (T16) indicates a behavioral characteristic identified based on user behavior information IF2, which is identified in activities A106 to A107 described below. The reference image group (T15) and the behavioral characteristic (T16) will be described in detail later.

[0058] Subsequently, when the authentication unit 234 executes the authentication process and the user X is permitted to enter, the management unit 233 associates the specific image IM1 with the identification information IF1 and manages them (activity A104). Specifically, the management unit 233 may store the specific image IM1 and the identification information IF1 in the storage unit 22 in association with each other.

[0059] If the authentication process is performed and the user X is permitted to enter, the information processing system 1 may further cooperate with an unlocking device at the entrance of the facility Z1 to execute a process of unlocking the entrance door.

[0060] Independently from the processing related to activities A101 to A104, the acquisition unit 232 acquires, as a second acquisition step, internal images IM2 captured inside the facility Z1 where the user is staying (activity A105). Specifically, multiple imaging devices 5 are installed inside the facility Z1. The internal images IM2 captured by the multiple imaging devices 5 are transmitted to the information processing device 2 via the communication unit 51. In this embodiment, the imaging devices 5 continuously capture internal images IM2 and transmit the internal images IM2 to the information processing device 2. The internal images IM2 acquired by the acquisition unit 232 of the information processing device 2 are stored in the storage unit 22, etc. The internal images IM2 are video data, and the management unit 233 may store the video data in the storage unit 22 in association with date and time information when the video data was captured. Alternatively, the internal images IM2 are multiple still images extracted from the video data at predetermined intervals, and may be stored in the storage unit 22 in association with date and time information when each of the multiple still images was captured.

[0061] FIG. 9 is a plan view of the interior of facility Z1. As shown in FIG. 9, facility Z1 includes multiple areas AR1 to AR5. Furthermore, multiple imaging devices 5 (5-1 to 5-5) are installed inside facility Z1. As shown in FIG. 9, imaging device 5-1 is installed so as to be able to capture images of users using area AR1. Preferably, identification information (hereinafter referred to as imaging device ID) that can identify imaging device 5 may be managed by information processing system 1 in association with the installation coordinates of imaging device 5 and the area that can be photographed by imaging device 5. In other words, the installation location (installation area) of imaging device 5 may be managed by information processing system 1.

[0062] In activities A106 to A108, a process for tracking the behavior of user X in facility Z1 (hereinafter referred to as user X tracking process) is executed. For example, the user X tracking process may be started when the authentication process is executed in activity A104 and user X is permitted to enter. That is, the user X tracking process is specifically a process for identifying and managing behavior information IF2 indicating the behavior of user X in facility Z1. The behavior information IF2 is information including the user's behavior history, which is identified based on the specific image IM1 and internal image IM2 of user X read from the storage unit 22. Next, as a management step, the management unit 233 manages the behavior information IF2 and the identification information IF1 in association with each other.

[0063] First, the identification unit 235 identifies an image (frame) in which the user X appears (hereinafter referred to as a specific internal image IM21) from the internal image IM2 acquired in activity A105 based on the specific image IM1 of the user X (activity A106).

[0064] Furthermore, the identification unit 235 identifies behavioral information IF2 of the user X based on information about the identified internal image IM21 (activity A107). The behavioral information IF2 includes information that can identify the area included in the facility Z1 and the user's stay time in the area. A specific method for identifying the behavioral information IF2 will be described with reference to FIGS. 10 and 11.

[0065] FIG. 10 is a diagram showing an example of an internal image IM2 captured by the imaging device 5. The internal image IM2 may be associated with the imaging device ID of the imaging device 5 that captured the image. Furthermore, the internal image IM2 may be associated with information that allows the time at which the image was captured to be ascertained. The internal image IM2 in FIG. 10 is assumed to be an image captured by the imaging device 5-1, capturing the state of area AR1. Furthermore, the internal image IM2 in FIG. 10 includes multiple person images M1-M3. In activity A106, the person images M1-M3 of the user are detected from the internal image IM2. Furthermore, in activity A107, the person images M1-M3 may be compared with the specific image IM1 of user X to calculate the degree of match between the person images M1-M3 included in the internal image IM2 and the specific image IM1 of user X. Furthermore, the person image M1 whose degree of match is equal to or greater than a predetermined threshold is identified as the image of user X. That is, if the internal image IM2 shown in FIG. 10 includes user X, it is a specific internal image IM21.

[0066] Next, an example of a method for identifying behavior information IF2 of user X from a specific internal image IM21 will be described. It is assumed that each specific internal image IM21 is associated with an image capture device ID and information that allows the time at which the specific internal image IM21 was captured to be known.

[0067] First, the identification unit 235 identifies the area (AR1) where user X stayed from the specific internal image IM21. The area may be identified based on, for example, the imaging device ID of the imaging device 5. Next, the identification unit 235 identifies the time that user X stayed in area AR1 based on date and time information indicating the date and time that specific internal image IM21 was captured. More specifically, the identification unit 235 identifies the time when capturing of the specific internal image IM21 began (the time when user X's figure began to appear in internal image IM2). Similarly, the identification unit 235 identifies the time when capturing of the specific internal image IM21 ended (the time when user X's figure disappeared from internal image IM2). This makes it possible to determine that user X stayed in area AR01, as well as the start and end times of their stay.

[0068] (Behavior history table T2) 11 is a schematic diagram of behavior history table T2. The behavior history table T2 is an example of a database that is referenced to understand the behavior history of user X at sports gym Z, for example. As shown in FIG. 11, the behavior history table T2 may include information such as a member ID (T21), a facility ID (T22), an entry time (T23), an exit time (T24), a stay area (T25), a start time (T26), and an end time (T27).

[0069] Here, the behavior history table T2 will be described. The behavior history table T2 is a database for managing user behavior information IF2, which includes the behavior history of the user within the facility Z1. For example, when the user behavior information IF2 is identified in the activity A107 of FIG. 6, a record corresponding to the behavior information IF2 may be added to the behavior history table T2 in the activity A108. Preferably, the user information table T1 is stored in the storage unit 22 of the information processing device 2.

[0070] The member ID (T21) is information that can identify a user who is a member of the gym Z, and corresponds to T11 in the user information table T1. The facility ID (T22) is information that can identify a facility provided by the gym Z.

[0071] The entry time (T23) is information indicating the time when the user entered the facility. Specifically, it may be the time when the authentication process was executed in activity A103 of FIG. 6 and the user was permitted to enter. The exit time (T24) is information indicating the time when the user left the facility. Specifically, it may be the time when the authentication process was executed again in activity A109 of FIG. 6 to allow the user to exit. According to this embodiment, the entry time and exit time of user X to the facility can be grasped as an example of the behavior information IF2 of user X.

[0072] Next, the stay area (T25) is information indicating the area where user X stayed while using facility Z1. Furthermore, the start time (T26) is information indicating the time when the stay in the corresponding stay area started. Furthermore, the end time (T27) is information indicating the time when the stay in the corresponding stay area ended. The behavior information IF2 relating to T25 to T27 is identified in the processing of activities A106 to A107 in FIG. 6.

[0073] According to the above process, for each user of sports gym Z, it is possible to grasp the behavior of user X within the facility, such as the type of exercise equipment used within the facility, the duration of use, or participation in lessons.

[0074] Here, when the authentication unit 234 executes the authentication process for user X again, the management unit 233 ends the identification of the behavior information IF2 based on the specific image IM1 of user X (activity A109). For example, when user X leaves facility Z1, the authentication process may be executed by having the reader RD read the two-dimensional code, as was done at the time of entry. In response to the execution of the authentication process again, the tracking process for user X may end.

[0075] To summarize the above, the information processing system includes at least one processor. The processor includes a program to perform the following functions: As a first acquisition step, the acquisition unit 232 acquires identification information IF1 that identifies a user and a captured image that shows at least a portion of the user, in association with each other. Here, the captured image is defined as a specific image IM1 related to the user. As a second acquisition step, the acquisition unit 232 acquires an internal image IM2 that captures the interior of a facility where the user is staying. As a management step, the management unit 233 manages the behavioral information IF2 and the identification information IF1 in association with each other. Here, the behavioral information IF2 is information including the user's behavior history, which is identified based on the specific image IM1 and the internal image IM2.

[0076] According to this aspect, it is possible to track the behavior of users within a facility. For example, even if there are no staff members present at the facility, it is possible to grasp the behavior of users.

[0077] Furthermore, by tracking the user's behavior based on an image taken immediately before the user enters the facility, the user can be identified from an image taken inside the facility based on the user's current clothing, hairstyle, etc. This improves the accuracy of identifying the user's behavior history. 3.2 Use of behavioral information IF2

[0078] As described above, each time user X uses a facility, behavioral information IF2 of user X at the facility is identified (activity A106). The behavioral information IF2 is stored in the behavior history table T2. In this section, an example of utilizing the accumulated behavioral information IF2 of the user will be described.

[0079] Furthermore, as an identification step, the identification unit 235 may identify a behavioral characteristic Ty of a user X (first user) who uses the facility Z1 based on the behavior information IF2. Furthermore, as a display control step (an example of a presentation step), the display control unit 238 may present a response guideline corresponding to the behavioral characteristic Ty to a second user who is an employee Y of the facility Z1. Note that the response guideline may be presented directly to the first user.

[0080] Here, employee Y may be an instructor working at sports gym Z. Employee Y may be, for example, a person who stays at facility Z1 during at least part of the opening hours of facility Z1. Employee Y may be, for example, a person who explains how to use facility Z1 to users, a person who gives training advice to users of facility Z1, or a person who is engaged in cleaning facility Z1.

[0081] (Proposal screen 9) 12 shows an example of a proposal screen 9 displayed on the display unit 44 of the employee terminal 4. For example, the display control unit 238 of the information processing device 2 may control the display unit 34 to display the proposal screen 9 by receiving a request from employee Y to display the proposal screen 9. The proposal screen 9 includes areas 91 to 93.

[0082] The area 91 includes information about user X. Specifically, the proposal screen 9 is displayed on the display unit 44 in a manner in which the information about user X is included in the area 91. The information about user X included in the area 91 may be, for example, information that can be identified based on the user information table T1 of FIG. 8.

[0083] The area 92 includes a behavioral characteristic Ty of the user X at the facility, which indicates the usage status of the facility by the user X. Specifically, the proposal screen 9 is displayed on the display unit 44 in a manner in which the behavioral characteristic Ty of the user X is included in the area 92. The behavioral characteristic Ty may be managed, for example, in association with each user. Specifically, the behavioral characteristic Ty may be associated with a member ID and stored in the user information table T1 of FIG. 8.

[0084] Furthermore, the behavioral characteristics Ty may include an index indicating the likelihood of a user canceling their membership. This allows employee Y to identify users who are likely to cancel their membership. Therefore, employee Y can respond to users who are likely to cancel their membership at an early stage, such as by interviewing them about their training-related concerns.

[0085] Here, an example of a method for identifying the behavioral characteristic Ty will be described. For example, the artificial intelligence unit 239 of the information processing device 2 may have a learning model LM. The learning model LM may be a learning model that generates and outputs the behavioral characteristic Ty of a user based on the behavioral information IF2 of the user included in the behavior history table T2.

[0086] This learning model LM may be a model generated by performing so-called supervised machine learning, in which the user information contained in the user information table T1 and the user behavior information IF2 contained in the behavior history table T2 are used as input data (explanatory variables) for learning, information indicating the user's actual behavior (including withdrawal) and suggestions received from employees are used as output data (objective variables) for learning, and the input data and output data are used as teacher data in which the input data and output data are associated in pairs.

[0087] The identification unit 235 may identify the behavioral characteristics Ty and response guidelines of the user X based on the information output by the learning model LM. Note that, in the present embodiment, an example in which the behavioral characteristics Ty of the user are identified by the learning model LM has been described, but this is not limitative. For example, the behavioral characteristics Ty of the user may be identified by a data table in which the behavioral information IF2 is associated with the behavioral characteristics Ty, etc.

[0088] According to this embodiment, the employee can grasp the behavioral characteristics of the facility users, and can also provide the users with appropriate training advice or propose changes to their contract plans. 4.Other The information processing system 1 according to the above embodiment may be configured as follows.

[0089] In the above embodiment, an example has been described in which a behavioral characteristic Ty is identified from the behavioral information IF2 of a user X who uses facility Z1, and appropriate advice is provided to the user X, but this is not limited to this. For example, the user whose behavioral information IF2 is identified may be a second user who is an employee of facility Z1. Furthermore, the evaluation unit 236 may evaluate the working status of the second user based on the behavioral information IF2 of an employee Y who stays at facility Z1.

[0090] An example of a method for evaluating employee Y will be described. As in the above embodiment, the areas where employee Y stayed in facility Z1 and the duration of stay are stored in the behavior history table T2. For example, the evaluation unit 236 may evaluate employee Y's work attitude by determining that employee Y stayed in the changing room area based on the behavior history table T2, and by assuming that employee Y cleaned the changing room area and checked for any leftover items. Alternatively, for example, the evaluation unit 236 may evaluate employee Y's work attitude by determining that employee Y stayed in the reception area for a predetermined period of time or longer, and by assuming that employee Y worked as a receptionist at the facility. Furthermore, the evaluation unit 236 may evaluate employee Y's work attitude by determining that employee Y stayed within a predetermined distance from the user's location, and by assuming that employee Y provided training support to the user.

[0091] According to this aspect, the facility manager can understand the work attitude of the staff when he / she is not at the facility, and the employees can be motivated to work.

[0092] Furthermore, in the above embodiment, the process of identifying the behavioral information IF2 of the authenticated user in activity A103 of FIG. 6 has been described. However, this is not limited to this. When the identification information IF1 includes predetermined information, the behavioral information IF2 of the specific user corresponding to the identification information IF1 may not be identified. For example, when a user who has not yet joined gym Z uses the facility to try it out, the user's identification information IF1 may include predetermined information that identifies the user as one whose behavioral information IF2 is not identified. Similarly, for example, the identification information IF1 of an employee with a predetermined employment status may include predetermined information that identifies the user as one whose behavioral information IF2 is not identified. The predetermined information may be, for example, a specific character or symbol included in the identification information IF1. In such a case, the captured image acquired in activity A102 does not need to be defined as a specific image IM1. According to this aspect, tracking is not performed on users who are using the facility for trial or temporary use. Therefore, the privacy of users or employees can be protected in cases where no contract regarding personal information is concluded, such as temporary or trial use.

[0093] Furthermore, the image of the user stored in the storage unit 22 during member registration may be replaced with the specific image IM1. For example, the user image may be stored in the user information table T1. Specifically, in the information processing system 1, user images (images of the user) are managed in association with identification information IF1, and the user image is an image showing the face of the user corresponding to the identification information IF1. Furthermore, when information related to the specific image satisfies a predetermined condition, the update unit 237 may update the image managed as the user image to an image specified as the specific image IM1. Here, the predetermined condition may be, for example, that the user's face is shown at a predetermined size or larger. Also, for example, the predetermined condition may be that a predetermined period of time (e.g., six months to one year) has passed since the user image was last changed. According to this embodiment, the facial image registered during member registration can be updated as needed. Therefore, the update process is performed as needed without the user having to perform a new procedure. Note that the image managed as the user image may include an image showing the user's entire body. According to this embodiment, employees can more easily verify the identity of users.

[0094] Furthermore, in the present embodiment, a case where a member ID is used for user authentication has been described, but this is not limited thereto. The management unit 233 further associates and manages a specific image IM1 with identification information IF1 each time a user enters or exits a facility, i.e., each time the first acquisition step is executed. Here, a plurality of specific images corresponding to the identification information IF1 are defined as a reference image group. Furthermore, if the reference image group satisfies a predetermined condition, the determination unit determines whether or not the user is permitted to enter the facility based on a new specific image newly acquired by the acquisition unit 232 and the reference image group. Here, the predetermined condition may be a state in which facial authentication is possible using the reference image group. Specifically, the predetermined condition may include, for example, a condition in which the number of images included in the reference image group is greater than or equal to a predetermined number. Furthermore, for example, the predetermined condition may include a condition in which the images included in the reference image group were acquired within a predetermined period.

[0095] According to this aspect, users whose reference images satisfying predetermined conditions are managed in the information processing system 1, for example, users who enter and exit the facility more frequently than a predetermined frequency, can undergo authentication procedures using facial recognition, eliminating the need to read their member ID when entering. On the other hand, by allowing authentication procedures using facial recognition only for users who satisfy predetermined conditions, it is possible to reduce unauthorized entry by users.

[0096] In addition, in the present embodiment, an example has been described in which the process of identifying the behavioral information IF2 of a user (user X or employee Y) is executed in real time, but this is not limited to this. For example, the acquisition unit 232 may acquire an internal image IM2 and a specific image IM1 of the user that are managed in advance in the storage unit 22 or the like at any timing, thereby identifying the behavioral information IF2 of the user. In such a case, the corresponding internal image IM2 and specific image IM1 may be matched based on date and time information associated with the internal image IM2 and the specific image IM1, respectively.

[0097] In one embodiment, the reception unit 231, the acquisition unit 232, the management unit 233, the authentication unit 234, the identification unit 235, the evaluation unit 236, the update unit 237, the display control unit 238, and the artificial intelligence unit 239 are described as functional units realized by the processor 23 of the information processing device 2, but at least some of these may be implemented as functional units realized by an external server not shown, or as functional units realized by the processor of the user terminal.

[0098] The information processing device 2 may be an on-premise type or a cloud type. As the information processing device 2 in the cloud type, the above functions and processes may be provided in the form of, for example, SaaS (Software as a Service) or cloud computing.

[0099] In the above embodiment, the information processing device 2 performs various storage and control operations, but multiple external devices may be used instead of the information processing device 2. That is, various information and programs may be distributed and stored in multiple external devices using block chain technology or the like.

[0100] The above-described embodiments may be implemented as an information processing method. The information processing method includes the steps of the information processing system. The above-described embodiments may also be implemented as a program. The program causes at least one computer to execute the steps of the information processing system.

[0101] Furthermore, it may be provided in the following manner.

[0102] (1) An information processing system comprising at least one processor, the processor configured to execute a program to perform the following steps: in a first acquisition step, identification information for identifying a user is associated with a captured image showing at least a portion of the user, and the captured image is defined as a specific image relating to the user; in a second acquisition step, an internal image of the interior of a facility where the user is staying is acquired; and in a management step, behavioral information is associated with the identification information and managed, wherein the behavioral information is information including the behavioral history of the user, identified based on the specific image and the internal image.

[0103] According to this aspect, the behavioral history of users at a facility can be managed together with the user's identification information, allowing the facility manager to grasp the behavior of each user.

[0104] (2) In the system described in (1) above, the specific image includes a part that shows characteristics of the user's appearance.

[0105] According to this aspect, it is possible to identify the portion in which the user appears from the internal image captured inside the facility.

[0106] (3) In the system described in (1) above, the behavioral information includes information that can identify the area included in the facility and the user's stay time in the area.

[0107] According to this aspect, it is possible to grasp the user's staying area and feeling of staying within the facility.

[0108] (4) In the system described in (1) above, the specific image is an image taken at the time the user enters the facility or a specified area included in the facility.

[0109] According to this aspect, it becomes easy to identify the user from the internal image.

[0110] (5) In the system described in (1) above, the user is a first user who uses the facility, and further, in the identification step, the behavioral characteristics of the first user are identified based on the behavioral information.

[0111] According to this aspect, it is possible to grasp the behavior of facility users within the facility.

[0112] (6) In the system described in (5) above, the presentation step further comprises presenting a response guideline corresponding to the behavioral characteristic to the first user or a second user who is an employee of the facility.

[0113] According to this embodiment, the staff can make suggestions to the user.

[0114] (7) In the system described in (6) above, the behavioral characteristics include the user's likelihood of canceling membership.

[0115] (8) In the system described in (1) above, if the identification information includes predetermined information, the behavioral information of a specific user corresponding to the identification information is not identified.

[0116] In this manner, trial or temporary users are not tracked, thus preserving privacy where necessary.

[0117] (9) In the system described in (1) above, the users include a first user who uses the facility and a second user who is an employee of the facility, and further, in the evaluation step, the working status of the second user is evaluated based on the behavioral information.

[0118] According to this aspect, the facility manager can evaluate the work performance of the staff.

[0119] (10) In the system described in (1) above, a user image is managed in association with the identification information, the user image is an image showing the face of the user corresponding to the identification information, and further, in the update step, if information relating to the specific image satisfies a predetermined condition, the image managed as the user image is updated to the specific image.

[0120] According to this aspect, it is possible to update the face photograph stored in advance as member information.

[0121] (11) In the system described in (1) above, in the management step, each time the first acquisition step is executed, the specific image and the identification information are further associated and managed, and multiple specific images corresponding to the identification information are defined as a group of reference images, and further, in the judgment step, if the group of reference images satisfies a predetermined condition, a judgment is made as to whether the user is allowed to enter the facility based on a new specific image newly acquired in the first acquisition step and the group of reference images.

[0122] According to this aspect, only users who have a large number of specific images stored, i.e., users who frequently use the facility, are allowed to enter through face authentication.

[0123] (12) An information processing method, comprising the steps of the information processing system according to any one of (1) to (11) above.

[0124] (13) A program that causes a computer to execute each step of the information processing system described in any one of (1) to (11) above. Of course, this is not the case.

[0125] Finally, while various embodiments of the present invention have been described, these are presented by way of example only and are not intended to limit the scope of the invention. The novel embodiments may be embodied in various other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. Such embodiments and modifications are intended to be included within the scope and spirit of the invention, as well as within the scope of the inventions and their equivalents as defined in the accompanying claims. [Explanation of symbols]

[0126] 1: Information processing system 11: Network 2: Information processing equipment 20: Communication bus 21: Communications Department 22: Storage section 23: Processor 231: Reception 232: Acquisition Department 233: Management Department 234: Authentication section 235: Specific part 236: Evaluation section 237: Update section 238: Display control unit 239: Artificial Intelligence Department 3: User terminal 30: Communication bus 31: Communications Department 32: Storage section 33: Processor 34:Display section 35: Input section 4: Employee terminal 41: Communications Department 42: Storage section 43: Processor 44: Display section 45: Input section 5: Imaging device 5-1: Imaging device 5-2: Imaging device 5-3: Imaging device 5-4: Imaging device 5-5: Imaging device 50: Communication bus 51: Communications Department 52: Storage section 53: Control unit 54: Imaging unit 6: Overview 7: Floor plan of facility Z1 8: Internal image 9: Suggestion screen 91: area 92: area 93 :Area AR01: Area AR02: Area AR03: Area AR04: Area AR05: Area IF1: Identification information IF2: Behavioral information IM1: Specific image IM2: Internal image IM21: Specific internal image LM: Learning Model M1: Person image M2: Portrait image M3: Portrait image RD: Reader T1: User information table T2: Action history table Ty: Behavioral characteristics X: User Y: Employee Z: Sports gym Z1: Facilities

Claims

1. An information processing system, at least one processor; The processor is configured to execute a program to perform the following steps: In the first obtaining step, identification information for identifying a user is obtained in association with a captured image in which at least a part of the user is shown, and the captured image is defined as a specific image related to the user; In the second acquisition step, an internal image of the facility where the user is staying is acquired, In the management step, behavioral information and the identification information are managed in association with each other, and the behavioral information is identified based on the specific image and the internal image, and includes information that can identify an area included in the facility and a stay time of the user in the area; In the identification step, a behavioral characteristic of a first user is identified based on the behavioral information of the user and a learning model; the first user is a user who is included in the users and uses the facility; the learning model is a model that has been trained in advance on a relationship between the behavioral information and the behavioral characteristics, In the presenting step, the behavioral characteristic of the first user is presented to a second user who is an employee of the facility. system.

2. 10. The system of claim 1, The behavioral characteristics include the first user's likelihood of unsubscribing.

3. 10. The system of claim 1, A system in which the specific image includes a part that indicates a characteristic of the user's appearance.

4. 10. The system of claim 1, The specific image is an image taken when the user enters the facility or a specified area included in the facility.

5. 10. The system of claim 1, A system in which, if the identification information includes predetermined information, behavioral information of a specific user corresponding to the identification information is not identified.

6. 10. The system of claim 1, the users include the second user; Furthermore, in the evaluation step, the system evaluates the working status of the second user based on the behavioral information.

7. 10. The system of claim 1, a user image is managed in association with the identification information; the user image is an image showing a face of a user corresponding to the identification information, Furthermore, in the updating step, if information relating to the specific image satisfies a predetermined condition, the system updates the image managed as the user image to the specific image.

8. 10. The system of claim 1, In the management step, each time the first acquisition step is performed, the specific image and the identification information are further associated with each other and managed, and a plurality of the specific images corresponding to the identification information are defined as a reference image group; Furthermore, in a determination step, if the group of reference images satisfies predetermined conditions, the system determines whether the user is allowed to enter the facility based on a new specific image newly acquired in the first acquisition step and the group of reference images.

9. An information processing method, comprising: A method comprising the steps of the information processing system according to any one of claims 1 to 8.

10. A program, A program that causes a computer to execute each step of the information processing system according to any one of claims 1 to 8.

Images