Information processing system, information processing device, information processing method, and program

Abstract

To provide an information processing system capable of increasing customer satisfaction in a store.SOLUTION: An information processing system includes image capturing means 2, storage means 41 for storing a captured face image as a registered face image in association with user information, storage means 42 for storing a plurality of avatar images and a plurality of interaction information sets, user identification means 33 for identifying a user by comparing the face image with the registered face image, avatar image selection receiving means 73, output data generation means 36 for generating output data including a combination of one selected avatar image and an interaction information set associated therewith, detection means 74 for detecting a prescribed action of a user, and control means 35 for determining a combination of one selected avatar image and an interaction information set associated therewith on the basis of at least one of the detected prescribed action and the user information stored in the storage means 41, and causing the output data generation means 36 to generate output data including the combination.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Patent Document 1 discloses a virtual space distribution apparatus including a display control unit that displays a virtual store and a plurality of virtual store clerks represented on a virtual space, and a selection unit that selects a virtual store clerk who serves an avatar of a user represented on the virtual space from among the plurality of virtual store clerks.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional technology including the technology disclosed in Patent Document 1, it is not possible to control the display of virtual store clerks according to the user, and it is not possible to improve the user satisfaction. Further, it is not possible to enjoy the interaction with the virtual store clerks, and the entertainment elements are scarce.

[0005] An object of the present invention is to provide an information processing system, an information processing apparatus, an information processing method, and a program capable of improving customer satisfaction with a store.

Means for Solving the Problems

[0006] To achieve the above object, one aspect of the present invention is imaging means for imaging a face image of a user; user data storage means for storing the captured face image as a registered face image in association with user information of the user; Avatar data storage means for storing a plurality of avatar images and a plurality of pieces of dialogue information, User identification means for identifying the user by comparing the captured face image with the registered face image, Selection reception means for receiving selection of one avatar image from the plurality of avatar images, Output data generation means for generating output data including a combination of the selected one avatar image and the dialogue information corresponding to the one avatar image, Detection means for detecting a predetermined action of the user with respect to the dialogue information of the output data output to the user, Based on at least one of the predetermined action detected by the detection means and the user information stored in the user data storage means, determine a combination of the selected one avatar image and the dialogue information corresponding to the one avatar image, and cause the output data generation means to generate the output data including the combination. Control means, It is an information processing system having.

[0007] An information processing apparatus, an information processing method, and a program corresponding to the information processing system according to one aspect of the present invention are also provided as an information processing apparatus, an information processing method, and a program corresponding to the information processing system according to one aspect of the present invention.

Effect of the Invention

[0008] According to the present invention, it is possible to provide an information processing system, an information processing apparatus, an information processing method, and a program capable of improving customer satisfaction with respect to a store.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 6A

Figure 6B

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Mode for Carrying Out the Invention

[0010] Hereinafter, this embodiment will be described with reference to the drawings. <Information Processing System S> FIG. 1 is a diagram showing an example of the overall configuration of an information processing system S according to an embodiment of the present invention. The information processing system S is an information processing system configured such that a management server 1 and store terminals 2-1 to 2-n (n is an integer value of 1 or more) are connected via a network N. The network N is, for example, the Internet, a LAN (Local Area Network), a VPN (Virtual Private Network), or the like. Hereinafter, when it is not necessary to individually describe each of the store terminals 2-1 to 2-n, these are also collectively referred to as "store terminal 2".

[0011] The information processing system S is an information processing system applied to a store that provides various services (for example, provision of food and drink) to users who visit the store. The information processing system S enables interaction between a user who visits a store (hereinafter also referred to as a "store") to which the information processing system S is applied and an avatar image output to a store terminal 2 arranged in the store. Thereby, the user can enjoy the interaction with the avatar image output to the display of the store terminal 2 while using various services provided in the store.

[0012] Specifically, when a user visits a store, the information processing system S causes a store terminal 2 having imaging means (described later) to image the user and acquire the face image thereof. The information processing system S stores in advance, as registered face images, the face images of users who have a record of using the store. The information processing system S identifies the user by comparing the face image of the user imaged at the time of visiting the store with a plurality of registered face images stored therein. When the face image of the imaged user matches the registered face image and the user can be identified, the information processing system S outputs, to the display of the store terminal 2, an avatar image that enables interaction with the user.

[0013] The avatar image is an image in a virtual space that serves as the alter ego of the customer service staff for users who visit the store. For example, it is an image such as a character that appears in movies, comics, anime, etc. Note that since the avatar image is an image in a virtual space output to the display of the store terminal 2, it does not become the provider of services that involve the provision of physical objects among the services provided in the store. Therefore, the avatar image is different from, for example, the person in charge of serving food and drinks ordered by the user in the store. The person in charge of serving food and drinks in the store may be, for example, a robot that can move within the store, or a real-world person such as a waiter or waitress.

[0014] The operation of the avatar image output to the display of the store terminal 2 is not uniform, and enables unique operations according to various types of information regarding the user (hereinafter also referred to as "user information") stored in advance and a predetermined operation by the user (described later). The user information may include information managed by the management server 1 (for example, the user's usage history), or information input to the store terminal 2 by the user who visited the store (for example, the user's attributes). Furthermore, it may include information (for example, the user's attributes) input to the smartphone (not shown) held by the user before or after the user visits the store. In this case, the user's smartphone transmits the input user information to the management server 1 via the network N in FIG. 1.

[0015] The user information includes, for example, information such as the user's attributes (for example, gender, age, hobbies, date of birth), and the usage history as the user's usage record (for example, the number of store visits, the most recent store visit date and time, the avatar image selected in the past, past dialogue information, products ordered, etc.). Also, the "predetermined operation" is, for example, at least one of the operations of selecting a selection button (described later) output to the display of the store terminal 2, speaking a voice, and changing the body (changes in the body such as gestures and waving). Note that the predetermined operation by the user may include the operations performed by the user during the execution of an electronic game.

[0016] 〔Management Server 1〕 The management server 1 that constitutes the information processing system S is an information processing device as a server that manages the entire information processing system S. The management server 1 can execute a predetermined application program that enables the use of the information processing system S. The management server 1 can transmit various types of information to each of the store terminals 2 and the outside, and enable the execution of various processes. Also, the management server 1 can acquire various types of information transmitted from each of the store terminals 2 and the outside, and execute various processes.

[0017] For example, the management server 1 identifies a user by comparing the face image of the user captured by the store terminal 2 with the registered face image stored and managed in the database in advance. When the captured face image of the user matches the registered face image and the user can be identified, the management server 1 treats the identified user as a user who has already been registered as a member (existing member). On the other hand, when the captured face image of the user does not match the registered face image, the management server 1 registers the face image of that user as a user who has come to the store for the first time (new member).

[0018] Also, the management server 1 can add the face image of the user captured by the store terminal 2 to the registered face image. Furthermore, the management server 1 can perform face image comparison by AI (artificial intelligence). In this case, the management server 1 can let the AI learn the errors in the comparison results.

[0019] The management server 1 stores and manages a plurality of avatar images. Also, the management server 1 stores and manages a plurality of dialogue information as information in which each of the plurality of avatar images speaks in the virtual space. The dialogue information includes, for example, text data output to the display of the store terminal 2 and voice data output from the speaker of the store terminal 2.

[0020] The management server 1 causes the display of the store terminal 2 used by the user who visits the store to output a plurality of avatar images in a selectable manner. The "selectable manner" is, for example, a manner in which information indicating each of the plurality of avatar images (for example, a thumbnail image, etc.) is output on the display of the store terminal 2, or a manner in which a selection button is output.

[0021] When one avatar image is selected from the plurality of avatar images output on the display of the store terminal 2, the management server 1 generates output data including a combination of the selected one avatar image and the dialogue information corresponding to the avatar image. The output data generated by the management server 1 includes at least one of text data and voice data as dialogue information, and further includes display data of selection buttons for the user to select and answer the dialogue information output to the user.

[0022] When a predetermined operation of the user with respect to the dialogue information output from the store terminal 2 is detected, the management server 1 determines a combination of the avatar image and the dialogue information to be output to the store terminal 2 based on at least one of the detected predetermined operation and the user information. That is, the management server 1 determines a combination of the avatar image and the dialogue information that is a reaction to the predetermined operation of the user.

[0023] The management server 1 generates output data including the combination determined as the output target, and transmits the generated output data to the store terminal 2. As a result, the avatar image and the dialogue information are output on the store terminal 2. By repeating such processing, communication between the user and the avatar image is realized.

[0024] Here, when determining the combination of the avatar image and the dialogue information to be output to the store terminal 2, the management server 1 determines a combination including dialogue information different from the dialogue information output within a predetermined period. That is, the management server 1 does not output the same content of dialogue information to the user, but gives changes to the dialogue information to be output. Thereby, the quality of communication with the user can be improved. The "predetermined period" is not particularly limited and can be any period such as seconds, minutes, hours, days, weeks, months, etc.

[0025] The management server 1 enables the user DB 41 to store the user's registered face image in association with an incentive given based on the user's usage record (usage history). The incentive given to the user is, for example, a predetermined point that enables the use of various services provided in the store. The functional configuration and processing flow of the management server 1 will be described later.

[0026] 〔Store Terminal 2〕 The store terminal 2 is an information processing device installed in the store and operated by a user who visits the store. The store terminal 2 is composed of, for example, digital signage, a tablet terminal, etc. The store terminal 2 can execute a predetermined application program that enables the use of the information processing system S. The store terminal 2 can perform various processes based on various information transmitted from each of the management server 1 and the outside, and various information input by the user. Also, the store terminal 2 can transmit various information to each of the management server 1 and the outside.

[0027] For example, the store terminal 2 captures a user who visits the store and transmits the face image of the user to the management server 1. Also, the store terminal 2 acquires information (for example, thumbnail images) indicating each of a plurality of avatar images from the management server 1 and outputs them to the display in a selectable manner.

[0028] When an operation is performed on the store terminal 2 to select one avatar image from a plurality of avatar images output on the display in a selectable manner, the input information is transmitted to the management server 1. Then, when the output data generated by the management server 1 is transmitted, the store terminal 2 acquires the transmitted output data and outputs it to the display. The output data to be output includes an avatar image and interaction information. Note that the functional configuration and processing flow of the store terminal 2 will be described later.

[0029] The above-described processing by each of the management server 1 and the store terminal 2 constituting the information processing system S is merely an example. That is, it is only necessary for the information processing system S as a whole to have a function for realizing the above-described processing. Therefore, part or all of the functions for realizing the above-described processing may be shared or cooperate within the information processing system S.

[0030] For example, part or all of the functions of the management server 1 may be the functions of other information processing devices within the information processing system S. Also, part or all of the functions of other information processing devices within the information processing system S may be the functions of the management server 1. Furthermore, part or all of the functions of the management server 1 may be transferred to other servers or the like not shown in the figure. Thereby, the processing of the information processing system S as a whole is promoted, and it becomes possible to complement each other's processing.

[0031] <Hardware Configuration> 〔Hardware Configuration of Management Server 1〕 FIG. 2 is a block diagram showing an example of the hardware configuration of the management server 1 constituting the information processing system S of FIG. 1. The management server 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, an output unit 16, an input unit 17, a storage unit 18, a communication unit 19, and a drive 20.

[0032] The CPU 11 executes various processes according to the programs recorded in the ROM 12 or the programs loaded from the storage unit 18 into the RAM 13. The RAM 13 also appropriately stores data and the like necessary for the CPU 11 to execute various processes. The CPU 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output interface 15 is also connected to this bus 14.

[0033] Connected to the input / output interface 15 are an output unit 16, an input unit 17, a storage unit 18, a communication unit 19, and a drive 20. The output unit 16 is composed of a display, a speaker, etc., and outputs various information as images, sounds, etc. The input unit 17 is composed of a keyboard, a mouse, a touch panel, etc., and accepts the input of various information. The storage unit 18 is composed of a hard disk, a DRAM (Dynamic Random Access Memory), etc., and stores various data. The communication unit 19 communicates with other devices via the above-mentioned network N composed of the Internet, etc.

[0034] A removable medium 21 composed of a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, etc. is appropriately mounted on the drive 20. The program read from the removable medium 21 by the drive 20 is installed in the storage unit 18 as necessary. Also, the removable medium 21 can store various data stored in the storage unit 18 in the same manner as the storage unit 18.

[0035] 〔Hardware Configuration of Store Terminal 2〕 FIG. 3 is a block diagram showing an example of the hardware configuration of the store terminal 2 constituting the information processing system S of FIG. 1. The store terminal 2 has the same configuration as the hardware configuration of the management server 1 shown in FIG. 2. That is, the store terminal 2 has a CPU 51, a ROM 52, a RAM 53, a bus 54, an input / output interface 55, an output unit 56, an input unit 57, a storage unit 58, a communication unit 59, a drive 60, and a removable media 61, each corresponding to the CPU 11, the ROM 12, the RAM 13, the bus 14, the input / output interface 15, the output unit 16, the input unit 17, the storage unit 18, the communication unit 19, the drive 20, and the removable media 21 in FIG. 2. Among these, the output unit 56 functions as output means.

[0036] In addition to these hardware configurations, the store terminal 2 has an imaging unit 62 as imaging means. The imaging unit 62 is also connected to the input / output interface 55. The imaging unit 62 is composed of a camera or the like, and images, for example, users who come to the store. Specifically, the imaging unit 62 images the face of the user, the movement of the user's body, etc. The imaging unit 62 may image as a still image or as a moving image when imaging the user. Also, a plurality of imaging units 62 may image the user from a plurality of angles.

[0037] <Functional Configuration> 〔Functional Configuration of Management Server 1〕 FIG. 4 is a functional block diagram showing an example of the functional configurations of the management server 1 and the store terminal 2. In the CPU 11 of the management server 1, when operating, an acquisition unit 31, a data management unit 32, a user identification unit 33 as user identification means, an output control unit 34, a determination unit 35 as control means, a generation unit 36 as output data generation means, and a transmission control unit 37 function.

[0038] Also, in the storage unit 18 of the management server 1, various databases are provided. For example, a user DB 41 as user data storage means for storing and associating registered face images and user information in a database, and an avatar DB 42 as avatar data storage means are provided.

[0039] The acquisition unit 31 acquires various types of information. For example, the acquisition unit 31 acquires various types of information transmitted from the store terminal 2 to the management server 1. Specifically, the acquisition unit 31 acquires a face image transmitted for collation with a registered face image. Further, the acquisition unit 31 acquires user information transmitted for storage in association with the registered face image. Also, the acquisition unit 31 acquires information input for selecting one avatar image from among a plurality of avatar images output on the display of the store terminal 2. Additionally, the acquisition unit 31 acquires information indicating the content of a predetermined action of the user detected by the store terminal 2.

[0040] The data management unit 32 stores and manages various types of information in the database of the storage unit 18. For example, the data management unit 32 stores and manages the face image acquired by the acquisition unit 31 as a registered face image in the user DB 41. Further, the data management unit 32 stores and manages the user information acquired by the acquisition unit 31 in the user DB 41 in association with the registered face image. Also, the data management unit 32 stores and manages combinations of a plurality of avatar images and a plurality of dialogue information in the avatar DB 42.

[0041] The user identification unit 33 identifies the user by collating the face image of the user captured by the imaging unit 62 of the store terminal 2 with the registered face image stored and managed in the user DB 41 by the data management unit 32. Also, the user identification unit 33 enables identification of the position of the user who uttered the voice. Furthermore, the user identification unit 33 enables identification of the user based on the face image of the user in which a predetermined action has been detected. Note that the process of identifying the user by the user identification unit 33 may be performed using a predetermined AI model.

[0042] The output control unit 34 performs control to cause the store terminal 2 to output various types of information. For example, the output control unit 34 performs control to cause a plurality of avatar images to be output on the display of the store terminal 2 in a selectable manner. Further, the output control unit 34 performs control to cause the store terminal 2 to output output data. In this case, the output control unit 34 performs control to cause the avatar image and the dialogue information (text data) among the output data to be output on the display of the store terminal 2, and performs control to cause the dialogue information (voice data) among the output data to be output from the speaker of the store terminal 2.

[0043] The output data includes menus for various services provided at the store. For example, if the store is a restaurant and the services provided at the store include the provision of food and drink, then the menu of food and drink is output to the store terminal 2 as output data. Further, the output data includes electronic game data. The electronic game is started by a predetermined operation of the user. The content of the electronic game is not particularly limited, but for example, it may be an electronic game with relatively simple content such as "rock-paper-scissors" or "Simon says" that can be enjoyed by the user and the avatar image together. Hereinafter, the operation of the user that serves as a trigger to start the electronic game, that is, the operation for starting the electronic game, is also referred to as a "game start operation".

[0044] For example, assume that the store is a restaurant and the services provided at the store include the provision of food and drink. In this case, in the detection unit 74 of the store terminal 2, the operation of the user ordering food and drink may be detected as the game start operation, and the electronic game may be automatically started. Further, even after the start of the electronic game, a predetermined operation of the user is detected as the game start operation. That is, the operation for starting the electronic game and the operation for outputting the avatar image and the dialogue information are respectively detected. Thereby, the user can enjoy the electronic game while communicating with the output avatar image.

[0045] When the detection unit 74 of the store terminal 2 detects a predetermined operation as the user's reaction to the output dialogue information, the determination unit 35 determines a combination of an avatar image to be output to the store terminal 2 and dialogue information corresponding to the avatar image based on at least one of the detected predetermined operation and user information. Then, the determination unit 35 causes the generation unit 36 to generate output data including the determined combination. In addition, the determination unit 35 can determine a combination of an avatar image and dialogue information based on the user's attributes and usage history. Further, the determination unit 35 can determine a combination including dialogue information different from the dialogue information output within a predetermined period.

[0046] The generation unit 36 generates output data including a combination of an avatar image and dialogue information that is the output target according to the determination of the determination unit 35. For example, the generation unit 36 generates output data in which the avatar image faces the position of the user specified by the user specifying unit 33. In addition, the generation unit 36 can generate output data further including electronic game data based on user information.

[0047] The transmission control unit 37 controls the communication unit 19 (see FIG. 2) to transmit various types of information to each of the store terminal 2 and the outside. For example, the transmission control unit 37 performs control to transmit control information for outputting various types of information to the store terminal 2. Specifically, the transmission control unit 37 performs control to transmit control information for outputting a plurality of avatar images in a selectable manner to the store terminal 2. In addition, the transmission control unit 37 performs control to transmit control information for outputting output data to the store terminal 2.

[0048] 〔Functional Configuration of Store Terminal 2〕 When operating, the CPU 51 of the store terminal 2 functions as an acquisition unit 71, an output control unit 72, a reception unit 73 as selection reception means, a detection unit 74 as detection means, and a transmission control unit 75.

[0049] The acquisition unit 71 acquires various types of information. For example, the acquisition unit 71 acquires control information for outputting various types of information transmitted from the management server 1. Specifically, the acquisition unit 71 acquires control information for outputting a plurality of avatar images in a selectable manner transmitted from the management server 1. Also, the acquisition unit 71 acquires control information for outputting output data.

[0050] The output control unit 72 performs control to output various types of information. For example, the output control unit 72 performs control to output various types of information to the output unit 56 (see FIG. 3) based on the control information for outputting various types of information transmitted from the management server 1. Specifically, the output control unit 72 performs control to output a plurality of avatar images in a selectable manner to the display of the output unit 56. Also, the output control unit 72 performs control to output output data to the store terminal 2. In this case, the output control unit 72 performs control to output the avatar image and the dialogue information (text data) among the output data to the display of the output unit 56, and performs control to output the dialogue information (voice data) among the output data from the speaker of the output unit 56.

[0051] The reception unit 73 receives an operation of selecting one avatar image from the plurality of avatar images output to the display of the output unit 56. The detection unit 74 detects a predetermined operation of the user with respect to the dialogue information output from the output unit 56. The detection unit 74 notifies the management server 1 (decision unit 35) that it has detected a predetermined operation of the user.

[0052] The transmission control unit 75 performs control to cause the communication unit 59 (see FIG. 3) to transmit various types of information to the management server 1 and each external entity. For example, the transmission control unit 75 performs control to cause the communication unit 59 to transmit the face image of the user captured by the imaging unit 62 to the management server 1. Also, the transmission control unit 75 performs control to cause the communication unit 59 to transmit the input information input for selecting one avatar image from among the plurality of avatar images to the management server 1.

[0053] <Flow of processing of the management server 1 and the store terminal 2> Figures 5A to 6B are flowcharts showing an example of the processing flow of the management server 1 and the store terminal 2. Figure 5A is a flowchart showing an example of the processing flow until output data including a combination of one avatar image selected by the user and corresponding dialogue information is generated and transmitted in the processing of the management server 1. The management server 1 acquires the face image of the user transmitted from the store terminal 2 and performs collation with the registered face image (step S11). When the face image of the user captured by the store terminal 2 matches the registered face image and the user can be identified (YES in step S12), the management server 1 acquires the user information associated with the registered face image (step S13). Then, the management server 1 extracts the user attributes and usage history from the acquired user information (step S14) and proceeds to the processing of step S16.

[0054] On the other hand, when the captured face image of the user does not match the registered face image and the user cannot be identified (NO in step S12), the management server 1 registers the face image as a user who has come to the store for the first time (new member) (step S15) and proceeds to the processing of step S16.

[0055] The management server 1 transmits information indicating each of the plurality of avatar images to the store terminal 2 (step S16). Here, when the user who has come to the store is not a new member, the management server 1 may transmit information indicating each of the plurality of avatar images in line with the hobbies of the user based on the user attributes and usage history extracted in step S14 to the store terminal 2.

[0056] When the user who has come to the store is a new member (YES in step S17), the management server 1 proceeds to the processing of step S19. On the other hand, when the user who has come to the store is not a new member, that is, an existing member (NO in step S17), the management server 1 further extracts the dialogue information output within a predetermined period from the usage history of the user extracted in step S14 (step S18) and proceeds to the processing of step S19.

[0057] The management server 1 generates output data including a combination of one avatar image selected by the user and the dialogue information corresponding to the avatar image, and transmits it to the store terminal 2 (step S19). At this time, when the management server 1 extracts the dialogue information in step S18, it generates and transmits output data including dialogue information different from the extracted dialogue information.

[0058] FIG. 5B is a flowchart showing an example of the processing flow until one avatar image selected by the user is output in the processing of the store terminal 2. The store terminal 2 images the user who has come to the store and transmits the face image of the user to the management server 1 (step S21).

[0059] The store terminal 2 acquires information (for example, thumbnail images) indicating each of a plurality of avatar images from the management server 1 and outputs them to the display in a selectable manner (step S22). When an operation of selecting one avatar image is performed from among the plurality of output avatar images (YES in step S23), the store terminal 2 transmits the input information to the management server 1 (step S24). On the other hand, when an operation of selecting one avatar image has not been performed (NO in step S23), the store terminal 2 repeats the determination process of step S23. Note that in step S23, when an operation of selecting one avatar image has not been performed even after a predetermined time has elapsed, the store terminal 2 may display, for example, a message or an image prompting the selection of an avatar image on the display.

[0060] The store terminal 2 acquires the output data transmitted from the management server 1 (step S25), and outputs the combination of the avatar image and the dialogue information included in the acquired output data (step S26).

[0061] FIG. 6(A) is a flowchart showing an example of the processing flow performed when a game start operation is detected at the store terminal 2 in the processing of the management server 1. When the management server 1 receives information indicating that it has detected a game start operation (YES in step S31), it determines a combination of an avatar image and dialogue information to be output to the store terminal 2 together with the start screen of the electronic game (step S32), and proceeds to the process of step S34. On the other hand, when the information transmitted from the store terminal 2 is the content of a predetermined operation other than the game start operation (NO in step S31), it determines a combination of an avatar image and dialogue information according to the content of the predetermined operation (step S33). Then, the management server 1 proceeds to the process of step S34.

[0062] The management server 1 generates output data including the combination of the avatar image and dialogue information determined in step S32 or step S33 (step S34), and transmits the generated output data to the store terminal 2 (step S35).

[0063] When the management server 1 receives information indicating that the screen of the electronic game has been transitioned (YES in step S36), it determines a combination of an avatar image and dialogue information to be output to the store terminal 2 together with the screen of the electronic game after the transition (step S37). Then, the management server 1 generates output data including the determined combination of the avatar image and dialogue information (step S38), and transmits the generated output data to the store terminal 2 (step S39). On the other hand, when the information indicating that the screen of the electronic game has been transitioned has not been transmitted (NO in step S36), the management server 1 repeats the determination process of step S36. Note that in step S36, when the information indicating that the screen of the electronic game has been transitioned has not been transmitted even after a predetermined time has elapsed, the management server 1 may perform processes such as sending a transmission request for information indicating that the screen of the electronic game has been transitioned to the store terminal 2.

[0064] FIG. 6(B) is a flowchart showing an example of the processing flow performed when the store terminal 2 detects a game start operation. When the store terminal 2 detects a predetermined operation of the user on the output avatar image and interaction information (YES in step S41), it proceeds to the determination process of step S42. On the other hand, when the store terminal 2 does not detect a predetermined operation of the user (NO in step S41), it repeats the determination process of step S41. Note that in step S41, when the store terminal 2 does not detect a predetermined operation of the user even after a predetermined time has elapsed, for example, it may display a message, an image, etc. so as to perform a predetermined operation that becomes a game start operation on the display.

[0065] When the detected predetermined operation is a game start operation (YES in step S42), the store terminal 2 outputs the start screen of the electronic game (step S43) and transmits information indicating that the game start operation has been detected to the management server 1 (step S44). On the other hand, when the detected predetermined operation is a predetermined operation other than the game start operation (NO in step S42), the store terminal 2 transmits the content of the detected predetermined operation to the management server 1 (step S45) and proceeds to the process of step S46.

[0066] The store terminal 2 acquires the output data transmitted from the management server 1 (step S46) and outputs the combination of the avatar image and the interaction information included in the acquired output data (step S47). When the store terminal 2 is outputting the start screen of the electronic game (YES in step S48), it changes the screen of the electronic game according to the user's operation (step S49) and transmits information indicating that the screen has been changed to the management server 1 (step S50). On the other hand, when the store terminal 2 is not outputting the start screen of the electronic game (NO in step S48), it ends the process (END).

[0067] <Specific Example> 〔Specific Example of Store〕 FIG. 7 is a diagram showing a specific example of a store to which the information processing system S of FIG. 1 is applied. FIG. 7 shows a restaurant T, which is a specific example of a store to which the information processing system S of FIG. 1 is applied. In the restaurant T, there are arranged a store terminal 2-1 for guiding a user who has come to the store to a seat, and a robot R for guiding the user to a seat and serving the ordered food and drink to the seat. The store terminal 2-1 is a digital signage capable of imaging a user who comes to the store and outputting a combination of an avatar image A1 and interaction information W1.

[0068] In the specific example of FIG. 7, the seat E1 used by users U1 and U2 is a booth that can be used by a plurality of users simultaneously, and store terminals 2-2 and 2-3 are arranged. The store terminal 2-2 is a digital signage capable of outputting a combination of an avatar image A1 and interaction information W1, and the store terminal 2-3 is a tablet terminal capable of outputting a combination of an avatar image A1 and interaction information W1.

[0069] Store terminals 2-4 and 2-5 are arranged at the seat E2 used by users U3 and U4. The store terminal 2-4 is a digital signage capable of outputting a combination of an avatar image A2 and interaction information W2, and the store terminal 2-5 is a tablet terminal capable of outputting a combination of an avatar image A2 and interaction information W2.

[0070] For example, when users U1 and U2 enter the restaurant T, so-called check-in processing is performed. Specifically, the store terminal 2-1 images users U1 and U2 and transmits the face images of users U1 and U2 to the management server 1 (see FIG. 1). Then, face image matching is performed in the management server 1. Here, it is assumed that users U1 and U2 have visited the restaurant T in the past, and as a result of face image matching, users U1 and U2 are identified.

[0071] When the check-in processes for users U1 and U2 are successfully completed at the store, the store terminal 2-1 guides users U1 and U2 to their seats by outputting avatar images and dialogue information. This guidance may be such that the store terminal 2-1 outputs a pre-determined number for each seat as the dialogue information, or the robot R may guide users U1 and U2 to seat E1 while moving within the store.

[0072] When users U1 and U2 arrive at seat E1, a plurality of avatar images are output (as selection buttons) in a selectable manner on the display of at least one of store terminal 2-2 and store terminal 2-3. Users U1 and U2 consult and then determine one avatar image A1, and select that one avatar image by performing an input operation on either store terminal 2-2 or store terminal 2-3. Then, the selected one avatar image A1 is output large on the display of at least one of store terminal 2-2 and store terminal 2-3. Figure 7 shows the state where the selected one avatar image A1 is output large on the display of store terminal 2-2.

[0073] Thereby, users U1 and U2 can utilize various services provided at the restaurant T while communicating with the selected one avatar image A1. For example, users U1 and U2 can order food from the avatar image A1 or enjoy an electronic game after ordering food. The food and beverages ordered by users U1 and U2 via the avatar image A1 are served to seat E1 by the robot R.

[0074] 〔Specific Examples of Avatar Images and Dialogue Information〕 Figures 8 to 13 are diagrams showing specific examples of avatar images and dialogue information. FIG. 8 shows a specific example of the screens output to the store terminals 2-2 and 2-3 at the customer seats E1 of the restaurant T (reference) in FIG. 7. Among these, on the screen output to the store terminal 2-2, a logo mark with the store name "Hero Cafe" of the restaurant T and a message for the users U1 and U2 in FIG. 7 are displayed. Specifically, a message "Welcome! This is a cafe where you can enjoy your meal while communicating with your hero. Please enjoy it to your heart's content." is displayed.

[0075] Also, on the screen output to the store terminal 2-3, together with the message "Which hero would you like to talk to today?", avatar images A1 to A3 representing three characters who are heroes for the users U1 and U2 are displayed in a selectable manner. Here, assuming that the avatar image A1 is selected by an input operation on the store terminal 2-3 by the user U1 or the user U2. Then, the avatar image A1 selected by the user U1 or the user U2 is displayed on the screen output to the store terminal 2-2.

[0076] FIG. 9(A) shows a specific example when the selected avatar image A1 is displayed on the screen of the store terminal 2-2. As shown in FIG. 9(A), on the screen output to the store terminal 2-2, the avatar image A1 and a message as dialogue information are displayed. Specifically, a message "Nice to meet you. My name is XX. Please call me 'XX'." is displayed.

[0077] FIG. 9(B) shows a specific example of the screen output to store terminals 2-2 and 2-3 when the avatar image A1 receives a menu order. As shown in FIG. 9(B), on the screen of store terminal 2-2, the avatar image A1 and a message as dialogue information are displayed. Specifically, a message saying "Thank you for coming today! It's early, but please choose the menu you want to eat. All the menus are delicious, but my recommendation is curry A." is displayed. Also, on the screen of store terminal 2-3, thumbnail images of the menus provided at the restaurant T are displayed in a selectable manner (as selection buttons) together with a message saying "Please select what you want to eat from the menus A to F."

[0078] Here, assume that two menus (A: curry) are selected by an input operation on store terminal 2-3 by user U1 or user U2. In this case, as shown in FIG. 10(A), on the screen output to store terminal 2-2, the avatar image A1 and a message as dialogue information are displayed. Specifically, a message saying "Thank you for your order! I remember what I ate when I went to the hero training camp. Wait while I place your order now. Also, if you want to place an additional order, please talk to the tablet terminal." is displayed. Thus, the recommended menu differs for each selected avatar image, and a message incorporating a story for each menu is displayed. For this reason, the user can enjoy the fun of choosing the avatar image and menu each time they visit the store, so monotony is suppressed and the repeat rate can be increased.

[0079] FIG. 10(B) shows a specific example of the start screen of an electronic game output to store terminal 2-2. As shown in FIG. 10(B), on the start screen of the electronic game, the avatar image A1 and a message as dialogue information are displayed. Specifically, a message saying "Order completed! While waiting for the food, what should we do? That's right, let's have a quiz showdown!" is output.

[0080] The electronic game may be proposed by the avatar image A1 based on the user information of the users U1 and U2. In this case, since an appropriate electronic game can be provided according to the user information, for example, it is also possible to re-challenge an electronic game that the users U1 and U2 have experienced in the past or newly challenge an unexperienced electronic game.

[0081] FIG. 11(A) shows a specific example of the screen output to each of the store terminals 2-2 and 2-3 after the electronic game is started and the start screen is transitioned. As shown in FIG. 11(A), on the screen output to the store terminal 2-2, the avatar image A1 and a message (quiz question) as dialogue information are displayed. Specifically, a message (quiz question) "Who won the World Hero Tournament?" is displayed.

[0082] Also, as shown in FIG. 11(A), on the screen output to the store terminal 2-3, together with the message "Please select one from the following three people.", thumbnail images of each of the three avatar images A1 to A3 that are quiz options are displayed in a selectable manner (as selection buttons). Here, it is assumed that the correct avatar image A2 is selected by an input operation on the store terminal 2-3 by the user U1 or the user U2.

[0083] Then, as shown in FIG. 11(B), on the screen output to the store terminal 2-2, the avatar image A1 and a message (quiz answer) as dialogue information are displayed. Specifically, a message (quiz answer) "Correct! However, △△ who won didn't seem very happy." is displayed. Also, on the screen output to the store terminal 2-3, a message "Correct!" is displayed as a message indicating that the quiz has been answered correctly.

[0084] Fig. 12(A) shows a specific example of the screen output to the store terminal 2-2 when the dishes ordered by users U1 and U2 are completed. As shown in Fig. 12(A), an avatar image A1 and a message as interactive information are displayed on the screen output to the store terminal 2-2. Specifically, a message "It should be almost time for the dishes to be completed. When the robot arrives at the customer's seat, please receive the dishes." is output.

[0085] Fig. 12(B) shows a specific example of the screen output to the store terminal 2-2 after the ordered dishes are served to the customer seat E1. As shown in Fig. 12(B), an avatar image A1 and a message as interactive information are displayed on the screen output to the store terminal 2-2. Specifically, a message "It looks extremely delicious! Let's eat together! Well, while we're eating, let's review my heroic deeds together!" is output.

[0086] Here, although not shown in the figure, a digest moving image of an anime, movie, etc. in which the avatar image A1 appears may be displayed on the store terminal 2-2, and further, an explanation, review, and response of the digest moving image by the avatar image A1 may be output as interactive information. Also, the higher the bidirectionality of the communication between the avatar image A1 and the users U1 and U2, the more enjoyable it can be for the users U1 and U2. However, since it is necessary to prevent the meals of the users U1 and U2 from being disturbed, the avatar image A1 may be set to prioritize listening to the conversations of the users U1 and U2. Also, communication may be taken on the assumption that any one of the avatar images A1 to A3 cooks the meals of the users U1 and U2. Thereby, the meal time can be enlivened.

[0087] Also, although not shown in the figures, the case where user U1 or user U2 places an additional order for a menu item will be described. In this case, user U1 or user U2 can place an additional order for a menu item by performing an action of speaking towards the store terminal 2-3 as a predetermined action. For example, assume that user U1 or user U2 speaks "Please give me two cups of coffee." towards the store terminal 2-3 in order to place an additional order for coffee after a meal. Then, the content of the speech is detected and the additional order is accepted. Specifically, avatar image A1 speaks a message indicating that the additional order has been accepted.

[0088] Figure 13(A) shows a specific example of the screen output to the store terminal 2-2 when a predetermined time has elapsed after the ordered dish has been served to the customer seat E1, or when user U1 or user U2 has performed an action indicating that they have finished eating the dish as a predetermined action. The action indicating that the dish has been eaten includes, for example, an action where user U1 or user U2 speaks "Thank you for the meal!" towards the store terminal 2-3. As shown in Figure 13(A), an avatar image A1 and a message as dialogue information are displayed on the screen output to the store terminal 2-2. Specifically, a message "Ah - I'm full! How was the curry?" is output.

[0089] Also, Figure 13(A) shows a specific example of the screen output to the store terminal 2-3 when a predetermined time has elapsed after the ordered dish has been served to the customer seat E1. As shown in Figure 13(A), a message "Please let me know your thoughts on the dish." and three selection buttons B1 to B3 are displayed on the screen output to the store terminal 2-3. Note that the screen shown in Figure 13(A) may be output when user U1 or user U2 has performed an action indicating that they have finished eating the dish as a predetermined action.

[0090] Here, assume that the selection button B1 labeled "It was delicious!" is pressed by the input operation of the user U1 or the user U2 on the store terminal 2-3. Then, as shown in FIG. 13(B), on the screen output to the store terminal 2-2, an avatar image A1 and a message as dialogue information are displayed. Specifically, a message "Thank you! I'm very happy!" is displayed.

[0091] FIG. 14(A) shows a specific example of the screen output to the store terminal 2-2 after the operation of inputting the user's U1 or U2's impression of the dish is completed. As shown in FIG. 14(A), on the screen output to the store terminal 2-2, an avatar image A1 and a message as dialogue information are displayed. Specifically, a message "Okay! Let's take a photo together!" is output. Thus, the user U1 or the user U2 can take a photo together with the avatar image A1 using the smartphone they possess or the store terminal 2-3. Note that when the user U1 or the user U2 takes a photo using the store terminal 2-3, the developed photo may be a sales target.

[0092] FIG. 14(B) shows a specific example of the screen output to the store terminal 2-2 when the user U1 or the user U2 who has finished taking a photo presses the payment button B4 displayed on the screen of the store terminal 2-3 as a predetermined operation. As shown in FIG. 14(B), on the screen output to the store terminal 2-2, an avatar image A1 and a message as dialogue information are displayed. Specifically, a message "Thank you so much for coming today! Come again! I'm looking forward to it!" is output.

[0093] After that, a message including a guidance for settlement of the charge from the restaurant T is displayed on the screen of the store terminal 2-2. Specifically, for example, as shown in FIG. 15(A), a logo mark with the store name "Hero Café" of the restaurant T written thereon and a message for the users U1 and U2 are displayed. In the example of FIG. 15(A), a message "Thank you for coming today. For settlement, please use the tablet terminal installed at your seat. We, Hero, look forward to your next visit." is displayed. The user U1 or the user U2 can perform the settlement of the charge by operating the store terminal 2-3. Note that the settlement of the charge may be enabled by using the smartphone possessed by the user U1 or the user U2.

[0094] <Advantageous Effects of the Present Embodiment> According to the above-described embodiment, since the user can select his / her own avatar image and further the dialogue based on the past usage history is performed, the customer satisfaction with respect to the store can be improved.

[0095] <Others> As described above, one embodiment of the present invention has been described. However, the present invention is not limited to the above-described embodiment, and modifications, improvements, etc. within the scope capable of achieving the object of the present invention are included in the present invention. For example, each of the specific examples such as the avatar images and messages shown in FIGS. 7 to 15 is merely an example for achieving the object of the present invention and is not particularly limited.

[0096] Further, the above-described series of processes can be executed by hardware or can be executed by software. In other words, the above-described functional configurations are merely illustrative and not particularly limited. That is, it is sufficient that the information processing system is provided with a function capable of executing the above-described series of processes as a whole, and the functional blocks used for realizing this function are not particularly limited to the above examples.

[0097] In addition, in FIGS. 7 to 13 described above, a restaurant, which is a specific example of a store to which the information processing system S is applied, is illustrated. However, the store to which the information processing system S is applied is not limited to a restaurant. For example, the information processing system S can also be applied to the guest room of a hotel where a user stays.

[0098] In addition, in the above-described embodiment, when the captured face image of the user does not match the registered face image, the user is not identified and the face image is registered as a first-time visitor (new member) to the store. However, the present invention is not limited to this. For example, even when the captured face image of the user does not match the registered face image, the user may be identified when the user information input at the time of visiting the store matches the registered user information. In this case, the management server 1 causes the display of the store terminal 2 to output an input field for inputting predetermined user information. Then, the management server 1 can identify the user by collating the predetermined user information input in the input field with the user information stored in advance. When collating the user information, the predetermined user information to be input in the input field is, for example, the user ID and password given to the user when visiting the store for the first time. The management server 1 collates the input user information. When the input user information matches the registered user information and the user can be identified, the identified user is treated as a user who has already been registered as a member (existing member). On the other hand, when the input user information does not match the registered user information and the user cannot be identified, the management server 1 registers the face image of the user as a first-time visitor (new member) to the store.

[0099] In addition, in the above-described embodiment, a configuration is adopted in which a user who visits a store operates the store terminal 2 provided in the store to use various services provided in the store. However, the present invention is not limited to this. For example, an application program that enables the use of the information processing system S may be installed on a smartphone or the like owned by the user so that it can be used as the store terminal 2. In this case, for example, the user may be able to make a reservation for using the store before visiting the store by operating a smartphone or the like.

[0100] Also, the location of the functional blocks is not particularly limited and may be arbitrary. For example, the functional blocks of the management server 1 may be transferred to other devices or the like, or the functional blocks of other devices may be transferred to a server or the like. Also, one functional block may be configured by hardware alone, software alone, or a combination thereof.

[0101] When a series of processes are executed by software, the program constituting the software is installed in a computer or the like from a network or a recording medium. The computer may be a computer incorporated in dedicated hardware. Also, the computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose smartphone or personal computer other than a server.

[0102] A recording medium containing such a program is not only constituted by a removable medium (not shown) distributed separately from the apparatus main body to provide the program to a user or the like, but also constituted by a recording medium or the like provided to a user or the like in a state pre-installed in the apparatus main body. Since the program can be distributed via a network, the recording medium may be mounted on or accessible to a computer connected to or connectable to the network.

[0103] Note that in this specification, the steps of describing a program recorded on a recording medium include not only processes performed in chronological order according to the order, but also processes that are not necessarily processed in chronological order and are executed in parallel or individually. Also, in this specification, the term "system" shall mean the entire apparatus composed of a plurality of devices, a plurality of means, and the like.

Explanation of Reference Numerals

[0104] 1: Management server, 2: Store terminal, 11: CPU, 16: Output unit, 17: Input unit, 18: Memory unit, 19: Communication unit, 31: Acquisition unit, 32: Data management unit, 33: User identification unit, 34: Output control unit, 35: Decision unit, 36: Generation unit, 41: User DB, 42: Avatar DB, 51: CPU, 56: Output unit, 57: Input unit, 58: Memory unit, 59: Communication unit, 62: Imaging unit, 71: Acquisition unit, 72: Output control unit, 73: Reception unit, 74: Detection unit, 75: Transmission control unit, S: Information processing system, N: Network

Claims

1. An imaging means for capturing a face image of a user; a user data storage means for storing the captured face image as a registered face image in association with user information of the user; an avatar data storage means for storing a plurality of avatar images and a plurality of pieces of dialogue information; a user identification means for identifying the user by comparing the captured face image with the registered face image; a selection receiving means for receiving a selection of one avatar image from the plurality of avatar images; an output data generating means for generating output data including a combination of the one selected avatar image and the dialogue information corresponding to the one avatar image; a detection means for detecting a predetermined action of the user in response to the dialogue information of the output data outputted to the user; a control means for determining a combination of the one selected avatar and the dialogue information corresponding to the one avatar, based on at least one of the predetermined action detected by the detection means and the user information stored in the user data storage means, and for causing the output data generation means to generate the output data including the combination; An information processing system having the above configuration.

2. The user information includes attributes and usage records of the user, The control means determines the combination based on the attributes and usage history of the user. The information processing system according to claim 1 .

3. the control means determines the combination including dialogue information different from the dialogue information output within a predetermined period of time. The information processing system according to claim 2 .

4. the output data includes at least one of text data and voice data, and further includes display data of selection buttons that the user selects to respond to the dialogue information output to the user. The information processing system according to claim 1 .

5. The predetermined action is at least one of the output of the selection button, a voice, and a body change.

5. The information processing system according to claim 4.

6. an output means for outputting the output data to a booth that can be used simultaneously by a plurality of the users; The user identification means further identifies the user based on the face image of the user in which the predetermined action is detected by the detection means, The control means determines the combination to be output based on the detected predetermined action of the user and the user information of the user. The information processing system according to claim 1 .

7. The user identification means identifies a position of the user who has spoken a voice, the output data generating means generates the output data in which the avatar image is oriented with respect to the identified position of the user. The information processing system according to claim 1 .

8. an incentive based on the user's usage record is associated with the registered face image of the user, and the associated incentive is stored in the user data storage means; The information processing system according to claim 2 .

9. When the detection means detects an action for starting a predetermined electronic game as the predetermined action by the user, the output data generating means generates the output data further including data of the electronic game based on the user information; The predetermined action by the user includes an action performed by the user during execution of the electronic game. The information processing system according to claim 1 .

10. a user data storage means for storing the captured face image as a registered face image in association with user information of the user; an avatar data storage means for storing a plurality of avatar images and a plurality of pieces of dialogue information; a user identification means for identifying the user by comparing the captured face image with the registered face image; an output data generating means for generating output data including a combination of an avatar selected from the plurality of avatars and the dialogue information corresponding to the one avatar; a control means for determining, when a predetermined action of the user in response to the dialogue information of the output data output to the user is detected, a combination of the selected one of the avatars and the dialogue information corresponding to the one of the avatars based on at least one of the predetermined action and the user information stored in the user data storage means, and causing the output data generation means to generate the output data including the combination; An information processing device having the above configuration.

11. Taking a facial image of a user; storing the captured face image as a registered face image in association with user information of the user; storing a plurality of avatar images and a plurality of pieces of dialogue information; identifying the user by matching the captured face image with the registered face image; accepting a selection of an avatar from the plurality of avatars; generating output data including a combination of the selected one of the avatar images and the interaction information corresponding to the selected one of the avatar images; detecting a predetermined action of the user in response to the dialogue information of the output data outputted to the user; determining a combination of the selected one of the avatar images and the dialogue information corresponding to the selected one of the avatar images based on at least one of the detected predetermined action and the stored user information, and generating the output data including the combination in the output data generating step; An information processing method comprising:

12. Information processing system, Taking a facial image of a user; storing the captured face image as a registered face image in association with user information of the user; storing a plurality of avatar images and a plurality of pieces of dialogue information; identifying the user by matching the captured face image with the registered face image; accepting a selection of an avatar from the plurality of avatars; generating output data including a combination of the selected one of the avatar images and the interaction information corresponding to the selected one of the avatar images; detecting a predetermined action of the user in response to the dialogue information of the output data outputted to the user; determining a combination of the selected one of the avatar images and the dialogue information corresponding to the selected one of the avatar images based on at least one of the detected predetermined action and the stored user information, and generating the output data including the combination in the output data generating step; A program for executing control processing including the above.

Images