Virtual-person dialogue system, virtual-person dialogue server, and virtual-person dialogue program

Abstract

To provide a virtual person interactive system, a server and a program for appropriately coping with a situation even when the situation which cannot be coped with by a virtual person occurs.SOLUTION: The virtual-person dialogue system SYS includes a virtual-person dialogue server 1 that generates a virtual person, a plurality of user terminals 70 used by users, and a plurality of operator terminals 400 used by operators who are real persons. When the virtual-person dialogue server 1 is accessed by the user terminal 70, the virtual-person dialogue server 1 generates a virtual person, transmits the virtual person to the user terminal 70, causes the virtual person displayed on the user terminal 70 to have a dialogue with the user, analyzes the content of the dialogue between the virtual person and the user, generates an analysis result, determines whether or not an action by an operator is necessary, calls the operator when it is determined that the action by the operator is necessary, transmits an image of the operator to the user terminal 70, and causes the operator displayed on the user terminal 70 to have a dialogue with the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a virtual person dialogue system, a virtual person dialogue server, and a virtual person dialogue program. [Background technology]

[0002] Systems that allow users to interact with virtual characters displayed on a screen have been put to practical use. For example, Patent Document 1 discloses a technology that can collect information about a user's tendencies without making the user feel uncomfortable or tense, and can accurately provide the user with information that they need or are interested in.

[0003] Specifically, Patent Document 1 states that the characters have personalities implemented in the program and have the ability to ask questions to the user, that surveys requested by the server operator are conducted in the form of conversations between the user and the virtual personality, and that because control is carried out in a conversational format via the characters, advertisements for products in which the user is interested can be made through conversations from the characters, allowing the user to accept information without resistance. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2007-334732 Summary of the Invention [Problem to be solved by the invention]

[0005] The virtual person dialogue system can be installed in various facilities such as commercial facilities, event venues, and government offices to provide information to users. Users can obtain necessary information by interacting with a virtual person displayed on a monitor.

[0006] On the other hand, users may need a variety of information. For example, there may be cases where a user needs more detailed explanation during a conversation with a virtual character, but the virtual character cannot provide the necessary information.

[0007] Therefore, an object of the present invention is to provide a technology that can appropriately respond to situations that cannot be handled by a virtual character. [Means for solving the problem]

[0008] A virtual character dialogue system according to a representative embodiment of the present invention includes a virtual character dialogue server that generates a virtual character, a user terminal used by a user, and an operator terminal used by an operator who is a real person. When accessed from the user terminal, the virtual character dialogue server generates a virtual character, transmits the generated virtual character to the user terminal, and allows the user to interact with the virtual character displayed on the user terminal. The virtual character dialogue server analyzes the content of the dialogue between the virtual character and the user and generates an interaction content analysis result. The virtual character dialogue server determines whether or not an operator's response is required, and if it determines that an operator's response is required, calls the operator, transmits video of the operator captured by the operator terminal to the user terminal, and allows the user to interact with the operator displayed on the user terminal.

[0009] The virtual person dialogue server may determine whether or not an operator response is required based on the dialogue content analysis results and the operator response determination conditions, and may determine that an operator response is required if the dialogue content satisfies the operator response determination conditions.

[0010] The operator response determination conditions may include whether the dialogue between the virtual character and the user is proceeding smoothly, items and / or keywords that indicate that the dialogue between the virtual character and the user is not proceeding smoothly, the number of times that the dialogue between the virtual character and the user is not proceeding smoothly, items that the user pointed out as not being a successful dialogue, and the number of times that the user pointed out that the dialogue is not being a successful dialogue.

[0011] The virtual person dialogue server may determine that a response by an operator is required when the user requests dialogue with an operator.

[0012] When the dialogue between the operator and the user ends, the virtual person dialogue server may switch the user's dialogue partner to the virtual person.

[0013] In a representative embodiment of the present invention, when accessed from a user terminal, the virtual character dialogue server generates a virtual character, transmits the generated virtual character to the user terminal, allows the user to interact with the virtual character displayed on the user terminal, analyzes the content of the dialogue between the virtual character and the user, generates a dialogue content analysis result, determines whether or not an operator's response is required, and if it determines that an operator's response is required, calls the operator, transmits video of the operator captured by the operator terminal to the user terminal, and allows the user to interact with the operator displayed on the user terminal.

[0014] A virtual character dialogue program in a representative embodiment of the present invention causes a processor to perform the following processes when accessed from a user terminal: generate a virtual character, send the generated virtual character to the user terminal, have the user interact with the virtual character displayed on the user terminal, analyze the content of the dialogue between the virtual character and the user, and generate a dialogue content analysis result; determine whether or not operator response is required; and, if it is determined that operator response is required, call the operator, send video of the operator captured by the operator terminal to the user terminal, and have the user interact with the operator displayed on the user terminal. [Effects of the Invention]

[0015] According to the present invention, it is possible to appropriately respond even when a situation occurs that cannot be handled by a virtual character. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a diagram illustrating an example of the configuration of a virtual person dialogue system according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of the basic configuration of each device included in the virtual person dialogue server. [Figure 3] FIG. 2 is a diagram illustrating a specific configuration of a virtual person dialogue server. [Figure 4] FIG. 2 is a diagram illustrating an example of the configuration of an operator terminal. [Figure 5] FIG. 10 is a diagram illustrating an example of the configuration of a user terminal. [Figure 6] FIG. 10 is a sequence diagram illustrating a method for determining a video model to be used. [Figure 7] FIG. 10 is a sequence diagram illustrating a method for generating a voice of a virtual character. [Figure 8] FIG. 10 is a sequence diagram illustrating a method for determining a personality model of a virtual person. [Figure 9] FIG. 10 is a sequence diagram illustrating an example of a method for executing a dialogue with a virtual person. [Figure 10] FIG. 10 is a diagram illustrating an example of a state in which a virtual person is displayed on a user terminal. [Figure 11] FIG. 10 is a sequence diagram illustrating a method for switching a user's conversation partner from a virtual person to an operator. [Figure 12] FIG. 10 is a diagram illustrating an example of a state in which the person displayed on the user terminal is switched from a virtual person to an operator. [Figure 13] FIG. 10 illustrates a method for resuming a dialogue with a virtual person. DETAILED DESCRIPTION OF THE INVENTION

[0017] (Embodiment 1) Hereinafter, embodiments of the present invention will be described with reference to the drawings. In each drawing for explaining the embodiments, the same components are generally designated by the same reference numerals, and repeated description thereof will be omitted as appropriate.

[0018] <Configuration of virtual person dialogue system> The virtual person dialogue system is a system in which a user can interact with a virtual person by accessing a virtual person dialogue server from a user terminal. The virtual person dialogue system is used, for example, for guidance services, consultation services, etc. in various indoor and outdoor facilities such as commercial facilities, event venues, and government offices. The virtual person dialogue system is not limited to these uses and may also be used for users to enjoy interacting with a virtual person. The virtual person dialogue system is also configured to allow the user's dialogue partner to be switched from a virtual person to an operator depending on the situation.

[0019] Fig. 1 is a diagram illustrating the configuration of a virtual person dialogue system according to the first embodiment of the present invention. As shown in Fig. 1, the virtual person dialogue system SYS includes a virtual person dialogue server 1, one or more user terminals 70 used by each user of the virtual person dialogue system SYS, and one or more operator terminals 400 used by each operator. The virtual person dialogue server 1 and the operator terminal 400, and the virtual person dialogue server 1 and the user terminal 70 are connected to each other via a network NET, enabling transmission and reception of information.

[0020] When accessed from a user terminal 70, the virtual person dialogue server 1 generates a predetermined virtual person and allows the generated virtual person to have a dialogue with the user who accessed the virtual person dialogue server 1. The virtual person who will have a dialogue with the user may be, for example, a virtual person selected randomly from a plurality of candidates, or may be a virtual person selected arbitrarily by the user. The virtual person who will have a dialogue with the user may also be a predetermined virtual person corresponding to the facility where the virtual person dialogue system is used, or a virtual person corresponding to the characteristics of the operator (e.g., age, gender, physique, clothing, accessories, etc.) described below.

[0021] <<Virtual Person Interaction Server>> As shown in FIG. 1, the virtual person dialogue server 1 includes a storage management device 10, a virtual person generation device 20, a video generation device 30, a dialogue content analysis device 40, and an operator control device 60. These devices may be separate devices, or some or all of the devices may be implemented as a single device. When the virtual person dialogue server 1 is configured with multiple devices, these devices may be connected to each other through API collaboration. Furthermore, the virtual person dialogue server 1 may be cloud-based.

[0022] Here, a case where these devices are configured as separate devices will be described. Figure 2 is a diagram illustrating the basic configuration of each device included in the virtual person dialogue server. The basic configurations of these devices are generally similar, and these basic configurations will be described using Figure 2. After that, the specific configuration of each device will be described.

[0023] 2, each device in the virtual person dialogue server 1 includes a processor 51, a main memory 53, a storage 55, a communication interface 57, a monitor 59, etc. The communication interface 57 is a communication device that transmits and receives various information to and from a user terminal 70 and an operator terminal 400 via a network NET. The monitor 59 displays various information such as management screens for each device, information related to the virtual person dialogue server 1 and the virtual person dialogue system SYS, and the user interface.

[0024] The storage 55 includes various storage areas, such as a program storage area 55a and a data storage area 55b. The program storage area 55a stores various programs, such as basic programs such as an OS (Operating System) that runs the corresponding devices, and programs that cause the processor 11 to realize various functions of the virtual person dialogue system. The program storage area 55a also stores parameters for each program. In this way, the storage 55 functions as a program recording medium.

[0025] The data storage area 55b stores various information related to the virtual person dialogue system. The data stored in the data storage area 55b differs for each device. For example, the data stored in the data storage area 55b includes information provided to the user (a user of a facility, etc.) in dialogue with the virtual person, information related to the generation of the virtual person and dialogue with the virtual person, and user information (account information, etc.). The data stored in the data storage area 55b also includes operator information and information on currently available operators (operator presence / absence information). The operator information includes, for example, each operator's account information (contact information including an operator ID, telephone number, and email address), and the services each operator can handle (facility, specialty, language used, gender, age, or age group, etc.). The information provided to the user includes, for example, information related to the facility (e.g., services, products, directions to stores), etc.

[0026] The main memory 53 temporarily stores programs and parameters read from the program storage area 55a, various types of information read from the data storage area 55b, and calculation results by the processor 51. The main memory 53 also temporarily stores various types of information received via the communication interface 57, information to be transmitted via the communication interface 57, and the like.

[0027] The processor 51 reads and executes programs stored in the main memory 53, thereby configuring, in software, functional blocks that drive the components of each device, functional blocks that realize the virtual person dialogue server 1 and ultimately the virtual person dialogue system SYS, etc. Note that some of the functions may be configured in hardware, or may be configured in a combination of software and hardware.

[0028] In addition to these, each device may also be equipped with input devices such as a keyboard and a mouse, drives for various memory cards and recording media, and the like.

[0029] <<<Storage management device>>> Fig. 3 is a diagram illustrating a specific configuration of a virtual person dialogue server. Fig. 3 mainly shows the functional configuration of each device that constitutes the virtual person dialogue server 1. As shown in Fig. 3, the storage management device 10 includes, as software resources, at least an image model DB 11, a personality model DB 12, a virtual person data storage unit 13, and a communication processing unit 19. In this specification, "DB" is an abbreviation for "database."

[0030] The video model DB11 is a storage unit that stores multiple types of video models of human movements. The video models are video templates used to generate images of virtual people. The video models are data that configure the shape and movement of the torso in particular. The video models are also configured to integrate face data, which will be described later, and move the integrated face data together with the torso image.

[0031] The video model includes the appearance of multiple types of people with different physiques depending on height, weight, age, etc. The video model also includes multiple types of clothing that can be worn by each person and played back. Furthermore, the video model includes various data on the movements of people with each appearance, such as data on movements that are commonly performed during conversation, such as nodding, folding arms, and raising hands.

[0032] The video model may be video of recorded data of an actual person or a facility mascot character (hereinafter referred to as "actual person, etc."), or video of modeling data modeled using CG, or may include both. Note that "actual person, etc." may include not only actual people but also non-actual people such as a facility mascot character. In this embodiment, the actual person, etc. serving as the video model may be, for example, a celebrity such as a talent or artist. In this case, the video model may be video of recorded data of a celebrity, or video of modeling data of a celebrity modeled using CG, etc. The recorded data may include various information about the person, for example, video of the person, video of the person in action, the person's possessions, information posted by the person on blogs or social media, etc.

[0033] The personality model DB12 is a storage unit that stores multiple types of personality models for people. The personality model includes, for example, characteristics of answers to questions, and determines the answer policy, such as whether the content is positive or negative, and the emotions expressed in the answer. Furthermore, the personality model is not limited to answers to questions from users, but may also be characteristics of messages depending on the season, time of day, etc. The personality model DB12 may also store responses to questions that are expected in advance and that are in line with each personality model. This configuration reduces the computational burden of generating responses in line with the personality model to standard questions.

[0034] The virtual character data storage unit 13 is a storage unit that stores information about an image model, a personality model, and a voice determined for each virtual character. The virtual character data storage unit 23 stores information known to the virtual character, such as episodes and personal experiences of the target character. The virtual character data is determined and stored by the virtual character generation device 20. The virtual character data is called up by the video generation device 30 when a video of the virtual character is played back.

[0035] <<<Virtual Person Generation Device>>> As shown in Fig. 3, the virtual person generation device 20 comprises at least a video processing unit 21, an audio processing unit 22, a personality processing unit 23, and a communication processing unit 29 as software resources.

[0036] The video processing unit 21 is a functional unit that extracts appearance data used to generate a virtual character from data of a target person. The appearance data includes data such as the target person's face, body, hairstyle, and clothing. The video processing unit 21 also selects a video model to use in generating the virtual character and determines the video data to use for the virtual character's video. The video processing unit 21 may extract the appearance data of the virtual character based on information sources registered via the information source registration unit of the user terminal 70, as well as information sources acquired from the Internet. The video processing unit 21 may also extract appearance data to be used to generate a single virtual character based on information sources registered from multiple user terminals 70. When many users, such as celebrities, interact with a common virtual character, each user registers an information source for one virtual character. This configuration allows a virtual character to be generated based on more information sources, enabling more realistic interactions.

[0037] The information sources include, for example, videos, still images, and audio sources that include the target person, as well as records such as diaries created by the target person, documents expressing the target person's hobbies and preferences, and text data such as social media. The information sources also include information about the target person's language, clothing, and other possessions. The information sources may be registered by the user or obtained via the Internet. The obtained information sources are transmitted to the virtual person generation device 20.

[0038] The video processing unit 21 includes a moving image acquisition unit 211 , a still image acquisition unit 212 , a trimming unit 213 , an image correction unit 214 , a video model selection unit 215 , and a face insertion unit 216 .

[0039] The video acquisition unit 211 is a functional unit that acquires video data. The video acquisition unit 211 acquires videos included in an information source registered in the user terminal 70. The video acquisition unit 211 can also prompt the user to shoot a video through the user terminal 70. Situations in which a video can be shot through the user terminal 70 include, for example, when the target person is someone close to the user and a virtual person is displayed on another user terminal 70, or when a virtual person is generated to enable interaction with the target person even after the person has passed away. In this case, the video acquisition unit 311 may display a tutorial on the user terminal 70 to encourage the user to shoot a video.

[0040] The still image acquisition unit 212 is a functional unit that acquires still image data. The still image acquisition unit 212 acquires still images included in an information source registered in the user terminal 70. The still image acquisition unit 212 can also prompt the user to take a still image through the user terminal 70. In this case, the still image acquisition unit 212 may cause the user terminal 70 to display a tutorial to encourage the user to take a still image, i.e., a photograph. The still image acquisition unit 212 also converts video data into still images and acquires them. The still image acquisition unit 212 extracts images of the target person from various angles and images of various facial expressions and converts them into still images.

[0041] The trimming unit 213 is a functional unit that trims and extracts data of a target person from a still image. The trimming unit 213 may have a face recognition function and be able to automatically extract only the face of the target person.

[0042] The image correction unit 214 performs color correction and resolution correction on the extracted images to equalize the quality of the extracted images. The image correction unit 214 may also determine whether the extracted images are clear or not, and exclude unclear images from the extracted data group. The image correction unit 214 may also exclude images with a resolution lower than a predetermined value from the extracted data group.

[0043] The video model selection unit 215 is a functional unit that selects a video model to be used for generating a virtual person from the data in the video model DB 11. The video model selection unit 215 may select a video model that is most similar to the target person based on the appearance data acquired by the video acquisition unit 211, or may present multiple video models to the user terminal 70 and allow the user to select a video model to use. With this configuration, a video of a virtual person can be created using video models without having to register a sufficient number of information sources showing the virtual person moving.

[0044] The video model selection unit 215 may determine the clothing of the virtual person to be generated based on appearance data or on possession information included in the information source. The video model selection unit 215 may also select the clothing of the virtual person from the video model DB 11. That is, if there is an information source in which the target person is wearing that clothing, a video of the virtual person can be generated based on that information source. Even if there is no information source for the target person, a video of the virtual person can be generated based on possession information. Since clothing data can be selected from the video model DB 11, a virtual person can be easily generated even if data regarding the target person's clothing is insufficient. The video model selection unit 215 may generate videos of virtual people wearing multiple types of clothing, and the clothing may be changeable based on the time of year, the time of day, or user selection.

[0045] The video model selection unit 215 may determine the hairstyle of the virtual person to be generated based on the appearance data, or may select the hairstyle of the virtual person from the video model DB 11. Furthermore, the video model selection unit 215 may create videos of virtual people with a plurality of different hairstyles, and the hairstyle may be changeable.

[0046] In the above explanation, it has been assumed that the video processing unit 21 extracts data of a virtual character based on the information source of the target person himself / herself. However, recorded data of newly captured video or still images of a person who resembles the target person, or modeling data modeled using CG, may also be used to generate a virtual character. Furthermore, partial appearance data of a similar person, such as hairstyle or clothing, may also be used to generate a virtual character. In other words, the user may be able to select elements of the appearance data to be used to generate a virtual character.

[0047] The face insertion unit 216 is a functional unit that integrates the face data extracted by the video acquisition unit 211, the still image acquisition unit 212, the trimming unit 213, and the image correction unit 214 into the used video model. The face insertion unit 216 integrates the face data into the torso configured by the used video model, and creates a full-body image of a virtual person.

[0048] The voice processing unit 22 is a functional unit that artificially generates the speaking voice of the virtual person. The voice processing unit 22 includes a voice extraction unit 221 and a voice generation unit 222.

[0049] The voice extraction unit 221 is a functional unit that extracts the voice of a target person from an information source. For example, the voice extraction unit 221 may identify the voice of the person that is included for the longest time among multiple types of voices included in the information source as the voice of the target person.

[0050] The voice generation unit 222 is a functional unit that generates the voice of the virtual character based on the voice extracted by the voice extraction unit 221. The voice generation unit 222 may trim the voice of the target character and edit it so that it can be played back as the voice of the virtual character. The voice generation unit 222 may also select a voice similar to the voice of the target character from voice data prepared in advance and determine it as the voice of the virtual character. Furthermore, the voice generation unit 222 may generate an artificial voice similar to the voice of the target character. Note that voice generation may not be necessary if a message from the virtual character is displayed as text.

[0051] The personality processing unit 23 is a functional unit that determines the personality model of the virtual character. The personality processing unit 23 includes a text data registration unit 231, a personality model selection unit 232, and a personality model correction unit 233.

[0052] The text data registration unit 231 is a functional unit that extracts text data from information sources and stores it in the virtual person data storage unit 13 of the storage management device 10. The text data registration unit 231 extracts electronic text data from the target person's blog, SNS, etc., and stores it in the virtual person data storage unit 13 according to predetermined rules. The text data registration unit 231 may also read a handwritten document by the target person, such as a diary, convert it into text data, and store it in the virtual person data storage unit 13. Furthermore, the text data registration unit 231 may convert the target person's voice contained in audio or video data into text data and store it in the virtual person data storage unit 13.

[0053] The personality model selection unit 232 is a functional unit that selects a personality model (hereinafter also referred to as a "used personality model") to be used in generating a virtual character from the personality model DB 12. The personality model selection unit 232 presents questions about the personality of the virtual character via the user terminal 70. When an answer to the question is input from the user terminal 70, the personality model selection unit 232 selects an used personality model to be used in generating a virtual character from the data in the personality model DB 12 based on the answer.

[0054] A plurality of personality-related questions may be presented. Alternatively, the questions may be presented according to a chart that links input answers to the next question. As the user answers the questions, a basic personality classification of the virtual character is performed based on basic personality classifications prepared in advance. If the personality were to be determined based on information about the actual conversation of the target character, a huge amount of conversation information would be required. According to the virtual character dialogue system SYS of this embodiment, the answers to personality-related questions can be classified into one of the prepared personality classifications, so that the personality of the virtual character can be determined with a simple configuration even if there is insufficient information.

[0055] The personality model of the virtual character may be determined for each scenario pattern according to the type of question from the user. Scenario patterns include, for example, everyday conversations or consultations about worries. Once personality models are determined for some scenario patterns, a configuration may be made in which a dialogue conforming to the scenario pattern is possible. With this configuration, a dialogue can be conducted by determining only the personality model for the necessary scenario pattern, making it easy to determine the personality of the virtual character.

[0056] The personality model correction unit 233 is a functional unit that corrects the personality model used by the personality model selection unit 232. The personality model correction unit 233 receives an evaluation of the response made by the virtual character from the user terminal 70 and corrects the personality model used based on the evaluation. For example, the user inputs an evaluation of whether the response was appropriate for the target person. The user may also evaluate the virtual character's actions accompanying the response. The personality model correction unit 233 performs automatic learning using AI or the like to correct the personality model. This configuration allows the virtual character's personality to be corrected to be more similar to the target person. Note that when multiple user terminals 70 simultaneously or at different times interact with a single virtual character, the evaluations from the multiple user terminals 70 may be used to correct the personality model of the single virtual character. This configuration allows for a large amount of feedback to be provided to the virtual character's personality model, making the virtual character's personality model closer to the target person's personality and improving interaction accuracy.

[0057] Furthermore, the personality model correction unit 233 may determine whether a message from a virtual character is appropriate based on the user's response to the message, rather than on the user's evaluation, and may correct the personality model. The personality model correction unit 233 may convert the content of the user's response into text data and analyze it, or may infer the user's level of satisfaction from the user's tone of voice.

[0058] The communication processing unit 29 is a functional unit that communicates with the user terminal 70, the storage management device 10, and the video generation device 30 via the network NET.

[0059] <<<Video Creation Device>>> The moving image generation device 30 is a device that displays a moving image of a virtual person generated by the virtual person generation device 20 on a user terminal 70. As shown in FIG. 3 , the moving image generation device 30 includes a video display processing unit 31, a dialogue processing unit 32, a translation processing unit 33, and a communication processing unit 39.

[0060] The video display processing unit 31 is a functional unit that generates a speech video in which the virtual person speaks. The video display processing unit 31 performs modeling processing on the face data extracted from the appearance data, and makes the face data move in accordance with the speech.

[0061] The dialogue processing unit 32 is a functional unit that generates a message to be spoken by the virtual character based on the usage personality model. The content of the message may be a response to a question from the user, or may be words generated based on external information such as the date, season, time of day, or weather forecasts or news on the Internet. Furthermore, when responding to the user, the response may be generated based on external information such as the date, season, time of day, or weather forecasts or news on the Internet in addition to the usage personality model. The dialogue processing unit 32 determines the optimal response using AI.

[0062] The message generated by the dialogue processing unit 32 is spoken in a voice generated by the audio processing unit 22 of the virtual person generation device 20, and is played on the user terminal 70 together with the spoken image generated by the image display processing unit 31. The voice of the virtual person may be the lines of the target person extracted by the audio extraction unit 221. Alternatively, it may be played based on sound source data of a similar voice determined in advance. Furthermore, an artificial voice may be generated and played.

[0063] The translation processing unit 33 is a functional unit that generates a translation of a message spoken by a virtual character when the language used by the virtual character differs from the language used by the user. The translation processing unit 33 references the account information of the user who is interacting with the virtual character, and when the language used by the virtual character differs from the language used by the user, translates the message spoken by the virtual character into the language used by the user. The generated translation is then displayed in the speaking video generated by the video display processing unit 31 in accordance with the voice of the virtual character.

[0064] The communication processing unit 39 is a functional unit that communicates with the user terminal 70, the storage management device 10, and the virtual person generation device 20 via the network NET.

[0065] <<<Dialogue Content Analysis Device>>> The dialogue content analysis device 40 is a device that analyzes the content of a dialogue between a virtual character and a user. As shown in FIG. 3, the dialogue content analysis device 40 includes a dialogue content analysis unit 41, a dialogue content determination unit 42, and a communication processing unit 49.

[0066] The dialogue content analysis unit 41 analyzes the dialogue content between the virtual character and the user and generates a dialogue content analysis result. While the dialogue is taking place, the dialogue content analysis unit 41 updates the dialogue content analysis result as needed. The dialogue content analysis result may be stored in the dialogue content analysis device 40 or the storage management device 10.

[0067] The results of the dialogue content analysis may include, for example, dialogue items (multiple items possible), keywords, whether the dialogue is going smoothly, the time from receiving a question from the user to responding (response time), items and / or keywords for which the dialogue went smoothly, items and / or keywords for which the dialogue did not go smoothly, the number of times the dialogue went smoothly, the number of times the dialogue did not go smoothly, items for which the user pointed out that the dialogue was not going well, the number of times the user pointed out that the dialogue was not going well, etc.

[0068] The dialogue content determination unit 42 determines whether or not an operator's response is required instead of the virtual character based on the dialogue content analysis result generated by the dialogue content analysis unit 41. If the dialogue content satisfies the operator response determination condition, the dialogue content determination unit 42 determines that an operator's response is required, and the user's dialogue partner is switched from the virtual character to the operator. In this case, the operator, who is a real person, is called and displayed on the monitor of the user terminal 70. The operator takes over the dialogue with the user from the virtual character.

[0069] Examples of situations in which an operator may be required to respond on behalf of a virtual character include when it is determined that the dialogue is not proceeding smoothly, when more detailed explanations are required during the dialogue with the virtual character, when a complex dialogue occurs that is difficult for the virtual character to handle, when the user requests dialogue with an operator, when a phase has been reached where a specific action such as a medical procedure that the virtual character cannot handle is required, and in emergency situations where processing related to the virtual character cannot be carried out properly due to a malfunction of the virtual character dialogue server 1.

[0070] The conditions for determining whether an operator should respond include, for example, the following conditions. The dialogue content determination unit 42 may determine that an operator's response is necessary when the number of items and / or keywords for which the dialogue was not conducted smoothly is equal to or greater than a predetermined number. The dialogue content determination unit 42 may also determine that an operator's response is necessary when the number of items for which the user has indicated that the dialogue was not established is equal to or greater than a predetermined number. The dialogue content determination unit 42 may also determine that an operator's response is necessary when the number of times the dialogue was conducted smoothly is equal to or greater than a predetermined number. The dialogue content determination unit 42 may also determine that an operator's response is necessary when the number of times the dialogue was not conducted smoothly is equal to or greater than a predetermined number. The dialogue content determination unit 42 may also determine that an operator's response is necessary when the number of times the user has indicated that the dialogue was not established is equal to or greater than a predetermined number. Note that any other conditions for determining whether an operator should respond can be set. The dialogue content determination unit 42 may also determine whether an operator's response is necessary by combining a plurality of these conditions.

[0071] On the other hand, if the dialogue content does not satisfy the operator response determination condition, the dialogue content determination unit 42 determines that no response by an operator is required. In this case, the dialogue between the virtual character and the user continues. The dialogue content determination unit 42 determines the dialogue content every time a dialogue content analysis result is generated, at predetermined intervals, or at any timing while the dialogue between the virtual character and the user is ongoing.

[0072] If the user requests a dialogue with an operator, the dialogue content determination unit 42 may determine that a response by an operator is necessary regardless of the determination result of the dialogue content. Such a request from the user may be extracted from the dialogue content by the dialogue content analysis unit 41 and accepted by the dialogue content determination unit 42, or the dialogue content determination unit 42 may accept an operation, for example, when the user touches or presses a predetermined button on the user terminal 70.

[0073] The communication processing unit 49 is a functional unit that communicates with the user terminal 70, the storage management device 10, the virtual person generation device 20, the moving image generation device 30, and the operator control device 60 via the network NET.

[0074] <<<Operator Control Device>>> The operator control device 60 is a device that performs processing related to the dialogue between the operator and the user. As shown in FIG.

[0075] When the dialogue content determination unit 42 determines that an operator response is required, the operator control unit 61 performs processing related to calling an operator. As a method of calling an operator, for example, the operator control unit 61 may send an operator call signal to the operator terminal 400 of an operator selected by referring to the operator information and operator presence / absence information to notify that an operator response is required, or the operator may be contacted by means of a telephone call, e-mail, or the like and instructed to respond.

[0076] When the operator responds by operating the operator terminal 400, the operator control unit 61 connects the virtual person dialogue server 1 and the operator terminal 400 via the network NET. As a result, an image of the operator captured by the operator terminal 400 is transmitted to the virtual person dialogue server 1. Then, the operator control unit 61 stops the generation and transmission of the image of the virtual person by the virtual person generation device 20 and the moving image generation device 30, and transmits the image of the operator to the user terminal 70. As a result, the image of the operator is displayed on the user terminal 70, and a dialogue between the operator and the user takes place. Note that an image of the user captured by the user terminal 70 may also be displayed on the operator terminal 400.

[0077] When the dialogue between the operator and the user ends, the operator control unit 61 performs a process to end the dialogue between the operator and the user (dialogue termination process). As the dialogue termination process, the operator control unit 61, for example, disconnects the connection between the virtual person dialogue server 1 and the operator terminal 400, and stops the transmission of the operator's video to the user terminal 70 and the transmission of the user's video to the operator terminal 400. Furthermore, as the dialogue termination process, the operator control unit 61 may store the operator's history information in the storage management device 10. The operator's history information may include various information, such as the connection time and disconnection time between the virtual person dialogue server 1 and the operator terminal 400 (which may also be the dialogue start time and dialogue end time between the operator and the user), user information, and corresponding operator information.

[0078] The operator control unit 61 updates the operator presence / absence information. For example, when an operator is on standby, that is, when the operator is available to converse with the user, the operator control unit 61 sets the operator presence / absence information of this operator to "present." On the other hand, when the operator is not at work or is conversing with another user, and is therefore unable to converse with the user who is trying to switch the conversation partner, the operator control unit 61 sets the operator presence / absence information of this operator to "absence." Then, when the conversation with the user ends, the operator presence / absence information of this operator is set to "present."

[0079] The communication processing unit 69 is a functional unit that communicates with the user terminal 70, the memory management device 10, the virtual person generation device 20, the moving image generation device 30, and the dialogue content analysis device 40 via the network NET.

[0080] <<Operator terminal>> The operator terminal 400 is a device used by an operator who interacts with a user on behalf of a virtual character. The operator terminal 400 is an information processing device equipped with a communication function, such as a smartphone, a mobile phone, a personal computer, or a tablet terminal.

[0081] Fig. 4 is a diagram illustrating the configuration of an operator terminal. As shown in Fig. 4, the operator terminal 400 includes a processor 401, a main memory 403, a storage 405, a communication interface 407, a monitor 409, a camera 411, a microphone 413, and a speaker 415. The communication interface 407 is a communication device that communicates with the virtual person dialogue server 1 via the network NET.

[0082] The storage 405 includes various storage areas such as a program storage area 405a, an account information storage area 405b, and a data storage area 405c.

[0083] The program storage area 405a stores various programs, such as basic programs such as an OS that operates the operator terminal 400, and programs that cause the processor 401 to realize various functions of the virtual person dialogue system. The program storage area 405a also stores parameters for each program. In this way, the storage 405 functions as a program recording medium.

[0084] The account information storage area 405b stores account information of the operator who uses this operator terminal 400 in the virtual person dialogue system SYS. The account information may include various information such as a registration number, an operator ID, a password, an operator name (e.g., name, title, etc.), gender, age (may be a generation), and used language (multiple selections allowed). The account information may also be stored in the data storage area 405c.

[0085] The data storage area 405c stores various information related to the virtual person dialogue system. For example, the data stored in the storage includes information such as the usage history of the virtual person dialogue system SYS and the history of responses to users.

[0086] The main memory 403 temporarily stores programs and parameters read from the program storage area 405a, various information read from the account information storage area 405b and the data storage area 405c, and calculation results by the processor 401. The main memory 403 also temporarily stores various information received via the communication interface 407, information to be transmitted via the communication interface 407, and the like.

[0087] The processor 401 reads and executes programs and parameters stored in the main memory 403, thereby configuring, in software, functional blocks that drive the components of the operator terminal 400, functional blocks that realize the virtual person dialogue system SYS, etc. Note that some functions may be configured in hardware, or may be configured in combination with software and hardware.

[0088] The monitor 409 displays, for example, screens related to basic operations of the terminal (including operation interfaces, still images, moving images, etc.), screens related to the virtual person dialogue system SYS, screens related to other applications, etc. As an example, the monitor 409 may display an image of a user engaged in dialogue. The monitor 409 may also have a touch input function, which allows the user to perform desired input operations by touching the monitor 409 while looking at the display screen. Note that the input device may be provided separately from the monitor 409.

[0089] The camera 411 is an imaging device that captures still and moving images of the operator. The microphone 413 and speaker 415 are used for calls and dialogue with the user. The microphone 413 is a device that converts the voice and sounds uttered by the operator into audio data. The audio data is transmitted to the terminal of the other party in the call and to the virtual person dialogue server 1. The speaker 415 is a device that converts audio data of the other party in the call or the user in the conversation, and audio data included in the video, into the original voice and sound.

[0090] <<User terminal>> Next, we will explain the configuration of the user terminal 70. The user terminal 70 is a device used by a user who interacts with a virtual character or an operator. The user terminal 70 is an information processing device equipped with a communication function, such as a smartphone, a mobile phone, a personal computer, or a tablet terminal.

[0091] Fig. 5 is a diagram illustrating the configuration of a user terminal. As shown in Fig. 5, the user terminal 70 includes a processor 71, a main memory 73, a storage 75, a communication interface 77, a monitor 79, a camera 81, a microphone 83, and a speaker 85. The communication interface 77 is a communication device that communicates with the virtual person dialogue server 1 via the network NET and with the operator terminal 400 via the virtual person dialogue server 1.

[0092] The storage 75 includes various storage areas such as a program storage area 75a, an account information storage area 75b, a data storage area 75c, etc. The storage 75 may also include a wallet 75d.

[0093] The program storage area 75a stores various programs, such as basic programs such as an OS that runs the user terminal 70, and programs that cause the processor 71 to realize various functions of the virtual person dialogue system. The program storage area 75a also stores parameters for each program. In this way, the storage 75 functions as a program recording medium.

[0094] The account information storage area 75b stores account information of the user who uses this user terminal 70 in the virtual person dialogue system SYS. The account information may include various information such as a registration number, a user ID, a password, a user name (e.g., a name, a nickname, etc.), a gender, an age (or a generation), and a language used (multiple selections are possible). The account information may also be stored in the data storage area 75c.

[0095] The data storage area 75c stores various information related to the virtual person dialogue system. For example, the data stored in the storage includes information such as the usage history of the virtual person dialogue system SYS. The data may also include information sources.

[0096] The wallet 75d stores, for example, crypto assets, etc. The crypto assets can be used to pay the usage fees for the virtual person dialogue system SYS, etc.

[0097] The main memory 73 temporarily stores programs and parameters read from the program storage area 75a, various types of information read from the account information storage area 75b and the data storage area 75c, and calculation results by the processor 71. The main memory 73 also temporarily stores various types of information received via the communication interface 77, information to be transmitted via the communication interface 77, and the like.

[0098] The processor 71 reads and executes programs and parameters stored in the main memory 73, thereby configuring, in software, functional blocks that drive the components of the user terminal 70, functional blocks that realize the virtual person dialogue system SYS, and the like. The functional blocks that realize the virtual person dialogue system SYS include, for example, an information source registration unit, which is a functional unit that acquires information about the target person, i.e., the information source of the target person. Note that some of the functions may be configured in hardware, or may be configured in a combination of software and hardware.

[0099] The monitor 79 displays, for example, screens related to basic operations of the terminal (including operation interfaces, still images, moving images, etc.), screens related to the virtual person dialogue system SYS, screens related to other applications, etc. The monitor 79 may be equipped with a touch input function, which allows the user to perform desired input operations by touching the monitor 79 while looking at the display screen. Note that the input device may be provided separately from the monitor 79.

[0100] The camera 81 is an imaging device that captures still images and moving images. The camera 81 can capture images of the scenery around the user terminal 70, the user himself, the virtual person dialogue system SYS, the virtual person dialogue server 1, and the like.

[0101] The microphone 83 and speaker 85 are used for calls and dialogue with virtual characters. The microphone 83 is a device that converts the voice and sounds made by the user into audio data. The audio data is sent to the terminal of the other party and the virtual character dialogue server 1. The speaker 85 is a device that converts the audio data of the other party and the virtual character during a call, and the audio data included in the video, back into the original voice and sound.

[0102] <Determining the video model to be used> Here, a method for determining a video model to be used by the virtual person generation device 20 will be described. FIG. 6 is a sequence diagram illustrating the method for determining a video model to be used. As shown in FIG. 6, first, recorded data of an actual person or the like and modeling data created by CG modeling are registered in the virtual person generation device 20 (step S100). Then, an information source of the target person is registered from the user terminal 70 and transmitted to the virtual person generation device 20 (step S101). Next, the virtual person generation device 20 extracts appearance data from the information source (step S102). Of the appearance data, moving images are converted into still images (step S103). Next, images of the target person are cropped for the registered still images and still images converted from the moving images, and the color tone and resolution of the images are corrected (step S104). The cropping and image correction can be performed in any order. At this time, if the data quality is below a predetermined level even after correction, it may be determined that the image will not be used in subsequent processes. When recording data and / or modeling data are used, steps S102 to S104 may be omitted as appropriate.

[0103] Next, the virtual person generation device 20 stores the trimmed and corrected image in the virtual person data storage unit 13 of the memory management device 10 (step S105). The virtual person generation device 20 refers to the video models stored in the video model DB 11 based mainly on information about the physique of the stored image (step S106), selects the video model that most resembles the appearance of the target person, and displays it on the user terminal 70 (step S107). At this time, multiple candidate video models may be displayed on the user terminal 70, allowing the user terminal 70 to select the video model to use. Alternatively, the user terminal 70 may be able to select a video model different from the presented video model.

[0104] Next, the user terminal 70 accepts input to individually change features of the used video model (step S108). Features include face contours, eyes, nose, mouth, etc. At this time, selections regarding the virtual model's hairstyle and clothing may also be input. Once the features of the used video model have been appropriately changed and the used video model of the virtual person has been determined, face data extracted from the appearance data is integrated into the used video model (step S109). Next, the used video model with the integrated face data is stored in the virtual person data storage unit 23 of the storage management device 20 (step S110).

[0105] <Generating a virtual character's voice> Next, a method by which the virtual person generation device 20 generates a voice of a virtual person will be described. FIG. 7 is a sequence diagram illustrating a method for generating a voice of a virtual person. When recorded data is registered (step S200) and an information source is registered from the user terminal 70 (step S201), the virtual person generation device 20 extracts voice data of the target person from the information source and recorded data (step S202). The virtual person generation device 20 generates a voice of the virtual person based on the voice data (step S203). Information about the voice of the generated virtual person is stored in the virtual person data storage unit 23 (step S204).

[0106] <Determining the personality model of a virtual character> Next, a method by which virtual person generation device 20 determines a personality model of a virtual person will be described. FIG. 8 is a sequence diagram illustrating a method for determining a personality model of a virtual person. When recorded data is registered (step S300) and an information source is registered from user terminal 70 (step S301), virtual person generation device 20 extracts text data such as blogs and SNS from the information source and recorded data (step S302). At this time, image data such as handwritten diaries is also extracted and converted into text data. Furthermore, sound source data is extracted and the voice of the target person is converted into text data. The extracted text data is stored in virtual person data storage unit 23 based on predetermined rules (step S303).

[0107] Next, the virtual person generation device 20 causes the user terminal 70 to display questions about the personality of the target person (step S304). At this time, the content of the questions may be determined based on the registered information source. Alternatively, the user may be prompted to select a scenario pattern to be registered, and questions corresponding to that scenario pattern may be displayed. The user terminal 70 then accepts input of answers to the questions (step S305). Note that multiple questions may be displayed at once, or steps S304 and S305 may be executed repeatedly.

[0108] Based on the answers to the personality questions, virtual person generation device 20 refers to the personality models stored in personality model DB 12 (step S306) and determines the personality model to be used (step S307).Then, the determined personality model to be used is stored in virtual person data storage unit 23 (step S308).

[0109] <Interaction with virtual characters> Next, a method for executing a dialogue between a user and a virtual character will be described. Here, a case where an operator is not called will be described. Fig. 9 is a sequence diagram for explaining an example of a method for executing a dialogue with a virtual character.

[0110] A user ID and password are transmitted from the user terminal 70 (step S401), and once authenticated by the virtual person generation device 20 (step S402), the user is able to converse with the virtual person. At this time, effects may be provided, such as an incoming chat message, a video call, a telephone call, or an email from the virtual person or the person modeled after the virtual person. Next, data of the virtual person to be conversed is retrieved from the virtual person data storage unit 13 of the storage management device 10, and becomes available for reference by the video generation device 30 (step S403). Then, an image of the virtual person based on the retrieved data is transmitted to the user terminal 70, and the image of the virtual person is displayed on the user terminal 70 (step S404).

[0111] Fig. 10 is a diagram illustrating a state in which a virtual person is displayed on a user terminal. In Fig. 10, the user terminal 70 is held in the hand of the user USR. A virtual person VIR, who will be a conversation partner, is called up from the virtual person data storage unit 13 and displayed on the monitor 79 of the user terminal 70. The virtual person VIR may speak or move when it is displayed on the monitor 79.

[0112] When a question is input to the virtual character from the user terminal 70 (step S405), the moving image generating device 30 generates a moving image in which the virtual character replies based on the data of the virtual character and information related to the facility.

[0113] Specifically, first, the moving image generation device 30 generates a response text to the question based on the personality model of the virtual character (step S406). The moving image generation device 30 then generates a response voice that reproduces the response text in the voice of the virtual character (step S407). The response voice may be stored audio data of the target character or an artificially generated artificial voice. The moving image generation device 30 then generates a response video to be played when the response voice is reproduced (step S408). The generated response voice and response video are transmitted to the user terminal 70 as a response video (step S409). The response voice and response video may be integrated into a single data file and transmitted to the user terminal 70, or separate data files may be transmitted to the user terminal 70. Next, a video of the virtual character is displayed on the user terminal 70 (step S410). That is, a dialogue with the virtual character is established when the virtual character responds to the user's question. The processes from step S404 to step S410 may be repeated multiple times. This configuration allows for natural interaction with the virtual person.

[0114] 9 illustrates the flow of generating a video of a virtual person triggered by inputting a question to the user terminal 70 in step S405. However, the video of a virtual person may be generated based on a predetermined date or time and displayed on the user terminal 70. The video may also be generated based on external information from the Internet, or based on an instruction from an administrator of the virtual person dialogue system SYS. The video may be displayed on the user terminal 70 immediately after it is generated, or the video may be generated in advance and displayed on the user terminal 70 in response to a question from the user, a date, a time, external information, an instruction, or the like.

[0115] During this time, the dialogue content analysis device 40 analyzes the dialogue content between the virtual character and the user, and generates and saves the dialogue content analysis result (step S421). The operator control device 60 determines that no response by the operator is required based on the dialogue content analysis result generated by the dialogue content analysis device 40 and the operator response determination condition (step S422). This allows the dialogue between the virtual character and the user to continue.

[0116] In addition to the above, the conditions for determining whether an operator should respond include the following: The dialogue content determination unit 42 may determine whether the dialogue is proceeding smoothly, for example, if the response time after receiving a question from the user exceeds a predetermined time, and determine that an operator response is necessary.

[0117] Furthermore, with regard to the items and / or keywords for which the dialogue did not proceed smoothly, the dialogue content determination unit 42 may determine whether the items and / or keywords for which the dialogue did not proceed smoothly in the past dialogue have been used in the current dialogue, for example. In this case, the dialogue content determination unit 42 may determine that an operator's response is required in combination with other determination conditions, for example, items for which the user indicated that the dialogue was not proceeding smoothly (whether the user indicated a predetermined item) or the number of times that the user indicated that the dialogue was not proceeding smoothly (whether the number of times exceeds a predetermined threshold).

[0118] Following step S410, when an evaluation of the dialogue or video is input from the user terminal 70 (step S411), the virtual person generation device 30 corrects the personality model and stores it in the virtual person data storage unit 23 of the storage management device 20 (step S412). Note that steps S410 to S412 may be omitted as appropriate.

[0119] <Switching from virtual person to operator> Next, a case where a user's conversation partner is switched from a virtual person to an operator will be described. Fig. 11 is a sequence diagram explaining a method for switching a user's conversation partner from a virtual person to an operator. Note that Fig. 11 mainly shows the processing related to switching the conversation partner, and parts that overlap with Fig. 9 are omitted as appropriate.

[0120] The operator control device 60 determines that an operator response is required based on the dialogue content analysis result generated by the dialogue content analysis device 40 and the operator response determination conditions (step S423). Then, a process is performed to switch the user's dialogue partner from the virtual person to the operator.

[0121] The operator control device 60 selects an operator to interact with the user and calls the selected operator (step S424). Upon receiving a response from the operator (step S425), the operator control device 60 connects the operator terminal 400 to the virtual person dialogue server 1 (step S426).

[0122] The operator terminal 400 captures an image of the operator using the camera 411 and transmits the captured real-time image of the operator to the user terminal 70 (step S427). Specifically, the image of the operator is transmitted to the user terminal 70 via the virtual person dialogue server 1. The real-time image of the operator transmitted from the operator terminal is displayed on the user terminal 70 (step S428).

[0123] 12 is a diagram illustrating a situation in which the person displayed on the user terminal is switched from the virtual person to the operator, in which the state before the switch and the state after the switch are displayed side by side.

[0124] Before the dialogue partner is switched, the virtual person VIR is displayed on the monitor 79 of the user terminal 70. Then, after the dialogue partner is switched, the image of the operator REA transmitted from the operator terminal 400 is displayed on the monitor 79 of the user terminal 70. In this way, when the dialogue partner is switched, the image of the person displayed on the monitor 79 is switched. The operator REA may start speaking or making a movement at the time when his / her own image is displayed on the user terminal 70.

[0125] Furthermore, the user's video captured by the user terminal 70 may be transmitted from the user terminal 70 via the virtual person dialogue server 1 (step S429), and the real-time video of the user may be displayed on the operator terminal 400 (step S430). The user may start speaking or making a movement at the time when the user's video is displayed on the operator terminal 400 (or when transmission of the user's video starts).

[0126] Thereafter, a question is asked by the user (step S431), and a reply is made by the operator (step S432). When the dialogue between the operator and the user ends, the operator terminal 400 notifies the operator control device 60 of the dialogue end (step S433), and the operator control device 60 performs a process to end the dialogue between the operator and the user (step S434).

[0127] Until the process of terminating the dialogue between the operator and the user is performed, a real-time video of the operator is displayed on the user terminal 70, and a real-time video of the user is displayed on the operator terminal 400. The user may be allowed to select whether or not to transmit the user's video to the operator terminal 400. If the user's video is not to be transmitted to the operator terminal 400, steps S429-S430 are omitted.

[0128] According to this embodiment, switching from the virtual character to the operator can be performed smoothly, so that even if a situation arises that the virtual character cannot handle, it is possible to respond appropriately. This also makes it possible to provide the information required by the user in a timely manner, thereby improving customer satisfaction.

[0129] (Embodiment 2) Next, a second embodiment will be described. After the user has finished interacting with the operator, there are cases where the user wants to obtain other information. In such cases, it would be inefficient for the user to access the virtual person interaction system SYS again. Therefore, in this embodiment, once the operator's response has finished, the user's interaction partner can be switched from the operator to a virtual person.

[0130] 13 is a sequence diagram illustrating a method for switching the user's conversation partner from an operator to a virtual character. Fig. 13 shows the process from a conversation with the operator (steps S431 and S432). When the user requests a conversation with the virtual character (step S433), the operator operates the operator terminal 400 to issue an instruction to switch the conversation partner to the virtual character (step S434).

[0131] The trigger for switching from the operator to the virtual character may be, for example, the dialogue content analysis device 40 extracts a dialogue request from the user with the virtual character from the dialogue content, generates a dialogue content analysis result including the dialogue request from the user with the virtual character, and the operator control device 60 switches the dialogue partner from the operator to the virtual character based on the dialogue content analysis result.Alternatively, the user may operate the user terminal 70 to make a dialogue request with the virtual character from the user terminal 70 to the virtual character dialogue server 1, which may serve as a trigger for switching the dialogue partner from the operator to the virtual character.

[0132] 13, thereafter, transmission of the operator's video to the user terminal 400 is stopped (step S435), and the video of the virtual person is transmitted to the user terminal 70, and the video of the virtual person is displayed on the user terminal 70 (step S436). Thereafter, a dialogue takes place between the virtual person and the user, with the user inputting a question (step S436) and the virtual person replying (step S437).

[0133] According to this embodiment, even after the operator's response has ended, the user's conversation partner is switched from the operator to the virtual character, and the conversation between the virtual character and the user continues. With this configuration, after the user has finished talking with the operator, the user can continue talking with the virtual character while remaining logged in.

[0134] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Those skilled in the art can easily modify, add, or convert each element of the above-described embodiments within the scope of the present invention. [Explanation of symbols]

[0135] 1...virtual person dialogue server, 10...storage management device, 20...virtual person generation device, 30...video generation device, 70...user terminal, 400...operator terminal, SYS...virtual person dialogue system, VIR...virtual person, REA...operator.

Claims

1. a virtual person interaction server that generates a virtual person; a user terminal used by a user; an operator terminal used by an operator who is a real person; Equipped with the virtual person dialogue server generates a virtual person when accessed from the user terminal, transmits the generated virtual person to the user terminal, and allows the user to interact with the virtual person displayed on the user terminal; the virtual person dialogue server analyzes the content of the dialogue between the virtual person and the user and generates a dialogue content analysis result; The virtual person dialogue server determines whether or not a response by the operator is necessary, and if it determines that a response by the operator is necessary, calls the operator, transmits an image of the operator taken by the operator terminal to the user terminal, and allows the user to have a dialogue with the operator displayed on the user terminal. Virtual person interaction system.

2. 2. The virtual person dialogue system according to claim 1, the virtual person dialogue server determines whether or not a response by the operator is necessary based on the dialogue content analysis result and an operator response determination condition, and determines that a response by the operator is necessary if the dialogue content satisfies the operator response determination condition; Virtual person interaction system.

3. 3. The virtual person dialogue system according to claim 2, The operator response determination conditions include any of whether the dialogue between the virtual character and the user is proceeding smoothly, items and / or keywords indicating that the dialogue between the virtual character and the user is not proceeding smoothly, the number of times that the dialogue between the virtual character and the user is not proceeding smoothly, items for which the user has indicated that the dialogue is not being established, and the number of times that the user has indicated that the dialogue is not being established. Virtual person interaction system.

4. 2. The virtual person dialogue system according to claim 1, The virtual person dialogue server determines that a response by the operator is necessary when the user requests dialogue with the operator. Virtual person interaction system.

5. 2. The virtual person dialogue system according to claim 1, When the dialogue between the operator and the user ends, the virtual person dialogue server switches the user's dialogue partner to the virtual person. Virtual person interaction system.

6. When accessed from a user terminal, a virtual character is generated, the generated virtual character is transmitted to the user terminal, and the user interacts with the virtual character displayed on the user terminal; Analyzing the content of a dialogue between the virtual person and the user, and generating a dialogue content analysis result; determining whether or not an operator's response is required, and if it is determined that an operator's response is required, calling the operator, transmitting an image of the operator taken by an operator terminal to the user terminal, and allowing the user to interact with the operator displayed on the user terminal; Virtual person interaction server.

7. When accessed from a user terminal, a virtual character is generated, the generated virtual character is transmitted to the user terminal, and the user interacts with the virtual character displayed on the user terminal; a process of analyzing the content of a dialogue between the virtual person and the user and generating a dialogue content analysis result; A process of determining whether an operator's response is required; If it is determined that a response by the operator is necessary, a process of calling the operator, transmitting an image of the operator taken by an operator terminal to the user terminal, and having the operator displayed on the user terminal interact with the user; causing a processor to execute Virtual person interaction program.

Images