Virtual person dialogue system, virtual person dialogue server, and virtual person dialogue program

Abstract

To provide a technique that enables having dialogue with a virtual person in relation to a specific product.SOLUTION: A virtual person dialogue system comprises: a virtual person dialogue server that generates a virtual person; and a user terminal. The virtual person dialogue server authenticates a product code transmitted from the user terminal, generates a virtual person related to the product code transmitted from the user terminal, transmits the generated virtual person to the user terminal, and enables a dialogue between the virtual person displayed on the user terminal and a user.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a virtual character dialogue system, a virtual character dialogue server, and a virtual character dialogue program. [Background technology]

[0002] Systems that allow users to interact with virtual characters displayed on a screen have been put to practical use. For example, Patent Document 1 discloses a technology that can collect information about a user's tendencies without making the user feel uncomfortable or tense, and can provide the user with information that he or she needs or is interested in.

[0003] Specifically, Patent Document 1 describes that the characters have personalities implemented through programming and have the ability to ask questions to the user, that surveys requested by the server operators are conducted in the form of conversations between the user and the virtual personality, and that because control is carried out in the form of conversations via the characters, products in which the user is interested can be advertised through conversations from the characters, allowing the user to accept information without resistance. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2007-334732 A Summary of the Invention [Problem to be solved by the invention]

[0005] In this way, the conversations with the characters in Cited Document 1 are intended to conduct surveys of users and provide product advertisements based on the survey results, and are not conducted in connection with any specific product.

[0006] Therefore, an object of the present invention is to provide a technique that enables a user to interact with a virtual person in relation to a specific product. [Means for solving the problem]

[0007] A virtual person dialogue system according to a representative embodiment of the present invention includes a virtual person dialogue server that generates a virtual person, and a user terminal. The virtual person dialogue server authenticates a product code transmitted from the user terminal, generates a virtual person associated with the product code transmitted from the user terminal, transmits the generated virtual person to the user terminal, and allows the user to dialogue with the virtual person displayed on the user terminal.

[0008] The system may include a product management server that manages product codes, and the virtual person dialogue server may refer to the product codes managed by the product management server and authenticate the product code transmitted from the user terminal.

[0009] When the virtual person dialogue server authenticates the product code sent from the user terminal, it may issue a talk ticket that enables a dialogue with the virtual person, and transmit the issued talk ticket to the user terminal that sent the product code.

[0010] The talk ticket may be rechargeable so that the number of times it can be used is increased.

[0011] Talk tickets may be tradable.

[0012] The virtual person interaction server may display the virtual person and the user's avatar in the metaverse space.

[0013] If the language used by the virtual character differs from the language used by the user, the virtual character dialogue server may translate messages spoken by the virtual character into the language used by the user.

[0014] In a representative embodiment of the present invention, the virtual character interaction server authenticates a product code transmitted from a user terminal, generates a virtual character associated with the product code transmitted from the user terminal, transmits the generated virtual character to the user terminal, and allows the user to interact with the virtual character displayed on the user terminal.

[0015] A virtual character dialogue program in a representative embodiment of the present invention causes a virtual character dialogue server to execute the following processes: authenticating a product code transmitted from a user terminal; generating a virtual character associated with the product code transmitted from the user terminal; and transmitting the generated virtual character to the user terminal. Effect of the Invention

[0016] According to the present invention, it is possible to provide a technique that enables a user to interact with a virtual person in relation to a specific product. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a virtual person dialogue system according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating an example of the basic configuration of each device included in the virtual person dialogue server. [Diagram 3] FIG. 2 is a diagram illustrating a specific configuration of a virtual person dialogue server. [Figure 4] FIG. 2 is a diagram illustrating a specific configuration of a product management server. [Diagram 5] FIG. 2 is a diagram illustrating a configuration of a user terminal. [Figure 6] FIG. 11 is a sequence diagram illustrating a method for determining a video model to be used. [Figure 7] FIG. 11 is a sequence diagram illustrating a method for generating a voice of a virtual character. [Figure 8] FIG. 11 is a sequence diagram illustrating a method for determining a personality model of a virtual person. [Figure 9] FIG. 2 is a sequence diagram illustrating a method for performing a dialogue with a virtual person. [Figure 10] FIG. 11 is a diagram illustrating an example of a state in which a virtual person is displayed on a user terminal. [Figure 11] FIG. 11 is a diagram illustrating an example of a configuration of a virtual person dialogue system according to a second embodiment of the present invention. [Figure 12] A diagram illustrating a specific configuration of a talk ticket charging server. [Figure 13] FIG. 11 is a sequence diagram illustrating a method for charging a talk ticket. [Figure 14] FIG. 11 is a diagram illustrating a configuration of a virtual person dialogue system according to a third embodiment of the present invention. [Figure 15] A diagram illustrating a specific configuration of a talk ticket buying and selling server. [Figure 16] 11 is a sequence diagram illustrating a method for buying and selling talk tickets. [Figure 17] FIG. 1 is a diagram illustrating a dialogue in a metaverse space. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] (Embodiment 1) Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In each drawing for explaining the embodiment, the same members are generally designated by the same reference numerals, and repeated description thereof will be omitted as appropriate.

[0019] <Configuration of Virtual Person Dialogue System> The virtual person dialogue system is a system in which a user who has purchased a product or the like can have a dialogue with a virtual person (e.g., a celebrity) associated with the product. FIG. 1 is a diagram illustrating a configuration of a virtual person dialogue system according to a first embodiment of the present invention. As shown in FIG. 1, the virtual person dialogue system SYS includes a virtual person dialogue server 1, a product management server 100, and a user terminal 70 used by each user of the virtual person dialogue system SYS. The virtual person dialogue server 1 and the product management server 100, the virtual person dialogue server 1 and the user terminal 70, and the product management server 100 and the user terminal 70 are connected to each other via a network NET, enabling transmission and reception of information.

[0020] The virtual person dialogue server 1 generates a virtual person corresponding to a product code transmitted from a user terminal 70, and realizes a dialogue between the generated virtual person and the user who transmitted the product code. The product management server 100 manages product codes that are assigned to products that can be interacted with by a virtual person. The product code is, for example, a combination of a common code (such as a JAN code) common to the same product and a serial number that identifies each product, but may be only the serial number. The user terminal 30 is a terminal for dialogue with a virtual person, such as transmitting the product code of a purchased product.

[0021] <<Virtual Person Dialogue Server>> As shown in Fig. 1, the virtual person dialogue server 1 includes a memory management device 10, a virtual person generation device 20, a video generation device 30, and a dialogue content analysis device 40. These devices may be separate devices, or some or all of the devices may be realized as a single device. The virtual person dialogue server 1 may be cloud-based. These devices may be connected to each other through API cooperation.

[0022] Here, a case where these devices are configured as separate devices will be described. Fig. 2 is a diagram illustrating an example of the basic configuration of each device included in the virtual person dialogue server. The basic configurations of these devices are roughly the same, and these basic configurations will be described using Fig. 2. After that, the specific configuration of each device will be described.

[0023] 2, each device in the virtual person dialogue server 1 includes a processor 51, a main memory 53, a storage 55, a communication interface 57, a monitor 59, etc. The communication interface 57 is a communication device that transmits and receives various information to and from a user terminal 70 via a network NET. The monitor 59 displays various information such as a management screen for each device, information related to the virtual person dialogue server 1 and the virtual person dialogue system SYS, and a user interface.

[0024] The storage 55 includes various storage areas, such as a program storage area 55a, a data storage area 55b, etc. The program storage area 55a stores various programs, such as a basic program such as an OS (Operating System) that operates each corresponding device, and a program that causes the processor 11 to realize various functions of the virtual person dialogue system. The program storage area 55a also stores parameters for each program. In this way, the storage 55 functions as a program recording medium.

[0025] The data storage area 55b stores various information related to the virtual person dialogue system. The data stored in the data storage area 55b differs for each device, but for example, the data stored in the storage includes product information related to dialogue with a virtual person, information related to the generation of a virtual person and dialogue with a virtual person, user account information, and the like.

[0026] The main memory 53 temporarily stores programs and parameters read from the program storage area 55a, various information read from the data storage area 55, calculation results by the processor 51, etc. The main memory 53 also temporarily stores various information received via the communication interface 57, information to be transmitted via the communication interface 57, etc.

[0027] The processor 51 reads and executes programs stored in the main memory 53, thereby configuring, with software, functional blocks for driving each component of each device, the virtual person dialogue server 1, the product management server 100, and ultimately the functional blocks for realizing the virtual person dialogue system SYS. Note that some of the functions may be configured with hardware, or may be configured with a combination of software and hardware.

[0028] In addition to the above, each device may also be equipped with input devices such as a keyboard and a mouse, drives for various memory cards and recording media, and the like.

[0029] <<<Storage management device>>> Fig. 3 is a diagram illustrating a specific configuration of a virtual person dialogue server. Fig. 3 mainly shows the functional configuration of each device constituting the virtual person dialogue server 1. As shown in Fig. 3, the storage management device 10 includes at least an image model DB 11, a personality model DB 12, a virtual person data storage unit 13, and a communication processing unit 19 as software resources. In this specification, "DB" is an abbreviation for "database."

[0030] The video model DB11 is a storage unit that stores a plurality of types of video models of human movements. The video models are video templates used to generate an image of a virtual person. The video models are data that configure the shape and movement of the torso in particular. The video models are also configured to integrate face data, which will be described later, and move the integrated face data together with the image of the torso.

[0031] The video model includes a variety of appearances of people with different physiques depending on height, weight, age, etc. The video model includes a variety of outfits that can be worn by each person and played back. Furthermore, the video model includes various data of actions of people with each appearance, such as data of actions that are often performed during conversation, such as nodding, folding arms, and raising hands.

[0032] The video model may be a video of recorded data of an actual person, or may be a video of modeling data modeled with CG, or may include both. In this embodiment, the actual person serving as the video model may be, for example, a so-called celebrity such as a talent or an artist. In this case, the video model may be a video of recorded data of a celebrity, or a video of modeling data of a celebrity modeled with CG, or the like. The recorded data may include various information about the person, such as, for example, a video of the person, a video of the person in action, the person's possessions, and information on blogs and SNS posted by the person.

[0033] The personality model DB12 is a storage unit in which a plurality of types of personality models of people are stored. The personality model includes, for example, characteristics of answers to questions, and determines the policy of the answer, such as whether the content is positive or negative, and the emotions expressed in the answer. The personality model is not limited to answers to questions from users, but may also be characteristics of messages according to seasons, time periods, etc. The personality model DB12 may also store responses to questions that are expected in advance and are in line with each personality model. This configuration can reduce the computational burden of generating responses in line with the personality model to standard questions.

[0034] The virtual character data storage unit 13 is a storage unit that stores information on an image model, a personality model, and a voice determined for each virtual character. The virtual character data storage unit 23 stores information known to the virtual character, such as episodes and personal experiences of the target character. The virtual character data is determined and stored by the virtual character generation device 20. The virtual character data is called up by the video generation device 30 when a video of the virtual character is played back.

[0035] <<<Virtual Person Generation Device>>> 3, the virtual person generation device 20 includes at least a video processing unit 21, a voice processing unit 22, a personality processing unit 23, and a communication processing unit 29 as software resources.

[0036] The video processing unit 21 is a functional unit that extracts appearance data used to generate a virtual person from the data of the target person. The appearance data includes the face, body, hairstyle, clothes, etc. of the target person. The video processing unit 21 also selects a video model to be used to generate the virtual person and determines the video data to be used for the video of the virtual person. The video processing unit 21 may extract the appearance data of the virtual person based on an information source registered via the information source registration unit of the user terminal 70, as well as an information source acquired from the Internet. The video processing unit 21 may also extract the appearance data to be used to generate one virtual person based on information sources registered from multiple user terminals 70. When many users, such as celebrities, interact with a common virtual person, each user registers an information source for one virtual person. According to this configuration, a virtual person can be generated based on more information sources, enabling more realistic interactions.

[0037] The information sources include, for example, videos, still images, and audio sources including the target person, as well as records such as diaries created by the target person, documents expressing the person's hobbies and preferences, and character data such as SNS. The information sources also include information regarding the target person's language used and possessions such as clothing. The information sources may be registered by the user or may be obtained via the Internet. The obtained information sources are transmitted to the virtual person generation device 20.

[0038] The video processing unit 21 includes a moving image acquiring unit 211 , a still image acquiring unit 212 , a trimming unit 213 , an image correcting unit 214 , a video model selecting unit 215 , and a face inserting unit 216 .

[0039] The video acquisition unit 211 is a functional unit that acquires video data. The video acquisition unit 211 acquires videos included in an information source registered in the user terminal 70. The video acquisition unit 211 can also prompt the user to shoot a video through the user terminal 70. As a situation in which a video can be shot through the user terminal 70, for example, a case where the target person is a person close to the user and a virtual person is displayed on another user terminal 70, or a case where a virtual person is generated so that the target person can be interacted with even after the target person dies, etc. are considered. In this case, the video acquisition unit 311 may cause the user terminal 70 to display a tutorial for encouraging the user to shoot a video.

[0040] The still image acquisition unit 212 is a functional unit that acquires still image data. The still image acquisition unit 212 acquires still images included in an information source registered in the user terminal 70. The still image acquisition unit 212 can also prompt the user to take a still image through the user terminal 70. In this case, the still image acquisition unit 212 may cause the user terminal 70 to display a tutorial for making the user take a still image, i.e., a photo. The still image acquisition unit 212 also converts video data into a still image and acquires it. The still image acquisition unit 212 extracts images of a target person from various angles and images of various facial expressions, and converts them into still images.

[0041] The trimming unit 213 is a functional unit that trims and extracts data of a target person from a still image. The trimming unit 213 may have a face recognition function and be capable of automatically extracting only the face of the target person.

[0042] The image correction unit 214 performs color correction and resolution correction on the extracted images to equalize the quality of the extracted images. The image correction unit 214 may also determine whether the extracted images are clear or not, and exclude unclear images from the extracted data group. The image correction unit 214 may also exclude images with a resolution equal to or lower than a predetermined value from the extracted data group.

[0043] The video model selection unit 215 is a functional unit that selects a video model to be used for generating a virtual person from the data in the video model DB 11. The video model selection unit 215 may select a video model that is most similar to the target person based on the appearance data acquired by the video acquisition unit 211, or may present a plurality of video models to the user terminal 70 and allow the user to select a video model to be used. With this configuration, a video of a virtual person can be composed using a video model even if a sufficient number of information sources showing the virtual person moving are not registered.

[0044] The image model selection unit 215 may determine the clothes of the virtual person to be generated based on the appearance data or based on the possession information included in the information source. The image model selection unit 215 may select the clothes of the virtual person from the image model DB11. That is, if there is an information source in which the target person is wearing that clothing, it is possible to generate an image of the virtual person based on the information source, and even if there is no information source of the target person, it is possible to generate an image of the virtual person based on the possession information. Since the clothing data can be selected from the image model DB11, even if the data on the clothing of the target person is insufficient, it is possible to easily generate a virtual person. Note that the image model selection unit 215 may configure images of virtual people wearing multiple types of clothing, and the clothing may be changed based on the season, time period, or user selection.

[0045] The video model selection unit 215 may determine the hairstyle of the virtual person to be generated based on the appearance data, or may select the hairstyle of the virtual person from the video model DB 11. Furthermore, the video model selection unit 215 may compose videos of virtual people with a plurality of different hairstyles, and the hairstyle may be changeable.

[0046] In the above description, it has been assumed that the image processor 21 extracts data of a virtual person based on the information source of the target person himself, but recorded data of a person similar to the target person newly shot in video or still images, or modeling data modeled using CG may be used to generate a virtual person. Also, the appearance data of a similar person, such as hairstyle and clothing, may be partially used to generate a virtual person. In other words, the user may be able to select elements of the appearance data to be used to generate a virtual person.

[0047] The face insertion unit 216 is a functional unit that integrates face data extracted by the video acquisition unit 211, the still image acquisition unit 212, the trimming unit 213, and the image correction unit 214 into the used video model. The face insertion unit 216 integrates the face data into the torso configured by the used video model, and creates a whole-body image of a virtual person.

[0048] The voice processing unit 22 is a functional unit that artificially generates the speaking voice of the virtual person. The voice processing unit 22 includes a voice extraction unit 221 and a voice generation unit 222.

[0049] The voice extraction unit 221 is a functional unit that extracts the voice of a target person from an information source. For example, the voice extraction unit 221 may identify the voice of a person that is included for the longest time among a plurality of types of voices included in the information source as the voice of the target person.

[0050] The voice generating unit 222 is a functional unit that generates the voice of the virtual character based on the voice extracted by the voice extracting unit 221. The voice generating unit 222 may trim the voice of the target character and edit it so that it can be played back as the voice of the virtual character. The voice generating unit 222 may also select a voice similar to the voice of the target character from voice data prepared in advance and determine it as the voice of the virtual character. Furthermore, the voice generating unit 222 may generate an artificial voice similar to the voice of the target character. Note that when a message from the virtual character is displayed as text, voice generation may not be necessary.

[0051] The personality processing unit 23 is a functional unit that determines a personality model of a virtual person. The personality processing unit 23 includes a text data registration unit 231, a personality model selection unit 232, and a personality model correction unit 233.

[0052] The text data registration unit 231 is a functional unit that extracts text data from an information source and stores it in the virtual person data storage unit 13 of the memory management device 10. The text data registration unit 231 extracts electronic text data such as blogs and SNS of the target person, and stores it in the virtual person data storage unit 13 according to a predetermined rule. The text data registration unit 231 may also read handwritten documents by the target person, such as diaries, convert them into text data, and store them in the virtual person data storage unit 13. Furthermore, the text data registration unit 231 may convert the voice of the target person contained in audio or video data into text data, and store it in the virtual person data storage unit 13.

[0053] The personality model selection unit 232 is a functional unit that selects a personality model (hereinafter also referred to as a "used personality model") to be used in generating a virtual character from the personality model DB 12. The personality model selection unit 232 presents a question about the personality of the virtual character via the user terminal 70. When an answer to the question is input from the user terminal 70, the personality model to be used in generating a virtual character is selected from the data in the personality model DB 12 based on the answer.

[0054] A plurality of questions regarding personality may be presented. Also, the questions may be presented along a chart in which the input answers are linked to the next question. As the user answers the questions, the basic personality of the virtual person is determined based on basic personality classifications prepared in advance. If the personality is determined based on information about the actual conversation of the target person, a huge amount of conversation information is required. According to the virtual person dialogue system SYS of this embodiment, the answer to the question regarding personality can be classified into one of the previously prepared personalities, so that the personality of the virtual person can be determined with a simple configuration even if information is insufficient.

[0055] The personality model of the virtual character may be determined for each scenario pattern according to the type of question from the user. The scenario pattern may be, for example, daily conversation or consultation about a problem. If a personality model is determined for some scenario patterns, a dialogue in accordance with the scenario pattern may be possible. With this configuration, it is possible to have a dialogue by determining only the personality model for the necessary scenario pattern, so that the personality of the virtual character can be determined easily.

[0056] The personality model correction unit 233 is a functional unit that corrects the personality model used selected by the personality model selection unit 232. The personality model correction unit 233 receives an evaluation of the reply made by the virtual person from the user terminal 70, and corrects the personality model used based on the evaluation. For example, the user inputs an evaluation of whether the reply was appropriate for the target person. The user may also evaluate the action of the virtual person made along with the reply. The personality model correction unit 233 performs automatic learning using AI or the like and corrects the personality model. With this configuration, the personality of the virtual person can be corrected to be closer to the target person. Note that, when multiple user terminals 70 simultaneously or at different times have a conversation with one virtual person, the evaluations from the multiple user terminals 70 may be used to correct the personality model of one virtual person. With this configuration, a lot of feedback can be given to the personality model of the virtual person, so that the personality model of the virtual person can be made closer to the personality of the target person, and the accuracy of the conversation can be improved.

[0057] Furthermore, the personality model correction unit 233 may determine whether a message from a virtual person is appropriate or not based on the user's response to the message, rather than on the user's evaluation, and may correct the personality model. The personality model correction unit 233 may convert the content of the user's response into text data and analyze it, or may infer the user's level of satisfaction from the user's tone of voice.

[0058] The communication processing unit 29 is a functional unit that communicates with the user terminal 70, the storage management device 10, and the video generation device 30 via the network NET.

[0059] <<<Video Creation Device>>> The moving image generating device 30 is a device that displays a moving image of a virtual person generated by the virtual person generating device 20 on a user terminal 70. As shown in FIG. 3 , the moving image generating device 30 includes a video display processing unit 31, a dialogue processing unit 32, a translation processing unit 33, and a communication processing unit 39.

[0060] The image display processing unit 31 is a functional unit that generates an utterance image in which the virtual person speaks. The image display processing unit 31 performs a modeling process on face data extracted from the appearance data, and makes the face data move in accordance with the utterance.

[0061] The dialogue processing unit 32 is a functional unit that generates a message to be spoken by the virtual person based on the usage personality model. The content of the message may be a response to a question from the user, or may be a word generated according to external information such as a date, season, or time period, or a weather forecast or news on the Internet. In addition to the usage personality model, a response to the user may be generated based on external information such as a date, season, or time period, or a weather forecast or news on the Internet. The dialogue processing unit 32 determines the optimal response using AI.

[0062] The message generated by the dialogue processing unit 32 is spoken by a voice generated by the audio processing unit 22 of the virtual person generation device 20, and is played on the user terminal 70 together with the spoken image generated by the image display processing unit 31. The voice of the virtual person may be the lines of the target person extracted by the audio extraction unit 221. It may also be played based on sound source data of a similar voice determined in advance. Furthermore, an artificial voice may be generated and played.

[0063] The translation processing unit 33 is a functional unit that generates a translation of a message spoken by a virtual person when the language used by the virtual person differs from the language used by the user. The translation processing unit 33 refers to the account information of the user who converses with the virtual person, and when the language used by the virtual person differs from the language used by the user, translates the message spoken by the virtual person into the language used by the user. The generated translation is then displayed in the speaking image generated by the image display processing unit 31 in accordance with the voice of the virtual person.

[0064] The communication processing unit 39 is a functional unit that communicates with the user terminal 70, the memory management device 10, and the virtual person generation device 20 via the network NET.

[0065] <<Product management server>> The product management server 100 manages product codes that are assigned to products that can interact with a virtual person. The product management server 100 may be realized as the same server as the virtual person dialogue server 1. The product management server 100 may also be cloud-based. The product management server 100 and the virtual person dialogue server 1 may also be connected to each other through API cooperation. The basic configuration of the product management server 100 is the same as that shown in FIG. 2. FIG. 4 is a diagram illustrating a specific configuration of the product management server.

[0066] As shown in FIG. 4, the product management server 100 includes a product code DB 111 and a communication processing unit 119. The product code DB 111 registers and stores, for example, product codes of manufactured or sold products to be managed. As already mentioned, the product code is, for example, a combination of a common code common to the same products and a serial number for identifying each product, but if the product can be distinguished from other types of products, only the serial number may be used. The product code is registered in the product code DB 111 based on management data transmitted from the manufacturer of each product. In addition, the product code DB 111 may store information such as whether or not a dialogue between the user and a virtual character has taken place for each product code, or the number of times a dialogue between the user and the virtual character has taken place.

[0067] The product management server 100 may store the contents of the dialogue between the user and the virtual character transmitted from the virtual person dialogue server 1. For example, the product management server 100 generates and stores a dialogue file that associates the product (which may include a product code), the user's account information, and the dialogue contents. The product management server 100 may acquire statistical data based on a plurality of dialogue files. The statistical data may include, for example, information such as the dialogue contents by gender and age (generation). The statistical data may be used as data for future product development.

[0068] <<User terminal>> Next, a configuration of the user terminal 70 will be described. The user terminal 70 is a device used by a user who interacts with a virtual person. The user terminal 70 is an information processing device equipped with a communication function, such as a smartphone, a mobile phone, a personal computer, or a tablet terminal.

[0069] Fig. 5 is a diagram illustrating an example of the configuration of a user terminal. As shown in Fig. 5, the user terminal 70 includes a processor 71, a main memory 73, a storage 75, a communication interface 77, a monitor 79, a camera 81, a microphone 83, and a speaker 85. The communication interface 77 is a communication device that communicates with the virtual person dialogue server 1 and the product management server 100 via the network NET.

[0070] The storage 75 includes various storage areas such as a program storage area 75a, an account information storage area 75b, a data storage area 75c, etc. The storage 75 may also include a wallet 75d.

[0071] The program storage area 75a stores various programs, such as a basic program such as an OS that operates the user terminal 70, and a program that causes the processor 71 to realize various functions of the virtual person dialogue system. The program storage area 75a also stores parameters for each program. In this way, the storage 75 functions as a program recording medium.

[0072] The account information storage area 75b stores account information of the user who uses this user terminal 30 in the virtual person dialogue system SYS. The account information may include various information such as a registration number, a user ID, a password, a user name (e.g., a name, a nickname, etc.), a gender, an age (may be a generation), a language used (multiple selections possible), etc. The account information may be stored in the data storage area 75c.

[0073] The data storage area 75c stores various information related to the virtual person dialogue system. For example, the data stored in the storage includes information such as the acquisition history of talk tickets for dialogue with virtual people, the use history of talk tickets (history of dialogue with virtual people), and the transaction history of talk tickets (or talk ticket NFTs). The data may also include information sources.

[0074] Wallet 75d stores, for example, talk tickets that allow interactions with virtual characters, talk ticket NFTs (Non-Fungible Tokens) that are tokenized talk tickets, crypto assets, etc.

[0075] Crypto assets are used to pay for the purchase of talk ticket NFTs, etc. Note that fiat currency may also be used to pay / receive the purchase price of talk ticket NFTs.

[0076] The main memory 73 temporarily stores programs and parameters read from the program storage area 75a, various information read from the account information storage area 75b and the data storage area 75c, and calculation results by the processor 71. The main memory 73 also temporarily stores various information received via the communication interface 77, information to be transmitted via the communication interface 77, and the like.

[0077] The processor 71 reads and executes programs and parameters stored in the main memory 73, thereby configuring, with software, functional blocks for driving the various components of the user terminal 70, functional blocks for realizing the virtual person dialogue system SYS, and the like. The functional blocks for realizing the virtual person dialogue system SYS include, for example, an information source registration unit, which is a functional unit for acquiring information on a target person, i.e., an information source of the target person, and the like. Note that some of the functions may be configured with hardware, or may be configured with a combination of software and hardware.

[0078] The monitor 79 displays, for example, a screen related to basic operations of the terminal (including an operation interface, still images, moving images, etc.), a screen related to the virtual person dialogue system SYS, a screen related to other applications, etc. The monitor 79 may be equipped with a touch input function. This allows the user to perform a desired input operation by touching the monitor 79 while looking at the display screen. Note that the input device may be provided separately from the monitor 79.

[0079] The camera 81 is an imaging device that captures still images and moving images. The camera 81 can capture images of the scenery around the user terminal 30, the user himself, product codes attached to products, etc. Various captured images of product codes, etc. are transmitted to the virtual person dialogue server 1 and the product management server 100.

[0080] The microphone 83 and speaker 85 are used for calling and interacting with a virtual character. The microphone 83 is a device that converts the voice and sounds emitted by the user into audio data. The audio data is sent to the other party's terminal and the virtual character dialogue server 1. The speaker 85 is a device that converts the audio data of the other party or virtual character during a call and the audio data included in the video into the original voice or sound.

[0081] <Determining the video model to be used> Here, a method for determining the video model to be used by the virtual person generation device 20 will be described. FIG. 6 is a sequence diagram for explaining a method for determining the video model to be used. As shown in FIG. 6, first, recorded data obtained by photographing an actual person and modeling data obtained by modeling using CG are registered in the virtual person generation device 20 (step S100). Then, an information source of the target person is registered from the user terminal 70 and transmitted to the virtual person generation device 20 (step S101). Next, the virtual person generation device 20 extracts appearance data from the information source (step S102). Among the appearance data, moving images are converted into still images (step S103). Next, the registered still images and still images converted from the moving images are trimmed to images of the target person, and the color tone and resolution of the images are corrected (step S104). The trimming and image correction can be performed in any order. At this time, if the quality of the data is below a predetermined level even after correction, it may be determined that the image is not to be used in the subsequent process. When recording data and / or modeling data are used, steps S102 to S104 may be omitted as appropriate.

[0082] Next, the virtual person generation device 20 stores the trimmed and corrected image in the virtual person data storage unit 13 of the memory management device 10 (step S105). The virtual person generation device 20 refers to the video models stored in the video model DB 11 based on information mainly related to physique among the stored images (step S106), selects a video model that is most similar to the appearance of the target person, and displays it on the user terminal 70 (step S107). At this time, a plurality of candidates for the video model may be displayed on the user terminal 70, and the user terminal 70 may be able to select the video model to be used. Also, the user terminal 70 may be able to select a video model different from the presented video model.

[0083] Next, the user terminal 70 accepts an input to change individual features of the used video model (step S108). Features include each of the contour, eyes, nose, and mouth, etc. At this time, a selection regarding the hairstyle and clothes of the virtual model may be input. When the features of the used video model are appropriately changed and the used video model of the virtual person is determined, face data extracted from the appearance data is integrated into the used video model (step S109). Next, the used video model with the integrated face data is stored in the virtual person data storage unit 23 of the memory management device 20 (step S110).

[0084] <Generating a voice for a virtual character> Next, a method in which the virtual person generation device 20 generates a voice of a virtual person will be described. FIG. 7 is a sequence diagram illustrating a method of generating a voice of a virtual person. When recorded data is registered (step S200) and an information source is registered from the user terminal 70 (step S201), the virtual person generation device 20 extracts voice data of the target person from the information source and recorded data (step S202). The virtual person generation device 20 generates a voice of a virtual person based on the voice data (step S203). Information on the voice of the generated virtual person is stored in the virtual person data storage unit 23 (step S204).

[0085] <Determining a personality model for a virtual character> Next, a method in which the virtual person generation device 20 determines a personality model of a virtual person will be described. FIG. 8 is a sequence diagram illustrating a method for determining a personality model of a virtual person. When recorded data is registered (step S300) and an information source is registered from the user terminal 70 (step S301), the virtual person generation device 20 extracts text data such as blogs and SNS from the information source and recorded data (step S302). At this time, image data such as handwritten diaries is also extracted and converted into text data. Furthermore, sound source data is extracted and the voice of the target person is converted into text data. The extracted text data is stored in the virtual person data storage unit 23 based on a predetermined rule (step S303).

[0086] Next, the virtual person generation device 20 causes the user terminal 70 to display questions regarding the personality of the target person (step S304). At this time, the content of the questions may be determined based on the information source to be registered. Also, the user may be allowed to select a scenario pattern to be registered, and questions corresponding to the scenario pattern may be displayed. The user terminal 70 accepts input of answers to the questions (step S305). At this time, multiple questions may be displayed at once, or steps S304 and S305 may be executed repeatedly.

[0087] Based on the answers to the questions about personality, virtual person generation device 20 refers to the personality models stored in personality model DB 12 (step S306) and determines the personality model to be used (step S307). Next, the determined personality model to be used is stored in virtual person data storage unit 23 (step S308).

[0088] <Interaction with virtual characters> Next, a method for executing a dialogue between a user and a virtual character will be described. Fig. 9 is a sequence diagram for explaining a method for executing a dialogue with a virtual character. When a user ID and a password are transmitted from the user terminal 70 (step S401), they are authenticated by the virtual character generation device 20 (step S402).

[0089] When a product code is transmitted from the user terminal 70 (step S403), the virtual person generation device 20 verifies whether the product code received from the user terminal 70 is valid by referring to information managed by the product management server 100 (step S404). When the product code is authenticated, the virtual person generation device 20 issues a ticket (talk ticket) that allows a conversation with the virtual person, and transmits the ticket to the user terminal that transmitted the product code (step S405). The issued ticket information is stored in the storage management device 10 (step S406). The ticket information includes, for example, information such as a user ID, a product code, a time of ticket issuance, and a number of times that the ticket can be used. In principle, a talk ticket can be used only once. That is, the number of times that the talk ticket can be used when issued is one, and the number of times that a used talk ticket can be used is 0. However, for example, the number of times that the talk ticket can be used when issued can be set to multiple times, so that one talk ticket can be used multiple times, or multiple talk tickets can be required for one use, etc., can be freely selected.

[0090] Ticket information (talk ticket) is transmitted from the user terminal 70 (step S407), and the ticket information received from the user terminal 70 is collated with the ticket information stored in the memory management device 10 and authenticated (step S408), enabling a conversation with the virtual person associated with the product code included in the ticket information. At this time, effects may be provided such as an incoming chat message, a video call, a phone call, or an e-mail from the virtual person or the person who is the model for the virtual person. Next, data of the virtual person to be talked to is called from the virtual person data storage unit 13 of the memory management device 10, and becomes available for reference by the video generation device 30 (step S409). That is, an image of the virtual person is displayed on the user terminal 70.

[0091] Fig. 10 is a diagram illustrating a state in which a virtual person is displayed on a user terminal. In Fig. 10, the user terminal 70 is held in the hand of the user USR. A virtual person VIR, who is to be a conversation partner, is displayed on a monitor 79 of the user terminal 70 and has been called up from the virtual person data storage unit 13. The virtual person VIR may speak or move when it is displayed on the monitor 79.

[0092] When a question is input to the virtual character from the user terminal 70 (step S410), the moving image generating device 30 generates a moving image in which the virtual character responds, based on the data of the virtual character.

[0093] Specifically, first, the video generation device 30 generates a response text to the question based on the personality model of the virtual person (step S411). The video generation device 30 also generates a response voice that reproduces the response text in the voice of the virtual person (step S412). The response voice may be stored sound source data of the target person, or may be an artificial voice that is artificially generated. Furthermore, the video generation device 30 generates a response video that is reproduced when reproducing the response voice (step S413). The generated response voice and response video are transmitted to the user terminal 70 as a response video (step S414). Note that the response voice and the response video may be integrated and transmitted to the user terminal 70 as one data file, or each data file may be transmitted to the user terminal 70. Next, a video of the virtual person is displayed on the user terminal 70 (step S415). That is, the virtual person responds to the question from the user, and a dialogue with the virtual person is established. The process from step S410 to step S415 may be repeated multiple times. This configuration allows for natural interaction with the virtual person.

[0094] 9, the flow of generating a video of a virtual person triggered by inputting a question to the user terminal 70 shown in step S410 has been described, but the video of a virtual person may be generated based on a predetermined date or time and displayed on the user terminal 70. The video may be generated based on external information from the Internet or the like, or based on an instruction from an administrator of the virtual person dialogue system SYS. The video may be displayed on the user terminal 70 immediately after it is generated, or the video may be generated in advance and displayed on the user terminal 70 in response to a question from the user, a date, a time, external information, an instruction, or the like.

[0095] Following step S415, when an evaluation of the video is input from the user terminal 70 (step S416), the virtual person generation device 30 corrects the personality model and stores it in the virtual person data storage unit 23 of the storage management device 20 (step S417). Note that steps S416 to S417 may be omitted as appropriate.

[0096] According to this embodiment, a video of a virtual person associated with a product code transmitted from the user terminal 70 is generated, and the generated video of the virtual person is displayed on the user terminal, allowing the user to have a conversation with the virtual person. This provides a technology that allows a conversation with a virtual person associated with a specific product.

[0097] (Embodiment 2) Next, a second embodiment will be described. In the first embodiment described above, it was explained that a talk ticket can be used a predetermined number of times (one or more times). If a particular product allows a user to converse with a virtual celebrity, then if the user wishes to converse with the virtual celebrity again, the user must purchase the same product anew. This may result in the purchased product being discarded without ever being used, which is undesirable in terms of resource management. Therefore, in this embodiment, it is made possible to increase the number of times a user can converse with one talk ticket.

[0098] Fig. 11 is a diagram illustrating a configuration of a virtual person dialogue system according to embodiment 2 of the present invention. As shown in Fig. 11, the virtual person dialogue system of embodiment 2 has a configuration in which a talk ticket charge server 200 is added to the configuration of Fig. 1.

[0099] <<Talk Ticket Charge Server>> The talk ticket charging server 200 collects a predetermined fee from the user and charges the number of times a talk ticket can be used. The talk ticket charging server 200 may be realized as the same server as the virtual person dialogue server 1 or the product management server 100. The talk ticket charging server 200 may also be cloud-based. The talk ticket charging server 200 and the virtual person dialogue server 1 may be connected to each other through API cooperation. The basic configuration of the talk ticket charging server 200 is the same as that shown in FIG. 2. FIG. 12 is a diagram illustrating an example of a specific configuration of the talk ticket charging server.

[0100] As shown in Fig. 12, the talk ticket charging server 200 includes a payment processing unit 211, a charge processing unit 212, and a communication processing unit 219. The payment processing unit 211 is a functional unit that collects a predetermined fee for charging a talk ticket. The predetermined fee may be set, for example, as a fee for one time or a fee for multiple times. Payment for charging a talk ticket may be made with cryptocurrency or with legal tender.

[0101] The charge processing unit 212 is a functional unit that charges (adds) the number of times a talk ticket can be used according to the fee paid by the user.

[0102] The communication processing unit 219 is a functional unit that communicates with the user terminal 70 and the virtual person dialogue server 1 through the network NET.

[0103] <<How to charge your talk ticket>> 13 is a sequence diagram explaining a method for charging a talk ticket. When a charge request is made from the user terminal 70 (step S501), the payment processing unit 211 inquires of the virtual person dialogue server 1 about the number of times the talk ticket to be charged can be used (step S502). Then, when a response is received from the virtual person dialogue server 1 (step S503), the payment processing unit 211 causes the user terminal 70 to display the number of times it can be used (step S504). When the number of times to charge is selected from the user terminal 70 (step S505), the payment processing unit 211 presents the charge fee (step S506). When the user agrees to the fee, a charge instruction is given from the user terminal 70 (step S507), and the payment processing unit 211 settles the charge fee (step S508). When the payment is completed, the charge processing unit 212 performs a charge process for the number of times the talk ticket can be used according to the settled fee (step S509), and the number of times the talk ticket can be used after the charge process is stored in the memory management device 10 of the virtual person dialogue server 1 (step S510).

[0104] According to this embodiment, it is possible to increase the number of times that a user can converse with a virtual character using one talk ticket. In addition, since there is no need to purchase the same product multiple times, it is possible to reduce the user's expenses and contribute to the effective use of resources.

[0105] (Embodiment 3) Next, a third embodiment will be described. In the above embodiments, it is assumed that the user who purchased the product interacts with the virtual character, but there are cases where a user other than the purchaser of the product wishes to interact with the virtual character, for example, when the user is unable to purchase the product. Therefore, in this embodiment, it is possible for users other than the purchaser of the product to interact with the virtual character.

[0106] Fig. 14 is a diagram illustrating a configuration of a virtual person dialogue system according to the third embodiment of the present invention. As shown in Fig. 14, the virtual person dialogue system according to the second embodiment has a configuration in which a talk ticket buying and selling server 300 is added to the configuration in Fig. 1.

[0107] <<Talk ticket trading server>> The talk ticket buying and selling server 300 buys and sells talk tickets. The talk ticket buying and selling server 300 may be realized as the same server as the virtual person dialogue server 1, the product management server 100, and the talk ticket charge server 200. The talk ticket buying and selling server 300 may be cloud-based. The talk ticket buying and selling server 300 and the virtual person dialogue server 1 may be connected to each other through API cooperation. The basic configuration of the talk ticket buying and selling server 300 is the same as that of FIG. 2. FIG. 15 is a diagram illustrating an example of a specific configuration of the talk ticket buying and selling server.

[0108] As shown in Fig. 15, the talk ticket buying and selling server 300 includes a payment processing unit 311, a buying and selling processing unit 312, and a communication processing unit 319. The payment processing unit 311 is a functional unit that collects a predetermined fee related to the buying and selling of talk tickets. The predetermined fee may include, for example, a predetermined buying and selling commission in the buying and selling price of the talk ticket. The payment of the predetermined fee related to the buying and selling of talk tickets may be made in crypto assets or in legal tender.

[0109] The buying and selling processing unit 312 is a functional unit that executes buying and selling of talk tickets. When buying and selling talk tickets, the talk ticket itself may be the subject of the transaction, but from the viewpoint of storing the buying and selling history and preventing tampering of the talk ticket, it is preferable to use, for example, a talk ticket NFT that is an NFT version of the talk ticket as the subject of the transaction. For this reason, it is preferable that the talk ticket buying and selling server 300 is connected to a blockchain network.

[0110] The communication processing unit 319 is a functional unit that communicates with the user terminal 70 and the virtual person dialogue server 1 through the network NET.

[0111] <<How to buy and sell talk tickets>> 16 is a sequence diagram explaining how to buy and sell talk tickets. When a sell request is made from the user terminal 70A of the seller (step S601), the buy and sell processing unit 312 converts the seller's talk ticket into an NFT to generate a talk ticket NFT and puts it up for sale (step S602). When a buyer finds a talk ticket NFT they wish to purchase among the ones put up for sale on the talk ticket buying and selling server 300, they make a purchase request from their user terminal 70B (step S603). The payment processing unit 311 displays the purchase cost, including the buying and selling price and the buying and selling fee, of the talk ticket NFT for which the purchase request has been made on the user terminal 70B (step S604). When the potential purchaser agrees to the purchase cost, a purchase instruction is made from the user terminal 70B (step S605). The payment processing unit makes payment for the purchase cost for the talk ticket NFT for which the purchase instruction has been made (step S606). The buying and selling processing unit 312 updates the owner of the talk ticket NFT for which the purchase fee has been paid to the current purchaser (step S607), and saves the buying and selling history and the updated owner information in the blockchain (step S608). The updated owner information is stored in the virtual person dialogue server 1 (step S609). The buying and selling processing unit 312 notifies the user terminal 70A that the sale of the talk ticket NFT has been completed (step S610), and notifies the user terminal 70B that the purchase of the talk ticket NFT has been completed (step S611).

[0112] According to this embodiment, even people other than product purchasers can converse with the virtual person.

[0113] (Embodiment 4) Next, a fourth embodiment will be described. In the above embodiments, the case where the user himself / herself interacts with a virtual character via the user terminal 70 has been described, but in the present embodiment, the case where the user interacts with a virtual character in the metaverse space will be described.

[0114] FIG. 17 is a diagram for explaining a dialogue in the metaverse space. In the metaverse space MET, a virtual person and a user communicate using their own avatars. In FIG. 17, VIR is a virtual person, and USR_AVA is a user's avatar. The user's avatar USR_AVA can be freely selected by the user. The user can also freely select hairstyles, clothes, accessories, etc. for the virtual person and his / her own avatar.

[0115] Although the metaverse space is a three-dimensional space, the virtual person VIR and the user's avatar USR_AVA may be configured in two dimensions. In this case, the posture of the virtual person VIR is controlled so that it always faces the direction of the user's avatar USR_AVA. This ensures that the virtual person VIR is not seen from the side, so that the user does not feel uncomfortable during the conversation. This also reduces the processing involved in generating and controlling the virtual person VIR and the user's avatar USR_AVA.

[0116] According to this embodiment, by using the virtual person VIR and the user's avatar USR_AVA in the metaverse space MET, the environment in which the conversation takes place can be freely changed, making it possible to enjoy conversation with the virtual person even more.

[0117] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-mentioned embodiments. Those skilled in the art can easily change, add, or convert each element of the above-mentioned embodiments within the scope of the present invention. [Explanation of symbols]

[0118] 1...virtual person dialogue server, 10...memory management device, 20...virtual person generation device, 30...video generation device, 100...product management server, 200...talk ticket charge server, 300...talk ticket buying and selling server, MET...metaverse space, SYS...virtual person dialogue system, USR_AVA...user avatar.

Claims

1. a virtual person dialogue server that generates a virtual person; A user terminal; Equipped with the virtual person dialogue server authenticates the product code transmitted from the user terminal, generates a virtual person associated with the product code transmitted from the user terminal, transmits the generated virtual person to the user terminal, and allows the user to dialogue with the virtual person displayed on the user terminal; Virtual person dialogue system.

2. 2. The virtual person dialogue system according to claim 1, A product management server that manages product codes is provided. the virtual person dialogue server refers to a product code managed by the product management server, and authenticates the product code transmitted from the user terminal; Virtual person dialogue system.

3. 2. The virtual person dialogue system according to claim 1, the virtual person dialogue server issues a talk ticket that enables a dialogue with a virtual person when authenticating the product code transmitted from the user terminal, and transmits the issued talk ticket to the user terminal that transmitted the product code; Virtual person dialogue system.

4. 4. The virtual person dialogue system according to claim 3, The talk ticket can be recharged to the number of times it can be used. Virtual person dialogue system.

5. 4. The virtual person dialogue system according to claim 3, The talk ticket is tradable. Virtual person dialogue system.

6. 2. The virtual person dialogue system according to claim 1, The virtual person dialogue server displays the virtual person and an avatar of the user in a metaverse space. Virtual person dialogue system.

7. 2. The virtual person dialogue system according to claim 1, the virtual person dialogue server translates a message spoken by the virtual person into a language used by the user when the language used by the virtual person is different from the language used by the user; Virtual person dialogue system.

8. Authenticate the product code sent from the user terminal, generating a virtual person associated with the product code transmitted from the user terminal; Transmitting the generated virtual person to the user terminal; allowing a user to interact with the virtual person displayed on the user terminal; Virtual person dialogue server.

9. A process of authenticating a product code transmitted from a user terminal; A process of generating a virtual person associated with the product code transmitted from the user terminal; A process of transmitting the generated virtual person to the user terminal; The virtual person dialogue server executes the above. Virtual person dialogue program.

Images