Information processing device, information processing method, and program

Abstract

This invention provides an information processing device, an information processing method, and a program that enable interaction using the user's video and audio. [Solution] The information processing device comprises: a receiving unit that receives user information and personality information of a user; a speech information creation unit that creates information indicating the user's speech to the conversation partner in the user's voice based on the user information and personality information of the user received by the receiving unit; and a video information creation unit that creates video information indicating the user to be displayed to the conversation partner based on the user information and personality information of the user received by the receiving unit.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Regarding the technology for communication between users via a network, there is known a technology for controlling and transmitting a user's voice or video to the other party as necessary without impairing the real-time performance (see, for example, Patent Document 1). A communication device has, for example, a function of automatically responding by a generative AI (artificial intelligence), a function of collecting, referring to, and reflecting personal information (personality, appearance, behavior, etc.), a function of a dialogue system (management / improvement of dialogue data, etc.), and a function of generating an avatar.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The above-described technology has the following problems. A response by a generative AI may not resonate with people's hearts in some cases. / / 这里原文中“生成AIによる応答だけでは、人の心に響かない場合がある。”翻译为“A response by a generative AI may not resonate with people's hearts in some cases.”更符合英文表达习惯 Although a user can generate an avatar that is individually suitable for each of their appearance, personality, and behavior, they cannot generate a virtual human (VH) that has all three elements. The avatar and the virtual human side cannot read deeper into "what real humans are seeking" based on a deeper inner aspect.

[0005] When users utilize avatars or virtual humans, they are required to input the actions and words of the avatar or virtual human displayed on the user interface (UI) screen, or to make selections from multiple options, resulting in a cumbersome operation. Avatars and virtual humans can only respond according to pre-programmed procedures. While users can reflect on their past through text and images, it is difficult for them to interact with their past or future selves through virtual humans, receive advice, or reflect on their past.

[0006] This invention has been made in view of these circumstances, and aims to provide an information processing device, an information processing method, and a program that enable interaction using the user's video and audio. [Means for solving the problem]

[0007] One aspect of the present invention that solves the above-mentioned problems is an information processing device comprising: a receiving unit that receives user information and personality information of a user; a speech information creation unit that creates information indicating the user's speech to the conversation partner in the user's voice based on the user information and personality information of the user received by the receiving unit; and a video information creation unit that creates video information indicating the user to be displayed to the conversation partner based on the user information and personality information of the user received by the receiving unit.

[0008] Furthermore, one aspect of the present invention is an information processing method executed by a computer, which receives user information and personality information of a user, creates information that indicates the user's speech to the conversation partner in the user's voice based on the received user information and personality information of the user, and creates video information indicating the user to be displayed to the conversation partner based on the received user information and personality information of the user.

[0009] Furthermore, one aspect of the present invention is a program that causes a computer to receive user information and personality information of a user, to create information that indicates the user's speech to the conversation partner in the user's voice based on the received user information and personality information of the user, and to create video information indicating the user to be displayed to the conversation partner based on the received user information and personality information of the user. [Effects of the Invention]

[0010] According to the present invention, the effect of enabling interaction using the user's video and audio can be obtained. [Brief explanation of the drawing]

[0011] [Figure 1] This diagram shows the configuration of a dialogue system relating to one embodiment of the present invention. [Figure 2] This figure shows the hardware configuration of the information processing device according to this embodiment. [Figure 3] This is a functional block diagram showing an example of an information processing device according to this embodiment. [Figure 4] This flowchart shows an example of the operation of the information processing device according to this embodiment. [Figure 5] This flowchart shows an example of the operation of the information processing device according to this embodiment. [Figure 6] This is a functional block diagram showing an example of an information processing device according to a modified embodiment. [Figure 7] This figure shows an example of a learning model creation device according to this embodiment. [Figure 8] This flowchart shows an example of the operation of the learning model creation device according to this embodiment. [Modes for carrying out the invention]

[0012] Next, the information processing apparatus, information processing method, and program according to this embodiment will be described with reference to the drawings. The embodiments described below are merely examples, and the embodiments to which the present invention is applied are not limited to the following embodiments. In all the figures for explaining the embodiments, those having the same function are denoted by the same reference numerals, and repeated explanations are omitted. In addition, “based on XX” as used in the present application means “at least based on XX”, and includes cases where it is based on another element in addition to XX. Also, “based on XX” is not limited to the case where XX is directly used, and includes cases where it is based on something obtained by performing arithmetic operations or processing on XX. “XX” is an arbitrary element (for example, arbitrary information).

[0013] [Overview of the dialogue system] FIG. 1 is a diagram showing the configuration of a dialogue system according to an embodiment of the present invention. As shown in FIG. 1, the dialogue system 1 includes an information processing apparatus 100. The information processing apparatus 100 includes an output device 21, and displays a virtual human VH on the output device 21. The virtual human VH is a photo-realistic three-dimensional avatar of a user U who is an actual person. The virtual human VH conducts a dialogue with an interlocutor INT. The information processing apparatus 100 creates the virtual human VH so that it cannot be distinguished from the user U. FIG. 1 shows a state in which the virtual human VH and the interlocutor INT are conducting a dialogue.

[0014] [Hardware configuration of the information processing apparatus] FIG. 2 is a diagram showing the hardware configuration of the information processing apparatus according to the present embodiment. As shown in FIG. 2, the information processing apparatus 100 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, and a RAM (Random Access Memory) 13. The information processing apparatus 100 also includes a bus 14, a bridge 15, an interface 16, an imaging device 17, a microphone 18, a communication device 19, an input device 20, and an output device 21.

[0015] The information processing apparatus 100 may have a processing circuit such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array) instead of or together with the CPU 11.

[0016] The CPU 11 functions as an arithmetic processing unit and a control unit, and controls all or part of the operations within the information processing apparatus 100 according to various programs recorded in the ROM 12 or the RAM 13. The ROM 12 stores programs, arithmetic parameters, etc. used by the CPU 11. The RAM 13 primary stores programs used in the execution of the CPU 11 and parameters that change as appropriate during the execution. The CPU 11, the ROM 12, and the RAM 13 are interconnected by a bus 14 constituted by an internal bus such as a CPU bus.

[0017] The imaging device 17 is, for example, a camera that images the real space using an imaging element such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device), and various members such as a lens for controlling the formation of a subject image on the imaging element, and generates an imaging image. The imaging device 17 may capture still images or moving images. The imaging device 17 may capture three-dimensional video. The imaging device 17 images the user U or the interlocutor INT and creates video information. The microphone 18 creates voice information by collecting voice. The microphone 18 creates voice information by collecting the voice of the user U or the interlocutor INT.

[0018] The communication device 19 is a communication interface, for example, consisting of a communication device for connecting to the communication network 50. The communication device 19 is, for example, a communication card for LAN (Local Area Network), Bluetooth (registered trademark), Wi-Fi, or WUSB (Wireless USB). The communication device 19 sends and receives signals, etc., to and from the Internet or other communication devices using a predetermined protocol such as TCP / IP. The communication network 50 connected to the communication device 19 is a network connected by wire or wireless, and may include, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

[0019] Communication device 19 communicates wirelessly with wallet WAL. Wallet WAL stores non-fungible tokens (NFTs). Non-fungible tokens are non-fungible digital tokens (certificates) issued on the blockchain that have unique values ​​and attributes. NFTs are linked to user information of user U. Communication device 19 receives NFTs sent by wallet WAL.

[0020] The input device 20 is a device operated by the user, such as a touch panel, physical buttons, switches, and levers. The input device 20 may be a remote control device that uses radio waves such as infrared, or an external connected device such as a smartphone or smartwatch that is compatible with the operation of the information processing device 100. The input device 20 includes an input control circuit that generates an input signal based on information input by the user U or the conversation partner INT and outputs it to the CPU 11. By operating this input device 20, the user U or the conversation partner INT inputs various data to the information processing device 100 or instructs it to perform processing operations. For example, user U inputs their own personality information by operating the input device 20. The input device 20 acquires the personality information input by user U.

[0021] The output device 21 is a device capable of notifying the user U or the conversation partner INT of information acquired from the CPU 11 using senses such as sight and hearing. The output device 21 may include, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, and an audio output device such as a speaker. The output device 21 outputs the results obtained from the processing of the information processing device 100 in the form of text, images, videos, and other visuals, as well as sound such as voice and acoustics.

[0022] Each of the above components may be constructed using general-purpose materials, or it may be constructed using hardware specialized for the function of each component. Such configurations may be modified as appropriate depending on the technological level at the time of implementation.

[0023] [Functional Block Configuration of Information Processing Device] Figure 3 is a functional block diagram showing an example of the information processing device 100 according to this embodiment. As shown in Figure 3, the information processing device 100 includes a reception unit 102, a speech information creation unit 104, a video information creation unit 105, a speech information analysis unit 107, a video information analysis unit 108, and an integration unit 109.

[0024] This section explains how to create a virtual human (VH) representing user U. The reception unit 102 receives and accepts user U's personality information from the input device 20. An example of user U's personality information may be represented by the five basic factors that describe human personality according to the Big Five theory. The following explanation continues, using the example of user U's personality information being represented by the Big Five theory. According to the Big Five theory, a person's personality consists of five factors: extraversion, conscientiousness, agreeableness, openness, and neuroticism. User U's personality information may also be a numerical value indicating the degree of each of the five factors.

[0025] The reception unit 102 obtains the NFT from the communication device 19, retrieves the user information of user U associated with the obtained NFT, and accepts it. Since the NFT can prove that it is owned by user U, the user information associated with the NFT belongs to user U. For example, based on the NFT, metadata such as a URL (Uniform Resource Locator) where user U's user information is stored may be obtained, and the user information may be retrieved. Examples of user information for user U include user U's personal information, user U's activity log (life log), user U's acquired qualifications, user U's payment information, user U's hobby information, and user U's behavioral history such as student ID. This information is linked to authentication certificates such as NFTs.

[0026] As a behavioral history recording system for recording behavioral history, distributed ledger technology and distributed ID technology may be used, and these may also be used as parameters for the Generative AI described later. For example, it may be a blockchain operated by a company or consortium, or it may be a public chain that is not operated by a company or consortium but is completely decentralized. In order to truly communicate with oneself, advice should be given based on public behavioral history that is outside the control of the company. By being a completely decentralized public chain, it is possible to use information from an open and tamper-proof public chain. Specifically, in an authentication method that performs authentication between a first device relating to the authenticated person and a second device relating to the authenticater, a blockchain system is used which is composed of multiple processing nodes and storage media connected to each other via a network, and each of the multiple processing nodes performs calculations to verify and approve transactions based on the electronic signature obtained from the private key information corresponding to the destination address, in order to record that an address based on private key information is the destination of a predetermined type of token, which is a digital asset that can be transferred between addresses, and the results of the calculations are distributed to multiple storage media to hold the information. By configuring the system in this way, it is possible to create a virtual human (VH) using the behavioral history provided by an individual without infringing on the individual's sovereignty. Furthermore, the created virtual human (VH) can then be invoked. As the distributed ledger technology, NFTs or SBTs (Soul Bound Tokens) may be used. As the distributed identity technology, decentralized identity (DID) or verifiable credentials may be used. Furthermore, information such as behavioral history and personality assessments can be stored locally, not just on the blockchain. This configuration allows for the combined use of information such as behavioral history and personality assessments stored on the blockchain and information stored locally. For example, information that changes over time, such as emotions and personality during conversations, can be recorded off-chain and used by the virtual human to change their speaking style in real time.

[0027] The reception unit 102 receives and accepts user U's voice information from the microphone 18. An example of user U's voice information is a reproduction of user U's own voice reading aloud. The reception unit 102 acquires and receives video information of user U from the imaging device 17. An example of video information of user U is an image showing user U's appearance.

[0028] The speech information creation unit 104 acquires user U's voice information from the reception unit 102. Based on the acquired user U's voice information, the speech information creation unit 104 synthesizes user U's voice. Furthermore, the speech information creation unit 104 acquires user U's personality information and user information from the reception unit 102. Based on the acquired user U's personality information and user information, the speech information creation unit 104 creates information indicating user U's tone of voice towards their conversation partner INT. Examples of tone of voice include a gentle tone and an irritated tone.

[0029] Specifically, the speech information creation unit 104 has a first tone of voice information derivation table that associates a combination of personality information and user information with information indicating tone of voice. The speech information creation unit 104 obtains information indicating tone of voice associated with the combination of personality information and user information of user U obtained from the reception unit 102 from the first tone of voice information derivation table. The speech information creation unit 104 is composed of a generation AI and generates various speech information. Based on the acquired tone information and the generated speech information, the speech information creation unit 104 creates voice information for user U and inputs it to the integration unit 109.

[0030] The video information creation unit 105 acquires video information of user U from the reception unit 102. Based on the acquired video information of user U, the video information creation unit 105 creates video information of user U's virtual human VH. Specifically, based on the video information of user U, the video information creation unit 105 creates a 3D computer graphics image of user U's appearance. The video information creation unit 105 creates video information of the virtual human VH that shows user U's appearance, including the 3D computer graphics image of user U's appearance. Furthermore, the video information creation unit 105 obtains user U's personality information and user information from the reception unit 102. Based on the obtained user U's personality information and user information, the video information creation unit 105 creates video information showing the virtual human VH with a modified facial expression.

[0031] Specifically, the video information creation unit 105 has a first facial expression information derivation table that associates combinations of personality information and user information with information indicating facial expressions. The video information creation unit 105 obtains information indicating facial expressions associated with combinations of personality information and user information of user U obtained from the reception unit 102 from the first facial expression information derivation table. Examples of information indicating facial expressions include expressions of sadness, anger, joy, fear, disgust, and surprise.

[0032] The video information creation unit 105 creates video information showing the virtual human VH with a modified facial expression based on the acquired facial expression information, and inputs the created video information showing the virtual human VH to the integration unit 109. The virtual human VH with a modified facial expression is, for example, a virtual human VH with a modified facial expression from the virtual human VH before the modification. The change in facial expression may include at least one of the following: blinking of the eyes, movement of the mouth, and change in the orientation of the face.

[0033] The integration unit 109 acquires synthesized voice information of user U from the speech information creation unit 104 and video information representing user U's virtual human VH from the video information creation unit 105. The integration unit 109 integrates the acquired synthesized voice information of user U and the information representing user U's virtual human VH by synchronizing them and outputs them to the output device 21. For example, the integration unit 109 integrates the mouth movements and facial expressions of the virtual human VH to make it appear as if user U's voice is being spoken.

[0034] The output device 21 acquires the synthesized voice information of user U and the video information representing user U's virtual human VH from the integration unit 109, displays user U's virtual human VH, and outputs user U's voice.

[0035] This section explains how a virtual human (VH) interacts with an interactive partner (INT). The reception unit 102 receives and processes speech information from the microphone 18. An example of speech information is what the conversation partner INT has said. The following explanation continues with the case where the speech information is what the conversation partner INT has said. The reception unit 102 acquires and receives video information from the imaging device 17. An example of video information is an image showing the appearance of the conversation partner INT. The following explanation will continue with the case where the video information shows the appearance of the conversation partner INT.

[0036] The speech information analysis unit 107 obtains speech information from the conversation partner INT from the reception unit 102. The speech information analysis unit 107 analyzes the obtained speech information from the conversation partner INT. For example, the speech information analysis unit 107 recognizes the content of the conversation partner INT's speech based on the conversation partner INT's speech information.

[0037] The speech information creation unit 104 obtains information from the speech information analysis unit 107 that shows the content of the dialogue partner INT's speech, which is the result of analyzing the dialogue partner INT's speech information. Based on the user U's personality information and user information, as well as the obtained information showing the content of the dialogue partner INT's speech, the speech information creation unit 104 uses a generating AI to create a response to the dialogue partner INT. The speech information creation unit 104 creates information indicating the tone of voice that user U uses when speaking to the conversation partner INT, based on user U's personality information and user information, as well as information indicating the content of the conversation partner INT's speech that has been acquired.

[0038] Specifically, the speech information creation unit 104 has a second tone of voice information derivation table that associates a combination of personality information, user information, and utterance content with information indicating tone of voice. The speech information creation unit 104 obtains information indicating tone of voice, which is associated with a combination of the personality information and user information of user U and the information indicating the utterance content of the conversation partner INT, from the second tone of voice information derivation table. The speech information creation unit 104 synthesizes the voice of user U based on the acquired tone information and the information indicating the response to the conversation partner INT created by the generating AI, and inputs the synthesized information of user U's response voice to the integration unit 109.

[0039] The video information creation unit 105 obtains information from the speech information analysis unit 107 that shows the content of the speech of the conversation partner INT, which is the result of analyzing the speech information of the conversation partner INT. Based on the personality information and user information of user U, as well as the information showing the content of the speech of the conversation partner INT, the video information creation unit 105 creates video information showing the virtual human VH with a modified facial expression.

[0040] Specifically, the video information creation unit 105 has a second facial expression information derivation table that associates a combination of personality information, user information, and information indicating the content of speech with information indicating facial expressions. The video information creation unit 105 obtains information indicating facial expressions associated with a combination of the personality information and user information of user U and information indicating the content of speech of the conversation partner INT from the second facial expression information derivation table. The video information creation unit 105 creates video information showing the virtual human VH of user U with a modified facial expression based on the acquired facial expression information, and inputs the created video information showing the virtual human VH of user U to the integration unit 109.

[0041] The integration unit 109 acquires synthesized voice information of user U from the speech information creation unit 104 and information indicating user U's virtual human VH from the video information creation unit 105. The integration unit 109 integrates the acquired synthesized voice information of user U and information indicating user U's virtual human VH by synchronizing them and outputs them to the output device 21. For example, the integration unit 109 integrates the mouth movements and facial expressions of the virtual human VH to make it appear as if user U's voice is being spoken.

[0042] The video information analysis unit 108 acquires video information of the conversation partner INT from the reception unit 102. The video information analysis unit 108 analyzes the acquired video information of the conversation partner INT. For example, the video information analysis unit 108 recognizes the facial expressions of the conversation partner INT based on the video information of the conversation partner INT.

[0043] The speech information creation unit 104 obtains the facial expression information of the conversation partner INT, which is the result of analyzing the video information of the conversation partner INT, from the video information analysis unit 108. Based on the personality information and user information of the user U, as well as the obtained facial expression information of the conversation partner INT, the speech information creation unit 104 generates a response to the conversation partner INT using a generating AI. The speech information creation unit 104 creates information indicating the tone of voice that user U uses when speaking to the conversation partner INT, based on user U's personality information and user information, as well as the facial expression information of the conversation partner INT that it has acquired.

[0044] Specifically, the speech information creation unit 104 has a second tone of voice information derivation table that associates combinations of personality information, user information, and facial expression information with information indicating tone of voice. The speech information creation unit 104 obtains information indicating tone of voice, which is associated with combinations of the personality information and user information of user U and the facial expression information of the conversation partner INT, from the second tone of voice information derivation table. The speech information creation unit 104 synthesizes the voice of user U based on the acquired tone information and the information indicating the response to the conversation partner INT created by the generating AI, and inputs the synthesized information of user U's response voice to the integration unit 109.

[0045] The video information creation unit 105 obtains the facial expression information of the conversation partner INT, which is the result of analyzing the video information of the conversation partner INT, from the video information analysis unit 108. Based on the personality information and user information of user U, as well as the obtained facial expression information of the conversation partner INT, the video information creation unit 105 creates video information showing the virtual human VH of user U with a modified facial expression to be displayed to the conversation partner INT.

[0046] Specifically, the video information creation unit 105 has a second facial expression information derivation table that associates combinations of personality information, user information, and facial expression information with information indicating facial expressions. The video information creation unit 105 obtains information indicating facial expressions associated with combinations of the personality information and user information of user U and the facial expression information of the conversation partner INT from the second facial expression information derivation table. The video information creation unit 105 creates video information showing the virtual human VH of user U with a modified facial expression based on the acquired facial expression information, and inputs the created video information showing the virtual human VH of user U to the integration unit 109.

[0047] The integration unit 109 acquires synthesized voice information of user U from the speech information creation unit 104 and information indicating user U's virtual human VH from the video information creation unit 105. The integration unit 109 integrates the acquired synthesized voice information of user U and information indicating user U's virtual human VH by synchronizing them and outputs them to the output device 21. For example, the integration unit 109 integrates the mouth movements and facial expressions of the virtual human VH to make it appear as if user U's voice is being spoken. The output device 21 acquires synchronized information from the integration unit 109, including synthesized voice information of user U and information indicating user U's virtual human VH. It then displays user U's virtual human VH and outputs user U's voice.

[0048] The reception unit 102, speech information creation unit 104, video information creation unit 105, speech information analysis unit 107, video information analysis unit 108, and integration unit 109 are realized, for example, by a hardware processor such as a CPU executing computer programs (software) stored in memory units such as ROM 12 and RAM 13. Furthermore, some or all of these functional components may be implemented by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC, FPGA, and GPU (Graphics Processing Unit), or by collaboration between software and hardware.

[0049] Computer programs may be stored in advance on a storage device such as an HDD or flash memory, or they may be stored on a removable storage medium such as a DVD (Digital Versatile Disc) or CD-ROM, and installed when the storage medium is inserted into a drive device.

[0050] (Operation of the information processing device 100) Figure 4 is a flowchart showing an example of the operation of the information processing device 100 according to this embodiment. The process of creating a virtual human (VH) will be explained with reference to Figure 4. (Step S1-1) The imaging device 17 captures images of user U and creates video information. (Step S2-1) The reception unit 102 acquires and accepts video information of user U from the imaging device 17. The video information creation unit 105 acquires video information of user U from the reception unit 102 and creates a 3D computer graphics image of user U's face based on the acquired video information of user U.

[0051] (Step S3-1) The video information creation unit 105 creates a virtual human VH that includes a 3D computer graphics image of the user U's appearance. (Step S4-1) The microphone 18 creates audio information by collecting sound. (Step S5-1) The reception unit 102 receives and accepts the voice information of user U from the microphone 18. The speech information creation unit 104 receives the voice information of user U from the reception unit 102. The speech information creation unit 104 synthesizes the voice of user U based on the acquired voice information of user U.

[0052] (Step S6-1-1) The communication device 19 receives the NFT sent by the wallet (WAL). (Step S6-1-2) The communication device 19 receives the behavioral history information transmitted by the behavioral history recording system. (Step S7-1) The reception unit 102 obtains an NFT from the communication device 19, retrieves user information of user U associated with the obtained NFT, and accepts it. The reception unit 102 obtains behavioral history information from the communication device 19, retrieves user information of user U based on the obtained behavioral history information, and accepts it. (Step S8-1) User U undergoes a personality assessment. (Step S9-1) The input device 20 acquires the personality information entered by user U. The reception unit 102 acquires and accepts the personality information of user U from the input device 20.

[0053] (Step S10-1) The speech information creation unit 104 acquires voice information of user U from the reception unit 102 and synthesizes user U's voice based on the acquired voice information of user U. Furthermore, the speech information creation unit 104 acquires personality information and user information of user U from the reception unit 102 and creates information indicating the tone of voice of user U towards the person they are talking to based on the acquired personality information and user information of user U. The speech information creation unit 104 generates speech information and creates user U's voice information based on the tone of voice information and the generated speech information, and inputs it to the integration unit 109.

[0054] The video information creation unit 105 acquires video information of user U from the reception unit 102 and creates a virtual human VH of user U based on the acquired video information of user U. Furthermore, the video information creation unit 105 acquires personality information and user information of user U from the reception unit 102 and creates a virtual human VH with modified facial expressions based on the acquired personality information and user information of user U, and inputs it into the integration unit 109.

[0055] The integration unit 109 acquires synthesized voice information of user U from the speech information creation unit 104 and video information representing user U's virtual human VH from the video information creation unit 105. The integration unit 109 integrates the acquired synthesized voice information of user U and the information representing user U's virtual human VH by synchronizing them and outputs them to the output device 21. In the flowchart shown in Figure 4, the process may be configured so that either step S6-1-1 or step S6-1-2 is performed.

[0056] Figure 5 is a flowchart showing an example of the operation of the information processing device 100 according to this embodiment. An example of the process of interacting with the conversation partner INT will be explained with reference to Figure 5. The conversation partner INT and the virtual human VH converse (speak). (Step S1-2) The reception unit 102 receives and processes the utterance information of the conversation partner INT from the microphone 18. (Step S2-2) The speech information analysis unit 107 obtains speech information from the conversation partner INT from the reception unit 102. The speech information analysis unit 107 then analyzes the obtained speech information from the conversation partner INT. The speech information creation unit 104 obtains information from the speech information analysis unit 107 that shows the content of the dialogue partner INT's speech, which is the result of analyzing the dialogue partner INT's speech information. Based on the user U's personality information and user information, as well as the obtained information showing the content of the dialogue partner INT's speech, the speech information creation unit 104 uses a generating AI to create a response to the dialogue partner INT.

[0057] The speech information creation unit 104 creates information indicating the tone of voice that user U uses when speaking to the conversation partner INT, based on user U's personality information and user information, as well as information indicating the content of the conversation partner INT's speech that has been acquired. The speech information creation unit 104 synthesizes user U's voice based on the acquired tone information and the information indicating the response to the dialogue partner INT created by the generating AI, and inputs the synthesized information of user U's response voice to the integration unit 109. For example, the speech information creation unit 104 synthesizes silence from user U and inputs the synthesized information of user U's response voice to the integration unit 109.

[0058] The video information creation unit 105 obtains information from the speech information analysis unit 107 that shows the content of the speech of the conversation partner INT, which is the result of analyzing the speech information of the conversation partner INT. Based on the personality information and user information of user U, as well as the information showing the content of the conversation partner INT's speech, the video information creation unit 105 creates video information showing a virtual human VH with a modified facial expression. For example, the video information creation unit 105 creates a virtual human VH that is nodding in agreement. The nodding virtual human VH is used to temporarily fill gaps in the conversation. The video information creation unit 105 inputs the information showing the created user U's virtual human VH to the integration unit 109.

[0059] (Step S3-2) The integration unit 109 acquires synthesized voice information of user U from the speech information creation unit 104 and information indicating user U's virtual human VH from the video information creation unit 105. The integration unit 109 integrates the acquired synthesized voice information of user U and information indicating user U's virtual human VH by synchronizing them and outputs them to the output device 21. In this way, the integration unit 109 creates a virtual human VH that is nodding in agreement based on the mood and other factors inferred from user U's facial expressions and speech patterns.

[0060] The output device 21 acquires synthesized voice information of user U and information indicating user U's virtual human VH from the integration unit 109, displays user U's virtual human VH, and outputs user U's voice. (Step S4-2) The speech information analysis unit 107 obtains speech information from the conversation partner INT from the reception unit 102. The speech information analysis unit 107 then analyzes the obtained speech information from the conversation partner INT. (Step S5-2) The speech information creation unit 104 obtains information from the speech information analysis unit 107 that shows the content of the dialogue partner INT's speech, which is the result of analyzing the dialogue partner INT's speech information. Based on the user U's personality information and user information, as well as the obtained information showing the content of the dialogue partner INT's speech, the speech information creation unit 104 uses a generating AI to create a response to the dialogue partner INT.

[0061] The speech information creation unit 104 creates information indicating the tone of voice that user U uses when speaking to the conversation partner INT, based on user U's personality information and user information, as well as information indicating the content of the conversation partner INT's speech that has been acquired. The speech information creation unit 104 synthesizes the voice of user U based on the acquired tone information and the information indicating the response to the conversation partner INT created by the generating AI, and inputs the synthesized information of user U's response voice to the integration unit 109.

[0062] (Step S6-2) The video information creation unit 105 obtains information from the speech information analysis unit 107 that shows the content of the speech of the conversation partner INT, which is the result of analyzing the speech information of the conversation partner INT. Based on the personality information and user information of user U, as well as the information showing the content of the conversation partner INT, the video information creation unit 105 creates video information showing the virtual human VH with a modified facial expression. Based on the acquired facial expression information, the video information creation unit 105 creates video information showing the virtual human VH of user U with a modified facial expression, and inputs the created video information showing the virtual human VH of user U to the integration unit 109.

[0063] (Step S7-2) The integration unit 109 acquires synthesized voice information of user U from the speech information creation unit 104 and information indicating user U's virtual human VH from the video information creation unit 105. The integration unit 109 integrates the acquired synthesized voice information of user U and information indicating user U's virtual human VH by synchronizing them and outputs them to the output device 21. The output device 21 acquires the synthesized voice information of user U and information indicating user U's virtual human VH from the integration unit 109, displays user U's virtual human VH, and outputs user U's voice. Then, the process returns to step S1-2.

[0064] In step S5-2 of the flowchart shown in Figure 5, the speech information creation unit 104 may, in addition to obtaining analysis information of the dialogue partner's speech information from the speech information analysis unit 107, also collect the dialogue partner's speech information. The speech information creation unit 104 may also create information indicating a response to the dialogue partner based on the acquired analysis information of the dialogue partner's speech information and the dialogue partner's speech information.

[0065] In the embodiment described above, a case was described in which a user U inputs personality information to the information processing device 100 by operating the input device 20. However, the invention is not limited to this example. For example, the information processing device 100 may be configured to perform a personality diagnosis. In the embodiments described above, we explained the case where user U's personality information is represented by the Big Five personality theory, but this is not the only example. For example, user U's personality information may be represented by Jung's personality typology.

[0066] In the embodiments described above, the virtual human VH may be configured to have the following functions. For example, it may have a generative AI function to generate standard answers, a function to create photorealistic virtual human (VH) models, a function to create synthesized voices, and a function to create virtual human (VH) models that are closer to the user by combining the user's appearance, personality, and behavior.

[0067] Furthermore, for example, the system may include features to create virtual human (VH) models of the past, present, and future; features to enable natural dialogue by using nods and other acknowledgments to bridge the gap until the user is ready to respond, without displaying the VH's status on the UI screen when reading the user's situation; and features to diagnose the user's personality. Furthermore, for example, the virtual human (VH) may have functions to instruct the user on improvements, guidance, and consultation; functions to acquire the user's face, heart rate, etc., and utilize this information for the virtual human's facial expressions and behavior; and functions to process and manage dialogue data.

[0068] Furthermore, for example, it may be equipped with functions to judge the reliability of old data, functions to react according to the user's personality, gestures, facial expressions, and actions, and functions to build a network for interaction between virtual humans (VHs). Furthermore, for example, the system may have functions to acquire data on changes in user behavior and utilize it in the behavior of the virtual human (VH), or to read the user's tone of voice and gestures and use them in the dialogue. Furthermore, for example, by increasing the amount of information available for the virtual human (VH), it may be possible to enable the user to interact with the display screen without prior preparation. In this case, for example, the information processing device 100 can train the virtual human (VH) from the history and logs of the interaction content.

[0069] In the embodiment described above, information may be collected and analyzed regarding the creation and interaction of avatars such as virtual humans (VH), and the system may be configured to perform the following processing. For example, the functions of the virtual human (VH) could be constantly updated and even edited by the user, or a community of virtual humans could be created to facilitate the building of friendships. Alternatively, the virtual human could predict the user's future actions and enable suggestions and simulations from the virtual human's side, or suggest things that are relevant or recommended to the user. Furthermore, the virtual human could use a speaking style and tone tailored to the user to elicit their true needs, or provide a more effective virtual space that allows users to utilize the virtual human while enhancing their sense of immersion.

[0070] (modified version) The dialogue system 1 according to a modified embodiment includes an information processing device 200 in place of, or together with, the information processing device 100. The hardware configuration of the information processing device 200 can be described using Figure 2, so a detailed explanation is omitted. [Functional Block Configuration of Information Processing Device] Figure 6 is a functional block diagram showing an example of an information processing device 200 according to a modified embodiment. As shown in paragraph 6, the information processing device 200 includes a reception unit 202, a speech information creation unit 204, a video information creation unit 205, a speech information analysis unit 207, a video information analysis unit 208, and an integration unit 209.

[0071] This section explains how to create a virtual human (VH) representing user U. The reception unit 202 can be used with the reception unit 102. The reception unit 202 obtains and accepts user U's personality information from the input device 20. The reception unit 202 obtains an NFT from the communication device 19, retrieves the user information of user U associated with the obtained NFT, and accepts it. Alternatively, the reception unit 102 may obtain behavioral history information from the communication device 19, and based on the obtained behavioral history information, retrieve and accept the user information of user U. The reception unit 202 receives and processes the voice information of user U from the microphone 18. The reception unit 202 acquires and receives video information of user U from the imaging device 17.

[0072] The speech information creation unit 204 acquires user U's voice information from the reception unit 202. Based on the acquired user U's voice information, the speech information creation unit 204 synthesizes user U's voice. Furthermore, the speech information creation unit 204 acquires user U's personality information and user information from the reception unit 202. Based on the acquired user U's personality information and user information, the speech information creation unit 204 creates information indicating user U's tone of voice towards their conversation partner INT.

[0073] Specifically, the speech information creation unit 204 has a first trained model 214. The first trained model 214 is machine-learned to recognize the relationship between a combination of personality information and user information and information indicating tone of voice. The method for creating the first trained model 214 will be described later. The speech information creation unit 204 performs the following processing on the combination of user U's personality information and user information obtained from the reception unit 202. The speech information creation unit 204 inputs the combination of personality information and user information into the first trained model 214 and obtains the information indicating tone of voice output by the first trained model 214.

[0074] The speech information creation unit 204 includes a generation AI and generates various speech information. Based on the acquired tone information and the generated speech information, the speech information creation unit 204 creates voice information for user U and inputs it to the integration unit 209.

[0075] The video information creation unit 205 can apply the video information creation unit 105. The video information creation unit 205 acquires video information of user U from the reception unit 202. Based on the acquired video information of user U, the video information creation unit 205 creates video information of user U's virtual human VH. Specifically, the video information creation unit 205 creates a 3D computer graphics image of user U's appearance based on user U's video information. The video information creation unit 205 creates video information of the virtual human VH showing user U's appearance, including the 3D computer graphics image of user U's appearance.

[0076] Furthermore, the video information creation unit 205 obtains user U's personality information and user information from the reception unit 202. Based on the obtained user U's personality information and user information, the video information creation unit 205 creates video information showing the virtual human VH with a modified facial expression.

[0077] Specifically, the video information creation unit 205 has a second trained model 215. The second trained model 215 is machine-trained to understand the relationship between a combination of personality information and user information and information indicating facial expressions. The method for creating the second trained model 215 will be described later. The video information creation unit 205 performs the following processing on the combination of user U's personality information and user information obtained from the reception unit 202. The video information creation unit 205 inputs the combination of personality information and user information into the second trained model 215 and obtains the information indicating facial expressions output by the second trained model 215.

[0078] The video information creation unit 205 creates video information showing the virtual human VH with a modified facial expression based on the acquired facial expression information, and inputs the created video information showing the virtual human VH to the integration unit 209.

[0079] The integration unit 209 can apply the integration unit 109. The integration unit 209 acquires synthesized voice information of user U from the speech information creation unit 204 and video information representing user U's virtual human VH from the video information creation unit 205. The integration unit 209 integrates the acquired synthesized voice information of user U and the information representing user U's virtual human VH by synchronizing them and outputs them to the output device 21.

[0080] The output device 21 acquires the synthesized voice information of user U and the video information representing user U's virtual human VH from the integration unit 209, displays user U's virtual human VH, and outputs user U's voice.

[0081] This section explains how a virtual human (VH) interacts with an interactive partner (INT). The reception unit 202 receives and processes the utterance information of the conversation partner INT from the microphone 18. The reception unit 202 acquires and receives video information of the conversation partner INT from the imaging device 17.

[0082] The speech information analysis unit 207 can apply the speech information analysis unit 107. The speech information analysis unit 207 obtains speech information of the conversation partner INT from the reception unit 202. The speech information analysis unit 207 analyzes the obtained speech information of the conversation partner INT. The speech information creation unit 204 obtains information indicating the content of the dialogue partner INT's utterances, which is the result of analyzing the dialogue partner INT's utterance information, from the speech information analysis unit 207. Based on the personality information and user information of user U, as well as the obtained information indicating the content of the dialogue partner INT's utterances, the speech information creation unit 204 generates a response to the dialogue partner INT using a generating AI. Based on the personality information and user information of user U, as well as the obtained information indicating the content of the dialogue partner INT's utterances, the speech information creation unit 204 generates information indicating the tone of voice of user U towards the dialogue partner INT.

[0083] Specifically, the speech information creation unit 204 has a third trained model 224. The third trained model 224 is a machine learning model that studies the relationship between a combination of personality information, user information, and utterance content, and information indicating tone of voice. The method for creating the third trained model 224 will be described later. The speech information creation unit 204 performs the following processing on the information indicating the utterance content of the conversation partner INT obtained from the speech information analysis unit 207. The speech information creation unit 204 inputs the combination of personality information, user information, and information indicating the utterance content of the conversation partner INT into the third trained model 224, and obtains the information indicating tone of voice output by the third trained model 224. The speech information creation unit 204 synthesizes user U's voice based on the acquired tone information and the information indicating the response to the conversation partner INT created by the generating AI, and inputs the synthesized information of user U's response voice to the integration unit 209.

[0084] The video information creation unit 205 obtains information from the speech information analysis unit 207 that shows the content of the speech of the conversation partner INT, which is the result of analyzing the speech information of the conversation partner INT. Based on the personality information and user information of user U, as well as the information showing the content of the conversation partner INT's speech, the video information creation unit 205 creates video information showing the virtual human VH with a modified facial expression.

[0085] Specifically, the video information creation unit 205 has a fourth trained model 225. The fourth trained model 225 is machine-trained to understand the relationship between a combination of personality information, user information, and utterance content, and information indicating facial expressions. The method for creating the fourth trained model 225 will be described later. The video information creation unit 205 performs the following processing on the information indicating the utterance content of the conversation partner INT obtained from the utterance information analysis unit 207. The video information creation unit 205 inputs the combination of personality information, user information, and information indicating the utterance content of the conversation partner INT into the fourth trained model 225, and obtains the information indicating facial expressions output by the fourth trained model 225. The video information creation unit 205 creates video information showing the virtual human VH of user U with a modified facial expression based on the acquired facial expression information, and inputs the created video information showing the virtual human VH of user U to the integration unit 209.

[0086] The integration unit 209 acquires synthesized voice information of user U from the speech information creation unit 204 and information indicating user U's virtual human VH from the video information creation unit 205. The integration unit 209 integrates the acquired synthesized voice information of user U and the information indicating user U's virtual human VH by synchronizing them and outputs them to the output device 21.

[0087] The video information analysis unit 208 can apply the video information analysis unit 108 to it. The video information analysis unit 208 acquires video information of the conversation partner INT from the reception unit 202. The video information analysis unit 208 analyzes the acquired video information of the conversation partner INT. For example, the video information analysis unit 208 recognizes the facial expressions of the conversation partner INT based on the video information of the conversation partner INT.

[0088] The speech information creation unit 204 obtains the facial expression information of the conversation partner INT, which is the result of analyzing the video information of the conversation partner INT, from the video information analysis unit 208. Based on the personality information and user information of user U, as well as the obtained facial expression information of the conversation partner INT, the speech information creation unit 204 generates a response to the conversation partner INT using a generating AI. The speech information creation unit 204 creates information indicating the tone of voice that user U uses when speaking to the conversation partner INT, based on user U's personality information and user information, as well as the facial expression information of the conversation partner INT that has been acquired.

[0089] Specifically, the speech information creation unit 204 has a fifth trained model 234. The fifth trained model 234 is a machine learning model that studies the relationship between a combination of personality information, user information, and facial expression information, and information indicating tone of voice. The method for creating the fifth trained model 234 will be described later. The speech information creation unit 204 performs the following processing on the facial expression information of the conversation partner INT obtained from the video information analysis unit 208. The speech information creation unit 204 inputs the combination of personality information, user information, and facial expression information of the conversation partner INT into the fifth trained model 234 and obtains the information indicating tone of voice output by the fifth trained model 234.

[0090] The speech information creation unit 204 synthesizes user U's voice based on the acquired tone information and the information indicating the response to the conversation partner INT created by the generating AI, and inputs the synthesized information of user U's response voice to the integration unit 209. The video information creation unit 205 obtains the facial expression information of the conversation partner INT, which is the result of analyzing the video information of the conversation partner INT, from the video information analysis unit 208. Based on the personality information and user information of user U, as well as the obtained facial expression information of the conversation partner INT, the video information creation unit 205 changes the facial expression of user U's virtual human VH to be displayed to the conversation partner INT, and creates video information showing user U's virtual human VH.

[0091] Specifically, the video information creation unit 205 has a sixth trained model 235. The sixth trained model 235 is machine-learned to recognize the relationship between a combination of personality information, user information, and facial expression information, and information indicating facial expressions. The method for creating the sixth trained model 235 will be described later. The video information creation unit 205 performs the following processing on the facial expression information of the dialogue partner INT obtained from the speech information analysis unit 207. The video information creation unit 205 inputs the combination of personality information, user information, and facial expression information of the dialogue partner INT into the sixth trained model 235, and obtains the information indicating facial expressions output by the sixth trained model 235.

[0092] The video information creation unit 205 creates video information showing the virtual human VH of user U with a modified facial expression based on the acquired facial expression information, and inputs the created video information showing the virtual human VH of user U to the integration unit 209. The integration unit 209 acquires synthesized voice information of user U from the speech information creation unit 204 and information indicating user U's virtual human VH from the video information creation unit 205. The integration unit 209 integrates the acquired synthesized voice information of user U and information indicating user U's virtual human VH by synchronizing them and outputs them to the output device 21.

[0093] The output device 21 acquires synchronized information from the integration unit 209, including synthesized voice information of user U and information indicating user U's virtual human VH. It then displays user U's virtual human VH and outputs user U's voice.

[0094] The reception unit 202, speech information creation unit 204, video information creation unit 205, speech information analysis unit 207, video information analysis unit 208, and integration unit 209 are realized, for example, by a hardware processor such as a CPU executing computer programs (software) stored in memory units such as ROM 12 and RAM 13. Furthermore, some or all of these functional components may be implemented by hardware such as LSIs, ASICs, FPGAs, and GPUs (including circuitry), or by the collaboration of software and hardware.

[0095] Computer programs may be stored in advance on a storage device such as an HDD or flash memory, or they may be stored on a removable storage medium such as a DVD or CD-ROM and installed when the storage medium is inserted into a drive device. The operation of the information processing device 200 can be described using Figures 4 and 5, so a detailed explanation is omitted.

[0096] (Learning model creation device) The method for creating the first trained model 214 to the sixth trained model 235 will be explained. The first trained model 214 to the sixth trained model 235 are created by the trained model creation device. In other words, the trained model creation device creates the first trained model 214 to the sixth trained model 235. Note that the information processing device 200 may include the trained model creation device. In other words, the information processing device 200 may create the first trained model 214 to the sixth trained model 235.

[0097] Figure 7 shows an example of a learning model creation device 300 according to this embodiment. The learning model creation device 300 is implemented using a device such as a personal computer, server, smartphone, tablet computer, or industrial computer. The learning model creation device 300 trains a first learning model using training data such as a first learning dataset, which uses a combination of personality information and user information as input samples and information indicating tone of voice as output samples, and creates a first trained model 214. Here, the first learning model is the model on which the first trained model 214 is based.

[0098] The learning model creation device 300 trains a second learning model using training data such as a second learning dataset, which uses a combination of personality information and user information as input samples and information showing facial expressions as output samples, and creates a second trained model 215. Here, the second learning model is the model on which the second trained model 215 is based.

[0099] The learning model creation device 300 trains a third learning model using training data such as a third learning dataset, which uses combinations of personality information, user information, and utterance content as input samples and information indicating tone of voice as output samples, thereby creating a third trained model 224. Here, the third learning model is the model on which the third trained model 224 is based.

[0100] The learning model creation device 300 trains the fourth learning model using training data such as the fourth learning dataset, which uses combinations of personality information, user information, and utterance content as input samples and information indicating facial expressions as output samples, thereby creating the fourth trained model 225. Here, the fourth learning model is the model on which the fourth trained model 225 is based.

[0101] The learning model creation device 300 trains the fifth learning model using training data such as the fifth learning dataset, which uses combinations of personality information, user information, and facial expression information as input samples and information indicating tone of voice as output samples, thereby creating the fifth trained model 234. Here, the fifth learning model is the model on which the fifth trained model 234 is based.

[0102] The learning model creation device 300 trains the sixth learning model using training data such as the sixth learning dataset, which uses combinations of personality information, user information, and facial expression information as input samples and information showing facial expressions as output samples, thereby creating the sixth trained model 235. Here, the sixth learning model is the model on which the sixth trained model 235 is based.

[0103] For example, the learning model creation device 300 constructs the first to sixth trained models 214 to 235 using algorithms such as CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), Random Forest, SVM (Support Vector Machine), and Neural Networks. Input samples are the data input to the input layer when training the learning model. Output samples are ground truth data, such as training data, used to compare with the output values ​​output from the output layer when training the learning model.

[0104] The learning model creation device 300 includes an input unit 302, a reception unit 304, a processing unit 306, an output unit 308, and a storage unit 310. The input unit 302 receives information. For example, the input unit 302 may have an operation unit such as a keyboard and a mouse. In this case, the input unit 302 receives information corresponding to the operations performed by the user on the operation unit. As another example, the input unit 302 may receive information from an external device. This external device may be, for example, a portable storage medium. The input unit 302 receives the first to sixth training datasets.

[0105] The reception unit 304 obtains the first to sixth training datasets from the input unit 302 and accepts the obtained first to sixth training datasets. The first to sixth training datasets contain input samples and output samples, and the input samples and output samples are paired. The first to sixth training datasets consist of multiple pairs.

[0106] The processing unit 306 calculates the error between the output value output from the output layer of the learning model 307 (which is created by inputting the input sample into the input layer) and the corresponding output sample for each pair. It then modifies the parameters of the learning model 307 to minimize the error and creates the first trained model 214 to the sixth trained model 235. Here, the output sample is an example of training data. The learning model 307 is trained by changing its parameters.

[0107] As described above, the first to sixth trained models 214 to 235 are received by the information processing device 200 via a network or medium from the output unit 308. The first trained model 214, the third trained model 224, and the fifth trained model 234 are acquired by the speech information creation unit 204, while the second trained model 215, the fourth trained model 225, and the sixth trained model 235 are acquired by the video information creation unit 205. If the learning model creation device 300 is included in the information processing device 200, the speech information creation unit 204 acquires the first learned model 214, the third learned model 224, and the fifth learned model 234 from the learning model creation device 300, and the video information creation unit 205 acquires the second learned model 215, the fourth learned model 225, and the sixth learned model 235 from the learning model creation device 300.

[0108] All or part of the input unit 302, the receiving unit 304, the processing unit 306, and the output unit 308 are functional units (hereinafter referred to as software functional units) that are realized, for example, by a processor such as a CPU executing a program stored in the memory unit 310. Furthermore, all or part of the input unit 302, reception unit 304, processing unit 306, and output unit 308 may be implemented by hardware such as an LSI, ASIC, or FPGA, or by a combination of software functions and hardware.

[0109] (Operation of the learning model creation device 300) Figure 8 is a flowchart showing an example of the operation of the learning model creation device 300 according to this embodiment. (Steps S1-3) The input unit 302 acquires the first training dataset. (Step S2-3) The reception unit 304 obtains the first training dataset from the input unit 302 and accepts the obtained first training dataset.

[0110] (Step S3-3) The processing unit 306 obtains the first training dataset from the reception unit 304. For all pairs of input samples and output samples included in the first training dataset, the processing unit 306 inputs the input sample into the input layer of the training model 307 and obtains the output value output from the output layer. The processing unit 306 calculates the error between the output value and the output sample corresponding to the input sample, and modifies the parameters of the training model 307 to minimize the error as much as possible, thereby creating the first trained model 214. (Step S4-3) The output unit 308 obtains the learning model 307 from the processing unit 306. The output unit 308 outputs the obtained learning model 307.

[0111] The method for creating the second to sixth trained models 215 to 235 is the same as the method for creating the first trained model 214, so the explanation will be omitted. The second to sixth trained models 215 to 235 are created by the trained model creation device 300. In other words, the trained model creation device 300 creates the second to sixth trained models 215 to 235. Note that the information processing device 200 may include the trained model creation device 300. In other words, the information processing device 200 may create the second to sixth trained models 215 to 235. The training method used to create the first trained model 214 to the sixth trained model 235 is not limited to supervised learning; it can be any of the following methods: unsupervised learning, semi-supervised learning, reinforcement learning, or deep learning. Furthermore, it can be a combination of these methods, and the training method for machine learning is not restricted.

[0112] According to the dialogue system 1 of the embodiment described above, the information processing device includes a reception unit that receives user information and personality information of a user; a speech information creation unit that creates information indicating the user's speech to the dialogue partner in the user's voice based on the user information and personality information of the user received by the reception unit; and a video information creation unit that creates video information indicating the user to be displayed to the dialogue partner based on the user information and personality information of the user received by the reception unit. By configuring it in this way, the information processing device can create information that represents the user's utterances to the conversation partner in the user's voice, and video information that represents the user to be displayed to the conversation partner, based on the user's user information and personality information, thus enabling interaction using the user's video and voice. Furthermore, the information processing device can display the words, facial expressions, gestures, emotions, and speech patterns of the virtual human (VH) and respond to their speech. Here, speech patterns include tone of voice, gestures, and body language. During the conversation, the information processing device can respond in real time without displaying messages such as "listening" or "thinking" on the virtual human (VH)'s screen, or displaying the conversation history. The information processing device can utilize the virtual human (VH) to improve the conversation by having them repeat the same questions or clarify difficult content during the conversation.

[0113] According to the information processing device, the reception unit receives the utterance information of the conversation partner. The information processing device further includes an utterance information analysis unit that analyzes the utterance information of the conversation partner received by the reception unit. The utterance information creation unit creates information that indicates the response to the conversation partner in the user's voice, based on the analysis information of the utterance information of the conversation partner analyzed by the utterance information analysis unit. By configuring it in this way, the information processing device can create information that indicates a response to the conversation partner in the user's voice, based on further analysis of the conversation partner's utterance information.

[0114] According to the information processing device, the reception unit receives speech information from the conversation partner. The information processing device further includes a speech information analysis unit that analyzes the speech information of the conversation partner received by the reception unit. The video information creation unit creates video information indicating the user to be displayed to the conversation partner, based on the analysis information of the conversation partner's speech information analyzed by the speech information analysis unit. By configuring it in this way, the information processing device can create video information indicating the user to be displayed to the conversation partner, based on further analysis of the conversation partner's speech information.

[0115] According to the information processing device, the reception unit receives video information of the person being spoken to. The information processing device further includes a video information analysis unit that analyzes the video information of the person being spoken to received by the reception unit. The speech information creation unit, based on the analysis of the video information of the person being spoken to by the video information analysis unit, creates information that indicates the response to the person being spoken to in the user's voice. By configuring it in this way, the information processing device can create information that indicates the response to the conversation partner in the user's voice, based on further analysis of the conversation partner's video information.

[0116] According to the information processing device, the reception unit receives video information of the conversation partner. The information processing device further includes a video information analysis unit that analyzes the video information of the conversation partner received by the reception unit. The video information creation unit creates video information indicating the user to be displayed to the conversation partner, based on the analysis information of the video information of the conversation partner analyzed by the video information analysis unit. By configuring it in this way, the information processing device can create video information indicating the user to be displayed to the conversation partner, based on further analysis of the conversation partner's video information.

[0117] According to the information processing device, the reception unit receives the user's voice information. The speech information creation unit then uses the user's voice information received by the reception unit to create information that represents the utterance to the conversation partner in the user's voice. By configuring it in this way, the information processing device can create information that represents the utterance to the conversation partner in the user's voice, based on the user's voice information.

[0118] According to the information processing device, the reception unit receives the user's video information. The video information creation unit then creates video information showing the user to be displayed to the conversation partner, based on the user's video information received by the reception unit. The user's video information includes changes in the user's facial expressions. By configuring it in this way, the information processing device can create video information indicating the user to be displayed to the conversation partner, based on the user's video information.

[0119] According to this embodiment and its modifications, by collecting and integrating four pieces of information—face data from an imaging device, voice data from a recording device such as a microphone, user information from an action processing system, and personality information from a personality diagnosis system—it is possible to create a 3DCG-based photorealistic virtual human (VH) with a photorealistic exterior and interior and conversational capabilities. To make interactions with virtual humans (VH) feel more like real-life conversations, we use a highly accurate facial measurement device to photograph the user and create 3DCG (3D computer graphics). Furthermore, the system uses AI to generate responses to the dialogue, and the user's voice data is used for the voice responses. This configuration makes automated dialogue possible.

[0120] The generation AI function creates responses to the conversation partner, and the creation of a photorealistic virtual human (VH) increases the familiarity and trustworthiness of the VH. As a result, the VH's words and actions can elicit the conversation partner's (user's) worldview and true feelings, allowing for advice and consultation that is more relatable to real people.

[0121] Because virtual humans (VHs) can be created based not only on the user's appearance but also on user information such as personality and behavior, it is possible to interact with virtual humans (VHs) without barriers between them and real people, allowing for more natural conversations and engaging discussions, and enabling the creation of relationships like those of a partner, spouse, or close friend.

[0122] When utilizing virtual humans (VH), both the interaction partner (user) and the virtual human VH can operate autonomously, thus enabling the use of virtual humans VH without limiting the application environment or users, and increasing the frequency and duration of use.

[0123] Beyond introspection and nostalgia limited to text and photographs, users could also create a virtual human (VH) representing their past self and a virtual human (VH) representing their future self predicted from their current self. In other words, lessons can be learned and improvements made for current users in the real world by examining the avatars of past and future users.

[0124] It would be a good idea to allow the virtual human (VH) to suggest "people who are compatible with the user (considering their personality and behavior)"—people that users might not otherwise encounter in real life. By structuring the system in this way, users can quickly build a wide network of contacts as needed. User information may be created using a highly accurate behavioral history recording system. By configuring it in this way, a certain level of reliability can be ensured not only for virtual humans (VH) currently being generated, but also for virtual humans (VH) generated in the past.

[0125] Examples of embodiments and modified versions will be described. Dialogue system 1 can be applied to schools, companies, nursing homes, cram schools, in-house training, etc. This section describes an example of its application in schools. For instance, a virtual human (VH) can be used when students seek advice on course selection or career guidance. This configuration allows students to receive highly accurate advice. Let's describe an example of how this could be applied in a company setting. For instance, if a user has multiple meetings that need to be attended at the same time, they can have a virtual human (VH) attend them on their behalf and report on the meeting content.

[0126] This section describes an example of its application to businesses. For instance, by having a virtual human (VH) handle reception duties, accurate guidance can be provided based on pre-registered visitor information. This contributes to reducing the need for human labor. This section describes an example of its application in a retail store. For instance, by applying a virtual human (VH) to the interface of an unmanned checkout system, friendly communication becomes possible.

[0127] This section describes examples of applications in public facilities. For instance, by applying virtual humans (VH) to the counters of city halls, smooth and comfortable communication becomes possible for the elderly, children, and others. This section describes examples of applications in airports, train stations, and other locations. By applying virtual humans (VH) to airports, train stations, and other locations, smooth communication becomes possible without causing resistance to the elderly, children, and others.

[0128] Although embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments, and design modifications and the like are also included within the scope of the present invention. For example, the information processing device 100 and the information processing device 200 may be combined in appropriate ways. Alternatively, for example, a computer program for realizing the functions of each of the above-mentioned devices may be recorded on a computer-readable recording medium, and the computer program recorded on this recording medium may be loaded into a computer system and executed. Note that the term "computer system" here may include hardware such as an operating system and peripheral devices.

[0129] Furthermore, "computer-readable recording media" refers to storage devices such as flexible disks, magneto-optical disks, writable non-volatile memory including ROM and flash memory, portable media such as DVDs, and hard disks built into computer systems. Furthermore, "computer-readable recording media" also includes volatile memory (such as DRAM (Dynamic Random Access Memory)) within computer systems that act as servers or clients when computer programs are transmitted via networks such as the Internet or communication lines such as telephone lines, which retain programs for a certain period of time.

[0130] Furthermore, the above program may be transmitted from a computer system that stores the program in a memory device or the like to another computer system via a transmission medium or by transmission waves within the transmission medium. Here, the "transmission medium" used to transmit the program refers to a medium that has the function of transmitting information, such as a network (communication network) like the Internet or a communication line (communication line) like a telephone line. Furthermore, the above program may be intended to implement only some of the functions described above. Furthermore, the aforementioned functions can be achieved in combination with programs already recorded in the computer system; these may be so-called differential files (differential programs). [Explanation of Symbols]

[0131] 1 Dialogue system, 11 CPU, 12 ROM, 13 RAM, 14 Bus, 15 Bridge, 16 Interface, 17 Imaging device, 18 Microphone, 19 Communication device, 20 Input device, 21 Output device, 50 Communication network, 102, 202 Reception unit, 104, 204 Speech information creation unit, 105, 205 Video information creation unit, 107, 207 Speech information analysis unit, 108, 208 Video information analysis unit, 109, 209 Integration unit, 300 Learning model creation device, 302 Input unit, 304 Reception unit, 306 Processing unit, 307 Learning model, 308 Output unit, 310 Memory unit

Claims

1. A reception unit that receives user information and personality information from users, Based on the user information and personality information of the user received by the reception unit, a speech information creation unit creates information that indicates the user's speech to the conversation partner in the user's voice. A video information creation unit creates video information indicating the user to be displayed to the conversation partner, based on the user information and personality information of the user received by the reception unit. An information processing device equipped with the following features.

2. The reception unit receives the speech information of the person with the conversation, The aforementioned information processing device is Speech Information Analysis Unit that analyzes the speech information of the conversation partner received by the reception unit. Furthermore, The information processing apparatus according to claim 1, wherein the speech information creation unit creates information indicating the user's voice response to the conversation partner, based on the analysis information of the conversation partner's speech information analyzed by the speech information analysis unit.

3. The reception unit receives the speech information of the person with the conversation, The aforementioned information processing device is Speech Information Analysis Unit that analyzes the speech information of the conversation partner received by the reception unit. Furthermore, The information processing apparatus according to claim 1, wherein the video information creation unit creates video information indicating the user to be displayed to the conversation partner, based on the analysis information of the conversation partner's speech information analyzed by the speech information analysis unit.

4. The reception unit receives video information of the person being spoken to. The aforementioned information processing device is Video Information Analysis Unit that analyzes the video information of the person being spoken to, received by the Reception Unit. Furthermore, The information processing apparatus according to claim 1, wherein the speech information creation unit creates information indicating the user's voice response to the conversation partner, based on the analysis information of the video information of the conversation partner analyzed by the video information analysis unit.

5. The reception unit receives video information of the person being spoken to. The aforementioned information processing device is Video Information Analysis Unit that analyzes the video information of the person being spoken to, received by the Reception Unit. Furthermore, The information processing apparatus according to claim 1, wherein the video information creation unit creates video information indicating the user to be displayed to the conversation partner, based on the analysis information of the video information of the conversation partner analyzed by the video information analysis unit.

6. The reception unit receives the user's voice information, The information processing apparatus according to claim 1, wherein the speech information creation unit creates information indicating the utterance to the dialogue partner in the user's voice, based on the voice information of the user received by the reception unit.

7. The reception unit receives the user's video information, The information processing apparatus according to claim 1, wherein the video information creation unit creates video information indicating the user to be displayed to the conversation partner, based on the video information of the user received by the reception unit.

8. The information processing apparatus according to claim 7, wherein the video information of the user includes at least one of the following: changes in the user's facial expression, blinking of the eyes, movement of the mouth, and changes in the orientation of the face.

9. We accept user information and personality information from users. Based on the user information and personality information of the user that has been received, information is created that represents the utterance to the conversation partner in the user's voice. A computer-based information processing method for creating video information representing the user to be displayed to the conversation partner, based on the user information and personality information of the user that has been received.

10. On the computer, The system accepts user information and personality information from the user. Based on the user information and personality information of the user that has been received, information is created that represents the utterance to the conversation partner in the user's voice. A program that creates video information representing the user to be displayed to the conversation partner, based on the user information and personality information of the user that has been received.

Images