Information processing device, information processing method, and information processing program
The information processing system addresses the burden of initial interactions by generating virtual dialogue and video interactions between Digital Twins, allowing users to simulate real interactions and gain insight into potential partners, thereby reducing anxiety and time consumption.
Patent Information
- Application Number
- PCT/JP2023/019407
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-24
- Publication Date
- 2024-11-28
AI Technical Summary
Users face a significant psychological or time-consuming burden when meeting someone for the first time, as they cannot gauge the other person's atmosphere, impressions, or reactions until they communicate, which is a limitation in matchmaking and similar services.
An information processing system that generates dialogue data and videos between Digital Twins simulating users, allowing users to interact virtually and reduce the burden by providing visual and utterance information based on user profiles and behavior, thereby simulating real interactions.
This system reduces the psychological and time burden on users by allowing them to experience interactions with potential partners virtually, providing insight into the partner's appearance, personality, and atmosphere before actual communication, thus making initial meetings less daunting.
Smart Images

Figure JP2023019407_28112024_PF_FP_ABST
Abstract
Description
Information processing device, information processing method, and information processing program
[0001] The present invention relates to an information processing device, an information processing method, and an information processing program.
[0002] Conventionally, there are matching services in the fields of matchmaking, sales, recruitment, etc. that present potential partners who are compatible with each other based on information such as a user's profile, attributes, and behavioral history (see Non-Patent Documents 1 and 2).
[0003] Best Dating Sites for Real Relationships in 2023, [online], [accessed March 9, 2023], Internet <https: / / chicagoreader.com / reader-partners / best-dating-sites-for-real-relationships / > Dating Apps Thrive in China, but Not Just for Romance, [online], [accessed March 9, 2023], Internet <https: / / www.nytimes.com / 2022 / 09 / 27 / technology / dating-apps-china.html>
[0004] However, the atmosphere (e.g., behavior, speaking reactions, etc., which cannot be seen in photographs) and impressions of the people presented in a matching service cannot be understood unless the user actually meets and communicates with the people. Therefore, there is a problem in that communicating for the first time with people presented in a matching service places a heavy psychological (or time) burden on the user. Therefore, an object of the present invention is to solve the above-mentioned problem and reduce the psychological (or time) burden of communicating with people on a first meeting.
[0005] In order to solve the above-mentioned problems, the present invention is characterized by comprising a dialogue generation unit that generates dialogue data between a DT (Digital Twin) simulating a user and a dialogue partner user or a DT simulating the dialogue partner user, an image generation unit that generates an image including visual information and speech information during a dialogue between the DT of the user and the dialogue partner user or the DT of the dialogue partner user based on at least one of the dialogue data, avatar information of the user conducting the dialogue, sensing information, and information indicating individuality, and an output processing unit that outputs the dialogue information including the generated image.
[0006] According to the present invention, it is possible to reduce the psychological (or time) burden on a user when communicating with someone they meet for the first time.
[0007] FIG. 1 is a diagram illustrating an overview of an information processing system according to a first embodiment. FIG. 2 is a diagram illustrating an example of a configuration of the information processing system according to the first embodiment. FIG. 3 is a diagram illustrating an example of a configuration in which the information processing system according to the first embodiment is implemented by a server and a user terminal. FIG. 4 is a diagram illustrating an example of a configuration in which the information processing system according to the first embodiment is implemented by a server and a user terminal. FIG. 5 is a diagram illustrating an example of a configuration in which the information processing system according to the first embodiment is implemented by a server and a user terminal. FIG. 6 is a flowchart illustrating an example of a processing procedure executed by the information processing system according to the first embodiment. FIG. 7 is a diagram illustrating an overview of an information processing system according to a second embodiment. FIG. 8 is a diagram illustrating an example of a configuration in which the information processing system according to FIG. 2 is implemented by a server and a user terminal. FIG. 9 is a diagram illustrating an information processing system according to the second embodiment. FIG. 10 is a diagram illustrating an information processing system according to the second embodiment. FIG. 11 is a diagram illustrating an information processing system according to the second embodiment. FIG. 12 is a flowchart illustrating an example of a processing procedure executed by the information processing system according to the second embodiment. FIG. 13 is a diagram illustrating a computer that executes an information processing program.
[0008] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a first embodiment and a second embodiment of the present invention will be described with reference to the accompanying drawings. The present invention is not limited to the respective embodiments.
[0009] [First embodiment] [Overview] An overview of an information processing system (information processing device) of the first embodiment will be described using Figure 1. The information processing system of the first embodiment generates and stores a dialogue video between a DT (Digital Twin) simulating a user and a DT simulating another user. Thereafter, in response to a request from the user, the information processing system outputs a dialogue video between the user's DT and the DT of another user.
[0010] For example, the information processing system accepts input of a questionnaire about the user's profile (name, hometown, hobbies, job, etc.) and personality / characteristics. The information processing system also accepts input of audio utterances of episodes related to each item of the user's profile and images of the user when speaking (user sensing information).
[0011] The information processing system then generates a DT for the user based on the input user information. The information processing system also generates DTs for other users in the same way. The information processing system then simulates a conversation between the user's DT and the DTs of multiple other users and stores the video. After a predetermined time has passed since the user entered their profile, etc., the user watches the video of the conversation between their own DT and the DTs of the other users.
[0012] This allows the user to obtain information equivalent to that obtained when communicating with a conversation partner (for example, the conversation partner's appearance, inner thoughts, profile, atmosphere, etc.) without actually communicating with the conversation partner, thereby reducing the psychological (or temporal) burden on the user when communicating with someone they meet for the first time.
[0013] [System Configuration Example] An example configuration of an information processing system will be described using Fig. 2. The information processing system includes, for example, a first user interaction unit (acquisition unit) 11, a personality extraction unit 12, a human DTDB 13, a dialogue partner search unit 14, a dialogue simulation unit (dialogue generation unit) 15, a multimodal (MM) visualization unit (image generation unit) 16, a dialogue information processing unit 17, a dialogue information DB 18, and a second user interaction unit (output processing unit) 19.
[0014] The first user interaction unit 11, personality extraction unit 12, dialogue partner search unit 14, dialogue simulation unit 15, MM visualization unit 16, dialogue information processing unit 17, and second user interaction unit 19 are realized, for example, by a CPU executing programs stored in a storage device. The human DTDB 13 and dialogue information DB 18 are realized by a computer storage device.
[0015] [First User Interaction Unit] The first user interaction unit 11 acquires information about the user through interaction with the user, for example, using audio and video prompting the user to input information about the user.
[0016] Information about the user includes, for example, information entered by the user (e.g., the user's profile or survey results), information sensed from the user (e.g., video and audio of the user when the user interacts with the information processing system (system)), etc.
[0017] The first user interaction unit 11 outputs the acquired information about the user (for example, information input by the user, information obtained by sensing the user) to the individuality extraction unit 12 .
[0018] The first user interaction unit 11 prompts the user to input user profile information such as name, gender, hometown, hobbies, job, etc. At this time, the first user interaction unit 11 may prompt the user to input search conditions for a dialogue partner and conditions (dialogue simulation conditions) for the dialogue simulation unit 15 to simulate a dialogue between the user and the dialogue partner.
[0019] The search conditions for a dialogue partner are, for example, an extroverted person, a person with similar hobbies, a person in the same occupation, etc. For example, the first user interaction unit 11 accepts an input such as "a man / woman with similar hobbies" as the search condition for a dialogue partner.
[0020] The conditions for the dialogue simulation are, for example, the number of dialogue turns in the dialogue simulation, criteria for selecting utterances that constitute the dialogue, etc. The criteria for selecting utterances that constitute the dialogue are, for example, criteria for selecting utterances that increase a value indicating the human relationship between the speakers who are engaged in the dialogue (for example, a value indicating the rapport between the two speakers, Rubin's Love Scale, etc.).
[0021] Furthermore, the first user interaction unit 11 prompts the user to answer a questionnaire regarding personality and characteristics. For example, the questionnaire described in the following document 1 is used.
[0022] Reference 1: https: / / www.jstage.jst.go.jp / article / personality / 21 / 1 / 21_40 / _pdf / -char / ja
[0023] The first user interaction unit 11 also causes the user to speak an episode based on the user's profile. The first user interaction unit 11 then outputs the user's profile, the questionnaire results, and video and audio of the user speaking the episode based on the user's profile to the individuality extraction unit 12, outputs the video and audio to the avatar generation unit 20 and the MM visualization unit 16, outputs search conditions for a dialogue partner and information about the user himself / herself to the dialogue partner search unit 14, and outputs conditions for dialogue simulation to the dialogue simulation unit 15.
[0024] [Personality Extraction Unit] The personality extraction unit 12 extracts (estimates) information indicating the personality of the user from the information about the user input by the user and information obtained by sensing the user.
[0025] For example, the individuality extraction unit 12 estimates personality traits such as Big-five from the questionnaire results entered by the user and output from the first user interaction unit 11. The individuality extraction unit 12 also converts the user's speech into text using voice recognition, matches keywords with the user's profile, and generates an episode talk.
[0026] Information indicating the individuality of a user is, for example, Big-five characteristics, a profile, information about the user's memories (for example, the user's origin, hobbies, work, and episodes related to those), and the like.
[0027] The individual characteristic extraction unit 12 outputs information indicating the estimated individual characteristics of the user to the dialogue simulation unit 15. The individual characteristic extraction unit 12 also outputs information indicating the estimated individual characteristics of the user to the human DTDB 13.
[0028] For example, the individuality extraction unit 12 estimates the Big-five characteristics of a user from the results of a questionnaire entered by the user. The information described in the following literature 2 is used to estimate the Big-five characteristics of the user.
[0029] Reference 2: https: / / www.jstage.jst.go.jp / article / personality / 21 / 1 / 21_40 / _pdf / -char / ja
[0030] In addition, the individuality extraction unit 12 outputs to the dialogue simulation unit 15 information (episode talk data) that links the text of the utterances made when the user interacts with the system with the profile that the user input to the system.
[0031] [Human DTDB] The human DTDB 13 stores information on the human DT of each user. The human DT information is, for example, information that associates the user's avatar (output of the avatar generation unit 20), information indicating the user's individuality (output of the individuality extraction unit 12), a model that reproduces the user's motion (output of the MM visualization unit 16), and a voice synthesis model that reproduces the user's voice (output of the MM visualization unit 16). Note that the model that reproduces the user's motion and the voice synthesis model that reproduces the user's voice may be models that reproduce the user's motion and voice.
[0032] The human DT information includes, for example, each user's 3D avatar, user individuality parameters such as Big-five, user memory information (episode talk), a generation model of the user's motion, a voice synthesis model of the user's voice, etc. The human DT-DB 13 stores the human DT information for each user in, for example, an RDB (Relational Database) or KVS (Key-Value Store). Note that for each model and avatar, the path to their storage destination is stored, for example.
[0033] [Dialogue Partner Search Unit] The dialogue partner search unit 14 acquires information on the human DT of the user's dialogue partner from the human DTDB 13. For example, the dialogue partner search unit 14 acquires information on the user's human DT stored in the human DTDB 13 in the order in which they are stored.
[0034] Furthermore, if the amount of human DT information for each user in the human DTDB 13 is large, the dialogue partner search unit 14 may search for and acquire human DT information for other users (dialogue partners) that matches information indicating the user's individuality (e.g., Big-five characteristics, episode talk, profile, etc.) acquired from the first user interaction unit 11.
[0035] For example, based on the user profile obtained from the first user interaction unit 11, it is possible to search for information on human DTs of users who share a profile with the user, or to search for information on human DTs of users who are from a similar hometown or who live in a similar area.
[0036] Furthermore, the dialogue partner search unit 14 may use past search history when performing the above search. Furthermore, if the dialogue partner search unit 14 has received search conditions input from the user, the dialogue partner search unit 14 may perform the search using the search conditions.
[0037] The dialogue partner search unit 14 outputs information indicating the individuality of the dialogue partner contained in the human DT information acquired from the human DTDB 13 to the dialogue simulation unit 15. In addition, the dialogue partner search unit 14 outputs to the MM visualization unit 16 the dialogue partner's avatar contained in the human DT information acquired from the human DTDB 13, and a model that reproduces the dialogue partner's motion and voice.
[0038] [Dialogue Simulator] The dialogue simulator 15 simulates a dialogue between users. For example, the dialogue simulator 15 simulates a dialogue between users by inputting information indicating the personalities of the user and the dialogue partner and utterances of the user or the dialogue partner into a natural language generation model.
[0039] For example, the dialogue simulation unit 15 generates a simulation result (dialogue data) of a dialogue between a user and a dialogue partner by inputting information indicating the user's individuality output from the individuality extraction unit 12, information indicating the dialogue partner's individuality output from the dialogue partner search unit 14, and dialogue simulation conditions obtained from the first user interaction unit 11.
[0040] The simulation result of the dialogue between the user and the dialogue partner includes, for example, the content of utterances made by the user / dialogue partner for each turn and non-verbal information (e.g., emotions) in each utterance. The first utterance in the dialogue is statically determined by the dialogue simulation unit 15.
[0041] An example of the process executed by the dialogue simulator 15 will be described below.
[0042] (Process 1) The dialogue simulation unit 15 constructs a natural language processing model A that can respond to an utterance based on the input information indicating the personalities of two speakers (e.g., profiles and personality traits) and an utterance of the dialogue partner by having the natural language processing model learn a combination of the information indicating the personalities of the two speakers (e.g., profiles and personality traits) and a dialogue log of the two speakers in advance. The natural language processing model A uses, for example, the model described in the following document 3-1.
[0043] Furthermore, the dialogue simulation unit 15 calculates a value indicating the interpersonal relationship between the user and the dialogue partner using a natural language processing model B that calculates a value indicating the interpersonal relationship between the speakers (for example, a value indicating the rapport between the two speakers, a value indicating Rubin's love scale, etc.) from information indicating the personalities of the two speakers in the dialogue and the dialogue history of the two speakers.
[0044] For example, the information processing system trains natural language processing model B in advance so that it outputs a value indicating the relationship between two speakers (for example, a value indicating the rapport between two speakers or Rubin's Love Scale) from a combination of information indicating the personalities of two speakers and the dialogue logs of the two speakers.The dialogue simulation unit 15 then uses the natural language processing model B to calculate a value indicating the relationship between the speakers at the time of the latest utterance in the dialogue log.The natural language processing model B uses, for example, the model described in the following literature 3-2.
[0045] Reference 3-1: https: / / ai.googleblog.com / 2020 / 02 / exploring-transfer-learning-with-t5.html Reference 3-2: https: / / huggingface.co / docs / transformers / model_doc / longformer
[0046] (Process 2) The dialogue simulation unit 15 inputs information indicating the personalities of the user / dialogue partner and a question on a topic common to the two people whose dialogue is being simulated into the natural language processing model A constructed above. Note that the question on a topic common to the two people whose dialogue is being simulated is determined, for example, using the profiles of each of the two people.
[0047] (Process 3) Thereafter, the dialogue simulator 15 estimates whether the output from the natural language processing model A is a question or not. For the estimation, for example, the technique described in the following document 4 is used.
[0048] Reference 4: https: / / cloud.google.com / dialogflow / es / docs / intents-overview?hl=ja, https: / / aisuite.jp / #technologies
[0049] The dialogue simulator 15 executes the following (Process 3-1) according to the estimation result, and then executes the following (Process 3-2).
[0050] (Process 3-1) When the output from the natural language processing model A is a question: The dialogue simulation unit 15 selects data that matches a keyword included in the question (for example, "hobbies" in the case of the question "What are your hobbies?") from the user's episode talk data acquired by the individuality extraction unit 12. Then, the dialogue simulation unit 15 sets the selected data as a reply candidate (a candidate for utterance).
[0051] (Process 3-2) The dialogue simulation unit 15 inputs information indicating the individuality of the user or dialogue partner and the output from the natural language processing model A described above into the natural language processing model A, and obtains k candidate replies (candidate utterances) to the input utterance.
[0052] (Process 3-3) The dialogue simulation unit 15 adds the utterance candidates obtained in Process 3-1 and Process 3-2 to the dialogue history between the user and the dialogue partner up to that point. These are called dialogue simulation candidates. If there are k utterance candidates, the dialogue simulation unit 15 creates k dialogue simulation candidates. Then, the dialogue simulation unit 15 inputs the created dialogue simulation candidates and information indicating the individuality of the user or the dialogue partner into the natural language processing model B, and obtains, for each dialogue simulation candidate, a value indicating the human relationships between the speakers in the dialogue of the dialogue simulation candidate.
[0053] (Process 3-4) Based on the human relationship values of each dialogue simulation candidate obtained above, the dialogue simulation unit 15 selects a dialogue simulation candidate that meets the dialogue simulation conditions obtained from the first user interaction unit 11. For example, if the condition of the dialogue simulation is an increase in the human relationship value between speakers in a dialogue, the dialogue simulation unit 15 selects the dialogue simulation candidate with the highest human relationship value from among the dialogue simulation candidates so as to meet the above condition. Then, the dialogue simulation unit 15 outputs the latest utterance of the selected dialogue simulation candidate.
[0054] The above-mentioned processes 3-3 and 3-4 may be omitted, and the dialogue simulator 15 may select and output any one utterance candidate from the plurality of utterance candidates obtained in process 3-2.
[0055] (Process 4) The dialogue simulator 15 performs the same process as Process 3 on the output (utterance) obtained in Process 3.
[0056] (Process 5) The dialogue simulation unit 15 repeats processes 3 and 4 N times in accordance with the conditions for dialogue simulation acquired from the first user interaction unit 11 .
[0057] (Process 6) The dialogue simulator 15 also performs emotion estimation for each of the above utterances. For emotion estimation, for example, the technique described in the following document 5 is used.
[0058] Reference 5: http: / / arakilab.media.eng.hokudai.ac.jp / ~ptaszynski / repository / mlask.htm
[0059] [MM Visualization Unit] The MM visualization unit 16 generates a video of the user's DT and the conversation partner's DT interacting while moving.
[0060] The MM visualization unit 16 generates a video including the user's DT motion and voice in the above dialogue based on, for example, the simulation results of the dialogue between the user and the dialogue partner (the spoken text and non-verbal information of each utterance in the dialogue), the user's avatar information, and sensing information.
[0061] For example, first, the MM visualization unit 16 creates a model that reproduces the user's motion and voice based on the sensed information about the user (e.g., the user's video and voice) acquired from the first user interaction unit 11. After that, the MM visualization unit 16 inputs each utterance of the user in the simulation results of the above dialogue into the model, thereby generating the user's voice and motion based on that voice.
[0062] In addition, the MM visualization unit 16 generates the voice of the dialogue partner and motion based on that voice by inputting the utterance of the dialogue partner in the simulation results of the above dialogue into the dialogue partner model obtained from the dialogue partner search unit 14.
[0063] The MM visualization unit 16 then generates a video of the user's human DT and the dialogue partner's human DT speaking while moving, and stores the generated video and audio (dialogue information) in the dialogue information DB 18. Note that this dialogue information does not necessarily need to include audio; instead of audio, it may be video with spoken text superimposed in the form of subtitles or speech bubbles. The MM visualization unit 16 also stores the spoken text and non-verbal information that are the source of the video in the dialogue information DB 18. Note that the MM visualization unit 16 may output the generated video (and the spoken text and non-verbal information that are the source of the video) to the second user interaction unit 19.
[0064] The MM visualization unit 16 also outputs the generated model that reproduces the user's motion and voice to the human DTDB 13 .
[0065] An example of the processing executed by the MM visualization unit 16 will be described below.
[0066] (Process 1) For example, the MM visualization unit 16 creates a model that reproduces the user's voice from the user's voice included in the user's sensing information acquired from the individuality extraction unit 12. The MM visualization unit 16 then reproduces the user's voice using the created model. For example, the technology described in the following literature 6 is used to create the model.
[0067] Reference 6: https: / / coestation.jp / consumer / coestationapp / , https: / / www.futurevoice.jp /
[0068] (Process 2) For example, the MM visualization unit 16 creates a model that reproduces the user's motion based on the user's voice, using the user's video and voice included in the user's sensing information acquired from the individuality extraction unit 12. The MM visualization unit 16 then reproduces the user's motion using the created model. For example, the technology described in the following literature 7 is used to create the model.
[0069] Document 7: https: / / openaccess.thecvf.com / content / CVPR2022 / papers / Ahuja_Low-Resource_Adaptation_for_Personalized_Co-Speech_Gesture_Generation_CVPR_2022_paper.pdf
[0070] (Process 3) Then, the MM visualization unit 16 imports the user avatar obtained from the avatar generation unit 20 and the dialogue partner avatar obtained from the dialogue partner search unit 14 into the 3D space constructed by Unity or Unreal Engine.
[0071] (Process 4) The MM visualization unit 16 assigns the motions and sounds reproduced in processes 1 and 2 to an avatar on Unity or Unreal Engine.
[0072] (Process 5) The MM visualization unit 16 may output a video of the dialogue between the user's DT and the dialogue partner's DT that has been reproduced above, or may use it as a 3D space like a metaverse.
[0073] [Dialogue Information Processing Unit] The dialogue information processing unit 17 processes the dialogue information acquired from the dialogue information DB 18. For example, the dialogue information processing unit 17 superimposes non-verbal information of each utterance on the video included in the dialogue information in the dialogue information DB 18, processing the video so that the dialogue taking place in the video is more visually and intuitively understandable. Furthermore, if the video is long, the dialogue information processing unit 17 may generate a short, summarized video.
[0074] [Dialogue Information DB] The dialogue information DB 18 stores dialogue information between human DTs (video, audio, spoken text, non-verbal information such as emotions accompanying each utterance).
[0075] [Second user interaction unit] The second user interaction unit 19 outputs information selected by the user from the dialogue information stored in the dialogue information DB 18. At this time, the user may select the type of dialogue information to output (video, audio, spoken text, non-verbal information such as emotions accompanying each utterance), or may select dialogue information based on whose DT's dialogue information they wish to view. Note that the second user interaction unit 19 may have an access control function that, when outputting dialogue information, allows only dialogue information that includes the user's DT to be viewed.
[0076] For example, after creating the user's DT or after a predetermined time has elapsed since the last video viewing, the user accesses the dialogue information DB 18 via the second user interaction unit 19 and views dialogue videos with the DT that the user has not yet viewed, among the dialogue videos that include the user's own DT. At this time, the user may select either edited video or unedited video from the videos stored in the dialogue information DB 18.
[0077] The second user interaction unit 19 may also sequentially output the images generated by the MM visualization unit 16 .
[0078] [Avatar Generation Unit] The avatar generation unit 20 generates an avatar (avatar image) of a user based on a video of the user. For example, the avatar generation unit 20 divides the video of the user included in the user sensing information output from the individuality extraction unit 12 into frames to generate images. For example, the technology described in the following literature 8 is used to generate the avatar.
[0079] Reference 8: https: / / portal.avapf.com / home
[0080] [Configuration Example of Information Processing System] The information processing system described above may be configured with a server and a user terminal. In this case, for example, as shown in Fig. 3, the user terminal may be equipped with a first user interaction unit 11 and a second user interaction unit 19, and the server may be equipped with a personality extraction unit 12, a human DTDB 13, a dialogue partner search unit 14, a dialogue simulation unit 15, an MM visualization unit 16, a dialogue information processing unit 17, a dialogue information DB 18, and an avatar generation unit 20.
[0081] Also, for example, as shown in Figure 4, the user terminal may be equipped with a first user interaction unit 11, a personality extraction unit 12, a dialogue information processing unit 17, and a second user interaction unit 19, and the server may be equipped with a human DTDB 13, a dialogue partner search unit 14, a dialogue simulation unit 15, an MM visualization unit 16, a dialogue information DB 18, and an avatar generation unit 20.
[0082] Also, for example, as shown in FIG. 5, the user terminal may be equipped with a first user interaction unit 11, a dialogue information processing unit 17, and a second user interaction unit 19, and the server may be equipped with a personality extraction unit 12, a human DTDB 13, a dialogue partner search unit 14, a dialogue simulation unit 15, an MM visualization unit 16, a dialogue information DB 18, and an avatar generation unit 20.
[0083] [Example of Processing Procedure] An example of processing procedure executed by the information processing system will be described below with reference to Fig. 6. First, the first user interaction unit 11 acquires information about the user input by the user, information obtained by sensing the user, search conditions for a dialogue partner, conditions for dialogue simulation, etc. (S1: acquire user information).
[0084] After S1, the individuality extraction unit 12 estimates information indicating the individuality of the user from the information input by the user and the information obtained by sensing the user, which are acquired in S1 (S2).
[0085] After S2, the avatar generation unit 20 divides the image of the user included in the information obtained by sensing the user into frames and generates an image of an avatar (S3: Generate avatar).
[0086] Next, the dialogue partner search unit 14 acquires information on the human DT of the user's dialogue partner (information indicating the individuality of the dialogue partner, an avatar, and a model that reproduces motion and voice) from the human DTDB 13 (S4).
[0087] After S4, the dialogue simulation unit 15 simulates a dialogue between the user and the dialogue partner by inputting the information indicating the user's individuality estimated in S2 and the information indicating the dialogue partner's individuality included in the dialogue partner's DT information acquired in S4 into a natural language generation model (S5).
[0088] After S5, the MM visualization unit 16 generates a video of the user's DT and the conversation partner's DT talking while moving (S6: generate a video of human DTs talking to each other).
[0089] For example, the MM visualization unit 16 generates a video including the motion and voice of the user's DT in the above dialogue based on the simulation results of the dialogue between the user and the dialogue partner obtained in S5, the user's avatar information generated in S3, the user's sensing information obtained in S1, and information indicating the user's individuality estimated in S2. Also, the MM visualization unit 16 generates a video including the motion and voice of the dialogue partner's DT in the above dialogue based on the simulation results of the dialogue between the user and the dialogue partner obtained in S5 and the information of the dialogue partner's human DT acquired in S4.
[0090] After S6, the MM visualization unit 16 stores the video generated in S6 in the dialogue information DB 18 (S7). After S7, the dialogue information processing unit 17 processes the video in the dialogue information DB 18 and stores it in the dialogue information DB 18 (S8: process video).
[0091] Thereafter, the second user interaction unit 19 outputs the video stored in the dialogue information DB 18 based on the user's instruction (S9).
[0092] By performing the above processing, the information processing system can generate and output a video of a conversation between the user's DT and the conversation partner's DT.
[0093] [Second embodiment] [Overview] An overview of the information processing system of the second embodiment will be explained using Figure 7. The same components as those in the first embodiment are assigned the same reference numerals and explanations will be omitted. The information processing system of the second embodiment generates and stores a video of a dialogue between a user's DT and another user (a living person). Thereafter, the information processing system outputs the video of the dialogue between the user's DT and another user in response to a request from the user.
[0094] For example, the information processing system of the second embodiment accepts input of a questionnaire regarding a user's profile (such as name, hometown, hobbies, and job) and personality and characteristics, as in the first embodiment. The information processing system also accepts input of voice utterances of episodes related to each item of the user's profile and images of the user when uttering (user sensing information).
[0095] The information processing system generates a user's DT based on the input user information. The information processing system also operates the user's (first user's) DT, and creates and stores images of interactions with various other users (second users).
[0096] For example, the information processing system allows various users (second users) to interact with the DT of the first user and stores the video of the interaction. After a predetermined time has passed, the first user then watches the video of the interaction between his / her own DT and another user (a second user selected by the first user).
[0097] Such an information processing system also enables users to obtain information equivalent to that obtained when communicating with a conversation partner, without communicating with each other, thereby reducing the psychological (or time) burden on users when communicating with someone they meet for the first time.
[0098] [Example of configuration of information processing system] When the information processing system of the second embodiment is configured by a server and a user terminal, for example, as shown in Figure 8, the user terminal may be equipped with a first user interaction unit 11 and a second user interaction unit 19, and the server may be equipped with a human DTDB 13, a dialogue partner search unit 14, a dialogue simulation unit 15, and an MM visualization unit 16.
[0099] An example of processing in which the information processing system stores video of a second user interacting with various users' DTs (including the first user's DT) will be described below.
[0100] First, an example of a process in which the information processing system outputs a video of a DT who is a conversation partner of the second user will be described with reference to FIG.
[0101] [First user interaction unit] The first user interaction unit 11 accepts input of search conditions for a dialogue partner (DT of dialogue partner) from a user (second user) and outputs the input dialogue partner search conditions to the dialogue partner search unit 14.
[0102] [Human DTDB, Dialogue Partner Search Unit] The dialogue partner search unit 14 uses the above search conditions to search for information on the human DT of the dialogue partner from the human DTDB 13. Then, the dialogue partner search unit 14 outputs information indicating the individuality of the dialogue partner (for example, the dialogue partner's profile and personality traits) contained in the dialogue partner's human DT information obtained from the human DTDB 13 to the dialogue simulation unit 15. In addition, the dialogue partner search unit 14 outputs to the MM visualization unit 16 the dialogue partner's avatar and a model that reproduces the dialogue partner's motion and voice contained in the dialogue partner's human DT information.
[0103] [MM Visualization Unit] The MM visualization unit 16 generates an avatar that reproduces the motion and voice of the dialogue partner using the dialogue partner avatar output from the dialogue partner search unit 14 and a model that reproduces the dialogue partner's motion and voice. The MM visualization unit 16 then generates a video showing the dialogue partner's avatar (dialogue partner DT) and outputs it to the second user interaction unit 19. The second user interaction unit 19 outputs the video of the dialogue partner DT. This allows the second user to view the video of the dialogue partner DT. The second user then begins a dialogue with the dialogue partner DT.
[0104] Next, an example of a processing procedure in which the information processing system stores a video of a conversation between a second user and a conversation partner DT will be described with reference to FIG.
[0105] [First user interaction unit, second user interaction unit, individuality extraction unit, dialogue simulation unit, MM visualization unit] When a second user starts a dialogue with the DT video of the dialogue partner output by the second user interaction unit 19, the first user interaction unit 11 captures an image of the second user during the dialogue. Then, the first user interaction unit 11 outputs a video including the voice of the second user during the dialogue to the individuality extraction unit 12.
[0106] Thereafter, the individuality extraction unit 12 outputs information obtained by converting the voice of the second user into text (text of the user's utterance) to the dialogue simulation unit 15. The individuality extraction unit 12 also outputs a video including the voice of the second user during the dialogue to the MM visualization unit 16.
[0107] Thereafter, the dialogue simulation unit 15 inputs the information indicating the personality of the dialogue partner acquired from the dialogue partner search unit 14 and the text of the second user's utterance into a natural language processing model to simulate the dialogue partner's DT response to the second user's utterance (response to the user utterance) and non-verbal information of that response (emotions, etc.). The dialogue simulation unit 15 then outputs the results of the simulation to the MM visualization unit 16.
[0108] As in the first embodiment, the MM visualization unit 16 generates video including the motion and voice of the dialogue partner DT (voice video of the dialogue partner DT) using the simulation results of the dialogue between the second user and the dialogue partner DT (speech text and non-verbal information of each utterance), avatar information of the dialogue partner, and a model that reproduces the dialogue partner's motion and voice. The MM visualization unit 16 then presents the generated video to the second user via the second user interaction unit 19.
[0109] Furthermore, the MM visualization unit 16 combines the video of the second user received from the individuality extraction unit 12 with the video of the dialogue partner DT (human DT), and stores this video as a dialogue video between these two (user and human DT) in the dialogue information DB 18. When storing the dialogue video, the MM visualization unit 16 may also store the utterance text of the human DT and non-verbal information of each utterance.
[0110] 11, the dialogue information processing unit 17 processes the dialogue video between the second user and the dialogue partner human DT in the dialogue information DB 18, and stores it in the dialogue information DB 18. Thereafter, the second user interaction unit 19 outputs the dialogue video between the user (second user) and the human DT.
[0111] For example, when the second user interaction unit 19 receives an instruction input to view a dialogue video from the user (first user) who is the source of the human DT, it acquires and outputs a dialogue video between the first user and the desired dialogue partner from the dialogue information DB 18. Note that the second user interaction unit 19 may select and output only unviewed dialogue videos from among the dialogue videos between the first user's DT and other users.
[0112] In addition, the second user interaction unit 19 may obtain and output processed dialogue video from the dialogue information DB 18, or obtain and output unprocessed dialogue video, based on instruction input from the first user.
[0113] [Example of Processing Procedure] An example of processing procedure executed by the information processing system of the second embodiment will be described below with reference to Fig. 12. First, the first user interaction unit 11 acquires search conditions for a dialogue partner input by the user (S11). Thereafter, the dialogue partner search unit 14 acquires information on the human DT of the dialogue partner from the human DTDB 13 using the search conditions acquired in S11 (S12).
[0114] After S12, the MM visualization unit 16 uses the information about the human DT of the dialogue partner acquired in S12 to generate an avatar that reproduces the motion and voice of the dialogue partner (S13).The MM visualization unit 16 then outputs a video showing the appearance of the dialogue partner avatar (DT) to the second user interaction unit 19 (S14).Then, the user (second user) starts a dialogue with the dialogue partner DT.
[0115] After S14, the first user interaction unit 11 acquires an image of the user (second user) during the dialogue with the dialogue partner DT (S15). After that, the dialogue simulation unit 15 simulates the response of the dialogue partner DT using information indicating the individuality of the dialogue partner (information indicating the individuality of the dialogue partner included in the information of the dialogue partner human DT acquired in S12) and the text of the user's utterance (the recognition result of the voice included in the video of the user acquired in S15) (S16).
[0116] After S16, the MM visualization unit 16 generates a video showing the appearance of the dialogue partner's avatar (DT) using the simulation result of the response of the dialogue partner's DT obtained in S16 and the information of the dialogue partner's human DT acquired in S12, and outputs this to the second user interaction unit 19 (S17). Thereafter, the user (second user) views the video of the dialogue partner's DT responding output by the second user interaction unit 19, and makes an utterance to the dialogue partner's DT.
[0117] As the information processing system repeats the above process, the user (second user) engages in a dialogue with the dialogue partner DT. The second user interaction unit 19 then stores in the dialogue information DB 18 an image of the user during the dialogue combined with an image of the dialogue partner DT (S18).
[0118] Thereafter, the second user interaction unit 19, based on instructions from the user (first user), acquires and outputs a video of the interaction between the user's DT and another user (second user) from the interaction information DB 18 (S19).
[0119] By performing the above processing, the information processing system can output to the user (first user) a video of the conversation between the user's DT and the conversation partner.
[0120] [System Configuration, etc.] The components of each unit shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program executed by the CPU, or can be realized as hardware using wired logic.
[0121] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.
[0122] [Program] The information processing system described above can be implemented by installing a program (information processing program) as package software or online software on a desired computer. For example, by executing the program on an information processing device, the information processing device can function as an information processing system. The information processing device referred to here includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as terminals such as PDAs (Personal Digital Assistants).
[0123] 13 is a diagram showing an example of a computer that executes an information processing program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.
[0124] The memory 1010 includes a read-only memory (ROM) 1011 and a random access memory (RAM) 1012. The ROM 1011 stores a boot program such as a basic input / output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.
[0125] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define the processes executed by the information processing system are implemented as program modules 1093 in which computer-executable code is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, the program modules 1093 for executing processes similar to those of the functional configuration in the information processing system are stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced with an SSD (Solid State Drive).
[0126] Data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. The CPU 1020 then reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary and executes them.
[0127] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may also be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.
[0128] REFERENCE SIGNS LIST 11 First user interaction unit 12 Personality extraction unit 13 Human DTDB 14 Dialogue partner search unit 15 Dialogue simulation unit 16 MM visualization unit 17 Dialogue information processing unit 18 Dialogue information DB 19 Second user interaction unit 20 Avatar generation unit
Claims
1. An information processing device comprising: a dialogue generation unit that generates dialogue data between a DT (Digital Twin) simulating a user and a dialogue partner user or a DT simulating the dialogue partner user; an image generation unit that generates an image including visual information and speech information during a dialogue between the DT of the user and the dialogue partner user or the DT of the dialogue partner user based on at least any of the dialogue data, avatar information of the user conducting the dialogue, sensing information, and information indicating individuality; and an output processing unit that outputs the dialogue information including the generated image.
2. The information processing device according to claim 1, characterized in that the dialogue generation unit estimates a value indicating the interpersonal relationship between the user and the dialogue partner user in the dialogue based on information indicating the individuality of the user and the dialogue partner user and dialogue data indicating a dialogue between the DT of the user and the dialogue partner user or the DT of the dialogue partner user, and generates utterances constituting the dialogue data so that the value indicating the interpersonal relationship satisfies a predetermined condition.
3. The information processing device according to claim 1, further comprising: a dialogue information database for storing the dialogue information; and a dialogue information processing unit for processing the dialogue information stored in the dialogue information database and storing the dialogue information in the dialogue information database.
4. The information processing device according to claim 3, characterized in that the output processing unit outputs information specified by a user from the dialogue information stored in the dialogue information database, or sequentially outputs the image generated by the image generation unit.
5. The information processing device according to claim 1, characterized in that the dialogue data includes video of the dialogue, spoken audio, and text indicating the content of the speech, as well as information indicating the emotions of the user and the dialogue partner user in the dialogue.
6. The information processing device according to claim 1, further comprising an acquisition unit that acquires an image of the other user speaking in response to an image of the user's DT, and the image generation unit generates an image of the interaction between the user's DT and the other user by combining the image of the user's DT and the image of the other user during the interaction.
7. The information processing device according to claim 1, characterized in that the user's sensing information includes visual information and audio information when the user interacts, and the information indicating the user's individuality includes at least one of the user's profile information and information indicating the user's characteristics.
8. The information processing device according to claim 1, characterized in that the image generation unit uses information indicating the individuality of the user to generate a model that reproduces the motion and voice of the user, and uses the generated model and avatar information of the user to generate an image including the motion and speech information of the DT of the user.
9. The information processing device according to claim 1, further comprising: a human DT database that stores avatar information for each user, information indicating individuality, and a model that reproduces the user's motion and voice; and a dialogue partner search unit that searches the human DT database for a model that reproduces the avatar information, information indicating individuality, and motion and voice of the dialogue partner user, and outputs the model; and the image generation unit generates an image including the DT motion and speech information of the dialogue partner user using the model that reproduces the avatar information, information indicating individuality, motion, and speech information of the dialogue partner user output by the dialogue partner search unit.
10. An information processing method comprising the steps of: generating dialogue data between a DT (Digital Twin) simulating a user and a partner user or a DT simulating the partner user; generating an image including visual information and speech information during a dialogue between the DT of the user and the partner user or the DT of the partner user based on at least any of the dialogue data, avatar information of the user conducting the dialogue, sensing information, and information indicating individuality; and outputting the dialogue information including the generated image.
11. An information processing program for causing a computer to execute the steps of: generating dialogue data between a DT (Digital Twin) simulating a user and a partner user or a DT simulating the partner user; generating an image including visual information and speech information during a dialogue between the DT of the user and the partner user or the DT of the partner user based on at least any of the dialogue data, avatar information of the user conducting the dialogue, sensing information, and information indicating individuality; and outputting the dialogue information including the generated image.
Citation Information
Patent Citations
Human relationship estimation system
JP2013182560A
Artificial intelligence system for supporting communication
WO2017217507A1
Dialog processing server, control method for dialog processing server, and terminal
WO2018134960A1
Information processing device, information processing method, and program
WO2021230101A1
Cited By
Large language model streaming response processing method and device, medium, equipment and product
CN121705361A
