Information processing device, information processing method, and information processing program

The information processing device enhances understanding of speakers' feelings in past conversations by converting audio to text and generating synchronized video, enabling users to relive and better interpret past dialogues through interactive text and video presentation.

WO2024228268A1PCT designated stage expired Publication Date: 2024-11-07NT T INC
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2023/017210
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-02
Publication Date
2024-11-07

AI Technical Summary

Technical Problem

Existing technologies face challenges in helping users understand the feelings of speakers in past conversations, as they rely on text transcriptions and video viewing, which can be difficult for users to interpret effectively.

Method used

An information processing device and method that converts audio from past dialogues into text data indicating each speaker's utterances and generates video data of the conversation partner, allowing users to relive the conversation by speaking along with the text, enhancing understanding of feelings through synchronized video and audio presentation.

Benefits of technology

Facilitates easier comprehension of speakers' feelings in past conversations by providing a relived experience through synchronized text and video output, allowing users to engage more effectively with the dialogue content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2023017210_07112024_PF_FP_ABST
    Figure JP2023017210_07112024_PF_FP_ABST
Patent Text Reader

Abstract

An information processing device (10) summarizes conversation data for a conversation that took place in the past, converts audio included in the summarized conversation data to text data that indicates the details of the speech of each speaker, and outputs the text data as a script. The information processing device (10) also generates video data that includes at least video and audio of a speaker that was the conversation partner of a speaker in the conversation who is to be the subject of a vicarious experience by a user (a person having an experience) from video and audio included in the conversation data. Then the information processing device (10) presents the generated data in accordance with the timing of the speech of the user. The information processing device (10) also performs situation determination from the state of the person having an experience who is seeing and hearing the data and performs video processing that indicates a state in which the conversation partner reacts to the speech of the person having an experience.
Need to check novelty before this filing date? Find Prior Art

Description

Information processing device, information processing method, and information processing program

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program that allow a user to relive a conversation that took place in the past.

[0002] A user may view a video of a speaker speaking in order to check the appearance, situation, content, etc., of a past meeting or conversation. Non-Patent Document 1 discloses a technology for automatically transcribing the content of a conversation. A user can check the content of the conversation by reading the text of the transcribed content of the conversation.

[0003] Kamado et al., "HTML5-compatible cloud speech recognition platform," Research Report Spoken Language Processing (SLP), 2015-SLP-108(3), pp.1-6 (2015)

[0004] The above-mentioned technology allows a user to quickly grasp the content of a conversation through text, but has the problem of making it difficult to understand the emotions of the people appearing in the conversation. Furthermore, when a user views a video of a conversation, they tend to skim over it, making it difficult to understand the emotions of the people appearing in the conversation. Therefore, an object of the present invention is to solve the above-mentioned problems and make it easier for a user to understand the emotions of speakers in past conversations.

[0005] In order to solve the above-mentioned problems, the present invention is characterized by comprising a video generation unit that converts audio contained in dialogue data of a past dialogue into text data indicating the content of the utterances of each speaker and outputs the text data, and generates video data from the video and audio contained in the dialogue data that includes video and audio of at least the speaker who is the dialogue partner of the speaker selected by the user as the subject of the re-experience, and a presentation unit that presents the generated video data in accordance with the timing of the user's utterances.

[0006] According to the present invention, it becomes easier for a user to understand the feelings of a speaker in a past conversation.

[0007] FIG. 1 is a diagram for explaining an overview of an information processing device. FIG. 2 is a diagram for explaining an overview of an information processing device. FIG. 3 is a diagram for explaining an overview of processing executed by the information processing device. FIG. 4 is a diagram showing an example of the configuration of an information processing device. FIG. 5 is a diagram showing an example of video generated by the information processing device. FIG. 6 is a flowchart showing an example of processing procedures executed by the information processing device. FIG. 7 is a diagram showing a computer that executes an information processing program.

[0008] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a description will be given of an embodiment of the present invention with reference to the drawings, but the present invention is not limited to the embodiment.

[0009] [Overview] First, an overview of the processing executed by the information processing device of this embodiment will be described using FIG. 1. The information processing device, for example, outputs text (scenario) indicating the content of utterances by each speaker from past dialogue data. The information processing device also outputs video and audio of the dialogue included in the dialogue data. Then, a user relives a past dialogue by uttering the content of the utterance of the speaker that the user wants to relive, as shown in the scenario, while watching and listening to the video and audio of the dialogue. This makes it easier for the user to understand the feelings of the speaker in the past dialogue.

[0010] The dialogue data includes, for example, video and audio data of each speaker in the dialogue, but may also include either video or audio data. When the dialogue data includes video and audio data, the dialogue data includes offset information for time synchronization of the video and audio data.

[0011] The dialogue data may be dialogue data between two speakers or between three or more speakers. For simplicity of explanation, the dialogue data will be described here as an example of dialogue data between two speakers, Persons A and B. The explanation will also be given as an example of a case where the information processing device allows the user to relive the speech of Person A between Persons A and B.

[0012] For example, as shown in Fig. 1, the information processing device outputs text data (script) indicating the content of the dialogue between persons A and B from the voices included in the dialogue data. The script may be a text version of the dialogue content itself, or a text version of a summary of the dialogue content.

[0013] The information processing device then extracts the video and audio of person B from the dialogue data and outputs them to the monitor. The user (the person experiencing the experience) then speaks the speech content of person A (themselves) shown in the script while watching the video and audio of person B displayed on the monitor screen.

[0014] For example, the participant, as shown in Figure 1, utters the script of person A (the participant) "How do you usually spend your holidays?" while watching the video (and audio) of person B. After that, the participant watches the video and hears the audio of person B (the other person) saying "I've been climbing mountains a lot lately. Last week..." In this way, the participant can relive the speech of person A in the dialogue between persons A and B. This makes it easier for the participant to understand the feelings of person A in the dialogue.

[0015] 1, the information processing device has been described as outputting scripts for both persons A and B, but it may also output only the script for the person (person A) whose experiencer is reliving the experience, as shown in Fig. 2. In this case, the information processing device may also display the script on the same screen as the screen on which the video of the interlocutor (for example, person B) is displayed.

[0016] The dialogue data to be processed by the information processing device may be dialogue data between real people, or may be the result of a dialogue simulation using avatars, etc. Furthermore, the people appearing in the video of the dialogue data may be real people, or may be avatars created using CG, etc.

[0017] Next, an example of processing executed by the information processing device 10 will be described with reference to Fig. 3. First, the information processing device 10 summarizes the dialogue data (video data and audio data). Then, the information processing device 10 generates a video (including audio) of the dialogue from the summarized dialogue data.

[0018] Furthermore, the information processing device 10 generates and outputs a script of speech for each speaker in the dialogue from the speech of the summarized dialogue data.

[0019] The information processing device 10 then presents the generated video to the user. As a result, the user speaks the script of the speaker (e.g., person A) selected by the user while watching the video of the dialogue. In this way, the information processing device 10 allows the user to relive the dialogue.

[0020] Furthermore, the information processing device 10 captures the experiencer's behavior (video and audio) as he or she watches the presented video and speaks using a microphone, camera, etc. Then, when the information processing device 10 determines from the experiencer's behavior that the experiencer has been speaking for a predetermined period of time or more, it processes the video of the experiencer's conversation partner (e.g., person B) to show the interaction with the experiencer (reaction generation). For example, the video of the conversation partner may be processed to show the interaction partner nodding in response to the experiencer's speech. Then, the information processing device 10 presents the processed video.

[0021] This allows the information processing device 10 to allow the user to relive the conversation in a more realistic manner.

[0022] [Configuration Example] Next, a configuration example of the system will be described with reference to Fig. 4. The system includes, for example, a personal computer (PC) of a user and an information processing device 10. The PC includes a monitor, a speaker, a microphone, a camera, etc. The PC and the information processing device 10 are connected to each other so as to be able to communicate with each other via a network, for example.

[0023] The information processing device 10 includes, for example, an input / output unit 11, a storage unit 12, and a control unit 13. The input / output unit 11 is an interface that controls the input and output of various data. The input / output unit 11 receives, for example, input of past dialogue data. The input dialogue data is stored, for example, in a dialogue video database (DB) in the storage unit 12.

[0024] In addition, the input / output unit 11 accepts input of the user's voice, video, etc. input via the user's PC, outputs image data generated by the control unit 13 to the monitor of the user's PC, and outputs audio data to the PC's speaker.

[0025] The storage unit 12 stores data, programs, etc. that are referenced when the control unit 13 executes various processes. The storage unit 12 is realized by a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. For example, the storage unit 12 includes an interactive video database, etc.

[0026] The control unit 13 is responsible for overall control of the information processing device 10. The functions of the control unit 13 are realized, for example, by a CPU (Central Processing Unit) executing a program stored in the storage unit 12.

[0027] The control unit 13 includes, for example, a selection input unit 131 , a summarizing unit 132 , a video generating unit 133 , a situation determining unit 134 , a reaction generating unit 135 , and a presenting unit 136 .

[0028] The selection input unit 131 receives from the user a selection input of a speaker in the dialogue data that the user desires to relive. The selection input unit 131 also receives a selection input of a pattern of an image to be generated by the image generation unit 133.

[0029] The image patterns (image formats) that the user can select are, for example, the following patterns 1 to 4. Patterns 1 to 4 will be explained with reference to Fig. 5. Here, the explanation will be given taking as an example a case where the image data (original image data) used by the image generation unit 133 to generate an image is image data (reference numeral 501) of persons A and B having a conversation.

[0030] ・Pattern 1: Video in which all people who participated in the dialogue (Person A and Person B) are present on the screen. ・Pattern 2: Video in which the video of the person (Person A) whom the user wants to relive (Person A), among the people who participated in the dialogue (Person A and Person B), has been removed from the screen. ・Pattern 3: Video in which the video of the person (Person A), whom the user wants to relive (Person A), among the people who participated in the dialogue (Person A and Person B), has been replaced with video of the person experiencing it (user). ・Pattern 4: Video in which the video of the person (Person A), whom the user wants to relive (Person A), among the people who participated in the dialogue (Person A and Person B), has been changed to an avatar of the user's choice.

[0031] [Summarization Unit] The summarization unit 132 summarizes the input dialogue data. For example, the summarization unit 132 reads dialogue data from a dialogue video DB in the storage unit 12. The summarization unit 132 then summarizes the dialogue data by extracting important parts from the video and audio included in the dialogue data using the technology described in the following document 1. Thereafter, the summarization unit 132 outputs the summarized dialogue data to the video generation unit 133.

[0032] Reference 1: Takayama et al., "Video summarization method to accurately convey the impression of group conversations," Vol. 2022-GN-117, No. 6 pp. 1-7 (2022)

[0033] [Video Generation Unit] The video generation unit 133 generates and outputs text data indicating the content of an utterance by each speaker from the audio included in the dialogue data (e.g., summarized dialogue data output from the summarization unit 132). The video generation unit 133 also generates data (video data) including video and audio of a pattern selected by the user from the video included in the dialogue data.

[0034] [Conversion of Speech Content into Text] First, an example of the flow in which the video generating unit 133 converts the voice included in the dialogue data into text indicating the speech content of each speaker will be described.

[0035] For example, the video generation unit 133 separates the voices (voice data) included in the dialogue data for each speaker and converts them into text data indicating the content of each speaker's speech. Next, the video generation unit 133 divides the converted text data into sections (speech sections) where sounds with a sound pressure below a predetermined threshold continue for a predetermined period of time or longer. After that, the video generation unit 133 adds the start and end times of the divided sections and identification information of the speakers in the sections to the text data and saves it.

[0036] Furthermore, the video generation unit 133 may delete duplicate words or backchannels from the converted text data, or summarize the converted text. When the video generation unit 133 summarizes the converted text, the earliest start time in the summarized text is set as the new start time, and the latest end time in the summarized text is set as the new end time.

[0037] The text data output by the video generation unit 133 may be presented as a dialogue script by the presentation unit 136, separately from the video of the speaker, or may be presented as part of the video of the speaker. The video generation unit 133 may also output to the presentation unit 136 text color-coded for each speaker or video with an icon of the speaker added.

[0038] [Video] Next, we will describe the video generated by the video generation unit 133. The video generation unit 133 acquires information indicating which area in the video corresponds to a speaker (e.g., person A, B) from the video included in the dialogue data. This information can basically be identified in advance based on rectangular area information including x and y coordinates and w and h in the video, time series information of the video, etc.

[0039] Furthermore, if the position of a person moves within the video, the information is updated. The area to be processed in the video is identified, for example, based on the area information of each person. Note that if people overlap within the video, the video generation unit 133 may set a closed area with more complex points specified.

[0040] The image generating unit 133 generates an image of a pattern selected by the user (for example, patterns 1 to 4 shown in FIG. 5) from the images included in the dialogue data.

[0041] In the case of pattern 1: the image generation unit 133 generates an image in which all people who participated in the dialogue are present on the screen from the image included in the dialogue data. Note that if the video data received by the image generation unit 133 corresponds to pattern 1, no processing is required.

[0042] In the case of pattern 2, the image generation unit 133 generates an image in which the part of the person the user is reliving (person A) is deleted from the image included in the dialogue data. Note that in the case of pattern 2, the image generation unit 133 may delete the part of person A from the image in the dialogue data, or may cut out the part of person A's conversation partner (person B).

[0043] Furthermore, when the dialogue data is an image of a synthesized space, such as an image including a speaker's avatar, the image generation unit 133 may delete the avatar of the person (person A) that the user is reliving and generate a new image. Furthermore, when the image generation unit 133 generates a new image, the angle of view may be adjusted to any angle.

[0044] In the case of pattern 3, the image generation unit 133 replaces the image of the person (person A) who is being re-experienced by the experiencer (user) with an image of the experiencer himself / herself. In the case of pattern 3, the image generation unit 133 erases person A from the image included in the dialogue data, as in pattern 2, and displays an image of the experiencer in the image. For example, the image generation unit 133 acquires an image of the experiencer who is reliving the experience via a camera and incorporates that image into the image.

[0045] In the case of pattern 4, the image generator 133 changes the image of person A that the experiencer (user) is reliving to an avatar. In the case of pattern 4, there are, for example, the following two ways of displaying the avatar.

[0046] In the first method, similar to pattern 2, the image generation unit 133 first erases person A from the screen. Then, the image generation unit 133 acquires the movements of the experiencer himself / herself during the re-experiencing from a camera and links the avatar to those movements. In the second method, similar to pattern 2, the image generation unit 133 erases person A from the image, but acquires the movements of person A through image processing and moves the avatar in conjunction with those movements.

[0047] In all of the above patterns, the image generating unit 133 can place a person at any position within the image.

[0048] Furthermore, if the image included in the dialogue data (dialogue image) is an image generated by CG as a result of dialogue simulation, the angle of view of the image generated by the image generation unit 133 can be changed to any angle of view.

[0049] For example, as in the above-mentioned pattern 2, when the image generation unit 133 generates an image showing only the person (person B) who is the conversation partner of the user, the image generation unit 133 generates an image from the viewpoint of person A from the conversation image. This allows the image generation unit 133 to generate an image in which person B is facing the direction of the user. Then, the presentation unit 136 presents the above-mentioned image to the user, allowing the user to relive the conversation more realistically.

[0050] Furthermore, if the dialogue video is video of a dialogue between people captured using multiple cameras, the video generation unit 133 may generate video from the viewpoint of person A using the technology described in the following document 2.

[0051] Reference 2: Okami et al., "Free viewpoint video synthesis technology that realizes a new video viewing experience," https: / / journal.ntt.co.jp / backnumber2 / 1710 / files / JN20171010.pdf

[0052] For example, the image generating unit 133 generates one or more patterns of images from one dialogue image. Note that, if the presentation unit 136 receives an instruction input from the user to switch to an image of a different pattern while the presentation unit 136 is presenting an image, the image generating unit 133 may present an image of a different pattern based on the instruction input.

[0053] Furthermore, the background of the person in the video generated by the video generation unit 133 may be the background that appears in the original dialogue video, or a different background may be generated.

[0054] The video generation unit 133 outputs the generated video (including audio) and text data to the presentation unit 136. The presentation unit 136 outputs the generated video and text data to a monitor or the like of the user's PC. This allows the user to view the video and text for reliving the dialogue. While the user is viewing the video and text, the information processing device 10 acquires the user's state (video and audio) from a camera, microphone, or the like of the PC.

[0055] [Situation Determination Unit] The situation determination unit 134 determines whether or not the user's speech while viewing information (video, text) presented by the presentation unit 136 continues for longer than a predetermined time.

[0056] For example, the situation determination unit 134 acquires the information (video, text) currently being presented by the presentation unit 136 and data on the state of the user (experiencer) while the information is being presented.

[0057] The information presented by the presentation unit 136 includes the part of the video currently being played back by the presentation unit 136, text data of speech received by the presentation unit 136, etc. The data on the experiencer's state includes the video and speech of the experiencer while the information is being presented, which are acquired by a camera, a microphone, etc.

[0058] First, the situation determination unit 134 determines, from the information presented by the presentation unit 136, which of the following situations the current situation corresponds to.

[0059] ・Situation 1: Person B is speaking, or within t1 seconds before Person B speaks, or within t2 seconds after Person B speaks (t1 and t2 can be set arbitrarily). ・Situation 2: Other than situation 1 (Person B is not speaking).

[0060] For example, the situation determination unit 134 determines whether or not the current situation corresponds to situation 1 based on the part of the video currently being played back by the presentation unit 136 and the text data of the speech content. If the situation determination unit 134 determines that the current situation corresponds to situation 1, it does not perform any further processing, and after receiving new data on the state of the experiencer, determines whether or not the situation corresponds to situation 1.

[0061] On the other hand, if the situation determination unit 134 determines that the current situation corresponds to situation 2, it determines the time until person B next speaks from the text data of the speech content. Then, if the situation determination unit 134 determines that the time until person B next speaks is longer than a predetermined time, that is, if it determines that the experiencer's speech has continued for longer than a predetermined time, it outputs to the reaction generation unit 135 the part of the video currently being played, the text data of the speech content, the time until person B next speaks, data on the experiencer's appearance, etc.

[0062] [Reaction Generation Unit] The reaction generation unit 135 generates a simulated reaction of the dialogue partner (e.g., person B) from data on the state of the user. For example, if the situation determination unit 134 determines that the user's utterance has continued for longer than a predetermined time, the reaction generation unit 135 processes the image of the dialogue partner while the user is speaking to include the state of the dialogue partner's reaction, such as nodding.

[0063] For example, when the user is speaking, the reaction generation unit 135 detects the timing at which the conversation partner (person B) takes a non-verbal action such as nodding in accordance with the user's speech. Then, the reaction generation unit 135 processes the video of person B so that it appears as if person B is nodding at the detected timing.

[0064] This processing can basically be performed in any of the above patterns 1 to 4, but for example, in pattern 1, the processing may be optionally selectable to be turned on or off.

[0065] If the data on the experiencer's behavior includes video and audio, the reaction generation unit 135 may determine the point at which person B should nod using the method described in the following document 3.

[0066] Reference 3: L. Huang, L.-P. Morency, and J. Gratch. “Virtual Rapport 2.0”, In Proceedings of the 10th International Conference on Intelligent Virtual Agents. (2011).

[0067] When the reaction generation unit 135 determines the location in the video where the conversation partner is nodding, it processes the video of the conversation partner at that location to make it appear as if the conversation partner is nodding. At this time, default values ​​are set in advance for information required to generate a video that makes the conversation partner appear to be nodding, such as the angle of the conversation partner's head when nodding and the duration of the nod.

[0068] The reaction generation unit 135 may adjust the processing of the nod based on information about the nod of the conversation partner himself / herself. The information about the nod of the conversation partner himself / herself is, for example, characteristics about the conditions for the occurrence of the nod of the conversation partner, such as whether the conversation partner is likely to nod or not. The reaction generation unit 135 processes the nod using, for example, the method described in the following literature 4. This enables the reaction generation unit 135 to make the person in the video nod naturally.

[0069] Reference 4: Kurauchi et al., "Adding arbitrary movements to facial images in real time," 2022-HCI-199(33), pp. 1-8 (2022).

[0070] The experiencer relives the speech of Person A while viewing the video and audio of the interlocutor (Person B) presented by the presentation unit 136, but the experiencer's speech may end up being longer than the original speech of Person A.

[0071] In such a case, the reaction generation unit 135 fixes the last frame of the time when person A speaks (for example, the video of person B at the time when person A finishes speaking in the original video) as the frame to be processed for the reaction until it determines that the person has finished speaking based on the acquired data on the state of the experiencer. Then, the reaction generation unit 135 processes the frame to be processed, and the presentation unit 136 presents the processed video.

[0072] Thereafter, when the reaction generation unit 135 determines from the acquired data on the experiencer's state that the experiencer has finished speaking (for example, when the experiencer has not spoken for a predetermined period of time or more), it causes the presentation unit 136 to resume playing the video. As a result, the speech in the video presented by the presentation unit 136 shifts to the speech of person B.

[0073] [Presentation Unit] The presentation unit 136 presents video data including video and audio generated by the video generation unit 133 in synchronization with the timing of the user's speech. When presenting the video data generated by the video generation unit 133, the presentation unit 136 may also present text indicating the content of the speech. The presentation of video data by the presentation unit 136 may involve, for example, outputting video (and text) to a monitor of the user's PC, outputting audio to a speaker, etc. The presentation unit 136 may also acquire the user's speech via, for example, a microphone on the user's PC.

[0074] For example, the presentation unit 136 displays text as subtitles in accordance with the timing of the speech of the experiencer while playing back the video of patterns 1 to 4 (see FIG. 5 ). The text displayed as subtitles here may be only the text of either person A or B, or may be the text of both persons A and B. The timing of displaying the text is the time when the speech indicated by the text starts, and the display time can be set to any time. In addition, the font size, color, and display position of the text used to display the text can be specified for each speaker.

[0075] When the image to be presented by the presentation unit 136 is an image of the above-mentioned patterns 2, 3, or 4 (see Figure 5), when the presentation unit 136 receives a processed image from the reaction generation unit 135 (for example, an image of person B nodding in response to the experiencer's utterance), the presentation unit 136 immediately displays the processed image.

[0076] On the other hand, when the image to be presented by the presentation unit 136 is the image of the above-mentioned pattern 1 (see Figure 5), and the processing of person B is turned ON by the reaction generation unit 135, the reaction generation unit 135 processes the image of person B, but does not process the image of person A.

[0077] For example, the presentation unit 136 stops playback of the video output from the video generation unit 133 in the range in which person A appears at the last frame of the target utterance (the utterance of person A).The presentation unit 136 then plays back the video of person A at the timing when the target utterance of the user ends.Incidentally, in the video output from the video generation unit 133, the area in which each person who is having a conversation exists is specified by the area for each person (w, h, or a closed area specified by more complex points) defined by the video generation unit 133.

[0078] Furthermore, if the experiencer's speech becomes longer than the original person A's speech, the presentation unit 136 may pause the playback of the video output from the video generation unit 133 until the experiencer finishes speaking.

[0079] For example, if the presentation unit 136 determines that the duration of the speech of the experiencer exceeds the duration of the speech of person A corresponding to the speech of the experiencer in the video output from the video generation unit 133, the presentation unit 136 stops playing the video until the experiencer finishes speaking. Then, after the experiencer finishes speaking, the presentation unit 136 resumes playing the video.

[0080] Furthermore, if the duration of the experiencer's speech is shorter than the duration of the original speech of person A, the presentation unit 136 may advance the timing at which person B next speaks in the video output from the video generation unit 133.

[0081] For example, if the presentation unit 136 determines that the duration of the experiencer's speech is shorter than the duration of the speech of person A corresponding to the experiencer's speech in the video output from the video generation unit 133, it skips the frame in which person A speaks and plays the video from the next frame in which person B speaks.

[0082] By performing the above processing, the presentation unit 136 can present the image output from the image generation unit 133 in accordance with the timing of the user's utterance. This allows the information processing device 10 to allow the user to relive the conversation more realistically.

[0083] [Example of Processing Procedure] Next, an example of processing procedure executed by the information processing device 10 will be described with reference to Fig. 6. First, the summarizing unit 132 of the information processing device 10 summarizes dialogue data acquired from the dialogue video DB (S1). Then, the selection input unit 131 accepts, from the experiencer, a selection input of a speaker to be the target of the re-experience (S2). Furthermore, the selection input unit 131 accepts, from the experiencer, a selection input of a pattern of video to be displayed (see Fig. 5) (S3).

[0084] The video generation unit 133 then generates text indicating the content of each speaker's utterance from the audio included in the dialogue data (video and audio) summarized in S1 (S4). The video generation unit 133 also generates video of the pattern selected in S3 from the video included in the dialogue data (S5). Here, we will explain an example in which the video of pattern 2 in FIG. 5 was selected in S3. In this case, the video generation unit 133 extracts video and audio of a speaker (e.g., person B) who is the conversation partner of the speaker (e.g., person A) selected by the user in S2. The video generation unit 133 then generates data including video and audio of the extracted conversation partner speaker.

[0085] Then, the presentation unit 136 presents the text generated in S4 and the video generated in S5 (S6).

[0086] After S6, the situation determination unit 134 captures the experiencer's speech as he or she views the text and video presented by the presentation unit 136 using a camera, microphone, etc., and determines whether the experiencer's speech has continued for more than a predetermined time (S7). If the situation determination unit 134 determines that the experiencer's speech has continued for more than a predetermined time (Yes in S7), the reaction generation unit 135 generates a reaction of person B to the experiencer's speech based on the image of person B in the image generated by the image generation unit 133, and processes the image into an image of person B (S8). The presentation unit 136 then presents the image processed in S8 (S9). On the other hand, if the situation determination unit 134 determines that the experiencer's speech has not exceeded the predetermined time (No in S7), the processes of S8 and S9 are skipped and the process ends.

[0087] [Other Embodiments] When there are three or more speakers in the dialogue data, it is also possible for the user to select in advance which speaker's utterance will be handled by the information processing device 10. Note that the user may make this selection by looking at the names of the speakers appearing in the dialogue data, or a script of the dialogue content may be generated in advance by the video generation unit 133, and the user may make the selection by referring to the script.

[0088] In addition, the user may be asked to fill out a questionnaire in advance to measure the user's characteristics, such as personality and values, and the speaker may be automatically selected based on the results of the questionnaire. When selecting a speaker automatically, a person with a personality or values ​​similar to the user may be selected, or conversely, a person completely different from the user may be selected.

[0089] The personalities and values ​​of the people who participated in the dialogue can be determined by having them answer a questionnaire similar to the one answered by the person who experienced the dialogue, or by using information such as video, audio, and speech content from the dialogue in which they participated to predict their personalities and values.

[0090] After the person to interact with has been selected by the user, the information processing device 10 performs processing equivalent to that of the above-described embodiment, but only the situation determination unit 134 and the reaction generation unit 135 have some different processing, so only these will be explained in excerpts.

[0091] The following description will be given taking as an example a case where the person making the utterance is person A, and persons B and C are participating in the dialogue.

[0092] The situation determination unit 134 acquires the information currently presented by the presentation unit 136 and data on the state of the experiencer. Then, the situation determination unit 134 determines, from the information presented by the presentation unit 136, which of the following situations the current situation corresponds to.

[0093] ・Situation 1: The person who is the conversation partner (in this case, person B or person C) is speaking, or within t1 seconds before person B or C speaks, or within t2 seconds after person B or C speaks (t1 and t2 can be set arbitrarily). ・Situation 2: Any situation other than situation 1.

[0094] For example, the situation determination unit 134 determines whether or not the current situation corresponds to situation 1 based on the part of the video currently being played back by the presentation unit 136 and the text data of the speech content. If the situation determination unit 134 determines that the current situation corresponds to situation 1, it does not perform the subsequent processing, and after receiving new data on the state of the experiencer, determines whether or not the situation corresponds to situation 1.

[0095] On the other hand, if the situation determination unit 134 determines that the current situation corresponds to situation 2, it determines the time until the next utterance by the dialogue partner (person B or person C) from the text data of the utterance content. If the situation determination unit 134 determines that the time until the next utterance by person B or person C is longer than a predetermined time, that is, if it determines that the experiencer's utterance has continued for longer than a predetermined time, it outputs the part of the video currently being played, the text data of the utterance content, the time until the next utterance by person B or person C, data on the experiencer's appearance, etc. to the reaction generation unit 135. However, in the above case, if the next utterance after either person B or person C's utterance is again by person B or C, the above data is not output to the reaction generation unit 135.

[0096] Thereafter, the reaction generation unit 135 generates pseudo reactions of the conversation partner (for example, persons B and C) from the data on the experiencer's behavior.

[0097] In this way, the information processing device 10 can also allow the user to relive the conversation.

[0098] [System Configuration, etc.] The components of each unit shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program executed by the CPU, or can be realized as hardware using wired logic.

[0099] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0100] [Program] The information processing device 10 can be implemented by installing a program (information processing program) as package software or online software on a desired computer. For example, by executing the program on an information processing device, the information processing device can function as the information processing device 10. The information processing device referred to here includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as terminals such as PDAs (Personal Digital Assistants).

[0101] 7 is a diagram showing an example of a computer that executes an information processing program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0102] The memory 1010 includes a read-only memory (ROM) 1011 and a random access memory (RAM) 1012. The ROM 1011 stores a boot program such as a basic input / output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0103] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define the processes executed by the information processing device 10 are implemented as program modules 1093 in which computer-executable code is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, the program modules 1093 for executing processes similar to those of the functional configuration of the information processing device 10 are stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced with an SSD (Solid State Drive).

[0104] Data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. The CPU 1020 then reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary and executes them.

[0105] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may also be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0106] REFERENCE SIGNS LIST 10 Information processing device 11 Input / output unit 12 Storage unit 13 Control unit 131 Selection input unit 132 Summarization unit 133 Image generation unit 134 Situation determination unit 135 Reaction generation unit 136 Presentation unit

Claims

1. An information processing device comprising: a video generation unit that converts audio contained in dialogue data of past dialogues into text data indicating the content of each speaker's utterances and outputs the text data, and generates video data from the video and audio contained in the dialogue data, including video and audio of at least the speaker who is the dialogue partner of the speaker selected by the user as the subject of the re-experience; and a presentation unit that presents the generated video data in accordance with the timing of the user's utterances.

2. An information processing device as described in claim 1, further comprising a situation determination unit which determines whether the user's speech during the presentation of the video data has continued for longer than a predetermined time, and a reaction generation unit which, when it is determined that the user's speech has continued for longer than the predetermined time, processes the image of the dialogue partner in the video data during the user's speech to show how the dialogue partner reacts to the user's speech, and the presentation unit presents the video data including the processed image of the dialogue partner.

3. The information processing device described in claim 1, characterized in that when the presentation unit determines that the duration of the user's utterance during the presentation of the video data exceeds the duration of the utterance of the speaker of the subject of the re-experience that corresponds to the utterance in the video data, it stops playback of the video data until the user finishes speaking, and resumes playback of the video data after the user finishes speaking.

4. The information processing device of claim 1, further comprising a selection input unit that accepts a selection input from the user as to whether the video data generated by the video generation unit is to be video in a format that includes all speakers appearing in the dialogue data, video in a format in which the speaker selected by the user to re-experience is deleted, or video in a format in which the speaker is replaced with the user's own video or avatar, and the video generation unit generates video data in the format selected by the selection input.

5. The information processing device according to claim 1, further comprising a summarizing unit that summarizes input dialogue data, wherein the image generating unit executes processing on the summarized dialogue data.

6. The information processing device according to claim 1, characterized in that the video data includes text data indicating the contents of speech for each of the speakers.

7. An information processing method executed by an information processing device, comprising the steps of: converting audio contained in dialogue data of a past dialogue into text data indicating the content of the speech for each speaker, outputting the text data, and generating, from the video and audio contained in the dialogue data, video data including video and audio of at least the speaker who is the dialogue partner of the speaker selected by the user as the subject of the re-experience; and presenting the generated video data in accordance with the timing of the user's speech.

8. An information processing program for causing a computer to execute the steps of: converting audio included in dialogue data of past dialogues into text data indicating the content of each speaker's utterances, outputting the text data, and generating, from the video and audio included in the dialogue data, video data including video and audio of at least the speaker who is the dialogue partner of the speaker selected by the user as the subject of the re-experience; and presenting the generated video data in accordance with the timing of the user's utterances.

Citation Information

Patent Citations

  • Moving image editing method and apparatus

    JP2004266578A

  • Conference system, minutes server, and minutes creation method

    JP2013183182A

  • Information processing unit

    JP2021184601A