Video presentation device, video presentation method, and video presentation program

Abstract

This video presentation device estimates the subjective value of each speaker at each time of day from a master file that is a file of video and audio of the dialog, and tags the master file. Thereafter, the video presentation device acquires effect data from a knowledge database that accumulates effect data corresponding to the subjective value of a user who views the master file on the basis of the result of estimation of the subjective value of each speaker at each time of day when the master file is tagged. Then, the video presentation device performs processing for adding the acquired effect data to a corresponding location of the master file. Thereafter, the video presentation device presents the processed master file.

Description

Image presentation device, image presentation method, and image presentation program

[0001] The present invention relates to an image presentation device, an image presentation method, and an image presentation program.

[0002] In order to check the state of a past meeting, conversation, discussion, etc., a user may view a video of the meeting, conversation, or discussion (hereinafter referred to as "dialogue video") that was filmed. In the prior art, a technique has been proposed for extracting only important parts of such a video so that a user can efficiently view the video (see Non-Patent Document 1).

[0003] Fumio Nihei and Yukiko I. Nakano: A Multimodal Meeting Browser that Implements an Important Utterance Detection Model based on Multimodal Information, In Proceedings of the 25th International Conference on Intelligent User Interfaces Companion,(2020)

[0004] However, in the conventional technology, it is not possible to know the type and level of emotion of the speaker shown in the dialogue video based on the user's own past experience. As a result, there is a problem that the user cannot deeply understand the content of the dialogue video. Therefore, an object of the present invention is to enable the user to deeply understand the content of the dialogue video.

[0005] In order to solve the above-mentioned problems, the present invention is characterized by comprising a memory unit that stores information indicating effect data corresponding to each subjective value of a user viewing a video and audio file of a dialogue; an acquisition unit that estimates the subjective value of a speaker of the dialogue at each time from the video and audio file of the dialogue and acquires from the memory unit effect data of the user that corresponds to the same subjective value as the estimated subjective value of the speaker; a processing unit that processes the acquired effect data by adding it to the file at the time corresponding to the speaker's subjective value; and a presentation unit that presents the processed master file.

[0006] According to the present invention, the user can deeply understand the content of the dialogue video.

[0007] FIG. 1 is a diagram showing display examples of patterns 1 to 3 for a subjective value Fa (fun: 1). FIG. 2 is a diagram for explaining an overview of an image presentation device. FIG. 3 is a diagram showing an example of the configuration of an image presentation device. FIG. 4 is a flowchart showing an example of a processing procedure executed by the image presentation device. FIG. 5 is a diagram showing display examples of patterns 1 to 3 for a subjective value Fa (fun: 8). FIG. 6 is a diagram showing an example of the display position of effect data on the screen of a master file. FIG. 7 is a diagram for explaining an example of a set of effect data. FIG. 8 is a diagram for explaining an example of a set of effect data. FIG. 9 is a diagram showing a computer that executes an image presentation program.

[0008] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a description will be given of an embodiment of the present invention with reference to the drawings, but the present invention is not limited to the embodiment.

[0009] [Overview] An overview of the video presentation device of this embodiment will be described with reference to Figure 1. In the following description, a media file in which a conversation between speakers is filmed and recorded will be referred to as a master file. Also, a database (DB) that stores information on effect data corresponding to each subjective value of a viewer (user) of the master file will be referred to as a knowledge DB.

[0010] The video presentation device estimates the subjectivity of a speaker in the dialogue in the master file, and acquires user effect data (e.g., characters or images expressing the subjectivity) corresponding to the subjectivity of the speaker from the knowledge DB to provide supplementary explanation of the subjectivity of the speaker to the user. The video presentation device then adds the acquired effect data to the master file and presents it.

[0011] For example, consider a case where the master file is a video of a conversation between persons A and B, and the viewer of the master file is person C. In this case, if the video presentation device estimates that person B's subjective opinion is "Enjoyment: 1," it obtains effect data corresponding to "Enjoyment: 1" from person C's knowledge DB and adds it to the master file.

[0012] For example, as shown in Figure 1, the video presentation device adds related images / videos (pattern 1), effects (pattern 2), text (pattern 3), etc. corresponding to "Fun: 1" to the master file. Then, the video presentation device presents the master file with the added effect data to person C.

[0013] This allows the video presentation device to process and present the subjectivity of the speakers (for example, persons A and B) in the master file in a way that is easy for the user (for example, person C) to understand.

[0014] As a result, the user can intuitively understand while watching how the psychological emotions and levels of the interlocutors in the master file correspond to the emotions and levels based on the user's own past experiences. This allows the user to compare what they are watching with their own experiences, deepening their understanding of the content of each speaker's speech.

[0015] For example, by viewing a master file that has been processed as described above, a user can more easily understand the content of the speaker's speech, even if the speaker in the master file (e.g., person A or B) is someone the user has never met before.

[0016] 2, the video presentation device includes a content understanding unit (acquisition unit) 131, a processing unit 132, a presentation unit 133, etc. An emotion identification unit 134 shown by a dashed line may or may not be provided, and the cases where it is provided will be described later.

[0017] The knowledge DB and the master file DB may be provided inside or outside the video presentation device. In the following, the case where the knowledge DB and the master file DB are provided inside the video presentation device will be described as an example.

[0018] [Configuration Example] Next, a configuration example of the image presentation device 10 shown in Fig. 2 will be described with reference to Fig. 3. The image presentation device 10 includes, for example, an input / output unit 11, a storage unit 12, and a control unit 13.

[0019] The input / output unit 11 is an interface that controls the input and output of various data, and outputs, for example, a master file processed by the control unit 13 to an external device.

[0020] The storage unit 12 stores data, programs, etc. that are referenced when the control unit 13 executes various processes. The storage unit 12 is realized by a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. For example, the storage unit 12 stores a master file DB, a knowledge DB, etc.

[0021] The knowledge DB stores information on effect data corresponding to each subjective value of a viewer (user) of a master file. The effect data includes, for example, video data, audio data, and image data.

[0022] The effect data information is, for example, information indicating the user's subjective value, the ID of the effect data corresponding to the subjective value (or the effect data itself), information about the user, the date of the effect data, the priority of the effect data, the shortest playback time of the effect data, metadata of the effect data, etc. (see symbol 101 in Figure 3).

[0023] The priority of effect data indicates the priority when selecting effect data to be displayed at a certain time from among multiple candidate effect data on the master file screen.

[0024] The priority value can be set by the user of the video presentation device 10 as an integer (1 or greater) for each piece of effect data. A value of 1 is the highest priority, and as the value increases, the priority decreases.

[0025] The priority value may be updated based on a predetermined rule. For example, the control unit 13 may increase the priority value by an arbitrary value for effect data whose date has passed an arbitrary amount of time.

[0026] For example, if a subjective value that is more closely related to a master file can be specified in advance, the priority of effect data associated with another subjective value may be increased by an arbitrary value in order to raise the priority of effect data associated with the arbitrary subjective value. In this case, the priority value before the change is also stored in the storage unit 12 so that the control unit 13 can reset the priority value of the effect data to the original priority value at any timing.

[0027] Furthermore, for example, the priority of effect data having an appropriate data volume may be increased depending on the processing performance of the video presentation device 10. In this case, for example, if the processing performance of the video presentation device 10 is high, the control unit 13 accordingly decreases the priority value of effect data exceeding a certain data volume (increases the priority). Also, if the processing performance of the video presentation device 10 is low, the control unit 13 accordingly increases the priority value of effect data exceeding a certain data volume (decreases the priority). Even in this case, the priority value before the change is also retained in the storage unit 12 so that the control unit 13 can reset the priority value of the effect data to the original priority value at any time.

[0028] The minimum playback time of the effect data indicates the minimum length of time for which the effect data is displayed (played back) when the effect data is displayed on the screen of the master file.

[0029] For example, consider a case where the processing unit 132 (described later) processes the video of the master file so that effect data corresponding to the subjective value of the speaker is continuously displayed at the timing when the speaker speaks while the video of the master file is being played back.

[0030] In this case, if the display time of the effect data on the master file is too short, it becomes difficult for the user to recognize the content of the effect data while the video of the master file is being played. In particular, if there are multiple effect data to be displayed in the master file, the multiple effect data will switch in a short time while the video of the master file is being played, making it even more difficult for the user to recognize the content of each effect data.

[0031] Therefore, the processing unit 132 selects and incorporates effect data into the master file based on the shortest playback time of each effect data, thereby ensuring a certain amount of display time for each effect data on the master file. As a result, the processing unit 132 can process the master file in a way that makes it easy for the user to recognize the content of each effect data.

[0032] The effect data itself may be stored in the knowledge DB or in a DB outside the knowledge DB. When the effect data is stored outside the knowledge DB, the control unit 13 acquires the effect data from outside the knowledge DB using the ID of the effect data as a key.

[0033] The control unit 13 is responsible for overall control of the image presentation device 10. The functions of the control unit 13 are realized, for example, by a CPU (Central Processing Unit) executing a program stored in the storage unit 12.

[0034] The control unit 13 includes, for example, a content understanding unit 131, a processing unit 132, and a presentation unit 133. The emotion identification unit 134 may or may not be provided, and the case where it is provided will be described later.

[0035] The content understanding unit 131 estimates the subjective value of each speaker at each time of the dialogue in the master file and tags the estimation result in the master file. Then, based on the estimation result of the subjective value of each speaker at each time of the dialogue tagged in the master file, the content understanding unit 131 acquires information on effect data corresponding to the subjective value from the knowledge DB.

[0036] The processing unit 132 performs processing to add effect data to a corresponding location in the master file based on the information on the effect data acquired by the content understanding unit 131. For example, the processing unit 132 adds the effect data indicated in the information on the effect data acquired by the content understanding unit 131 to a location in the master file at a time corresponding to the subjective value of the speaker.

[0037] In addition, when the processing unit 132 adds the effect data to the corresponding location in the master file based on the information of the acquired effect data, it sets the start time and end time of the effect data so that the playback time is equal to or longer than the shortest playback time of the effect data (details will be described later).

[0038] The presenting unit 133 presents the master file processed by the processing unit 132. For example, the presenting unit 133 outputs the processed master file to an external device.

[0039] [Example of Processing Procedure] Next, an example of the processing procedure executed by the video presentation device 10 will be described with reference to Figure 4. First, the content understanding unit 131 of the video presentation device 10 acquires a master file from the master file DB (S1). Then, the content understanding unit 131 estimates the subjective values ​​of each speaker at each time in the acquired master file (S2). Then, the content understanding unit 131 tags the master file with the results of the subjective value estimation for each speaker (S3).

[0040] The content understanding unit 131 then acquires, from the knowledge DB, effect data information corresponding to the subjective values ​​of each speaker at each time tagged in the master file (S4). The content understanding unit 131 then outputs the acquired effect data information to the processing unit 132. The processing unit 132 then adds the effect data indicated in the output effect data information to the master file (S5). The presentation unit 133 then presents the master file with the added effect data (S6).

[0041] [Details of the Content Understanding Unit] Next, the processing executed by the content understanding unit 131 will be described in detail with specific examples. In the following description, it is assumed that a user is viewing a video of a conversation between multiple people. The video recording of this conversation is referred to as a master file.

[0042] The speakers of the master file dialogue are persons A and B, and the user viewing the master file is person C. Persons A and B are meeting each other for the first time, introducing themselves to each other, and engaging in a dialogue about their respective hobbies and past experiences.

[0043] The master file may be stored in advance in the master file DB, or may be managed separately without being stored in the master file DB. If there are multiple master files, the master file to be processed may be one selected by the user or one selected by the video presentation device 10 based on some criteria.

[0044] The content understanding unit 131 estimates the speech content of each speaker (person A, B) in the master file and the subjective values ​​of persons A and B.

[0045] For example, if the master file is data including audio, the content understanding unit 131 tags each utterance in the master file to indicate whether it is being spoken by person A or person B. The content understanding unit 131 also records the start time (t1) and end time (t2) of each utterance in the master file. The start time (t1) and end time (t2) of each utterance are used as delimiter information when selecting effect data information from the knowledge DB.

[0046] The content understanding unit 131 may also extract noun information from the utterances extracted from the master file and store the extracted nouns with information on t1 and t2 as utterance noun information. Furthermore, if the master file is data including video, the content understanding unit 131 may extract nods, facial expressions, etc. of persons A and B from the movements, etc. of persons A and B. Then, using the extracted information, the content understanding unit 131 may estimate what emotions persons A and B are feeling at each time in the video of the master file and what their level is, using the technology described in the following literature 1.

[0047] Reference 1: Shumpei Otsuchi, Yoko Ishii, Momoko Nakatani, and Kazuhiro Otsuka. 2021. Prediction of Interlocutors' Subjective Impressions Based on Functional Head-Movement Features in Group Meetings. In Proceedings of the 2021 International Conference on Multimodal Interaction (ICMI '21). Association for Computing Machinery, New York, NY, USA, pp.352-360.

[0048] The content understanding unit 131 estimates a subjective value for each speaker who appears in the master file. For example, if the speakers of a conversation are persons A and B, the content understanding unit 131 outputs the subjective value Fa of person A and the subjective value Fb of person B in the master file in chronological order.

[0049] The content understanding unit 131 may output the subjective values ​​for each speaker by dividing them into a plurality of categories (for example, fun, nostalgic, etc.). Any category can be set.

[0050] In this case, for example, the subjective values ​​may be values ​​such as F1: enjoyment and F2: nostalgia. In this case, the content understanding unit 131 outputs, for example, F1a and F2a as the subjective values ​​of person A, and F1b and F2b as the subjective values ​​of person B.

[0051] Next, the content understanding unit 131 refers to the knowledge DB to determine what the situation of the output subjective values ​​Fa and Fb is for person C. The content understanding unit 131 may refer to the knowledge DB using either the subjective values ​​Fa or Fb, or may refer to the knowledge DB using both the subjective values ​​Fa and Fb. The content understanding unit 131 may also look at the audio tag in the master file and use only the information about the person speaking.

[0052] As mentioned above, the knowledge DB stores, for each user's subjective value, information on effect data indicating the ID of the effect data (or the effect data itself) corresponding to that subjective value, information on the user, the date of that effect data, the priority of that effect data, the shortest playback time of that effect data, metadata of that effect data, etc. (see symbol 101 in Figure 3).

[0053] The user can set any value for the metadata of the effect data when saving the data in the knowledge DB. For example, if the effect data is an image of a fireworks display, noun information related to the effect data, such as "fireworks display" or "summer," is set as the metadata. Note that multiple pieces of metadata can be set for one piece of effect data.

[0054] Furthermore, the subjective value may be, for example, a numerical value indicating the strength (level) of the subjectivity for each subjective category (F1: fun, F2: nostalgia, etc.) (hereinafter referred to as "save format 1" as appropriate), or a numerical range indicating the strength (level) of the subjectivity for each subjective category (hereinafter referred to as "save format 2" as appropriate).

[0055] For example, when the subjective values ​​Fa and Fb are given as input, the content understanding unit 131 refers to the knowledge DB and outputs one or more pieces of information on the effect data corresponding to the subjective values ​​Fa and Fb.

[0056] For example, when the subjective value is in storage format 1, the content understanding unit 131 acquires from the knowledge DB and outputs information on effect data that has the same category and numerical value as the Fa or Fb given as input.

[0057] Also, for example, if the value of the subjective value is storage format 2, the content understanding unit 131 acquires and outputs information about effect data that has the same category as the Fa or Fb given as input and whose numerical value is within the range of the subjective value in the knowledge DB.

[0058] In either case of storage format 1 or 2, when the content understanding unit 131 acquires information on multiple pieces of effect data from the knowledge DB, it may output all of the acquired information, or it may output a predetermined number of pieces of acquired information in descending order of priority. Furthermore, the content understanding unit 131 may output all information of a certain priority or higher.

[0059] [Specific Example] The input and output processing of the content understanding unit 131 will be described. First, the content understanding unit 131 extracts the time t1-t2 during which person A is speaking from the tags attached to the master file. For example, consider a case where the content understanding unit 131 detects a subjective value of "fun feeling of 8" from the tags of the master file during t1-t2. In this case, the content understanding unit 131 receives the value "8 for the subjective value of fun" as input, and obtains from the knowledge DB still image data pic1, which is effect data corresponding to the "8 for the subjective value of fun" for person C.

[0060] As a result, the content understanding unit 131 associates pic1 with the time t1-t2 of the master file. The content understanding unit 131 also outputs the time series information (t1-t2), the data of pic1 itself, the priority of pic1 acquired from the knowledge DB, the shortest playback time, and the master file itself to the processing unit 132.

[0061] The effect data associated with each subjective value may have, for example, the following three patterns.

[0062] Pattern 1: Still images, images, and video files Pattern 2: Files intended for multiple display, such as effects Pattern 3: Graphs and text

[0063] The effect data may be a file containing only audio.

[0064] Display examples of patterns 1 to 3 for the subjective value Fa (fun: 1, fun: 8) are shown in FIGS.

[0065] The content understanding unit 131 may output the time series information (t1-t2) and the ID of pic1 to the processing unit 132. In this case, the processing unit 132 uses the ID of pic1 as a key to obtain the data, priority, shortest playback time, etc. of pic1 from the knowledge DB.

[0066] The processing unit 132 may also set one or more patterns of effect data in the master file. For example, if multiple patterns of effect data corresponding to a certain subjective value are registered in the knowledge DB, the content understanding unit 131 sets an arbitrary pattern of effect data from among them.

[0067] [Details of the Processing Unit] Next, a detailed description will be given, with specific examples, of the processing executed by the processing unit 132. The processing unit 132 performs processing to add effect data to the master file.

[0068] For example, when the processing unit 132 receives effect data for a still image (pattern 1 described above) from the content understanding unit 131, the still image is displayed in a set location on the screen of the master file at a set size (see Figure 6).

[0069] For example, in Figure 6, the upper left corner of the master file screen is set as the origin, and the display position of the effect data is specified by x, y, and the image size by w, h. If there is more than one piece of effect data to display, x, y, w, h can be specified for each piece.

[0070] Furthermore, if the effect data is the effect of the above-mentioned Pattern 2, the processing unit 132 displays, for example, a set number of images of the effect data in a set size at a set location on the screen of the master file (see the example of Pattern 2 shown in FIG. 1). In this case, too, as in the case of Pattern 1, the above x, y, w, and h may be set for the number of pieces of effect data.

[0071] When the effect data is a graph of the above-mentioned pattern 3, the processing unit 132 displays, for example, a graph showing the subjective values ​​together with the names of the subjective values. The color and type of the graph (for example, a pie chart, a line graph, etc.) are assumed to be set in the information of the effect data in the knowledge DB. In this case, the processing unit 132 may also display the graph by specifying the above x, y, w, and h, as in the case of pattern 1.

[0072] The processing unit 132 selects, for example, effect data to be added to each time of the master file from the group of effect data output from the content understanding unit 131 in the following manner.

[0073] For example, the processing unit 132 selects, from the group of effect data output from the content understanding unit 131, effect data having t1 closest to the start time of the master file, effect data with a high priority, effect data having metadata related to the content currently being spoken by the speaker in the master file, or one or more effect data randomly. Which of these methods is to be selected can be set in advance.

[0074] This section explains how to select effect data that contains metadata about what is being spoken in the master file.

[0075] This method can be selected when the master file is a file containing audio and the spoken noun information has been extracted in advance along with t1 and t2.

[0076] For example, the processing unit 132 calculates the semantic distance between the spoken noun information at t1-t2 in the master file and the metadata of the effect data at t1-t2. For example, the processing unit 132 calculates the Euclidean distance between words using a method for representing words as vectors, such as word2vec, and sets the calculated Euclidean distance as the semantic distance.

[0077] Here, if there are multiple pieces of spoken noun information and metadata, the processing unit 132 may, for example, find the distances among them in a round-robin manner and use the shortest distance as the final distance.The processing unit 132 then selects the effect data having metadata that has the shortest semantic distance from the spoken noun information at t1-t2 in the master file.

[0078] [Example of Setting Effect Data] The processing unit 132 sets the next effect data after the shortest playback time of the effect data set in the master file has elapsed (this time is assumed to be t3). An example of setting effect data will be described with reference to FIG. 7.

[0079] Here, an example will be described in which the candidates for effect data to be set in the master file are still images 1, 2, and 3 shown in Fig. 7. The values ​​t1, t2, and shortest playback time of still images 1, 2, and 3 are assumed to be the values ​​shown in Fig. 7.

[0080] Of still images 1, 2, and 3, still image 1 has t1 closest to the start time of the master file (t1 of still image 1 = 10 seconds). Therefore, the processing unit 132 sets still image 1 for 5 seconds from 10 seconds in the master file. The end time of still image 1 in the master file is 15 seconds. The still image with t1 closest to 15 seconds is still image 3 (t1 of still image 3 = 12 seconds). However, t2 of still image 3 is 13 seconds, which is earlier than the end time of still image 1 in the master file, which is 15 seconds. Therefore, the processing unit 132 excludes still image 3 from the candidates.

[0081] Next, after still image 3, the still image having t1 closest to the end time of still image 1, which is 15 seconds, is still image 2 (t1 of still image 2 = 11 seconds). The shortest playback time of still image 2 is 5 seconds. Therefore, the processing unit 132 sets still image 2 for 5 seconds from the time 15 seconds in the master file.

[0082] When the processing unit 132 sets effect data, it may either apply a rule that effect data that has been used once is not to be reused or allow it to be reused.

[0083] Furthermore, the processing unit 132 may display the effect data on the master file for a time longer than the shortest playback time set for the effect data.

[0084] Furthermore, the processing unit 132 may set the start time of the display of the effect data in the master file earlier or later than t1. Similarly, the processing unit 132 may set the end time of the display of the effect data earlier or later than t2.

[0085] When there are multiple pieces of effect data to be displayed, the start and end times of displaying this effect data can be set for each of them.

[0086] For example, when the effect data is pattern 1 (e.g., a still image), the processing unit 132 may simultaneously display multiple still images side by side on the master file, or may display the still images one by one in order. Furthermore, when starting and ending the display of the effect data on the master file, the processing unit 132 may display various effects, such as dissolve or drop-in, which are used as transitions in moving images.

[0087] 8 is a diagram illustrating an example in which the candidates for effect data to be set in the master file are still images 1, 2, 3, 4, and 5. In the example shown in FIG. 8, multiple still images remain as candidates based only on the time series information t1-t2 and the shortest playback time. Therefore, the processing unit 132 narrows down the still images to be set by also referring to the priority order of each still image.

[0088] 8, still images 1 and 4 (both t1=10) have t1 closest to the start time of the master file. Therefore, the processing unit 132 compares the priorities of still images 1 and 4 and selects still image 1, which has the highest priority (priority=1).

[0089] Then, since t1 of still image 1 = 10 seconds, the processing unit 132 sets still image 1 for 5 seconds from 10 seconds in the master file. The end time of still image 1 in the master file = 15 seconds. The still images with t1 closest to 15 seconds are still images 2, 3, 4, and 5. Of these still images, t2 of still image 3 = 13 seconds, which is a time before 15 seconds, the end time of still image 1 in the master file. Therefore, the processing unit 132 excludes still image 3 from the candidates. The remaining still images 2, 4, and 5 all have t2 times later than 15 seconds, the end time of still image 1 in the master file. Therefore, the processing unit 132 leaves still images 2, 4, and 5 as candidates.

[0090] Next, the processing unit 132 compares the priorities of still images 2, 4, and 5, and selects still image 5, which has the highest priority (priority=2).The processing unit 132 then sets still image 5 in the master file for 5 seconds starting from 15 seconds into the master file.

[0091] Although still image 3 has a higher priority than still image 5 (priority of still image 3=1), both t1 and t2 of still image 3 are before the target time (15 seconds in the master file). Therefore, the processing unit 132 does not select still image 3.

[0092] If still image 5 is played for the shortest playback time (5 seconds) from 15 seconds in the master file, this will exceed the t2 (16 seconds) of still image 5 itself. However, t2 is merely information used to determine the timing for playing the still image. Therefore, the processing unit 132 sets still image 5 between 15 seconds and 20 seconds in the master file.

[0093] By performing the above processing, the processing unit 132 can ensure the shortest playback time for each piece of effect data, while playing back each piece of effect data at the appropriate time in the master file.

[0094] Other Embodiments The video presentation device 10 may further include an emotion identification unit 134 (see FIG. 3 ). The emotion identification unit 134 identifies the emotion of the viewer (user) of the master file and outputs the identification result of the user's emotion to the processing unit 132. For example, the emotion identification unit 134 acquires video and audio of the user watching the video of the master file using a camera, a microphone, or the like. Then, the emotion identification unit 134 identifies the user's emotion using the acquired video and audio of the user.

[0095] The method for identifying a user's emotion is, for example, the method described in the above-mentioned literature 1. For example, a model is prepared in advance that has been trained on data such as a user's video and audio, and the value of the user's emotion. The emotion identification unit 134 then provides the model with the video, audio, etc. of the user viewing the video as input data, thereby obtaining the category of the user's emotion and its value as F2c. The emotion identification unit 134 then outputs the F2c to the processing unit 132.

[0096] F2c is, for example, concentration level, motivation, etc. When F2c output by the emotion identification unit 134 is concentration level, the processing unit 132 may add an effect to the master file to improve the user's concentration level, for example, when F2c decreases.

[0097] For example, the processing unit 132 compares the F2c values ​​output from the emotion identification unit 134 in chronological order, and when it determines that the F2c value has decreased compared to the previous time, performs processing such as increasing the size of the effect data displayed on the master file or playing a sound. By performing the above processing by the processing unit 132, when the user views the master file, it is possible to elicit the user's awareness or deepen their understanding according to the user's viewing situation.

[0098] [System Configuration, etc.] The components of each unit shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program executed by the CPU, or can be realized as hardware using wired logic.

[0099] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0100] [Program] The image presentation device 10 can be implemented by installing a program (image presentation program) as package software or online software on a desired computer. For example, by executing the program on an information processing device, the information processing device can function as the image presentation device 10. The information processing device referred to here includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as terminals such as PDAs (Personal Digital Assistants).

[0101] 9 is a diagram showing an example of a computer that executes an image presentation program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0102] The memory 1010 includes a read-only memory (ROM) 1011 and a random access memory (RAM) 1012. The ROM 1011 stores a boot program such as a basic input / output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0103] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define the processes executed by the image presentation device 10 are implemented as program modules 1093 in which computer-executable code is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, the program modules 1093 for executing processes similar to those of the functional configuration of the image presentation device 10 are stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced with an SSD (Solid State Drive).

[0104] Data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. The CPU 1020 then reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary and executes them.

[0105] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may also be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0106] 10 Image presentation device 11 Input / output unit 12 Memory unit 13 Control unit 131 Content understanding unit 132 Processing unit 133 Presentation unit 134 Emotion identification unit

Claims

1. A video presentation device comprising: a memory unit that stores information indicating effect data corresponding to a subjective value of each user viewing a video and audio file of a dialogue; an acquisition unit that estimates a subjective value of a speaker of the dialogue at each time from the video and audio file of the dialogue and acquires from the memory unit effect data of the user corresponding to the same subjective value as the estimated speaker's subjective value; a processing unit that processes the acquired effect data by adding it to the file at a time corresponding to the speaker's subjective value; and a presentation unit that presents the processed file.

2. The image presentation device according to claim 1, characterized in that the effect data is one or a combination of still images, videos, and graphs corresponding to the categories and values ​​of the subjective values.

3. The video presentation device of claim 1, further comprising an emotion identification unit that acquires the behavior of a user viewing the presented file and identifies a category and value of the user's emotion based on the acquired behavior of the user, wherein the processing unit processes the effect data in accordance with the identified category and value of the user's emotion.

4. The video presentation device of claim 1, characterized in that the information indicating the effect data corresponding to the subjective value includes a shortest playback time of the effect data in the file, and when the processing unit adds the acquired effect data to the file at a time location corresponding to the speaker's subjective value, it sets the start time and end time of the effect data so that the playback time of the effect data is equal to or longer than the shortest playback time of the effect data.

5. A video presentation method executed by a video presentation device, wherein the video presentation device has a memory unit that stores information indicating effect data corresponding to a subjective value for each subjective value of a user viewing a video and audio file of a dialogue, the video presentation method comprising the steps of: estimating the subjective value of a speaker of the dialogue at each time from the video and audio file of the dialogue, and acquiring from the memory unit effect data of the user corresponding to the same subjective value as the estimated speaker's subjective value; processing the acquired effect data by adding it to the file at the time corresponding to the speaker's subjective value; and presenting the processed file.

6. A video presentation program for causing a computer to execute the steps of: estimating a subjective value of a speaker of a dialogue at each time from a video and audio file of the dialogue; acquiring effect data of the user corresponding to the same subjective value as the estimated speaker's subjective value from a memory unit that stores information indicating effect data corresponding to the subjective value for each user viewing the video and audio file of the dialogue; processing the file by adding the acquired effect data to the location in the file at the time corresponding to the speaker's subjective value; and presenting the processed file.

Images