Analysis system, analysis method, and program

Abstract

In dialogues using multimodal information, the goal is to enable understanding of the relationship between multimodal information and a common platform. [Solution] The analysis system includes: an acquisition unit that acquires the order of images extracted from video, rearranged chronologically by each interlocutor based on a dialogue using multimodal information between the interlocutors; an extraction unit that extracts the multimodal information from the dialogue; and an analysis unit that analyzes the relationship between the multimodal information and the common ground in the dialogue based on the order of the images.

Description

Technical Field

[0001] The present invention relates to an analysis system, an analysis method, and a program.

Background Art

[0002] In human communication, it is important to convey exactly what one wants to convey to the other party. In communication, the content mutually understood by the interlocutors is called a common ground, but how this common ground is constructed is not yet known.

[0003] Regarding this common ground, a technique for analyzing the relationship between the interaction in text chat and the common ground using a problem called the common figure placement problem is known (see, for example, Non-Patent Document 1).

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the method disclosed in Non-Patent Document 1, the relationship between text chat and the common ground can be examined, but there is a problem that the relationship between multimodal information such as gestures and expressions in face-to-face dialogue and the common ground cannot be examined for interactions involving such multimodal information.

[0006] Embodiments of the present invention have been made in view of the above problems, and enable the grasping of the relationship between multimodal information and the common ground in dialogue using multimodal information. [Means for solving the problem]

[0007] To solve the above problems, the analysis system according to an embodiment of the present invention includes: an acquisition unit that acquires the order of images extracted from video, rearranged chronologically by each interlocutor based on a dialogue using multimodal information between interlocutors; an extraction unit that extracts the multimodal information from the dialogue; and an analysis unit that analyzes the relationship between the multimodal information and the common base in the dialogue based on the order of the images. [Effects of the Invention]

[0008] According to embodiments of the present invention, in dialogue using multimodal information, it becomes possible to understand the relationship between multimodal information and a common platform. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the system configuration of the analysis system according to this embodiment. [Figure 2] This figure shows an example of the functional configuration of the analysis device according to this embodiment. [Figure 3] This figure shows an example of the functional configuration of the terminal device according to this embodiment. [Figure 4] This figure shows an example of an image extracted from video according to this embodiment. [Figure 5] This figure shows an example of the experimental environment according to this embodiment. [Figure 6] This flowchart shows an example of the processing performed by the analysis system according to this embodiment. [Figure 7] This figure shows an example of the display screen for the practice phase according to this embodiment. [Figure 8] This figure shows an example of the display screen for the confirmation phase according to this embodiment. [Figure 9] This figure shows the experimental process (transfer phase) according to this embodiment. [Figure 10] This figure shows an example of the display screen for the work phase according to this embodiment. [Figure 11] This flowchart shows an example of the analysis process according to this embodiment. [Figure 12] This figure shows an example of the analysis results according to this embodiment. [Figure 13] This diagram shows an example of a computer hardware configuration. [Modes for carrying out the invention]

[0010] Hereinafter, embodiments of the present invention (this embodiment) will be described with reference to the drawings. The embodiments described below are merely examples, and the embodiments to which the present invention is applied are not limited to the embodiments described below.

[0011] <About the analysis system> The analysis system according to this embodiment is a system that analyzes the relationship between multimodal information and a common ground in a dialogue using multimodal information. Multimodal information is, for example, multiple types of information such as gestures and facial expressions. In a dialogue, the content that the dialoguers mutually understand is called the common ground.

[0012] The method disclosed in Non-Patent Document 1 uses a task called the collaborative figure arrangement task. In this task, two speakers communicate remotely via text chat. Each speaker is presented with the same set of figures, but with different arrangements, and they work together to align these arrangements through text chat dialogue. The degree of agreement in the arrangement of the figure set can be considered as the arrangement that the two speakers have agreed upon and understood, and therefore can be seen as a quantitative representation of a common foundation. Through this task, it is possible to confirm how a common foundation is built with each utterance, and to investigate what kinds of utterances are useful in building a common foundation.

[0013] However, with the method disclosed in Non-Patent Document 1, although the relationship with the common base of text chat can be examined, there is a problem that in a dialogue using multimodal information such as face-to-face dialogue, the relationship between the multimodal information and the common base cannot be examined. There is also prior research on conducting the common figure arrangement task through face-to-face dialogue (for example, see Reference 1), but there is a problem that the expression of multimodal information is not sufficiently carried out for the operation of moving figures.

[0014] Therefore, the analysis system 1 according to the present embodiment enables the relationship between multimodal information and the common base to be grasped in a dialogue using multimodal information. Specifically, an image rearrangement task cut out from a predetermined video is performed between interlocutors. In this task, for each speaker, an image randomly arranged from different time points in a predetermined video is held, and while repeating the dialogue and rearrangement phases at regular intervals, the order of the images is appropriately rearranged. At this time, the degree of coincidence of the arrangement of the images can be regarded as the common base. In the present embodiment, this task is called the "common story rearrangement task".

[0015] The analysis system 1 acquires the order of the images arranged in time series by each interlocutor in the common story rearrangement task, extracts multimodal information from the dialogue, and analyzes the relationship between the multimodal information and the common base in the dialogue based on the acquired order of the images. For example, the analysis system 1 quantifies the influence of the multimodal information on the common base based on the acquired order of the images.

[0016] <System Configuration> FIG. 1 is a diagram showing an example of the system configuration of the analysis system according to the present embodiment. In the example of FIG. 1, the analysis system 1 includes terminal devices 10a, 10b, cameras 20a, 20b, 20c, and an analysis device 100, etc. In the following description, when indicating an arbitrary terminal device among the terminal devices 10a, 10b, "terminal device 10" is used. Also, when indicating an arbitrary camera among the cameras 20a, 20b, 20c, "camera 20" is used.

[0017] Terminal device 10 is an information terminal used by the interlocutor to engage in dialogue using multimodal information, such as a PC (Personal Computer), tablet device, or smartphone.

[0018] Camera 20a is a camera that photographs the person using terminal device 10a from the front. Camera 20b is a camera that photographs the person using terminal device 10b from the front. Camera 20c is a camera that photographs two people from the side.

[0019] The analysis device 100 is an information processing device equipped with a computer configuration, or a system including multiple computers. The analysis device 100 obtains the order of images arranged chronologically by each interlocutor in the aforementioned collaborative story rearrangement task, extracts multimodal information from the dialogue, and analyzes the relationship between the multimodal information and the common ground in the dialogue based on the obtained order of images.

[0020] Note that the system configuration of analysis system 1 shown in Figure 1 is just one example. For example, the functions of the analysis device 100 may be provided by terminal device 10a or terminal device 10b. Also, the functions of the analysis device 100 may be distributed among multiple devices.

[0021] <Functional Configuration> (Functional configuration of the analysis device) Figure 2 shows an example of the functional configuration of the analysis device 100 according to this embodiment. The analysis device 100 realizes each of the functional configurations shown in Figure 2 by executing a predetermined program on the computer provided in the analysis device 100. In the example in Figure 2, the analysis device 100 has each of the functional configurations such as a communication unit 201, an acquisition unit 202, an extraction unit 203, an analysis unit 204, an output unit 205, and a storage unit 206. At least a part of each of the above functional configurations may be realized by hardware.

[0022] The communication unit 201 performs communication processing to communicate with terminal devices 10a, 10b, and cameras 20a to 20c, etc.

[0023] The acquisition unit 202 performs an acquisition process to obtain the order of images extracted from the video and arranged chronologically by each participant based on the dialogue using multimodal information between the participants. Preferably, the acquisition unit 202 obtains the order of images arranged chronologically by each participant, who has cut out images at different points in time for each participant and arranged them randomly, by repeating the dialogue and rearrangement phases at predetermined time intervals, from the terminal device 10 or the like.

[0024] The extraction unit 203 performs an extraction process to extract multimodal information from the dialogue between the interlocutors. For example, the extraction unit 203 extracts multimodal information such as gestures and / or facial expressions from the video footage of the interlocutors captured by cameras 20a to 20c.

[0025] The analysis unit 204 performs an analysis process that analyzes the relationship between the multimodal information extracted by the extraction unit 203 and the common foundation in the dialogue, based on the order in which the images acquired by the acquisition unit 202 are acquired.

[0026] For example, the analysis unit 204 quantifies the impact of the multimodal information extracted by the extraction unit 203 on the common base based on the order of the images acquired by the acquisition unit 202. Preferably, the analysis unit 204 quantifies the impact of the multimodal information extracted by the extraction unit 203 on the common base using Kendall's rank correlation coefficient obtained from the order of the images acquired by the acquisition unit 202.

[0027] Preferably, the analysis unit 204 visualizes, during the dialogue, the areas where a common infrastructure has been established and / or where a common infrastructure has not been established.

[0028] Preferably, the analysis unit 204 analyzes multimodal information useful for constructing a common foundation by associating the feature quantities of multimodal information obtained from a predetermined section of the dialogue with the quantified common foundation.

[0029] The output unit 205 performs output processing to output the analysis results from the analysis unit 204. The output unit 205 may display the analysis results on the display unit of the analysis device 100, or it may transmit the analysis results to an external device such as the terminal device 10.

[0030] The memory unit 206 stores, for example, the order of images arranged chronologically by each interlocutor, acquired by the acquisition unit 202, and video footage of the interlocutors taken by the camera 20.

[0031] Note that the functional configuration of the analysis device 100 shown in Figure 2 is just one example. For example, at least some of the functional configurations of the analysis device 100 may be provided by the terminal device 10. Also, the functional configurations of the analysis device 100 may be distributed among multiple devices. In short, each functional configuration of the analysis device 100 shown in Figure 2 only needs to be provided by any of the devices included in the analysis system 1.

[0032] (Functional configuration of terminal devices) Figure 3 shows an example of the functional configuration of the terminal device 10 according to this embodiment. The terminal device 10 realizes each of the functional configurations shown in Figure 3 by executing a predetermined program on the computer provided in the terminal device 10. In the example in Figure 3, the terminal device 10 has a communication unit 301, a display control unit 302, an operation reception unit 303, and a storage unit 304, etc. At least a part of each of the above functional configurations may be realized by hardware.

[0033] The communication unit 301 performs communication processing to communicate with the analysis device 100, etc. The display control unit 302 performs display control processing to display various display screens, described later, on a display device such as a display provided by the terminal device 10. The operation reception unit 303 performs operation reception processing to receive operations on the display screens etc. displayed by the display control unit 302. The storage unit 304 stores various information, data, programs, etc.

[0034] Note that the functional configuration of the terminal device 10 shown in Figure 3 is just one example. For example, the terminal device 10 may have at least some of the functional configurations of the analysis device 100 shown in Figure 2.

[0035] (Summary of the process) In the collaborative narrative rearrangement task, the objects manipulated by the interlocutors are images captured from a predetermined video.

[0036] Figure 4 shows an example of images extracted from video according to this embodiment. In the example in Figure 4, the image 400 extracted from the video contains 20 images 401 captured from the same video. Each interlocutor is given 10 images, which are alternately selected from the 20 images 401. Here, in order to prevent the order from being determined from the image features, each speaker is given different images.

[0037] In the collaborative narrative rearrangement task, participants infer the original video through dialogue and rearrange the images assigned to them to match the chronological order of the original video. Participants must use their imagination and communicate about parts of the video they do not have. This facilitates communication to resolve misunderstandings. Furthermore, the communication phase and the image rearrangement phase are clearly separated. This eliminates the suppression of physical expression in communication that can occur when work and communication are performed simultaneously.

[0038] Furthermore, when comparing the image rearrangement results of each participant, the degree of agreement in the image order is evaluated. The degree of agreement in the image order can be calculated, for example, using the well-known Kendall rank correlation coefficient. For example, analysis system 1 calculates how well each participant's image order matches the correct order and averages them. Alternatively, since 20 images are selected alternately in this case, it can be considered that the 20 images are divided into 10 scenes, and one image is assigned to each scene. In this case, the similarity of the scene rearrangements between the two participants can be measured using Kendall rank correlation coefficient. In this case, other measures that can quantify the rearrangement may also be used. Here, the value of Kendall rank correlation coefficient can be considered as a quantification of the common ground.

[0039] (Example of an experimental environment) Figure 5 shows an example of the experimental environment according to this embodiment. Each participant sits at the desk 501 in the center of image 500 and performs the practice phase, confirmation phase, and work phase, which will be described later. A partition 502 is placed between the desks 501 to prevent dialogue during the practice phase, confirmation phase, and work phase. During the transmission phase, participants stand at the positions of the marks 503 on the floor and engage in dialogue with other participants using multimodal information. In addition, a camera 20a is placed between the two marks 503 to photograph the participant using terminal device 10a from the front, a camera 20b is placed to photograph the participant using terminal device 10b from the front, and a camera 20c is placed to photograph the two participants from the side.

[0040] Furthermore, an experimental tool was created for conducting the experiment. The experimental tool displays 10 images on the browser of terminal device 10 and allows users to rearrange the images by dragging and dropping them with a mouse. The experimental tool records the operation process in a file. The information recorded in the file includes, for example, the image dragged, the start time of the drag, the order of the images during the drag, the end time of the drop, the order of the images when dropped, and timestamps for both the drag and drop. The experimental tool corresponds to, for example, the display control unit 302 and the operation reception unit 303 in Figure 3.

[0041] <Processing flow> Next, the processing flow of the analysis method according to this embodiment will be described in detail. Figure 6 is a flowchart showing an example of the processing of the analysis system according to this embodiment.

[0042] In step S601, the analysis system 1 executes the practice phase. For example, the display control unit 302 of terminal devices 10a and 10b displays a practice phase display screen 700 as shown in Figure 7. This display screen 700 shows numbered images 701 from (1) to (10) arranged in a random order. Each participant can practice rearranging the order of the images 701 on this display screen 700. Each participant can also start the experiment by selecting the start button 702.

[0043] In step S602, the analysis system 1 executes the verification phase. For example, the display control unit 302 of the terminal devices 10a and 10b displays a display screen 800 for the verification phase, as shown in Figure 8. This display screen 800 displays 10 images 801, which are alternately selected from 20 images 401, as shown in Figure 4, in a random order. During the verification phase, each participant verifies the content of the given images 801 for, for example, one minute. Note that during the verification phase, participants cannot rearrange the images 801.

[0044] In step S603, the analysis system 1 initializes counter i to 1 and executes the processing from step S604 onward.

[0045] In step S604, the analysis system 1 executes the transmission phase. In the transmission phase, for example, as shown in Figure 9, each participant 901a and 901b stands at the position of the mark 503 on the floor and transmits information necessary for rearranging the images to the other participant for two minutes through dialogue using multimodal information such as gestures and / or facial expressions. At this time, the communication unit 201 of the analysis device 100 acquires video footage of each participant 901a and 901b taken by cameras 20a to 20c and stores it in the storage unit 206, etc.

[0046] In step S605, the analysis system 1 executes the work phase. For example, the display control unit 302 of the terminal devices 10a and 10b displays a work phase display screen 1000 on the display unit, as shown in Figure 10. This display screen 1000 displays 10 images 1001, similar to the confirmation phase display screen 800 described in Figure 8, which can be rearranged. In the work phase, each participant rearranges the 10 images 1001 in chronological order for one minute based on the information transmitted in the transmission phase.

[0047] In step S606, the analysis system 1 determines whether the value of counter i is equal to or greater than a predetermined number of executions N (for example, N=5). If the value of i is not equal to or greater than N, the analysis system 1 proceeds to step S607. On the other hand, if the value of i is equal to or greater than N, the analysis system 1 proceeds to step S608.

[0048] When the process moves to step S607, the analysis system 1 adds 1 to counter i and returns to step S604.

[0049] On the other hand, when the process moves to step S608, the analysis system 1 performs an analysis process that analyzes the relationship between multimodal information between the dialoguers and the common ground in the dialogue. For example, the analysis system 1 performs an analysis process as shown in Figure 11.

[0050] Figure 11 is a flowchart illustrating an example of the analysis process according to this embodiment. This process shows, for example, an example of the analysis process performed by the analysis device 100 in step S608 of Figure 6.

[0051] In step S1101, the acquisition unit 202 obtains the order of the images sorted by each participant in the processing steps S604 to S607 in Figure 6. For example, the acquisition unit 202 obtains the order of the images sorted by each participant from a file in which the aforementioned experimental tool has recorded the process of operation.

[0052] In step S1102, the extraction unit 203 extracts multimodal information from the dialogue using multimodal information between the interlocutors. For example, in the transmission phase of step D604 in Figure 6, the extraction unit 203 extracts the number and types of physical interactions such as gestures and / or facial expressions of each interlocutor 901a, 901b from the video stored in the memory unit 206 as multimodal information. This multimodal information can be extracted, for example, by a machine learning model built on training data to which the video has been labeled into the following six classes.

[0053] As a concrete example, according to previous research, gestures can be classified into six classes: Adaptor, Emblem, Deictic, Iconic, Metaphoric, and Beat (see, for example, Reference 2). The extraction unit 203 extracts the number of expressions of these six classes of gestures from a dialogue using multimodal information between interlocutors.

[0054] In step S1103, the analysis unit 204 analyzes the relationship between the multimodal information extracted by the extraction unit 203 and the common foundation in the dialogue, based on the order of the images acquired by the acquisition unit 202. For example, the analysis unit 204 uses Kendall's rank correlation coefficient obtained from the order of the images acquired by the acquisition unit 202 to quantify the influence of the multimodal information extracted by the extraction unit 203 on the common foundation.

[0055] For example, the analysis unit 204 calculates how well the sequence of images for each interlocutor matches the correct sequence and averages them. Alternatively, the analysis unit 204 may measure the similarity of the scene rearrangements for each interlocutor using Kendall's rank correlation coefficient. The analysis unit 204 may also use other measures that can quantify the rearrangements.

[0056] Now, let's return to Figure 6 and continue explaining the flowchart in Figure 6. In step S609 of Figure 6, the output unit 205 outputs the analysis results from the analysis unit 204. For example, the output unit 205 outputs a graph 1200 as an analysis result, as shown in Figure 12. The output destination for the analysis results may be a display device such as the display of the analysis device 100, or it may be the storage unit 206. Alternatively, the output destination for the analysis results may be an external device such as the terminal device 10.

[0057] Figure 12 shows an example of the analysis results according to this embodiment. This graph 1200 represents the process of building a common ground between certain interlocutors, with the horizontal axis representing elapsed time and the vertical axis representing Kendall's rank correlation coefficient as described above.

[0058] In the example in Figure 12, the first minute is the verification phase 1201, followed by a 2-minute transmission phase 1202 and a 1-minute work phase 1203, which are repeated five times each. Since no image swapping occurs between the verification phase 1201 and the transmission phase 1202, Kendall's rank correlation coefficient 1204 does not change.

[0059] In this way, by observing the number and types of physical interactions (multimodal information) that occur in each phase and comparing them with Kendall's rank correlation coefficient score for that phase, we can consider the physical elements that facilitate the construction of a common infrastructure.

[0060] Observations of physical interactions can be obtained using recordings from cameras 20a-20c. For example, according to previous research (Reference 2), gestures are divided into six classes: Adaptor, Emblem, Deictic, Iconic, Metaphoric, and Beat. By examining the relationship between the number of expressions of these gestures and Kendall's rank correlation coefficient, it is possible to clarify which gestures are useful for building a common infrastructure. Furthermore, by presenting the phases in which Kendall's rank correlation coefficient improves, it is expected that useful insights for building a common infrastructure can be obtained.

[0061] In this experiment, when participants successfully established a common ground, or when the establishment of a common ground between participants had progressed, an increase in the positive change in Kendall's rank correlation coefficient was observed when they engaged in dialogue to confirm the contextual relationships between scenes they assumed, or the overall flow. This suggests that estimating the original footage was equivalent to an attempt to establish a common ground, and therefore led to an increase in the score. On the other hand, pairs or phases that showed a negative change in Kendall's rank correlation coefficient exhibited dialogue where participants communicated about each other's scenes, but did not reach the stage of integrating them, or attempted to integrate them but could not reach an agreement. By analyzing these situations in detail, it is expected that effective communication using multimodal information will become possible.

[0062] Thus, according to this embodiment, in a dialogue using multimodal information, it is possible to visualize the areas where a common infrastructure has been established and / or where such a common infrastructure has not been established. Furthermore, according to this embodiment, by associating the feature quantities of multimodal information obtained from a predetermined section of the dialogue with the quantified common infrastructure, it is possible to perform analysis of multimodal information that is useful for constructing a common infrastructure.

[0063] <Hardware Configuration> The analysis device 100 and terminal device 10 according to this embodiment have, for example, a computer 1300 hardware configuration as shown in Figure 13. The analysis device 100 may be composed of multiple computers 1300. Furthermore, the computers constituting the analysis device 100 are not limited to physical machines, but may be, for example, virtual machines on the cloud.

[0064] Figure 13 shows an example of a computer hardware configuration. In the example in Figure 13, the computer 1300 includes a drive unit 1301, an auxiliary storage device 1302, a memory unit 1303, a CPU 1304, an interface device 1305, a display device 1306, an input device 1307, and an output device 1308, all of which are interconnected via bus B. The computer 1300 may also include other processors such as a GPU (Graphics Processing Unit).

[0065] The program that enables processing on the computer 1300 is provided on a recording medium 1311, such as a CD-ROM or memory card. When the recording medium 1311 containing the program is set in the drive device 1301, the program is installed from the recording medium 1311 to the auxiliary storage device 1302 via the drive device 1301. However, the program does not necessarily have to be installed from the recording medium 1311; it may also be downloaded from another computer via a network. The auxiliary storage device 1302 stores the installed program as well as necessary files and data.

[0066] The memory device 1303 reads and stores a program from the auxiliary storage device 1302 when a program startup command is received. The CPU 1304 implements the functions of the analysis device 100 or the terminal device 10 according to the program stored in the memory device 1303. The interface device 1305 is used as an interface for connecting to a communication network, etc. The display device 1306 displays a GUI (Graphical User Interface) etc., generated by a program. The input device 1307 consists of a keyboard, mouse, buttons, and / or touch panel etc., and is used to input various operation commands. The output device 1308 outputs the calculation results.

[0067] Note that CPU1304 may be other processors such as a DSP (Digital Signal Processor), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array).

[0068] <Effects of the Embodiment> According to this embodiment, in dialogue using multimodal information, it becomes possible to understand the relationship between multimodal information and a common platform.

[0069] For example, the analysis system 1 (analysis unit 204) can quantify the impact on the common infrastructure of multimodal information based on the order in which each participant rearranges images in chronological order, based on the dialogue between participants using multimodal information. Specifically, the analysis system 1 can quantify the impact on the common infrastructure of multimodal information using Kendall's rank correlation coefficient obtained from the order in which each participant rearranges images in chronological order.

[0070] Furthermore, the analysis system 1 (analysis unit 204) can visualize, during the dialogue, the areas where a common infrastructure has been built and / or where a common infrastructure has not been built. As a specific example, the analysis system 1 can visualize the areas where a common infrastructure has been built and the areas where a common infrastructure has not been built by outputting a graph 1200 as an analysis result, as shown in Figure 12.

[0071] Furthermore, according to the analysis system 1 (analysis unit 204), by associating the feature quantities of multimodal information obtained from a predetermined section of the dialogue with the quantified common foundation, it is possible to perform analysis of multimodal information that is useful for constructing the common foundation.

[0072] Thus, according to this embodiment, it becomes possible to specifically investigate the impact on the common foundation of multimodal information such as gestures and facial expressions in face-to-face communication between people. This makes it possible to improve the performance of dialogue systems that support human-to-human communication and interact with humans using multimodal information.

[0073] <Summary of Embodiments> This specification discloses at least the following analytical systems, analytical methods, programs, and analytical devices. (Section 1) An acquisition unit that obtains the order in which images extracted from video are rearranged chronologically by each participant based on a dialogue using multimodal information between participants, An extraction unit that extracts the multimodal information from the aforementioned dialogue, An analysis unit analyzes the relationship between the multimodal information and the common platform in the dialogue based on the order of the images, An analysis system having the following features. (Section 2) The analysis system according to paragraph 1, wherein the analysis unit quantifies the impact of the multimodal information on the common platform based on the order of the images. (Section 3) The analysis device according to paragraph 2, wherein the analysis unit quantifies the influence of the multimodal information on the common platform using Kendall's rank correlation coefficient obtained from the order of the images. (Section 4) The analysis system according to paragraph 1, wherein the analysis unit visualizes, in the dialogue, the locations where the common infrastructure has been constructed and / or where the common infrastructure has not been constructed. (Section 5) The analysis device described in paragraph 1, wherein the analysis unit analyzes multimodal information useful for constructing the common infrastructure by associating the feature quantities of multimodal information obtained from a predetermined section of the dialogue with the quantified common infrastructure. (Section 6) The acquisition unit acquires the order of the images, which are cut out at different points in time for each interlocutor and arranged randomly, by having each interlocutor rearrange the images in chronological order while repeating the dialogue and rearrangement phases at predetermined intervals, as described in any of paragraphs 1 to 5. (Section 7) The process involves extracting images from video and obtaining the order in which each participant rearranges the images chronologically based on a dialogue using multimodal information between participants. The process of obtaining the multimodal information of the aforementioned dialogue, A process to analyze the relationship between the multimodal information and the common platform in the dialogue based on the order of the images, A method of analysis performed by a computer. (Section 8) A program that causes a computer to execute the analysis method described in paragraph 7, or a storage medium that stores such a program. (Section 9) An acquisition unit that obtains the order in which images extracted from video are rearranged chronologically by each participant based on a dialogue using multimodal information between participants, An extraction unit that extracts the multimodal information from the aforementioned dialogue, An analysis unit analyzes the relationship between the multimodal information and the common platform in the dialogue based on the order of the images, An analytical device having the following features.

[0074] Although this embodiment has been described above, the present invention is not limited to this specific embodiment, and various modifications and changes are possible within the scope of the gist of the invention as described in the claims.

[0075] (Reference 1) Yuki Furuya, Mitsuteru Saito, Takahiro Ogura, Wataru Mitsuda, Ryuichiro Higashinaka, and Kazuki Takashio, "Analysis of the Influence of Modality and Interpersonal Relationships on the Common Infrastructure Construction Process," Cloud Network Robotics Research Group (CNR), pp. 25-31, 2022. (Reference 2) M. Kipp, "Gesture generation by imitation: From human behavior to computer character animation", Universal-Publishers (2005). [Explanation of Symbols]

[0076] 1. Analysis System 10, 10a, 10b Terminal devices 20, 20a~20c Camera 100 Analyzer 202 Acquisition Department 203 Extraction part 204 Analysis Department 205 Output section 206 Memory section 1200 graphs 1300 Computers

Claims

1. An acquisition unit that obtains the order in which images extracted from video are rearranged chronologically by each participant based on a dialogue using multimodal information between participants, An extraction unit that extracts the multimodal information from the aforementioned dialogue, An analysis unit analyzes the relationship between the multimodal information and the common platform in the dialogue based on the order of the images, An analysis system having the following features.

2. The analysis system according to claim 1, wherein the analysis unit quantifies the influence of the multimodal information on the common platform based on the order of the images.

3. The analysis system according to claim 2, wherein the analysis unit quantifies the influence of the multimodal information on the common platform using Kendall's rank correlation coefficient obtained from the order of the images.

4. The analysis system according to claim 1, wherein the analysis unit visualizes, in the dialogue, the locations where the common infrastructure has been constructed and / or where the common infrastructure has not been constructed.

5. The analysis system according to claim 1, wherein the analysis unit analyzes multimodal information useful for constructing the common foundation by associating the feature quantities of multimodal information obtained from a predetermined section of the dialogue with a quantified common foundation.

6. The analysis system according to any one of claims 1 to 5, wherein the acquisition unit acquires the order of the images, which are cut out at different points in time for each interlocutor and arranged randomly, by having each interlocutor rearrange the images in chronological order while repeating the dialogue and rearrangement phases at predetermined time intervals.

7. The process involves extracting images from video and obtaining the order in which each participant rearranges the images chronologically based on a dialogue using multimodal information between participants. The process of obtaining the multimodal information of the aforementioned dialogue, A process to analyze the relationship between the multimodal information and the common platform in the dialogue based on the order of the images, A method of analysis performed by a computer.

8. A program that causes a computer to execute the analysis method described in claim 7.

Images