Information processing device, information processing system, information processing method, and program

Abstract

To enable visualization tailored to the unique information of each participant in the meeting. [Solution] The information processing device includes: an acquisition unit that acquires unique data indicating unique information based on the actions of each participant in the meeting; an image generation unit that generates corresponding images for each participant based on the unique data acquired by the acquisition unit and a machine learning model trained using the unique data indicating the unique information and training data including images; and an emotion analysis unit that analyzes the emotions of each participant based on the unique data acquired by the acquisition unit.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing system, an information processing method, and a program.

Background Art

[0002] In a situation where multiple people participate in an interaction such as a meeting or other gathering, for example, the content of discussions can be represented graphically using graphic recording, which can enhance shared understanding regarding the content of the gathering.

[0003] Patent Document 1 discloses, with the aim of providing a communication system, a display device, a display control method, and a display control program capable of presenting appropriate visual information related to information transmission as the information transmission via voice, text input or the like progresses, language information input means for receiving input of language information, recognition means for recognizing the input language information, and image display means for displaying an image corresponding to the input language information on a display unit that displays images, based on the recognition result from the recognition means.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the technique of Patent Document 1 only retrieves and outputs illustrations and images corresponding to language information recognized from voice or text.

[0005] The present invention has been made in view of the above points, and an object thereof is to enable visualization corresponding to the unique information of each participant in a gathering.

Means for Solving the Problems

[0006] To solve the above problems, the information processing device includes: an acquisition unit that acquires unique data indicating unique information based on the actions of each participant in the meeting; an image generation unit that generates corresponding images for each participant based on the unique data acquired by the acquisition unit and a machine learning model trained using the unique data indicating the unique information and training data including images; and an emotion analysis unit that analyzes the emotions of each participant based on the unique data acquired by the acquisition unit. [Effects of the Invention]

[0007] This makes it possible to visualize information tailored to the unique characteristics of each participant in the meeting. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the configuration of the information processing system 1 in the first embodiment. [Figure 2] This figure shows an example of the hardware configuration of the terminal device 10 in the first embodiment. [Figure 3] This figure shows an example of the functional configuration of the information processing system 1 in the first embodiment. [Figure 4] This is a sequence diagram illustrating an example of a processing procedure performed by the information processing system 1 in the first embodiment. [Figure 5] This figure shows an example of a screen displayed in the first embodiment. [Figure 6] This is a diagram showing the first specific example on screen 530. [Figure 7] This is a diagram showing the second specific example of screen 530. [Figure 8] This is a diagram showing the third specific example of screen 530. [Figure 9] This figure illustrates an example of a processing procedure performed by the information processing system 1 in the second embodiment. [Figure 10] This diagram shows an example of the screen for a collaborative meeting system. [Figure 11] This figure shows the first example of the display of the collaborative meeting system screen, including the corresponding image. [Figure 12] This figure shows a second example of the display of the collaborative meeting system screen, including the corresponding image. [Figure 13] This is a diagram showing an example of how meeting minutes data is displayed. [Figure 14] This figure shows an example of the functional configuration of the information processing system 1 in the third embodiment. [Figure 15] This figure shows an example of the hardware configuration of the terminal device 10 in the fourth embodiment. [Figure 16] This figure shows an example of the functional configuration of the information processing system 1 in the fourth embodiment. [Figure 17] This figure shows an example of the functional configuration of the information processing system 1 in the fifth embodiment. [Figure 18] This figure shows an example of displaying a corresponding image, etc., in the fifth embodiment. [Figure 19] This figure shows a second example of displaying a corresponding image or the like in the fifth embodiment. [Figure 20] This figure shows an example of displaying the probability of each emotion in a visually apparent way, with each corresponding image being linked to it. [Modes for carrying out the invention]

[0009] The embodiments of the present invention will be described below with reference to the drawings. In this embodiment, a meeting will be described as an example of a gathering, but this embodiment is also effective for other forms of gatherings, such as training sessions or classes. Figure 1 is a diagram showing an example of the configuration of the information processing system 1 in the first embodiment. Figure 1 shows two configuration examples, (A) and (B), for the information processing system 1.

[0010] (A) is a configuration that supports online meetings such as web conferences or video conferences. In (A), a terminal device 10 is provided for each participant in the meeting. Each terminal device 10 connects to a server device 40 via a communication network 100 such as the Internet.

[0011] The server device 40 is one or more computers that generate an image based on voice data collected by each terminal device 10.

[0012] (B) is a configuration corresponding to an in-person meeting. In (B), one terminal device 10 is arranged for a plurality of participants attending the meeting. A microphone 109b is connected to or built in the terminal device 10. The number of microphones 109b may be one or more. When a plurality of microphones 109b are arranged, a highly directional microphone may be arranged for each participant, so that the participant who spoke can be identified according to the microphone 109b that collected the voice. In the case of (B), the terminal device 10 generates an image based on the voice data collected by the microphone 109b.

[0013] In the following, the configuration of (A) is referred to as "configuration A", and the configuration of (B) is referred to as "configuration B".

[0014] Further, in the present embodiment, "meeting" refers to a discussion such as a conversation or dialogue held with two or more people participating, and its form is not limited to a specific one.

[0015] Figure 2 is a diagram illustrating an example of a hardware configuration of the terminal device 10 according to the first embodiment. In Figure 2, the hardware configuration of the terminal device 10 will be described, and the description of the hardware configuration of the server device 40 will be omitted since it is the same. In Figure 2, the reference numerals of the components of the terminal device 10 start with "1", and the reference numerals of the components of the server device 40 start with "4".

[0016] The terminal device 10 is constructed by a computer, and as shown in Figure 2, includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, an HD (Hard Disk) 104, an HDD (Hard Disk Drive) controller 105, a display I / F 106, and a communication I / F 107.

[0017] Of these components, the CPU 101 controls the overall operation of the terminal device 10. The ROM 102 stores programs used to drive the CPU 101, such as the IPL (Initial Program Loader). The RAM 103 is used as the work area for the CPU 101.

[0018] HD104 stores various data such as programs. The HDD controller 105 controls the reading or writing of various data to HD10 according to the control of the CPU 101.

[0019] The display I / F 106 is a circuit that displays images on the display 106a. The display 106a is a type of display unit such as a liquid crystal or organic EL (electroluminescence) that displays various information such as cursors, menus, windows, characters, or images. The communication I / F 107 is an interface used for communication with other devices.

[0020] Communication I / F107 is, for example, a NIC (Network Interface Card) that supports TCP (Transmission Control Protocol) / IP (Internet Protocol).

[0021] Furthermore, the terminal device 10 is equipped with a sensor I / F 108, an audio input / output I / F 109, an input I / F 110, a media I / F 111, and a DVD-RW (Digital Versatile Disk Rewritable) drive 112.

[0022] The sensor interface 108 is an interface for receiving detection information from various sensors. The sound input / output interface 109 is a circuit that processes the input and output of sound signals between the speaker 109a and the microphone 109b according to the control of the CPU 101. The input interface 110 is an interface for connecting a predetermined input means to the terminal device 10.

[0023] Keyboard 110a is a type of input device equipped with multiple keys for inputting characters, numbers, various instructions, etc. Mouse 110b is a type of input device used for selecting and executing various instructions, selecting processing targets, moving the cursor, and operating on the display screen, etc.

[0024] The media interface 111 controls the reading or writing (storage) of data to or from a recording medium 111a such as flash memory. The DVD-RW drive 112 controls the reading or writing of various types of data to or from a DVD-RW 112a, which is an example of a removable recording medium. Note that this is not limited to DVD-RW; DVD-R or other types may also be used. Furthermore, the DVD-RW drive 112 may be a Blu-ray drive that controls the reading or writing of various types of data to or from a Blu-ray Disc (registered trademark).

[0025] Furthermore, the terminal device 10 is equipped with a bus line 113. The bus line 113 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 101.

[0026] Furthermore, recording media such as HDs and CD-ROMs on which the above programs are stored can be provided domestically or internationally as program products. The terminal device 10 realizes the information processing method according to the present invention by executing, for example, the program according to the present invention.

[0027] Figure 3 shows an example of the functional configuration of the information processing system 1 in the first embodiment. In Figure 3, the terminal device 10 includes an input unit 11, an input data transmission unit 12, and a display control unit 13, etc. Each of these units is realized by processing that one or more programs installed in the terminal device 10 cause the CPU 101 to execute.

[0028] The input unit 11 receives input from the user (meeting participant) of the terminal device 10. For example, the input unit 11 receives information indicating the meeting agenda (topic) and the number of participants (hereinafter referred to as "meeting information"), and inputs audio data of the meeting participants' speech collected by the microphone 109b during the meeting. The input unit 11 may also receive input from participants, for example, images related to the meeting agenda (hereinafter referred to as "input images").

[0029] The input data transmission unit 12 transmits the meeting information or data received by the input unit 11 to the server device 40.

[0030] The display control unit 13 controls the display of screens and the like based on the display information transmitted from the server device 40.

[0031] On the other hand, the server device 40 includes an input data acquisition unit 41, a text generation unit 42, a sentiment analysis unit 43, an image generation unit 44, a relevance calculation unit 45, a display information generation unit 46, and a display information transmission unit 47, etc. Each of these units is realized by processing that one or more programs installed on the server device 40 cause the CPU 401 to execute. The server device 40 also utilizes an input data storage unit 451 and a model storage unit 452. Each of these storage units can be realized using, for example, an HD 404 or a storage device that can be connected to the server device 40 via a network.

[0032] The input data acquisition unit 41 acquires (receives) conference information or data transmitted from the input data transmission unit 12 and records the acquired conference information or data in the input data storage unit 451. Note that when the server device 40 has the input data acquisition unit 41 as shown in Figure 3, the input data acquisition unit 41 has a receiving function. However, in configuration B, that is, when the terminal device 10 has the input data acquisition unit 41, the input data acquisition unit 41 has an input interface function.

[0033] The text generation unit 42 generates text data (i.e., text data indicating the content of what each participant in the meeting said) from the audio data acquired by the input data acquisition unit 41, for each participant.

[0034] The emotion analysis unit 43 analyzes the emotions of each participant and generates emotion data based on the text data (an example of unique data) representing the content of the participant's utterances, which is generated by the text generation unit 42 for each participant, and the emotion model m1 stored in the model storage unit 452. Emotion data is, for example, data that shows the probability of each type of emotion (e.g., anger, disgust, anxiety, calmness, sadness, joy, etc.).

[0035] The emotion model m1 is a machine learning model that generates emotion data from unique data (text data) that represents the unique information of each participant in a meeting. The emotion model m1 is trained, for example, using text data that represents the content of speech and training data that includes emotions. This training data includes, for example, training text data as input and emotions as the correct answers for the output. For example, the emotion model m1 is trained so that the emotion data it generates when inputting text data contained in the training data maximizes the probability of the correct emotions contained in the training data. The emotion model m1 may also be a machine learning model that generates emotion data using physical features such as "voice intonation" and "voice volume" that can be extracted from the audio data acquired by the input data acquisition unit 41 as input. In this case, the training data is a pair of these features and the correct emotions. The emotion analysis unit 43 may extract physical features such as "voice intonation" and "voice volume" from the audio data acquired by the input data acquisition unit 41 and generate emotion data based on these features and the emotion model m1.

[0036] The image generation unit 44 generates an image corresponding to the text data (hereinafter referred to as the "corresponding image") for each participant, based on the text data generated by the text generation unit 42 for each participant and the image generation model m2 stored in the model storage unit 452. Specifically, the image generation unit 44 inputs the text data pertaining to that participant into the image generation model m2 for each participant, and acquires the image generated by the image generation model m2 based on the text data as the corresponding image.

[0037] Here, the text showing the content of the participants' speech is an example of unique information based on each participant's behavior at that particular meeting (gathering), and the text data showing the content of the participants' speech is an example of unique data showing each participant's unique information.

[0038] The unique information of each meeting participant may include, in addition to, or instead of, text indicating the content of the participant's speech, at least one of the following: characters or actions entered by the participant, image information of each participant captured by a camera, and biometric information of each participant detected by a biometric information detection unit.

[0039] Image generation model m2 is an example of a machine learning model (generative AI) that generates images from unique data representing the unique information of each meeting participant. Specifically, it is a machine learning model (generative AI) that generates images from text data. Image generation model m2 is trained using training data that includes text data and images. This training data includes, for example, training text data as input and images as the correct output. For example, the image generation model m2 may be trained so that the image it generates when inputting text data from the training data approaches the correct image from the training data. Commercially available models such as Stable Diffusion and DALL·E, or independently developed models, may be used as image generation model m2.

[0040] The image generation unit 44 also generates an integrated corresponding image based on the text data of each of the multiple participants and the image generation model m2. Specifically, the image generation unit 44 inputs the set of multiple text data into the image generation model m2 and obtains the image that the image generation model m2 generates based on the set as the integrated corresponding image. In other words, a corresponding image is generated for each participant, whereas a single integrated corresponding image is generated for all participants.

[0041] Furthermore, if an input image (an image entered in relation to the meeting) is entered, the image generation unit 44 may generate a corresponding image for each participant based on the text data for each participant, the input image, and the image generation model m2, and may also generate an integrated corresponding image corresponding to the set of text data for multiple participants and the input image. In this case, the image generation unit 44 inputs the text data and input image pertaining to each participant into the image generation model m2, and obtains the image generated by the image generation model m2 based on the text data and input image as the corresponding image. In addition, the image generation model m2 inputs a set of multiple such text data into the image generation model m2, and obtains the image generated by the image generation model m2 based on the set as the integrated corresponding image. In this case, the corresponding image and integrated corresponding image may be, for example, an image in which modifications have been made to the input image according to the content of the participant's speech.

[0042] The relevance calculation unit 45 calculates the degree of relevance between each corresponding image generated by the image generation unit 44 and other images. Relevance is an index that indicates the degree of relevance. "Other images" for a given corresponding image are, for example, other corresponding images. In this case, the degree of relevance between the corresponding images is calculated. "Other images" may also be integrated corresponding images. In this case, the degree of relevance between each corresponding image and the integrated corresponding image is calculated. "Other images" may also be input images or images generated in advance based on predetermined text data.

[0043] The display information generation unit 46 generates display information that displays each corresponding image (for example, display information of a screen including the corresponding images), and associates each participant's emotional data with each corresponding image. In this case, if the display information generation unit 46 can understand the correspondence between each participant and the corresponding image, it may generate display information that associates participant identification information to identify the participant, the corresponding image, and the emotional data. Associating emotional data with a certain corresponding image is done, for example, by processing the content of the corresponding image based on the emotional data. Processing means, for example, blurring the image or correcting the color of the image. The display information generation unit 46 may also generate the display information that shows the degree of relevance in association with the participant identification information and the corresponding image.

[0044] The display information transmission unit 47 transmits the display information generated by the display information generation unit 46 to the terminal device 10.

[0045] Furthermore, if the information processing system 1 has configuration B, then the parts that the server device 40 has in Figure 3 may be provided by the terminal device 10.

[0046] The following describes the processing procedures performed by the information processing system 1. Figure 4 is a sequence diagram illustrating an example of the processing procedures performed by the information processing system 1 in the first embodiment. Figure 4 shows an example of the processing procedures when the information processing system 1 has configuration A, but if the information processing system 1 has configuration B, the terminal device 10 may also perform the processing procedures of the server device 40.

[0047] In step S101, the terminal device 10 receives input of meeting information from one of the meeting participants (for example, the organizer) via the displayed screen. The meeting information includes, for example, the agenda and the number of participants.

[0048] Figure 5 shows an example of a screen displayed in the first embodiment. In step S101, screen 510 is displayed on the terminal device 10. Screen 510 may be displayed based on display information received by the terminal device 10 from the server device 40 before the execution of step S101. Screen 510 includes an input area 511, an input area 512, and a start button 513, etc. The input area 511 is an area for receiving input of agenda items. The input area 512 is an area for receiving input of the number of participants. The start button 513 is a button for receiving a request to start a meeting.

[0049] When an agenda item is entered in input area 511, the number of participants is entered in input area 512, and the start button 513 is pressed, the input data transmission unit 12 transmits meeting information, including the entered agenda item and number of participants, to the server device 40 (S102). An input image may also be entered via screen 510. For example, a file for storing the input image may be selected via screen 510, and the input image stored in that file may be transmitted to the server device 40 along with the meeting information.

[0050] When the input data acquisition unit 41 of the server device 40 receives the conference information, it records the conference information in the input data storage unit 451 (S103).

[0051] Next, the display information transmission unit 47 transmits, for example, display information indicating the start of a meeting to each terminal device 10 (S104). The display control unit 13 of each terminal device 10 displays the screen 520 in Figure 5 based on the display information. The screen 520 contains a message instructing the start of a discussion on the agenda. After referring to the message, each participant starts the meeting (hereinafter referred to as the "target meeting") and speaks about their understanding and thoughts on the agenda. If one microphone 109b is provided for multiple people in the target meeting, it is possible to apply known techniques to separate the voice of each participant who speaks. However, considering the ease of separating the voice of each participant's speech, the meeting may proceed so that each participant speaks one at a time. If a microphone 109b is provided for each participant in the target meeting (including in the case of an online meeting), it may be possible to allow multiple people to speak at once.

[0052] Note that steps S101 to S104 may be skipped, and the meeting may begin without them being executed.

[0053] During the meeting, the input unit 11 of the terminal device 10 receives audio data input from the microphone 109b (S105). The input data transmission unit 12 transmits the audio data to the server device 40 (S106). The input data acquisition unit 41 of the server device 40 records the received audio data in the input data storage unit 451 (S107). Steps S105 to S107 are repeated until the meeting ends. Therefore, the input data storage unit 451 records audio data until the meeting ends. If one microphone 109b is provided for multiple people, one audio data is recorded for multiple participants. On the other hand, if a microphone 109b is provided for each participant (including in the case of an online meeting), audio data is recorded for each participant. In this case, the input data acquisition unit 41 may record audio data from the terminal device 10 or microphone 109b related to the participant in the input data storage unit 451 in association with the participant's identification information (hereinafter referred to as "participant identification information"). The correspondence between each terminal device 10 or each microphone 109b and participant identification information can be set in advance.

[0054] Furthermore, the duration of the meeting may be pre-set in the server device 40, or it may be a variable time until the power to the microphone 109b or terminal device 10 is turned off.

[0055] Upon the conclusion of the meeting, the text generation unit 42 generates text data from the audio data by performing transcription, text formatting, etc., using natural language processing (S108). In this case, if the audio data is distinguished by participant, the text generation unit 42 generates text data for each participant by generating text data for each audio data. If the audio data is not distinguished by participant, the text generation unit 42 also performs speaker separation on the audio data and generates text data for each participant. In this case, speaker separation may be performed using the number of participants included in the meeting information. Text formatting includes, for example, inserting punctuation marks and deleting unnecessary characters and spaces.

[0056] Next, the emotion analysis unit 43 generates emotion data for each participant based on text data representing the participant's utterances and the emotion model m1. Specifically, the emotion analysis unit 43 inputs text data representing the participant's utterances into the emotion model m1 for each participant and obtains the emotion data output by the emotion model m1. The emotion analysis unit 43 may also generate overall emotion data for all participants. In this case, the emotion analysis unit 43 may input the set of text data for all participants into the emotion model m1 and obtain the emotion data output by the emotion model m1 as overall emotion data.

[0057] Next, the image generation unit 44 generates prompts to be input to the image generation model m2 based on the agenda and text data for each participant (S110). At this time, the image generation unit 44 generates prompts for each piece of text data (hereinafter referred to as "corresponding prompts") and prompts based on the set of all text data (hereinafter referred to as "integrated corresponding prompts"). A corresponding prompt based on a given piece of text data is text data that causes the image generation model m2 to generate a corresponding image based on the agenda and that text data. An integrated corresponding prompt is text data that causes the image generation model m2 to generate an integrated corresponding image based on the agenda and the utterances of all participants. In addition, for both corresponding prompts and integrated corresponding prompts, the image generation unit 44 may generate prompts using words of relatively high importance (hereinafter referred to as "important words") and audio data by performing morphological analysis and important word extraction on the source text data. Furthermore, if the image generation model m2 used supports prompts in a specific language (for example, English), the image generation unit 44 may perform language conversion (translation) for each prompt and convert the prompts to the language supported by the image generation model m2. In addition, the image generation unit 44 may weight each important word in the prompt based on the importance of each important word obtained in important word extraction. By doing so, the influence of important words on the generated image data can be increased.

[0058] For example, a prompt is generated that combines the topic and key words, as shown below.

[0059] "<Agenda> inspired by ('work': 2.692), ('companion': 1.923), 'new': 1.538), ('future': 1.538), ('ai': 1.154), ('robot': 1.154)" Here, <topic> is a string indicating the topic recorded as meeting information. "work", "companion", "new", "future", "ai", and "robot" are important words. The number paired with an important word indicates its importance level. Note that the above prompt format follows the grammar of the text that the image generation model m2 is expected to input as a prompt. That is, the image generation model m2 already knows that the words and their importance levels are contained within the parentheses. Also, this format differs depending on the image generation model used.

[0060] Important words can be extracted using various publicly known techniques, such as TF-IDF (Term Frequency-Inverse Document Frequency), TextRank, RAKE (Rapid Automatic Keyword Extraction), and Transformer-based analysis.

[0061] Next, the image generation unit 44 inputs the corresponding prompt to the image generation model m2 for each corresponding prompt and obtains the image (corresponding image) generated by the image generation model m2 based on the corresponding prompt (S111). Thus, a corresponding image is generated for each corresponding prompt (for each participant).

[0062] Next, the image generation unit 44 inputs an integrated response prompt to the image generation model m2 and obtains the image (integrated response image) generated by the image generation model m2 based on the integrated response prompt (S112).

[0063] The processes for generating corresponding images for each participant (S110, S111) and generating an integrated corresponding image (S110, S112) may be executed in parallel for each image.

[0064] Alternatively, only one of the corresponding image or the integrated corresponding image may be generated. In this case, the user may be able to configure which one is generated. When a corresponding image is generated for each participant, the seed value of the image generation model m2 is fixed so that exactly the same image data is generated when exactly the same prompt is entered.

[0065] If corresponding images are generated, the relevance calculation unit 45 calculates the degree of relevance of each corresponding image with other images (S113). Here, the other images are, as described above, other corresponding images, integrated corresponding images, input images, or images generated in advance based on predetermined text data. For example, if there are three corresponding images, images 1 to 3, and the degree of relevance between the corresponding images is calculated, then three different degrees of relevance will be calculated: between image 1 and image 2, between image 1 and image 3, and between image 2 and image 3.

[0066] As a measure of relevance, for example, the correlation coefficient may be used. There are various methods for calculating the correlation coefficient, such as MSE (Mean Squared Error), SSIM (Structural Similarity Index), histogram comparison, feature vector comparison using a neural network, and comparison of objects detected in the image, and any of these may be used, or a value obtained by combining the results of multiple calculation methods using a linear sum may be adopted.

[0067] Furthermore, the correlation value may be normalized to a range of 0 to 100 for ease of understanding for the user. In this case, a larger value indicates a higher correlation (stronger association).

[0068] Furthermore, the relevance calculation unit 45 calculates the average of each relevance level as the common understanding level. The common understanding level is an indicator that shows the degree to which each participant has a common understanding of the agenda item.

[0069] If no corresponding image is generated (i.e., only one integrated corresponding image is generated), step S113 does not need to be performed.

[0070] Next, the display information generation unit 46 generates display information (for example, a screen) for displaying the generated images (each corresponding image or an integrated corresponding image, or each corresponding image and an integrated corresponding image) in association with emotion data (S114). At this time, if a degree of relevance has been generated for each corresponding image, the display information generation unit 46 displays the degree of relevance associated with each corresponding image, and also generates display information that displays the degree of common understanding. Furthermore, if the correspondence between each corresponding image and each participant (participant identification information) is clear, the display information generation unit 46 may generate display information that further associates the participant identification information with each corresponding image. In addition, the display information generation unit 46 processes the generated images (each corresponding image or an integrated corresponding image, or each corresponding image and an integrated corresponding image) based on the emotion indicated by the emotion data generated for the participant corresponding to the image (the emotion with the highest probability in the emotion data).

[0071] Next, the display information transmission unit 47 transmits the generated display information to each terminal device 10 (S115). When the display control unit 13 of each terminal device 10 receives the display information, it displays screen 530-1 or screen 530-2 (hereinafter referred to as "screen 530" when neither screen is distinguished) of Figure 5 based on the display information (S116). Screen 530 may be displayed on the display 106a or on a projector connected to the terminal device 10.

[0072] Screen 530-1 is an example of a screen displayed when there are three participants in the target meeting and a corresponding image is generated for each participant. Screen 530-1 includes, in addition to the corresponding images, the degree of relevance between the corresponding images (i.e., the degree of relevance for each of the three pairs of corresponding images) and the degree of common understanding as display elements. If the correspondence between each corresponding image and the participant can be understood, participant identification information may be associated with each corresponding image and displayed.

[0073] Screen 530-2 is an example of the screen displayed when a single unified corresponding image is generated for all participants.

[0074] If both a corresponding image and an integrated corresponding image are generated, a screen containing the content displayed on screen 530-1 and the content displayed on screen 530-2 may be displayed.

[0075] Here is a specific example of screen 530, which displays corresponding images or integrated corresponding images generated when three people hold a meeting on the topic of "The Office of the Future."

[0076] Figure 6 shows the first specific example of screen 530. Screen 530-1 shown in Figure 6 has the same configuration as screen 530-1 shown in Figure 5. Figure 6 shows an example of how screen 530-1 would look if three people said exactly the same thing. In this case, the content of the three corresponding images would be the same, and the degree of relevance between the corresponding images would all be 100%. The average degree of relevance, which is the common understanding, would also be 100%. However, each corresponding image has been processed to match the emotion indicated by the emotion data of the corresponding participant. For example, since the emotion of the participant corresponding to image 1 was "anxiety," image 1 has been corrected to a blurred image. Although not shown in the illustration, images 2 and 3 have also been corrected to match the colors of the emotions they correspond to. For example, if the emotion of the participant corresponding to image 2 is "joy," the color of image 2 will be corrected to a warm color. For example, if the emotion of the participant corresponding to image 3 is "calm," the color of image 3 will be corrected to a cool color.

[0077] Figure 7 shows a second specific example of screen 530. Screen 530-1, shown in Figure 7, has the same configuration as screen 530-1 shown in Figure 5, similar to Figure 6. However, Figure 7 shows an example of how screen 530-1 is displayed when there are discrepancies in the statements of the three participants. In this case, the content of the three corresponding images differs according to the statements of each participant. Therefore, the degree of relevance between the corresponding images is not 100%, and the degree of common understanding is 76%. In this example, the degree of common understanding was calculated by normalizing the linear sum of the correlation coefficient of the feature vectors and the correlation coefficient when the grayscale image is converted to one dimension.

[0078] From the content and relevance (85%) of the corresponding images in Figure 7, it can be seen that Image 2 (center) and Image 3 (right) are corresponding images based on similar statements. Furthermore, although the content of Image 1 (left) and Images 2 and 3 appear quite different at first glance, the commonality in the brightness patterns when converted to grayscale indicates that the perspective on "the office of the future" is divided into two directions: an overview (exterior view of the office) and a subjective view (interior view of the office), which can be useful for subsequent discussions. In addition, in Figure 7, each corresponding image is processed to reflect the emotions indicated by the corresponding participant's emotion data.

[0079] Figure 8 shows a third specific example of screen 530. Screen 530-2 shown in Figure 8 has the same configuration as screen 530-2 shown in Figure 5. That is, the single image included in screen 530-2 in Figure 8 is an integrated corresponding image based on the statements of three people. Since this image is generated with the statements of the three people mixed together, it is possible to stimulate or deepen discussions based on the visualized information, for example, by saying, "This part is different from what I imagined!" or "I can relate to this part!" Furthermore, if sentiment data is generated for the whole, the integrated corresponding image may be processed in a manner that corresponds to the sentiment indicated by that sentiment data.

[0080] Participants who view screens 530 as shown in Figures 6 to 8 can quantitatively grasp their level of shared understanding numerically (in the case of Figure 6 or Figure 7), and can also share their respective perspectives on understanding in a visible form as images.

[0081] As a variation of this embodiment, the degree of common understanding may be displayed without showing it as a numerical value, and corresponding images may be displayed instead. Even in this case, by visually comparing the corresponding image based on one's own statements with the corresponding images related to the statements of others or integrated corresponding images, it is possible to visually recognize the difference between the impression of one's own statements and the impression of the statements of others or the statements as a whole.

[0082] As described above, according to the first embodiment, an image (such as a corresponding image or an integrated image) is generated using text data representing the content spoken by each participant in a meeting and the image generation model m2, and the image is displayed. In other words, the image is dynamically generated according to the content of the speech. Furthermore, each image is processed based on the emotions of each participant. Therefore, it is possible to visualize information that is specific to each participant in a meeting or other gathering. For example, it is possible to visualize qualitative events contained in speeches in meetings or other gatherings. Qualitative events refer to qualitative, invisible, and ambiguous events, such as a person's thoughts, psychology, or emotions contained in their speech.

[0083] Next, a second embodiment will be described. The differences between the second embodiment and the first embodiment will be described. Therefore, points not specifically mentioned may be the same as in the first embodiment. In addition, the functions and configurations described in the second embodiment may be combined with the functions and configurations described in the first embodiment.

[0084] In the second embodiment, we will describe an example in which the information processing system 1 collaborates with an online meeting system such as Zoom® or Teams®, generates prompts based on text data from each speaker at certain time intervals generated by the transcription function provided by the collaborating online meeting system (hereinafter referred to as the "collaborating meeting system"), and generates corresponding images for each speaker.

[0085] Figure 9 is a diagram illustrating an example of a processing procedure performed by the information processing system 1 in the second embodiment. In Figure 9, steps that are the same as or corresponding to those in Figure 4 are indicated in parentheses with the step number from Figure 4.

[0086] In step S201, the input data acquisition unit 41 waits for a certain period of time (for example, 20 seconds). The terminal device 10 of each participant in the online meeting displays the screen of the collaborative meeting system.

[0087] Figure 10 shows an example of a collaborative meeting system screen. In Figure 10, screen 610 is an example of a collaborative meeting system screen. Screen 610 includes a window that displays the video of each participant, and a text area 611 that contains the content of each participant's speech, transcribed into text by a transcription function, in chronological order.

[0088] After a certain period of time has elapsed, the input data acquisition unit 41 acquires text data generated for each participant in the online meeting during the most recent fixed time interval from the collaborative meeting system, associating the content of the text data with the participant identification information of the participant who spoke (S202). The text data acquired here corresponds to a string obtained by dividing the text data obtained from the start to the end of the online meeting into chronological order.

[0089] In the subsequent steps S203-S207, the same processing as in steps S109-S113 in Figure 4 is performed for each acquired text data. As a result, an integrated corresponding image for the most recent period of time and corresponding images for each participant in the online meeting are generated. Note that either or both of steps S205 and S206 may be omitted. In other words, the integrated corresponding image may not be generated. Also, the relevance score may not be calculated.

[0090] Next, the display information generation unit 46 adds the corresponding image to the screen 610 of the collaborative meeting system (S208).

[0091] Figure 11 shows a first example of the display of the collaborative conferencing system screen, including the corresponding image.

[0092] In the screen 610 shown in Figure 11, each participant's window displays a corresponding image related to that participant, which has been processed based on the participant's emotions. Each window contains the participant identification information for each participant. Therefore, in screen 610, the corresponding image and the participant identification information are displayed in association.

[0093] Figure 12 shows a second example of the display of the collaborative meeting system screen, including the corresponding images. On screen 610 shown in Figure 11, a pop-up screen 612 containing the corresponding image and participant identification information for each participant is superimposed and displayed. Note that each corresponding image has been processed based on the emotions of each participant.

[0094] Furthermore, if an integrated corresponding image or relevance score is generated, the integrated corresponding image or relevance score may also be displayed in Figures 11 and 12. For relevance scores, it is sufficient that the correspondence between the corresponding image and the relevance score is clearly shown.

[0095] Steps S201 to S208 are repeated until the end of the online meeting. Therefore, each participant can view the corresponding image of each participant at regular intervals as the meeting progresses and participate in the discussion. As a result, it is expected that discussions will become more active and new ideas will be generated.

[0096] When the online meeting ends (Yes in S209), the display information generation unit 46 adds (pastes) a thumbnail of the corresponding image generated at the time the corresponding image was generated, processed based on the emotions of the participant corresponding to that image, to the location corresponding to the timing at which each repeatedly generated corresponding image was generated (S210). The meeting minutes data is an example of display information containing strings of text data obtained from the start to the end of the online meeting, divided into time series. Alternatively, the display information generation unit 46 may generate the meeting minutes data based on the text data acquired in step S202 without using the meeting minutes system.

[0097] The meeting minutes data is transmitted to the terminal device 10 in response to a predetermined operation on the terminal device 10, and the display control unit 13 of the terminal device 10 controls the display of the meeting minutes data.

[0098] Figure 13 shows an example of how meeting minutes data is displayed. As shown in Figure 13, the meeting minutes data 620 contains text data (strings separated by time) corresponding to the content of speech during each time interval, in chronological order. In addition, at the end of each time interval (i.e., the timing of the generation of the corresponding image), the meeting minutes data 620 includes a thumbnail of the corresponding image for each participant, based on the text data and emotion data corresponding to the content of speech of each participant during that time interval. Clicking on the thumbnail image may allow for enlargement and saving of the corresponding image.

[0099] This type of meeting minutes data (620) allows for a visual review of the meeting and can serve as a memory trigger.

[0100] In addition, in the first embodiment as well, corresponding images and integrated corresponding images may be generated at regular intervals, and the corresponding images and integrated images may be displayed. Whether or not to generate corresponding images at regular intervals may be configurable by the user. In other words, the portion of the text data from the start to the end of the meeting that the image generation unit 44 uses to generate corresponding images may be changed based on the user's specification of the "start" and "end" points, or based on predetermined conditions set by the system.

[0101] If corresponding images or other elements are generated at regular intervals, and the level of relevance or shared understanding remains significantly low for an extended period, a message prompting the user to review the agenda may be displayed.

[0102] Next, a third embodiment will be described. The differences between the third embodiment and the first embodiment will be described. Therefore, points not specifically mentioned may be the same as in the first embodiment. In addition, the functions and configurations described in the third embodiment may be combined with the functions and configurations described in the first embodiment.

[0103] Figure 14 shows an example of the functional configuration of the information processing system 1 in the third embodiment. In Figure 14, the same parts as in Figure 3 are denoted by the same reference numerals, and their descriptions are omitted.

[0104] In Figure 14, the server device 40 further includes a virtual data generation unit 48. The model storage unit 452 also stores a language model m3. The virtual data generation unit 48 is realized by a process in which one or more programs installed on the server device 40 are executed by the CPU 401.

[0105] The m3 language model is a machine learning model that takes text as input and outputs a text-based document corresponding to that text. For example, an existing large-scale language model (LLM) can be used as the m3 language model.

[0106] The virtual data generation unit 48, like a chatbot, makes virtual (simulated) utterances in the target meeting as a virtual participant (hereinafter referred to as "virtual participant"). For example, at regular intervals (hereinafter referred to as "virtual utterance intervals") during the target meeting (hereinafter referred to as "virtual utterance timing"), the virtual data generation unit 48 generates unique data (text data indicating the content of utterances) that represents the virtual unique information (text indicating the content of utterances) of the virtual participant, based on the unique data (text data indicating the content of utterances) and sentiment data of other participants. The virtual data generation unit 48 uses the language model m3 when generating the unique data.

[0107] For example, each time a virtual utterance timing arrives, the virtual data generation unit 48 generates text data by performing the same processing as in step S108 for each voice data recorded in the input data storage unit 451 in step S107, which was executed during the preceding virtual utterance interval (the entire time interval or the last 1-3 minutes of the virtual utterance interval). The sentiment analysis unit 43 generates sentiment data for each participant based on the text data and the sentiment model m1. In this case, if specific sentiment data (sentiment data indicating anger) is not generated, the virtual data generation unit 48 inputs a prompt to the language model m3 that includes the text data and the instruction, "As a participant in the meeting, speak based on the agenda and the content of other participants' utterances." On the other hand, if specific sentiment data (sentiment data indicating anger) is generated, the virtual data generation unit 48 inputs a prompt to the language model m3 that includes the text data and the instruction, "As a participant in the meeting, speak based on the agenda, the content of other participants' utterances and {emotion}, in a way that suppresses {emotion} (for example, to regain composure)." Here, {emotion} is replaced with the emotion indicated by specific emotion data. By inputting such a prompt, it is expected that emotions that may disrupt the progress of the meeting, such as anger, will be suppressed. The virtual data generation unit 48 generates audio data using the text data generated by the language model m3 based on the prompt as the utterance content, outputs this audio data to each terminal device 10, and records the text data in the input data storage unit 451 in association with the identification information of the virtual participants. Note that known speech synthesis technology can be used to generate the audio data using the text data as the utterance content.

[0108] In the third embodiment, from step S109 onward, text data (speech content) relating to the virtual participant is treated in the same way as other participants. That is, emotion data and corresponding prompts are also generated for the text data relating to the virtual participant (S109, S110), and corresponding images are generated based on the corresponding prompts. Furthermore, each corresponding image and the integrated corresponding image are processed based on their respective corresponding emotion data. At this time, emotion data may also be generated for the virtual participant. The process thereafter may be the same as in the first embodiment.

[0109] Furthermore, virtual utterance timing does not necessarily have to be at fixed intervals. For example, the timing when utterances decrease in the target meeting (for example, when the number of utterances in the most recent period falls below a threshold) may be considered the virtual utterance timing.

[0110] As described above, according to the third embodiment, by adding virtual participants, it is possible to expect to revitalize discussions in the target meeting.

[0111] Next, a fourth embodiment will be described. The differences between the fourth embodiment and the first embodiment will be described. Therefore, points not specifically mentioned may be the same as in the first embodiment. In addition, the functions and configurations described in the fourth embodiment may be combined with the functions and configurations described in the first embodiment.

[0112] Figure 15 shows an example of the hardware configuration of the terminal device 10 in the fourth embodiment. In Figure 15, the terminal device 10 is connected to an imaging device 114 and a biological information detection unit 115.

[0113] The imaging device 114 is a camera that captures images including the face or posture of the user (participant in the target meeting) of the terminal device 10. The images captured by the imaging device 114 may be video or still images taken at regular time intervals.

[0114] The biometric information detection unit 115 is a sensor or the like that measures the biometric information (blood pressure, pulse, etc.) of the user of the terminal device 10 (participant of the target meeting).

[0115] In the fourth embodiment, the unique data includes a first type of unique data containing a first type of unique information, and a second type of unique data containing a second type of unique information. The first type of unique information is text indicating at least one of the characters entered by the participant during the meeting and the content of the participant's speech. The second type of unique information refers to unique information based on the reaction of any other participant to the first type of unique information. The second type of unique information includes operation information (e.g., entered characters) indicating the operation entered by the participant who reacted to the first type of unique information of any other participant, captured image information of the participant taken by the camera 114, and at least one of the respective biometric information of the participant detected by the biometric information detection unit 115. The captured image information can be displayed on the display unit or not, depending on the settings.

[0116] In other words, the imaging device 114 and the biological information detection unit 115 are used to detect responses to speech and other actions by other participants.

[0117] Figure 16 shows an example of the functional configuration of the information processing system 1 in the fourth embodiment. In Figure 16, the same reference numerals are used for parts that are the same as those in Figure 3, and their descriptions are omitted.

[0118] In Figure 16, the model memory unit 452 also stores an image generation model m5. The image generation model m5 is a machine learning model that takes as input the first type of unique data of the first participant and emotion data (the emotion indicated by it) based on the second type of unique data of the second participant, and outputs corresponding images (i.e., corresponding images related to the response to the utterance of another participant) corresponding to the first type of unique data and the second type of unique data. Therefore, the corresponding image is an image related to the response of the second participant to the utterance of the first participant.

[0119] Specifically, the image generation model m5 is trained using training data that includes emotion data based on information including text (text of what other participants have said), operation information of a participant who has responded to another participant (hereinafter referred to as "Participant R") (operation information for the terminal device 10), captured image information of Participant R (face, posture, etc.), and Participant R's biometric information (blood pressure, pulse, etc.) (hereinafter, Participant R's operation information, captured image information, and biometric information are referred to as "reaction information") as input data, and is trained using training data that includes a correct image (hereinafter referred to as "correct image"). In other words, the image generation model m5 is trained so that the image it outputs when it receives input data from the training data approaches the correct image in that training data. Here, if the emotion data related to the input data of the training data shows a positive response to another participant's utterance, the image that has a relatively high degree of relevance to the corresponding image generated by the image generation model m2 for that other participant's utterance becomes the correct image in that training data. On the other hand, if the sentiment data related to the input data of the training data shows a negative reaction to the utterances of other participants, the image that has a relatively low correlation with the corresponding image generated by the image generation model m2 for the utterances of those other participants will be considered the correct image in the training data.

[0120] Therefore, the image generation model m5 will generate a corresponding image with a relatively high degree of relevance to the corresponding image of the input text of the other participant's utterance if the input sentiment data (sentiment data based on participant R's response information) is positive towards the other participant's utterance, and will generate a corresponding image with a relatively low degree of relevance to the corresponding image of the input text of the other participant's utterance if the input sentiment data (sentiment data based on participant R's response information) is negative towards the other participant's utterance.

[0121] Furthermore, in Figure 16, the emotion analysis unit 43 uses emotion model m4 instead of emotion model m1. Emotion model m4 is a machine learning model that generates emotion data using reaction information as input. Emotion model m4 is trained, for example, using training data that includes reaction information and emotions. This training data includes, for example, training reaction information as input and emotions as correct answers for the output. For example, emotion model m4 is trained so that the emotion data it generates when inputting reaction information contained in the training data maximizes the probability of the correct emotion contained in the training data.

[0122] The processing procedure in the fourth embodiment will be explained using Figure 4. The input data transmission unit 12 of each terminal device 10 transmits the response information of the user of the terminal device 10 to the server device 40 in response to the utterance of a participant related to a user of another terminal device 10. The input data acquisition unit 41 of the server device 40 records the received response information in the input data storage unit 451 for each participant. Therefore, throughout the period until the target meeting ends, the response information of other participants is stored in the input data storage unit 451 for each audio data related to the utterance of each participant.

[0123] In step S109, the emotion analysis unit 43 generates emotion data for each participant based on the response information stored for each participant in the input data storage unit 451 and the emotion model m4.

[0124] In step S111, the image generation unit 44 further inputs emotion data generated based on the participant's reaction information to the other participant, and the text data of the other participant's speech, for each participant to the image generation model m5, and obtains the image output by the image generation model m5 as the corresponding image related to the participant's reaction.

[0125] For example, if person B smiles (a positive reaction) in response to person A's statement, then because B agrees with A's statement (is positive towards A's statement), a corresponding image with a high degree of relevance (high correlation) to the corresponding image related to A's statement will be generated for person B.

[0126] On the other hand, if B shows a dissatisfied expression (negative reaction) to what A says, then B does not agree with what A says (is negative towards what A says), and therefore, a corresponding image with a low degree of relevance to the corresponding image related to what A says will be generated for B.

[0127] Here, we have shown an example where facial expressions are used as response information. However, by labeling positive and negative responses to participants' posture, pulse rate, etc., and training the image generation model m5 with these labels, corresponding images can be generated in a similar manner to facial expressions.

[0128] Other aspects may be the same as in the first embodiment.

[0129] As a result of the above, in the fourth embodiment, corresponding images related to responses to the speech of other participants are also displayed.

[0130] The fourth embodiment may be combined with any of the above embodiments.

[0131] As described above, according to the fourth embodiment, the responses of other participants to an utterance made by one participant can also be represented by images.

[0132] Next, a fifth embodiment will be described. The differences between the fifth embodiment and the third embodiment will be described. Therefore, points not specifically mentioned may be the same as those in the third embodiment. In addition, the functions and configurations described in the fifth embodiment may be combined with the functions and configurations described in the third embodiment.

[0133] Figure 17 shows an example of the functional configuration of the information processing system 1 in the fifth embodiment. In Figure 17, the same reference numerals are used for parts that are the same as those in Figure 14, and their descriptions are omitted.

[0134] In Figure 17, the server device 40 further includes a layout determination unit 49. The layout determination unit 49 is realized by a process that one or more programs installed on the server device 40 cause the CPU 401 to execute.

[0135] The layout determination unit 49 determines the arrangement of multiple corresponding images, the size of each corresponding image, and at least one of the decorations, based on unique information (text or audio of the utterance) and corresponding images, as well as emotion data generated by the emotion analysis unit 43 for each participant.

[0136] Specifically, the layout determination unit 49 determines the arrangement of multiple corresponding images when displaying them, based on the relevance calculation unit 45 calculates the relevance between each corresponding image and other images.

[0137] The layout determination unit 49 also determines the decoration or size of the corresponding image based on at least one of the following, which are included in the unique information: the amount and volume of the participant's speech, image information showing the facial expressions of other participants captured by the camera 114, changes in the biometric information of other participants detected by the biometric information detection unit 115, and emotion data generated for each participant by the emotion analysis unit 43. In this case, the terminal device 10 only needs to have the hardware configuration shown in Figure 15, and as described in the fourth embodiment, the input data storage unit 451 should be configured to record reaction information.

[0138] In the fifth embodiment, the display information generation unit 46 generates display information that displays a plurality of corresponding images in at least one of the arrangement, size, and decoration determined by the layout determination unit 49.

[0139] Specifically, in step S114 of Figure 4, before the display information generation unit 46 generates the display information, the layout determination unit 49 determines the placement position (placement order) of each corresponding image for each participant (including virtual participants) based on the degree of relevance (degree of relevance with other images) calculated in step S113. For example, the layout determination unit 49 determines the placement position of each corresponding image such that corresponding images with a relatively high degree of relevance to a given image are relatively close to that given image. That given image may be any one of the corresponding images, a combined corresponding image, or an agenda image. An agenda image is an image generated by inputting text data indicating the agenda into the image generation model m2.

[0140] The layout determination unit 49 also increases the size of the dialogue image or adds decorations (such as highlighting) to make the corresponding image stand out, depending on the amount of speech from the participant related to that dialogue image, or the volume of speech. The layout determination unit 49 also increases the size of the corresponding image or adds decorations (such as highlighting) to make the corresponding image stand out, depending on the size of the corresponding image, depending on the size of the response (i.e., the greater the impact on other participants) indicated by the response information to the speech related to that corresponding image.

[0141] Furthermore, the amount of speech a participant delivers in a given dialogue image can be evaluated based on the number of characters in the text data corresponding to the corresponding image (or participant), or the length of the audio data corresponding to the text data. The volume of speech a participant delivers in a given dialogue image can be evaluated based on the audio data corresponding to the corresponding image (or participant). For example, the average or maximum volume of the audio data may be used as the volume of speech in that audio data. In addition, the magnitude of the response indicated by the response information can be evaluated using publicly known technology (for example, the technology disclosed at https: / / aicam.jp / tech / optical_flow).

[0142] The layout determination unit 49 may also structure the conversation based on the flow of the conversation (using utterance order and natural language processing) based on the text data (utterance content) corresponding to each corresponding image, and determine the placement of the corresponding images based on that structure. Structuring refers to, for example, (1) and (2) below.

[0143] (1) The structure of the conversation in chronological order, that is, the order of utterances. In this case, the layout determination unit 49 determines the layout based on the chronological flow of utterances corresponding to each corresponding image.

[0144] (2) This refers to creating a network structure of a conversation using a graph generated by natural language processing (knowledge graph, co-occurrence network). In this graph, each utterance becomes a node, and related utterances are connected by edges. In this case, the layout determination unit 49 determines the layout of the corresponding images based on the degree of relevance of the utterances.

[0145] In step S114, the display information generation unit 46 generates display information (for example, a screen) for displaying the generated images (each corresponding image or an integrated corresponding image, or each corresponding image and an integrated corresponding image) based on the display mode determined by the layout determination unit 49.

[0146] As a result, in the fifth embodiment, each corresponding image, etc., is displayed on each terminal device 10 in the display configuration determined by the layout determination unit 49.

[0147] Figure 18 shows an example of displaying corresponding images in the fifth embodiment. In Figure 18, the vertical direction corresponds to the time axis (passage of time). The horizontal direction corresponds to the degree of relevance to the agenda image.

[0148] In this case, the placement of each corresponding image is determined based on chronological order and the degree of relevance to the topic image. Furthermore, the size and other characteristics of each corresponding image are determined by the method described above.

[0149] In the first embodiment, an example was described in which steps S108 and onward are executed in accordance with the end of the meeting. However, steps S108 and onward may be repeated at regular intervals during the meeting. Figure 18 shows an example in which corresponding images, etc., generated at regular intervals are arranged on a time axis in such a case. Also, although Figure 18 shows an example in which agenda images are arranged at regular intervals, the agenda items corresponding to each agenda image may be the same or different. In other words, the agenda may be changed during the meeting.

[0150] In the above example, the layout determination unit 49 determines the placement position of each corresponding image based on the degree of relevance of each corresponding image. However, the layout determination unit 49 may also determine the placement position of each corresponding image and the integrated corresponding image using emotion data generated for each corresponding image (and the participant associated with it).

[0151] Figure 19 shows a second example of the display of corresponding images in the fifth embodiment. In Figure 19, the vertical direction corresponds to the time axis (passage of time), similar to Figure 18. The horizontal direction corresponds to the probability of a particular emotion. In Figure 19, the horizontal axis shows an example where corresponding images with a lower probability of "calmness" (degree of calmness) are placed to the right (i.e., corresponding images that correspond to emotional participants).

[0152] In this case, the layout determination unit 49 determines the placement of each corresponding image in chronological order and based on the degree of calmness of each participant. Furthermore, for integrated corresponding images, the layout determination unit 49 determines the placement of the integrated corresponding image by using the average probability of calmness of each participant as the probability of calmness for that integrated corresponding image. The size of each corresponding image is determined by the method described above. In addition, the display information generation unit 46 processes the display manner of each image based on the emotion data corresponding to each corresponding image and integrated corresponding image.

[0153] The fifth embodiment may be combined with each of the above embodiments. In Figure 18, the corresponding image with the string "(Not spoken)" indicates the corresponding image generated based on the response information in the fourth embodiment.

[0154] As described above, according to the fifth embodiment, the display mode of each corresponding image can be varied according to its impact on the meeting. As a result, the magnitude of the impact of each corresponding image can be visually represented.

[0155] In addition, in each of the above embodiments, the display information generation unit 46 may generate display information in association with each corresponding image so that the probability of each emotion indicated by the emotion data corresponding to the corresponding image can be visually perceived.

[0156] Figure 20 shows an example of visually displaying the probability of each emotion in association with each corresponding image. In Figure 20, (1) is the corresponding image of person A and a radar chart showing person A's emotion data. (2) is the corresponding image of person B and a radar chart showing person A's emotion data. Each radar chart shows the probability for each type of emotion, such as anger, disgust, anxiety, calmness, sadness, and joy. In this case, each corresponding image may be processed based on the emotion data.

[0157] Each function of this embodiment can be implemented by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute the functions described above.

[0158] Furthermore, the apparatus described in this embodiment represents only one of several computing environments for carrying out the embodiments disclosed herein.

[0159] In one embodiment, the server device 40 includes a plurality of computing devices, such as a server cluster. The plurality of computing devices are configured to communicate with each other via any type of communication link, including a network and shared memory, and perform the processing disclosed herein.

[0160] In this embodiment, the terminal device 10 and the server device 40 are examples of information processing devices.

[0161] Although embodiments of the present invention have been described in detail above, the present invention is not limited to these specific embodiments, and various modifications and changes are possible within the scope of the gist of the present invention as described in the claims.

[0162] Examples of the present invention are as follows: <1> An acquisition unit that acquires unique data indicating unique information based on the actions of each participant in the meeting, An image generation unit generates corresponding images for each participant based on the unique data acquired by the acquisition unit and a machine learning model trained using the unique data and images that represent the unique information, Based on the unique data acquired by the acquisition unit, an emotion analysis unit analyzes the emotions of each participant, An information processing device characterized by having the following features. <2> Based on the analysis results of the emotion analysis unit, a display information generation unit generates display information to display the corresponding image. It is further characterized by having <1> The information processing device described. <3> The system further includes a virtual data generation unit that generates virtual unique data for a virtual participant based on the unique data of other participants and the emotional data analyzed by the emotional analysis unit, The image generation unit, Based on the virtual unique data and the machine learning model, corresponding images corresponding to the virtual unique data are generated for the virtual participants. <1> or <2> The information processing device described. <4> The unique data includes a first type of unique data containing a first type of unique information, and a second type of unique data containing a second type of unique information based on a response to any of the first type of unique information, The image generation unit, Based on the first type of unique data of the first participant, the sentiment data analyzed by the sentiment analysis unit based on the second type of unique data of the second participant, and a machine learning model, the corresponding image corresponding to the first type of unique data and the sentiment data is generated for the second participant. <1> ~ <3> Any of the information processing devices described above. <5> The first type of unique information is text that shows at least one of the characters entered by the participant and the content of the participant's speech. The second type of unique information includes at least one of the following: operation information indicating an operation entered by the participant, image information of the participant captured by the imaging device, and biometric information of the participant detected by the biometric information detection unit. <4> The information processing device described. <6> A display information generation unit generates display information that associates participant identification information that identifies the participant with the corresponding image generated for that participant. Characterized by having <1> ~ <5> Any of the information processing devices described above. <7> The display information generation unit generates the display information that displays the emotion data analyzed by the emotion analysis unit in association with the participant identification information and the corresponding image. Characterized by <6> The information processing device described. <8> A layout determination unit determines, based on at least one of the unique information and the corresponding image, and the sentiment data analyzed by the sentiment analysis unit, the arrangement of the multiple corresponding images when displaying them, the size of each corresponding image, and at least one of the decorations. A display information generation unit generates display information for displaying a plurality of corresponding images in at least one of the arrangement, size, and decoration determined by the layout determination unit, Characterized by having <1> ~ <7> Any of the information processing devices described above. <9> The layout determination unit determines the decoration or size of the corresponding image based on at least one of the following included in the unique information: the amount and volume of the participant's speech, image information showing the facial expressions of other participants captured by the camera, changes in the biometric information of other participants detected by the biometric information detection unit, and the emotion data. <8> The information processing device described. <10> The image generation unit generates corresponding images for each participant based on the strings obtained by separating the text data contained in the unique data acquired by the acquisition unit into time-series strings and the machine learning model, The display information generation unit generates the display information which displays the strings in chronological order and in an arrangement corresponding to the emotion data. Characterized by <8> The information processing device described. <11> An acquisition unit that acquires unique data indicating unique information based on the actions of each participant in the meeting, An image generation unit generates corresponding images for each participant based on the unique data acquired by the acquisition unit and a machine learning model trained using the unique data and images that represent the unique information, Based on the unique data acquired by the acquisition unit, an emotion analysis unit analyzes the emotions of each participant, An information processing system characterized by having the following features. <12> A procedure for obtaining unique data that shows unique information based on the actions of each participant in the meeting, An image generation procedure that generates corresponding images for each participant based on the unique data acquired by the acquisition procedure and a machine learning model trained using the unique data and images that represent the unique information, Based on the unique data acquired by the acquisition procedure, an emotion analysis procedure is performed to analyze the emotions of each participant, An information processing method characterized by a computer executing the following. <13> A procedure for obtaining unique data that shows unique information based on the actions of each participant in the meeting, An image generation procedure that generates corresponding images for each participant based on the unique data acquired by the acquisition procedure and a machine learning model trained using the unique data and images that represent the unique information, Based on the unique data acquired by the acquisition procedure, an emotion analysis procedure is performed to analyze the emotions of each participant, A program characterized by causing a computer to execute something. [Explanation of Symbols]

[0163] 10 Terminal devices 11 Input section 12 Input data transmission unit 13 Display Control Unit 40 Server Devices 41 Input data acquisition unit 42 Text generation unit 43 Emotion Analysis Department 44 Image generation unit 45. Relevance Calculation Unit 46 Display information generation section 47 Display Information Transmission Unit 48 Virtual Data Generation Unit 49 Layout Determination Section 451 Input data storage unit 452 Model Memory Unit m1 Emotion Model m2 image generation model m3 language model m4 emotion model m5 image generation model [Prior art documents] [Patent Documents]

[0164] [Patent Document 1] Japanese Patent Publication No. 2022-64301

Claims

1. An acquisition unit that acquires unique data indicating unique information based on the actions of each participant in the meeting, An image generation unit generates corresponding images for each participant based on the unique data acquired by the acquisition unit and a machine learning model trained using the unique data and images that represent the unique information, Based on the unique data acquired by the acquisition unit, an emotion analysis unit analyzes the emotions of each participant, An information processing device characterized by having the following features.

2. Based on the analysis results of the emotion analysis unit, a display information generation unit generates display information to display the corresponding image. The information processing apparatus according to claim 1, further comprising the above.

3. The system further includes a virtual data generation unit that generates virtual unique data for a virtual participant based on the unique data of other participants and the emotional data analyzed by the emotional analysis unit, The image generation unit, The information processing apparatus according to claim 1, which generates corresponding images for the virtual participants based on the virtual unique data and the machine learning model.

4. The unique data includes a first type of unique data containing a first type of unique information, and a second type of unique data containing a second type of unique information based on a response to any of the first type of unique information, The image generation unit, The information processing apparatus according to claim 1, which generates corresponding images for the second participant that correspond to the first type of unique data and the emotional data, based on the first type of unique data of the first participant, the emotional data analyzed by the emotional analysis unit based on the second type of unique data of the second participant, and a machine learning model.

5. The first type of unique information is text that shows at least one of the characters entered by the participant and the content of the participant's speech. The information processing apparatus according to claim 4, wherein the second type of unique information includes at least one of operation information indicating an operation entered by the participant, captured image information of the participant taken by the imaging device, and each of the participant's biological information detected by the biological information detection unit.

6. A display information generation unit generates display information that associates participant identification information that identifies the participant with the corresponding image generated for that participant. The information processing apparatus according to claim 1, characterized by having the following features.

7. The display information generation unit generates the display information that displays the emotion data analyzed by the emotion analysis unit in association with the participant identification information and the corresponding image. The information processing apparatus according to claim 6, characterized in that it is a product of the same name.

8. A layout determination unit determines, based on at least one of the unique information and the corresponding image, and the emotion data analyzed by the emotion analysis unit, the arrangement of the multiple corresponding images when displaying them, the size of each corresponding image, and at least one of the decorations. A display information generation unit generates display information for displaying a plurality of corresponding images in at least one of the arrangement, size, and decoration determined by the layout determination unit, The information processing apparatus according to claim 1, characterized by having the following features.

9. The information processing apparatus according to claim 8, wherein the layout determination unit determines the decoration or size of the corresponding image based on at least one of the following included in the unique information: the amount and volume of the participant's speech, image information showing the facial expressions of other participants captured by the camera, changes in the biometric information of other participants detected by the biometric information detection unit, and the emotion data.

10. The image generation unit generates corresponding images for each participant based on the strings obtained by separating the text data contained in the unique data acquired by the acquisition unit into time-series strings and the machine learning model, The display information generation unit generates the display information which displays the strings in chronological order and in an arrangement corresponding to the emotion data. The information processing apparatus according to claim 8, characterized in that it is a product of the present invention.

11. An acquisition unit that acquires unique data indicating unique information based on the actions of each participant in the meeting, An image generation unit generates corresponding images for each participant based on the unique data acquired by the acquisition unit and a machine learning model trained using the unique data and images that represent the unique information, Based on the unique data acquired by the acquisition unit, an emotion analysis unit analyzes the emotions of each participant, An information processing system characterized by having the following features.

12. A procedure for obtaining unique data that shows unique information based on the actions of each participant in the meeting, An image generation procedure that generates corresponding images for each participant based on the unique data acquired by the acquisition procedure and a machine learning model trained using the unique data and images that represent the unique information, Based on the unique data acquired by the acquisition procedure, an emotion analysis procedure is performed to analyze the emotions of each participant, An information processing method characterized by a computer executing the following.

13. A procedure for obtaining unique data that shows unique information based on the actions of each participant in the meeting, An image generation procedure that generates corresponding images for each participant based on the unique data acquired by the acquisition procedure and a machine learning model trained using the unique data and images that represent the unique information, Based on the unique data acquired by the acquisition procedure, an emotion analysis procedure is performed to analyze the emotions of each participant, A program characterized by causing a computer to execute something.

Images