Information processing systems, information processing devices, programs, information processing methods, terminal devices

Abstract

Information that assists in the progress of the event will be displayed within the space. [Means for solving the problem] The system receives input of voice data including speech from a person in the space transmitted from a terminal device, and transmits a prompt to a large language model that includes event information relating to the content of an event in the space and the content of the voice data, and requests the output of assist information to assist in the progress of the event based on the voice data. Based on the response data received from the large language model, the system generates screen information for a screen that includes assist information to assist in the progress of the event and transmits the screen information to the terminal device, and the terminal device causes a display device included in the space to display a screen that includes the assist information.

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing apparatus, a program, an information processing method, and a terminal device.

Background Art

[0002] Conventionally, a technology is known that distributes utterance information, which is obtained by converting voice uttered by meeting participants or a smart speaker into a character string, to one or a plurality of information processing terminals, thereby causing the content of utterances by participants in a meeting to be shared by appropriate sharers during the meeting.

Summary of Invention

Problem to be Solved by the Invention

[0003] The above-described conventional technology visualizes the utterance content of meeting participants during a meeting, and does not take assisting the progress of the meeting into consideration. For this reason, the above-described conventional technology cannot present information that assists the progress of an event in the space where the event is held, for example, when an event or the like is held.

[0004] The disclosed technique has been made in view of the above circumstances, and aims to display information that assists the progress of an event in a space.

Means for Solving the Problem

[0005] The disclosed technology is an information processing system including an information processing device, a terminal device, a sound collection device and a display device included in a space, wherein the information processing device includes an input receiving unit that receives input of voice data including speech of a person in the space transmitted from the terminal device, a transmitting unit that transmits a prompt to a large language model, the prompt including event information relating to the content of an event in the space and the content of the voice data, and the prompt requests the output of assist information to assist the progress of the event based on the voice data, a response receiving unit that receives response data transmitted from the large language model, and based on the response data, The device includes a screen information generation unit that generates screen information for a screen that includes assist information to assist in the progress of the aforementioned event, and a screen information transmission unit that transmits the screen information generated by the screen information generation unit to the terminal device, the terminal device having a communication control unit that receives audio data collected by the sound collection device from the sound collection device, transmits the audio data to the information processing device, and when it receives a screen display instruction based on the screen information transmitted from the information processing device, transmits the screen information and the display instruction to a display device included in the space, and the display device included in the space having a display control unit that displays the screen including the assist information. [Effects of the Invention]

[0006] Information that assists in the progress of the event can be displayed within the space. [Brief explanation of the drawing]

[0007] [Figure 1] This figure shows an example of the system configuration of the information processing system according to the first embodiment. [Figure 2] This diagram illustrates the general operation of an information processing system. [Figure 3] This figure shows an example of the hardware configuration of the information processing device according to the first embodiment. [Figure 4] This is a hardware configuration diagram of the display device according to the first embodiment. [Figure 5]This diagram illustrates the functional configuration of each device in the information processing system of the first embodiment. [Figure 6] This figure shows an example of an event storage unit according to the first embodiment. [Figure 7] This figure shows an example of a spatial memory unit. [Figure 8] This figure shows an example of a template memory unit. [Figure 9] This is the first sequence diagram illustrating the operation of the information processing system in the first embodiment. [Figure 10] This is a second sequence diagram illustrating the operation of the information processing system in the first embodiment. [Figure 11] This is the first figure showing an example of the display of the terminal device according to the first embodiment. [Figure 12] The second figure shows an example of the display of the terminal device according to the first embodiment. [Figure 13] This is a third sequence diagram illustrating the operation of the information processing system according to the first embodiment. [Figure 14] This figure shows an example of the first prompt in the first embodiment. [Figure 15] This is the first figure showing an example of the display of the display device according to the first embodiment. [Figure 16] This figure shows an example of the second prompt in the first embodiment. [Figure 17] The second figure shows an example of the display of the display device according to the first embodiment. [Figure 18] This figure shows an example of the third prompt in the first embodiment. [Figure 19] This is the third figure, showing an example of the display of the display device according to the first embodiment. [Figure 20] This figure shows the event space where the assist information is displayed in the first embodiment. [Figure 21] This is a sequence diagram illustrating the operation of the information processing system in the second embodiment. [Figure 22] This figure shows an example of a prompt in the second embodiment. [Figure 23]This is a diagram showing a display example of the display device according to the second embodiment. [Figure 24] This is a diagram for explaining the functional configuration of each device included in the information processing system according to the third embodiment. [Figure 25] This is a sequence diagram for explaining the operation of the information processing system according to the third embodiment. [Figure 26] This is a diagram showing a display example of the display device according to the third embodiment. [Figure 27] This is a sequence diagram for explaining the operation of the information processing system according to the fourth embodiment. [Figure 28] This is a first diagram showing a display example of the terminal device according to the fourth embodiment. [Figure 29] This is a second diagram showing a display example of the terminal device according to the fourth embodiment. [Figure 30] This is a diagram showing an example of the system configuration of the information processing system according to the fifth embodiment. [Figure 31] This is a diagram for explaining the functional configuration of each device included in the information processing system according to the fifth embodiment. [Figure 32] This is a diagram showing an example of the space storage unit according to the fifth embodiment. [Figure 33] This is a diagram showing an example of the template storage unit according to the fifth embodiment. [Figure 34] This is a diagram showing an example of the summary management storage unit according to the fifth embodiment. [Figure 35] This is a diagram showing another example of the space storage unit according to the fifth embodiment. [Figure 36] This is a diagram showing a display example of the display device according to the fifth embodiment. [Figure 37] This is a diagram showing an event space in which assist information is displayed according to the fifth embodiment. [Figure 38] This is a sequence diagram for explaining the operation of the information processing system according to the fifth embodiment. [Figure 39] This is a diagram for explaining the functions possessed by each device of the information processing system according to the sixth embodiment. [Figure 40] This is a diagram showing an example of the system configuration of the information processing system according to the seventh embodiment. [Figure 41] This diagram illustrates the functional configuration of each device in the information processing system of the seventh embodiment. [Figure 42] This figure shows an example of an event storage unit according to the seventh embodiment. [Figure 43] This figure shows an example of a spatial memory unit according to the seventh embodiment. [Figure 44] This figure shows an example of a template storage unit according to the seventh embodiment. [Figure 45] This figure shows an example of a procedure storage unit in the processing procedure management device of the seventh embodiment. [Figure 46] This is a sequence diagram illustrating the operation of the information processing system according to the seventh embodiment. [Figure 47] This is the first figure illustrating an example of the display of the seventh embodiment of the display device. [Figure 48] This is the second figure illustrating an example of the display of the seventh embodiment of the display device. [Modes for carrying out the invention]

[0008] (First embodiment) This embodiment will be described below with reference to the drawings. Figure 1 is a diagram showing an example of the system configuration of the information processing system of the first embodiment.

[0009] The information processing system 100 of this embodiment includes an information processing device 200, a terminal device 300, a sound collection device 600, and display devices 700A, 700B, and 700C. In the information processing system 100, the information processing device 200 and the terminal device 300 are connected via a communication network N1. The information processing device 200 is also connected to a server device 400 via the communication network N1.

[0010] The server device 400 in this embodiment has a Large Language Model (LLM) and implements Artificial Intelligence (AI). The Artificial Intelligence generates text data in response to input prompts, such as ChatGPT, and outputs it to the information processing device 200. The prompts in this embodiment are text data that express instructions or questions to the Artificial Intelligence implemented by the server device 400. Details of the server device 400 will be described later.

[0011] Furthermore, in the information processing system 100, the terminal device 300 is connected to the sound collection device 600, display devices 700A, 700B, 700C, etc., via the communication network N2.

[0012] In this embodiment, communication network N1 is a wide-area network such as the Internet or WAN, and communication network N2 is a LAN, Wi-Fi (registered trademark), Wide Area Ethernet (registered trademark), 4G, 5G, 6G, or other mobile phone network.

[0013] The terminal device 300, sound collection device 600, and display devices 700A, 700B, and 700C of this embodiment are installed in the space where the event is held. Note that the terminal device 300, sound collection device 600, and display devices 700A, 700B, and 700C may each be portable devices; in that case, these devices only need to be within the space where the event is held, and do not need to be installed. In the following description, the space where the event is held will be referred to as the event space.

[0014] The sound collection device 600 collects voices such as speech from people present in the event space and transmits audio data representing the collected voices to the terminal device 300. The sound collection device 600 may be a microphone worn by multiple people, or it may be a device integrated with a voice output function, such as a headset. Alternatively, the sound collection device 600 may be a single device present in the space that can collect voices within that space. The sound collection device 600 can also be a wearable microphone. An example of a wearable microphone is a wristband-type microphone. The sound collection device 600 may also be configured to transcribe (convert to text) or summarize the audio data and transmit the transcribed text data or the summarized text data to the terminal device 300 or the information processing device 200.

[0015] Each of the display devices 700A, 700B, and 700C displays information that assists in the progress of an event being held in the event space, based on screen information received from the terminal device 300. In the following description, unless distinguished, each of the display devices 700A, 700B, and 700C will simply be referred to as the display device 700. The display device 700 in this embodiment may be, for example, a large display placed in the event space, or a portable tablet that can be carried around within the event space. The display device 700 also includes a projection device such as a projector that projects image data onto a projection area. In this embodiment, the display device 700 and the sound collection device 600 may be integrated into a single device having both sound collection and display functions.

[0016] The terminal device 300 in this embodiment receives operations related to the progress of the event from people in the event space. Furthermore, when the terminal device 300 acquires audio data from the sound collection device 600, it transmits the audio data to the information processing device 200. Also, when the terminal device 300 receives screen information from the information processing device 200, it transmits the received screen information to the display device 700. In other words, the terminal device 300 in this embodiment functions as a relay device that relays communication between the display device 700 and the information processing device 200. The terminal device 300 may be integrated with the display device 700. Alternatively, the terminal device 300 may be integrated with the sound collection device 600.

[0017] In this embodiment, the information processing device 200, upon receiving audio data from the terminal device 300, converts the audio data into text data. The information processing device 200 also generates a prompt containing the text data converted from the audio data, which requests the generation of information to assist in the progress of the event, and outputs this prompt to the server device 400. Furthermore, upon receiving response data to the prompt from the server device 400, the information processing device 200 generates screen information containing this response data and transmits it to the terminal device 300. In the following description, the information that assists in the progress of the event will be referred to as assist information.

[0018] Here, the information processing device 200 generates and outputs multiple types of prompts to the server device 400 to generate multiple types of assist information. The multiple types of prompts include information that identifies the display device 700 on which the assist information is displayed.

[0019] Next, the information processing device 200 obtains multiple types of response data generated from the server device 400 based on multiple types of prompts. The multiple types of response data obtained here include multiple types of assist information and information that identifies the display device 700 on which each type of assist information is displayed.

[0020] When the information processing device 200 acquires multiple types of response data, it displays the assist information contained in each response data on the display device 700 identified based on the information that identifies the display device 700 contained in each response data.

[0021] In this embodiment, in response to speech by a person in the event space, assist information that helps the progress of the event can be displayed on the display device 700 located within the event space. Furthermore, according to this embodiment, different types of assist information can be displayed on each of the display devices 700A, 700B, and 700C located within the event space.

[0022] In the example in Figure 1, the information processing device 200 and the terminal device 300 are treated as separate devices, but this is not limited to this. The information processing device 200 may also function as the terminal device 300. Similarly, in the example in Figure 1, the information processing device 200 and the server device 400 are treated as separate devices, but this is not limited to this. The server device 400 may be included within the information processing device 200.

[0023] Furthermore, in Figure 1, the information processing device 200 may be implemented by one or more information processing devices, or it may be implemented by multiple information processing devices.

[0024] The following describes the general operation of the information processing system 100 of this embodiment with reference to Figure 2. Figure 2 is a diagram illustrating the general operation of the information processing system.

[0025] In the example shown in Figure 2, an event is being held in event space R, and participant P1 and participant P2 are engaging in dialogue within event space R. In this embodiment, an event includes various types of events such as experiential lectures (workshops), training sessions, award ceremonies, and reporting sessions in which participants actively participate.

[0026] Event space R is equipped with display devices 700A, 700B, and 700C. Participants P1 and P2 are each wearing headsets 600A and 600B, which have sound collection functions. Headsets 600A and 600B each include a sound collection device 600 and a speaker. In addition, a terminal device 300 is located in event space R (not shown) and relays communication between the display devices 700A, 700B, and 700C, as well as the headsets 600A and 600B, and the information processing device 200.

[0027] In this embodiment, when the information processing device 200 receives audio data indicating a conversation between participant P1 and participant P2 via the terminal device 300, it converts the utterances contained in this audio data into text data. The information processing device 200 then embeds the text data into each of the template information that shows pre-prepared templates for multiple types of prompts, generates multiple types of prompts, and transmits them to the server device 400.

[0028] Each of the multiple types of prompts generated here is, in other words, a prompt that makes a different request to the large-scale language model provided by the server device 400. In this embodiment, by making the content of the requests in the prompts different in this way, response data corresponding to the content of the requests can be obtained from the large-scale language model. In this embodiment, each of the multiple types of prompts generated by the information processing device 200 includes identification information that identifies a display device 700 included in the event space R, and the response data corresponding to the prompt is displayed on the display device 700 identified by the identification information of the display device 700. If there are multiple display devices 700, which display device 700 to display the data on is controlled by the terminal device 300 or the information processing device 200.

[0029] In the example shown in Figure 2, the information processing device 200 includes identification information that identifies the display device 700A, generates a first prompt requesting interaction with a person present in the event space R, and displays the response data to the first prompt on the display device 700A as first assist information. The information processing device 200 also includes identification information that identifies the display device 700B, generates a second prompt requesting the retrieval of information based on the interaction with a person present in the event space R, and displays the response data to the second prompt on the display device 700B as second assist information. The information processing device 200 also includes identification information that identifies the display device 700A, generates a third prompt requesting the creation of a summary of the interaction between participants P1 and P2 present in the event space R, and displays the response data to the third prompt on the display device 700C as third assist information.

[0030] In this embodiment, a number of prompts, each containing the content of a person's speech in the event space, identification information for the display device 700, and a request for the large-scale language model, are sent to the server device 400. These prompts differ in the content of the requests. In this embodiment, various assist information corresponding to the requests of each prompt can be displayed on each of the display devices 700 included in the space. In other words, in this embodiment, by outputting prompts of different types to the large-scale language model, the large-scale language model can be made to function as a generative AI agent that realizes functions according to the requests.

[0031] In this embodiment, prompt requests can be arbitrarily set according to the events held in the event space. Therefore, according to this embodiment, assistance information corresponding to the type and content of the events held in the space can be displayed on each display device 700 in the space.

[0032] In the example shown in Figure 2, various types of assistance information, inferred from the dialogue of participants P1 and P2, are displayed on the display devices 700A, 700B, and 700C installed in the event space. Therefore, according to this embodiment, participants P1 and P2 can view the various types of assistance information displayed on the display devices 700A, 700B, and 700C, and the event can proceed smoothly. To put it another way, participants can feel as if the space itself is an AI. The space's sound collection device collects the speech of event participants in that space, and various types of assistance information are displayed on large displays installed on walls or other surfaces in response to the speech, providing participants with a seamless AI-powered event experience.

[0033] In the example shown in Figure 2, the sound collection device 600 is assumed to be worn by participants P1 and P2, but it is not limited to this. The sound collection device 600 may be installed anywhere in the event space where it can collect speech from people. In the following explanation, people present in the event space may be referred to as event participants or simply participants.

[0034] Next, an example of the hardware configuration of each device in the information processing system 100 of this embodiment will be described.

[0035] Figure 3 shows an example of the hardware configuration of an information processing device according to the first embodiment. The information processing device 200 shown in Figure 3 is built by a computer (information processing device) and includes a CPU 201, ROM 202, RAM 203, HD 204, HDD (Hard Disk Drive) controller 205, display 206, external device connection I / F (Interface) 208, network I / F 209, bus line 210, keyboard 211, pointing device 212, DVD-RW (Digital Versatile Disk Rewritable) drive 214, and media I / F 216.

[0036] Of these, the CPU 201 controls the operation of the entire information processing unit 200. The ROM 202 stores programs used to drive the CPU 201, such as IPL. The RAM 203 is used as the work area for the CPU 501. The HD 204 stores various data such as programs. The HDD controller 205 controls the reading or writing of various data to the HD 204 according to the control of the CPU 201. The display 206 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 208 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory and printers. The network I / F 209 is an interface for data communication using a communication network. The bus line 210 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 201 shown in Figure 4.

[0037] The keyboard 211 is a type of input means equipped with multiple keys for inputting characters, numbers, and various instructions. The pointing device 212 is a type of input means for selecting and executing various instructions, selecting processing targets, and moving the cursor. The DVD-RW drive 214 controls the reading or writing of various data to the DVD-RW 213, which is an example of a removable recording medium. Note that it is not limited to DVD-RW, but may also be DVD-R, etc. The media I / F 216 controls the reading or writing (storage) of data to the recording medium 215, such as flash memory.

[0038] Next, the hardware configuration of the display device 700 in this embodiment will be described with reference to Figure 4. In Figure 4, an electronic whiteboard is used as an example of the display device 700. Note that the display device 700 is not limited to an electronic whiteboard, but may be a general-purpose display.

[0039] Figure 4 is a hardware configuration diagram of the display device according to the first embodiment. As shown in Figure 4, the display device 700 includes a CPU (Central Processing Unit) 701, ROM (Read Only Memory) 702, RAM (Random Access Memory) 703, SSD (Solid State Drive) 704, network I / F 705, and external device connection I / F (Interface) 706.

[0040] Of these components, the CPU 701 controls the overall operation of the display device 700. The ROM 702 stores programs used to drive the CPU 701, such as the CPU 701 and the IPL (Initial Program Loader). The RAM 703 is used as the work area for the CPU 701. The SSD 704 stores various data, such as programs for the electronic whiteboard. The network controller 705 controls communication with the communication network N1. The external device connection interface 706 is an interface for connecting various external devices. In this case, external devices include, for example, a USB (Universal Serial Bus) memory 730 and external devices (microphone 740, speaker 750, camera 760).

[0041] The display device 700 also includes a capture device 711, a GPU 712, a display controller 713, a contact sensor 714, a sensor controller 715, an electronic pen controller 716, a short-range communication circuit 719, and an antenna 719a for the short-range communication circuit 719, a power switch 722, and selector switches 723.

[0042] Of these, the capture device 711 displays video information as a still image or video on the display of an external PC (Personal Computer) 770. The GPU (Graphics Processing Unit) 712 is a semiconductor chip specializing in graphics. The display controller 713 controls and manages screen display in order to output the output image from the GPU 712 to the display 780, etc. The contact sensor 714 detects when an electronic pen 790 or the user's hand H, etc., touches the display 780. The sensor controller 715 controls the processing of the contact sensor 714. The contact sensor 714 performs coordinate input and coordinate detection using an infrared blocking method. This method of coordinate input and coordinate detection involves two light-emitting and receiving devices installed at both ends of the upper side of the display 780 emitting multiple infrared rays parallel to the display 780, and receiving the light that is reflected by reflective materials provided around the display 780 and returns along the same optical path as the light emitted by the light-receiving element. The contact sensor 714 outputs the infrared ID emitted by two light-receiving devices that are blocked by an object to the sensor controller 715, which then identifies the coordinate position of the object's contact. The electronic pen controller 716 communicates with the electronic pen 790 to determine whether the pen tip or the pen end has touched the display 780. The near-field communication circuit 719 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 722 is a switch for turning the power of the display device 700 ON / OFF. The selection switches 723 are a group of switches for adjusting, for example, the brightness and color of the display 780.

[0043] Furthermore, the display device 700 is equipped with a bus line 710. The bus line 710 is an address bus, data bus, etc., for electrically connecting each component, such as the CPU 701 shown in Figure 4.

[0044] Furthermore, the contact sensor 714 is not limited to the infrared blocking method. Various detection means may be used, such as a capacitive touch panel that identifies the contact position by detecting changes in capacitance, a resistive touch panel that identifies the contact position by voltage changes between two opposing resistive films, or an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by contact between an object and the display. In addition, the electronic pen controller 716 may determine whether or not there is a touch not only on the tip and end of the electronic pen 790, but also on the part of the electronic pen 790 that the user holds and other parts of the electronic pen.

[0045] Next, with reference to Figure 5, the functional configuration of each device in the information processing system 100 of this embodiment will be described. Figure 5 is a diagram illustrating the functional configuration of each device in the information processing system of the first embodiment.

[0046] First, the functional configuration of the information processing device 200 will be described. The information processing device 200 in this embodiment has a storage unit 230 and a control unit 270.

[0047] The memory unit 230 stores text data and other information that shows the history of interactions between event participants and the large-scale language model. The memory unit 230 also includes an event memory unit 240, a spatial memory unit 250, and a template memory unit 255. The event memory unit 240 stores event information related to the event. The spatial memory unit 250 stores spatial information that shows the characteristics of the space where the event is held. The template memory unit 255 stores template information that shows a template of a prompt corresponding to the type of assist information to be displayed on the display device 700. Details of each memory unit included in the memory unit 230 will be described later.

[0048] The control unit 270 is realized when the CPU 20 of the information processing device 200 reads and executes programs stored in the ROM 202 and HD 204.

[0049] The control unit 270 includes an input receiving unit 271, a person identification unit 272, a speech recognition unit 273, a prompt generation unit 274, a prompt transmission unit 275, a response receiving unit 276, a screen information generation unit 277, a screen information transmission unit 278, an event setting unit 279, and a mode identification unit 280.

[0050] The input receiving unit 271 receives various types of information for the information processing device 200. These types of information include, for example, voice data transmitted via the terminal device 300. Other types of information include, for example, information indicating the operation performed on the terminal device 300.

[0051] The person identification unit 272 identifies the speaker based on the voice data received by the input reception unit 271. Specifically, the person identification unit 272 determines, for example, whether the speaker of the voice indicated by the voice data received by the input reception unit 271 matches the speaker of the voice indicated by voice data previously received by the input reception unit 271, and identifies the person according to the determination result. In addition, if information associating speakers with their voice data is stored in the information processing device 200 in advance, the person identification unit 272 may identify the speaker by referring to this information.

[0052] The speech recognition unit 273 converts the speech data received by the input reception unit 271 into text data.

[0053] The prompt generation unit 274 generates a prompt to send to the server device 400 based on the text data converted from the speech data by the speech recognition unit 273 and the template information stored in the template storage unit 255. The prompt transmission unit 275 sends the prompt generated by the prompt generation unit 274 to the server device 400. The response receiving unit 276 receives response data to the prompt from the server device 400.

[0054] The screen information generation unit 277 generates screen information indicating the screen to be displayed on the display device 700 based on the response data received by the response receiving unit 276. The screen information transmission unit 278 transmits the screen information generated by the screen information generation unit 277 to the terminal device 300.

[0055] When the event setting unit 279 is notified that an event will be held, it refers to the event storage unit 240 to obtain event information for the event to be held and makes settings related to holding the event. These settings related to holding the event include, for example, displaying a start screen on the event space display device 700 indicating that the event has been held.

[0056] The mode identification unit 280 identifies the assist mode to be adopted from among a plurality of assist modes that indicate the degree of assistance in the progress of events using the large-scale language model 411. For example, when a prompt including a request to select an assist mode to be adopted from among the plurality of assist modes is sent to the server device 400, the mode identification unit 280 identifies the assist mode to be adopted based on the response data to this prompt.

[0057] Next, the functions of the terminal device 300 will be described. The terminal device 300 includes an input receiving unit 340, a display control unit 350, and a communication control unit 360. The input receiving unit 340 receives input to the terminal device 300. The display control unit 350 controls the display on the terminal device 300. The communication control unit 360 controls communication between the terminal device 300 and the information processing device 200. The communication control unit 360 also controls communication between the terminal device 300 and the sound collection device 600 and the display device 700. Furthermore, the communication control unit 360 transmits the audio data collected by the sound collection device 600 to the information processing device 200. In addition, the communication control unit 360 transmits a screen display instruction to the display device 700 based on the screen information received from the information processing device 200.

[0058] The input receiving unit 340, the display control unit 350, and the communication control unit 360 may be implemented by a Web browser of the terminal device 300 in this embodiment, or by means other than a Web browser.

[0059] Next, we will describe the various components of the server device 400 on which generation AIs such as Chat GPT and Bard operate.

[0060] The large-scale language model memory unit 410 contains the large-scale language model 411. The storage and reading unit 420 is executed by instructions from the CPU and the HDD, and performs processes such as storing various data in the large-scale language model memory unit 410 and reading various data from the large-scale language model memory unit 410.

[0061] The large-scale language model 411 is a computer language model that is generated by performing a training process using a vast amount of unlabeled text as training data, and consists of an artificial neural network with numerous parameters.

[0062] The large-scale language model 411 can capture much of the syntax and meaning of human language by being sufficiently trained with context-learning techniques such as next sentence prediction, which understands context by determining whether sentence 1 and sentence 2 are consecutive, and the masked language model, which understands context by masking words in a sentence and predicting the masked words from the words before and after them.

[0063] The generation unit 430 is implemented by instructions from the CPU and generates various types of information, such as text information. The communication unit 440 is implemented by instructions from the CPU and a communication interface and communicates with other terminals, devices, or systems via a communication network, such as sending and receiving various types of data (or information).

[0064] The sound collection device 600 of this embodiment includes a communication control unit 610. The communication control unit 610 transmits the collected audio data to an external device. The external device may be a terminal device 300 or an information processing device 200.

[0065] The display device 700 of this embodiment includes a communication control unit 755 and a display control unit 765. The communication control unit 755 receives screen information from the terminal device 300. The communication control unit 755 may also receive screen information from the information processing device 200. The display control unit 765 displays the screen based on the screen information.

[0066] Next, with reference to Figures 6 to 8, each storage unit included in the storage unit 230 of the information processing device 200 of this embodiment will be described.

[0067] Figure 6 shows an example of an event storage unit in the first embodiment. The event information stored in the event storage unit 240 of this embodiment includes, as information items, event identification information, spatial ID, event name, event type, duration, number of participants, event content, event purpose, event results, etc. In the event information, the value of the item "event identification information" is associated with the values ​​of the other items.

[0068] The value of the "Event Identification Information" field indicates identification information used to identify the event. The value of the "Space ID" field indicates identification information that identifies the space where the event is held. The value of the "Event Name" field indicates the name of the event. The value of the "Event Type" field indicates the type of event. The value of the "Duration" field indicates the duration from the start to the end of the event. The value of the "Number of Participants" field indicates the number of participants in the event. The value of the "Event Content" field indicates the content of the event. The value of the "Event Purpose" field indicates the purpose of the event. The value of the "Event Results" field is information entered after the event has ended and indicates information summarizing the results of holding the event.

[0069] Figure 7 shows an example of a spatial memory unit. The spatial information stored in the spatial memory unit 250 of this embodiment includes, as information items, spatial ID, name, device, request content, etc. In addition, the spatial information may also have an item called spatial characteristics. In the spatial information, the value of the item "Spatial ID" is associated with the values ​​of the other items.

[0070] The value of the "Spatial ID" field indicates identification information for identifying a space. The value of the "Name" field indicates the name of the space. The value of the "Device" field indicates identification information for the display devices contained in the space, and information about those display devices. The value of the "Request Content" field indicates the content of the request associated with the display device. If the field has a "Spatial Characteristics" field, its value will be text or tags that describe the characteristics of that space. For example, "A space that receives afternoon sun" or "A space with soundproofing." By including such information in the spatial information and sending it to a large-scale language model along with the prompt, response data that takes the spatial characteristics into account can be obtained. For example, if the space has soundproofing, a response data such as "Because there is soundproofing, let's speak a little louder" may be obtained.

[0071] In the examples in Figures 6 and 7, it can be seen that the "Science of Taste Workshop," identified by event identification information "I0001," is being held in "Conference Room 1," which is identified by spatial ID "1." Furthermore, "Conference Room 1," identified by spatial ID "1," has three displays and a camera, and the identification information for the three displays is "DY1," "DY2," and "DY3," respectively. It can also be seen that the display with identification information "DY1" is associated with the request content "Dialogue" included in the prompt, the display with identification information "DY2" is associated with the request content "Information Retrieval" included in the prompt, and the display with identification information "DY3" is associated with the request content "Summary Creation." In this embodiment, the identification information of the displays may be associated with each request content in this way, or the terminal device 300 or information processing device 200 may decide which display corresponds to which request content and which assist information each time a display is processed.

[0072] Figure 8 shows an example of a template storage unit. The template information stored in the template storage unit 255 shown in Figure 8 includes the content of the request and the prompt template as information items. In the template information, the value of the item "Content of the request" corresponds to the value of the item "Prompt template". Since the content of each request is different, when a prompt corresponding to the content of each request is sent to the large-scale language model 411, the response from the large-scale language model 411 will be different. For this reason, the content of the request can also be said to be the type of prompt.

[0073] The value of the item "Request Details" indicates the content of the request for the large-scale language model. In other words, the value of the item "Request Details" indicates the type of assistance information to be displayed on the display device 700. The value of the item "Prompt Template" is the text data that serves as a template for generating a prompt containing the request details.

[0074] In the information processing device 200 of this embodiment, the prompt generation unit 274 refers to the event storage unit 240 and the spatial storage unit 250 to identify the content of a request associated with a display device included in the event space where the event is held. Then, the prompt generation unit 274 generates a prompt in the template storage unit 255 using the prompt template associated with the identified request content and the text data converted into speech data by the speech recognition unit 273.

[0075] Furthermore, the value of the item "Request Content" in the template storage unit 255 is not limited to the example shown in Figure 8; any content that assists in the progress of the event can be added by the administrator of the information processing system 100 or the like.

[0076] For example, possible values ​​for the item "Request Content" include interaction with a large-scale language model, video generation, facilitator, idea suggestion based on dialogue, topic suggestion, document data search, and display of relationships between past events. The template storage unit 255 stores text data that serves as a template for generating prompts containing the content of these requests.

[0077] In other words, the template information stored in the template storage unit 255 includes text data for generating prompts that cause the large-scale language model 411 to function as a command center that identifies requests to the large-scale language model 411 in response to dialogue. In other words, the template information stored in the template storage unit 255 includes text data for generating prompts that request a determination of the type of assist information to be displayed on the display device 700 in response to dialogue.

[0078] Furthermore, the template information stored in the template storage unit 255 includes text data for generating prompts to make the large-scale language model 411 function as an AI that generates image videos and plays the generated videos, and text data for generating prompts to make the large-scale language model 411 function as a facilitating AI. Furthermore, the template information stored in the template storage unit 255 includes text data for generating prompts to make the large-scale language model 411 function as an AI that generates ideas and diverges discussions, and text data for generating prompts to make the large-scale language model 411 function as an AI that brainstorms based on conversations. Furthermore, the template information stored in the template storage unit 255 includes text data for generating prompts to make the large-scale language model 411 function as an AI that searches for and presents document data, and text data for generating prompts to make the large-scale language model 411 function as an AI that organizes the relationships between past events.

[0079] Next, the operation of the information processing system 100 in this embodiment will be described. First, with reference to Figures 9 to 12, the operation of the information processing system 100 when an event is held will be described.

[0080] Figure 9 is a first sequence diagram illustrating the operation of the information processing system in the first embodiment. Figure 9 shows the operation when an event is initiated by a participant's utterance.

[0081] In the information processing system 100, when the sound collection device 600 collects audio data representing speech made by event participants, the communication control unit 610 transmits the audio data to the terminal device 300 (step S901). The speech in question is, for example, a speech declaring the start of the event, such as, "We will now begin the XX event."

[0082] The terminal device 300 transmits the audio data received from the sound collection device 600 to the information processing device 200 via the communication control unit 360 (step S902).

[0083] When the information processing device 200 receives audio data input via the input receiving unit 271, the speech recognition unit 273 converts the audio data into text data (step S903). Next, the information processing device 200 uses the event setting unit 279 to search the event storage unit 240 and spatial storage unit 250 of the storage unit 230 (step S904) to identify an upcoming event and the space in which the event will be held. The event setting unit 279 then obtains the event information of the identified event and the spatial information of the event space in which the identified event will be held (step S905).

[0084] Next, the event setting unit 279 identifies the display device 700 included in the event space based on the acquired event information and the spatial information of the event space where the event is held (step S906). Subsequently, the information processing device 200 generates screen information for a start screen that notifies the holding of the event using the screen information generation unit 277 (step S907), and transmits it to the terminal device 300 as a display instruction including the screen information of the start screen using the screen information transmission unit 278 (step S908). The screen information of the start screen transmitted here includes identification information that identifies the display device 700 identified in step S906.

[0085] When the terminal device 300 receives a display instruction including screen information for the start screen via the communication control unit 360, it transmits the screen information to the display device 700 identified by the identification information included in the display instruction (step S909). When the display device 700 receives the screen information via the communication control unit 755, it causes the display control unit 765 to display the start screen based on the screen information (step S910). The screen information for the start screen in this embodiment is an example of first screen information, and the start screen is an example of first screen.

[0086] Figure 10 is a second sequence diagram illustrating the operation of the information processing system according to the first embodiment. Figure 10 shows the operation when an event is initiated by an operation of a terminal device 300 by an event administrator or the like.

[0087] In the information processing system 100, when the terminal device 300 receives an operation from the input receiving unit 340 to instruct the display of the event list screen, the communication control unit 360 sends a request to the information processing device 200 to display the event list screen (step S1001). The information processing device 200 receives this display request from the input receiving unit 271 and, using the event setting unit 279, refers to the event storage unit 240 of the storage unit 230 (step S1002) to obtain a list of event information (step S1003). Subsequently, the information processing device 200 sends an instruction to the terminal device 300 to display the event list screen (step S1004).

[0088] When the terminal device 300 receives a list of event information from the communication control unit 360, the display control unit 350 displays a list of event information screen, and the input reception unit 340 accepts an operation to select an event (step S1005). Subsequently, the terminal device 300 transmits the event identification information of the selected event to the information processing device 200 via the communication control unit 360 (step S1006).

[0089] The information processing device 200, using the event setting unit 279, searches the event storage unit 240 and the spatial storage unit 250 based on the received event identification information (step S1007), and obtains event information corresponding to the event identification information and spatial information corresponding to the event information (step S1008).

[0090] The process from step S1009 to step S1013 in Figure 10 is the same as the process from step S906 to step S910 in Figure 9, so the explanation is omitted.

[0091] Next, with reference to Figures 11 and 12, an example of the display of the terminal device 300 will be described. Figure 11 is the first diagram showing an example of the display of the terminal device according to the first embodiment.

[0092] The screen 110A shown in Figure 11(A) is an example of a screen displayed on the terminal device 300 in step S1005 of Figure 10.

[0093] Screen 110A has a display area 111. Screen 110A is an event information list screen. The display area 111 displays event names based on a list of event information acquired from the information processing device 200, and each event is associated with an operation button for selecting it. The example in Figure 11(A) shows the case where the event name "Deliciousness Science Workshop" is selected.

[0094] The screen 110B shown in Figure 11(B) is an example of a screen displayed on the terminal device 300 after the event name "Deliciousness Science Workshop" is selected on screen 110A. Screen 110B has a display area 112 and operation buttons 113. The display area 112 displays event information for the event selected on screen 110. The event information displayed in the display area 112 may be displayed in a state that allows the administrator or other person to edit it.

[0095] Operation button 113 is an operation button used to give instructions for configuring the selected event.

[0096] In the example shown in Figure 11, when the operation button 113 is selected on screen 110B, processing by the event setting unit 279 begins.

[0097] Figure 12 is a second diagram showing an example of the display of the terminal device according to the first embodiment. The screen 120 shown in Figure 12 shows another example of the event information list screen displayed on the terminal device 300 in step S1005 of Figure 10. Screen 120 has a display area 121. The display area 121 displays a list of event types and operation buttons for selecting each event type in association with each other. The example in Figure 12 is an example in which event type templates are provided even if the administrator has not individually set event information for each event in advance, as in Figure 11. In this way, event types are prepared, and event information corresponding to each event type is pre-set. For example, as event information corresponding to "training" among the event types in the display area 121, the event content includes information indicating that it is "training for participants," and the event purpose includes information indicating that it aims to "improve skills."

[0098] In the example of screen 120, when an event type is selected, a screen for entering event information may be displayed. By displaying such a screen 120 on the terminal device 300, administrators can store and set the event type as an event held in space in the event storage unit 240 before the event starts.

[0099] Next, with reference to Figure 13, the operation of the information processing system 100 after the event has started will be described. Figure 13 is a third sequence diagram illustrating the operation of the information processing system in the first embodiment. The processing shown in Figure 13 may be executed each time a participant in the event space speaks, or it may be executed at predetermined intervals.

[0100] Furthermore, the example in Figure 13 describes a case in which the information processing device 200 generates a first prompt for generating first assist information to be displayed on the display device 700A, a second prompt for generating second assist information to be displayed on the display device 700B, and a third prompt for generating third assist information to be displayed on the display device 700C.

[0101] In the information processing system 100 of this embodiment, the sound collection device 600 collects voice data from event participants, and the communication control unit 610 transmits the voice data to the terminal device 300 (step S1301). The terminal device 300 transmits the voice data received from the sound collection device 600 to the information processing device 200 via the communication control unit 360 (step S1302).

[0102] When the information processing device 200 receives voice data input via the input receiving unit 271, the person identification unit 272 identifies the speaker of the voice data (step S1303). Subsequently, the information processing device 200 converts the voice data into text data via the speech recognition unit 273 (step S1304), and stores the information indicating the speaker identified in step S1303 and the text data as dialogue text data indicating the history of the conversation (step S1304).

[0103] Next, the information processing device 200 uses the dialogue text data accumulated in step S1305 to generate a prompt requesting the generation of assist information to be displayed on the display device 700 in the event space. At this time, the prompt generation unit 274 of the information processing device 200 refers to the event information and spatial information to identify one or more prompt types (request content) corresponding to the event space, and generates and transmits those prompts based on templates corresponding to the identified prompt types.

[0104] The information processing device 200 generates a first prompt including identification information for the display device 700A using the prompt generation unit 274 (step S1306). Specifically, the prompt generation unit 274 refers to the event information and spatial information acquired when the event is held and identifies the content of the prompt request associated with the display device 700A included in the event space. Then, the prompt generation unit 274 refers to the template storage unit 255 and acquires a prompt template corresponding to the content of the identified request. Then, the prompt generation unit 274 adds dialogue text data and identification information for the display device 700A to the acquired prompt template to make it the first prompt.

[0105] Next, the information processing device 200 transmits a first prompt to the server device 400 using the prompt transmission unit 275 (step S1307). Subsequently, the information processing device 200 receives first response data from the server device 400 using the response receiving unit 276 (step S1308). The response data received here includes first assist information corresponding to the content of the request contained in the first prompt, and identification information of the display device 700A. The response data may also include the assist type (content of the request).

[0106] Next, the information processing device 200 updates the screen information, including the response data received from the screen information of the screen displayed on the display device 700A (first display device), using the screen information generation unit 277 (step S1309). If no screen is displayed on the display device 700A, the screen information generation unit 277 generates screen information for a screen to be displayed on the display device 700A. The screen information updated here, or the screen information generated here, is associated with the first assist information included in the first response data received in step S1310 and the identification information of the display device 700A.

[0107] Next, the information processing device 200 transmits screen information along with a screen display instruction to the terminal device 300 via the screen information transmission unit 278 (step S1310). The terminal device 300, via the communication control unit 360, transmits the screen information (first display screen information) and a screen display instruction to the display device 700A based on the identification information of the display device 700A contained in the screen information (step S1311).

[0108] When the display device 700A receives screen information and a screen display instruction from the communication control unit 755, the display control unit 765 displays the screen on the display 720 based on the screen information (step S1312). The processing from steps S1313 to S1326 in Figure 13 is the same as the processing from steps S1306 to S1312, except that the first prompt, first response data, and first assist information become the second prompt, second response data, second assist information, third prompt, third response data, and third assist information, respectively, and the identification information of the display device 700A becomes the identification information of the display device 700B (second display device) and the identification information of the display device 700C (third display device), so the explanation is omitted. Note that the screen information transmitted from the information processing device 200 to the display device 700B via the terminal device 300 is an example of the second display screen information, and the screen information transmitted from the information processing device 200 to the display device 700C via the terminal device 300 is an example of the third display screen information. Furthermore, the information on the second display screen is associated with the identification information of the display device 700B, and the information on the third display screen is associated with the identification information of the display device 700C.

[0109] Furthermore, in this embodiment, the assist mode can be identified by the mode identification unit 280 before the processing shown in Figure 13 begins. In this case, the prompt generation unit 274 generates a prompt including a request to select an appropriate assist mode from a plurality of assist modes and sends it to the server device 400, and the prompt transmission unit 275 obtains response data to this prompt. Then, the mode identification unit 280 identifies the assist mode to be adopted based on the response data.

[0110] The assist modes include, for example, an assist mode that helps with the progress of an event when the participants are the main drivers of the event, and a mode in which the large-scale language model 411 takes the lead in facilitating the event. The assist mode is selected by the large-scale language model 411 based on the voice tone of the participants' voice data, etc. Furthermore, if an imaging device is included in the event space, the assist mode may be selected by the large-scale language model 411 based on the image data of the participants captured by the imaging device.

[0111] This allows the system to automatically switch from assist mode to facilitator mode if discussions at an event reach an impasse.

[0112] Once the assist mode is identified, the prompts generated in steps S1306, S1313, and S1320, respectively, will contain text data indicating the content of the request corresponding to the assist mode. For example, the facilitator mode is set to the content of a "command center" request, and the text data for "command center" is selected. In the support mode, the content of an "information retrieval" or "summary generation" request is set, and the text data for "information retrieval" or "summary generation" is selected.

[0113] In this embodiment, by doing so, the amount of information in the assistance information, the timing of updating the assistance information, and the content of the assistance information can be changed according to the degree of assistance in the progress of the event.

[0114] Next, the operation of the information processing system 100 will be specifically explained with reference to Figures 14 to 20. In the following explanation, it will be assumed that the event being held is a "Science of Taste Workshop," and that the information processing device 200 has acquired event information identified by event identification information "I0001" and spatial information identified by spatial ID "1," which is associated with event identification information "I0001."

[0115] Furthermore, Conference Room 1, which will serve as the event space, will include display devices 700A, 700B, and 700C, with the identification information for display device 700A being DY1, the identification information for display device 700B being DY2, and the identification information for display device 700B being DY3.

[0116] Figure 14 shows an example of a first prompt in the first embodiment. The prompt 140 shown in Figure 14 is an example of a first prompt generated in step S1306 of Figure 13.

[0117] The prompt generation unit 274 generates a prompt requesting a "dialogue" from the large-scale language model 411 based on the item "request content" included in the spatial information identified by spatial ID "1". Specifically, the prompt generation unit 274 generates a prompt 140 in the template storage unit 255 that includes text data 1 where the value of the item "request content" is "dialogue" and the dialogue text data accumulated in step S1305.

[0118] Prompt 140 includes text data 141, 142, and 143. Text data 141 is text data indicating event information for an event being held. Text data 142 is dialogue text data stored in the information processing device 200. Text data 143 is text data included in text data 143 in the template storage unit 255 where the value of the item "Content of Request" is "Dialogue".

[0119] Text data 141 includes event identification information "I0001" and the event name. Text data 143 includes the content of the request and identification information DY1 of the display device 700A, which identifies the display destination of the assist information included in the response data. A prompt based on text data where the value of the item "Content of Request" is "Dialogue" is a prompt that enables dialogue with the AI ​​agent, responding to the user's utterance.

[0120] In this embodiment, by sending prompts to the server device 400 in this manner, the display 780 of the display device 700A can be used as a display area for assistance information provided by the AI ​​agent that realizes the dialogue function.

[0121] Figure 15 is a first diagram showing an example of the display of the display device of the first embodiment. The screen 150 shown in Figure 15 is an example of the screen displayed on the display device 700A in step S1312 of Figure 13. The screen 150 includes display areas 151 and 152. Display area 151 displays a portion of the dialogue text data. Display area 152 displays the first assist information.

[0122] The first type of assistance information is text data representing the dialogue content generated by the large-scale language model 411 based on the dialogue text data. Specifically, in response to a user's statement, "What other opinions do you think there are?", the first type of assistance information might be text data such as "I have this opinion." Alternatively, the first type of assistance information may also be text data representing the response generated by the large-scale language model 411 (the AI ​​agent that implements the dialogue function) to a question posed by an event participant to the large-scale language model 411.

[0123] Figure 16 shows an example of a second prompt in the first embodiment. The prompt 160 shown in Figure 16 is an example of a second prompt generated in step S1313 of Figure 13.

[0124] The prompt generation unit 274 generates a prompt requesting "information retrieval" from the large-scale language model 411 based on the item "request content" included in the spatial information identified by spatial ID "1". Specifically, the prompt generation unit 274 generates a prompt 160 in the template storage unit 255 that includes text data 2 where the value of the item "request content" is "information retrieval" and the dialogue text data accumulated in step S1305.

[0125] Prompt 160 includes text data 141, 142, and 161. Text data 161 is text data included in text data 2 in the template storage unit 255 where the value of the item "Request Content" is "Information Retrieval". Text data 161 includes the request content and the identification information DY2 of the display device 700B which identifies the display destination of the assist information included in the response data. A prompt based on text data where the value of the item "Request Content" is "Information Retrieval" is a prompt that causes the user to perform an information retrieval based on their utterance.

[0126] In this embodiment, by sending a prompt to the server device 400 in this manner, the display 780 of the display device 700B can be used as a display area for assist information provided by the AI ​​agent that realizes the information retrieval function.

[0127] Figure 17 is a second diagram showing an example of the display of the display device of the first embodiment. The screen 170 shown in Figure 17 is an example of the screen displayed on the display device 700B in step S1319 of Figure 13. The display area 171 of screen 170 displays the second assist information.

[0128] The second type of assist information is information that shows the search results obtained when the large-scale language model 411 performs information retrieval based on the utterance content indicated by the dialogue text data. In addition to text data, the second type of assist information includes figures and tables, and URLs (Uniform Resource Locator) for displaying the search results on the display device 700B.

[0129] Figure 18 shows an example of a third prompt in the first embodiment. The prompt 180 shown in Figure 18 is an example of a third prompt generated in step S1320 of Figure 13.

[0130] The prompt generation unit 274 generates a prompt requesting the large-scale language model 411 to "create a summary" based on the item "Content of request" included in the spatial information identified by spatial ID "1". Specifically, the prompt generation unit 274 generates a prompt 180 in the template storage unit 255 that includes text data 3 where the value of the item "Content of request" is "create a summary" and the dialogue text data accumulated in step S1305.

[0131] Prompt 180 includes text data 141, 142, and 181. Text data 181 is text data included in text data 3 in the template storage unit 255 where the value of the item "Request Content" is "Create Summary". Text data 181 includes the request content and the identification information DY3 of the display device 700C which identifies the display destination of the assist information included in the response data. A prompt based on text data where the value of the item "Request Content" is "Generate Summary" is a prompt that generates a summary based on the user's utterance.

[0132] In this embodiment, by sending a prompt to the server device 400 in this manner, the display 780 of the display device 700C can be used as a display area for assistance information provided by the AI ​​agent that realizes the summary creation function.

[0133] Figure 19 is a third figure showing an example of the display of the display device of the first embodiment. The screen 190 shown in Figure 19 is an example of the screen displayed on the display device 700C in step S1326 of Figure 13.

[0134] Screen 190 includes display areas 191 and 192. Display area 191 displays a summary created by the large-scale language model 411 based on the dialogue text data. The summary displayed in display area 191 is information that visually represents the content and ideas of the discussions among event participants as diagrams and illustrations, based on the dialogue text data. Display area 192 displays text data that assists in the progress of the event based on the summary displayed in display area 191.

[0135] Thus, the third type of assistance information includes information that visually represents the information indicated by the dialogue text data as a diagram, and information that provides advice on the progress of the event, derived from the information that visually represents the information indicated by the dialogue text data as a diagram.

[0136] In this embodiment, the display processing of the communication control unit 360 of the terminal device 300 in steps S1312, S1319, and S1326 may be performed for each browser window. Specifically, the terminal device 300 opens multiple browser windows, and the communication control unit 360 controls each window to be displayed on the display devices 700A, 700B, and 700C. In the terminal device 300, the user specifies the type of assistance for each browser window, and according to the specified type of assistance, the information processing device 200 sends screen information containing response data to the prompt corresponding to the type of assistance for each browser window to the terminal device 300, and displays it in the corresponding window.

[0137] In this embodiment, the first to third prompts include event information, which allows for the acquisition of assistance information appropriate to the event. This is because event information can be provided as prerequisite information for the request (prompt) input to the large-scale language model 411. In other words, response data can be obtained that takes into account event information such as the event name, purpose, content, description, number of participants, duration, and location. Furthermore, the first to third prompts in this embodiment may also include spatial information of the event space where the event is held. By including spatial information in the first to third prompts, information indicating the characteristics of the event space can be provided to the large-scale language model 411, allowing for the acquisition of assistance information appropriate to the event space.

[0138] Furthermore, in this embodiment, response data corresponding to the content of requests in the first to third prompts can be obtained from the large-scale language model 411. In other words, in this embodiment, assistance information corresponding to the content of requests in the first to third prompts can be obtained. The assistance information in this embodiment includes at least one of the following: information that visually summarizes the utterances in the event, information that is the result of searching for information related to the utterances in the event, information that facilitates the event, and information that responds to the utterances of people in the event.

[0139] Figure 20 shows the event space where the assist information is displayed in the first embodiment.

[0140] In Conference Room 1, which is an event space identified by spatial ID "1", when an event is held, screen 150 containing the first assist information, screen 170 containing the second assist information, and screen 190 containing the third assist information are displayed on the display devices 700A, 700B, and 700C installed in Conference Room 1, respectively.

[0141] The assistance information displayed on each screen is updated as needed, based on what event participants say or as a certain amount of time passes. Therefore, according to this embodiment, information that assists the progress of the event can be displayed within the event space, and various event-related information can be provided to participants present in the event space.

[0142] (Second Embodiment) A second embodiment will be described below with reference to the drawings. The second embodiment differs from the first embodiment in that the large-scale language model 411 determines the type of assist information to be displayed on each of the display devices 700 in the event space. In the following description of the second embodiment, the differences from the first embodiment will be explained, and components having the same functional configuration as the first embodiment will be given the same reference numerals as those used in the description of the first embodiment, and their explanation will be omitted.

[0143] Figure 21 is a sequence diagram illustrating the operation of the information processing system in the second embodiment. In Figure 21, it is assumed that in the spatial information identified by spatial ID "1", the identification information of the display device 700A and the request content "command center" are associated. The request content "command center" indicates the determination of the type of assist information to be displayed on each of the display devices 700 in the event space for the large-scale language model 411.

[0144] Therefore, in the example shown in Figure 21, the display device 700A displays the response data resulting from sending a prompt to the large-scale language model 411 requesting it to determine the type of assist information to be displayed on the display devices 700B and 700C.

[0145] The processes from step S2101 to step S2105 in Figure 21 are the same as the processes from step S1301 to step S1305, so the explanation is omitted. The information processing device 200 generates a prompt requesting a determination of the type of assist information to be displayed on the display device 700 in the event space using the prompt generation unit 274 (step S2106). In other words, the prompt generation unit 274 generates a prompt requesting a determination of a combination of identification information of the display device 700 in the event space and information indicating the type of assist information to be displayed on the display device 700 identified by the identification information.

[0146] Next, the information processing device 200 transmits the generated prompt to the server device 400 using the prompt transmission unit 275 (step S2107), and the response receiving unit 276 receives the response data from the server device 400 (step S2108). The received response data includes identification information for the display device 700 in the event space and the type of assist information to be displayed on the display device 700 identified by the identification information.

[0147] Next, the information processing device 200, using the prompt generation unit 274, refers to the response data and obtains identification information of the display device 700 to be displayed and information indicating the type of assist information to be displayed on the display device 700 (step S2109). In other words, the prompt generation unit 274 identifies the display device 700 to be displayed and the content of the request to the large-scale language model 411 based on the response data.

[0148] Next, the prompt generation unit 274 generates a prompt that includes identification information of the identified display device 700 and the content of the request corresponding to the type of assist information (step S2110).

[0149] Here, it is assumed that the display device 700B is identified as the display device 700 to which the assist information will be displayed, and that the content of the request is information retrieval. Therefore, the prompt generation unit 274 generates a prompt that includes the identification information of the display device 700B and the dialogue text data, based on the prompt template corresponding to the information retrieval.

[0150] Next, the information processing device 200 sends the prompt generated in step S2110 to the server device 400 via the prompt transmission unit 275 (step S2111), and receives response data from the server device 400 via the response receiving unit 276 (step S2112). The processing from steps S2113 to S2116 in Figure 21 is the same as the processing from steps S1316 to S1319 in Figure 13, so the explanation is omitted.

[0151] Next, the prompt generation unit 274 generates a prompt that includes the identification information of the identified display device 700 and the content of the request corresponding to the type of assist information (step S2117). Here, it is assumed that the display device 700C is identified as the display device 700 to which the assist information will be displayed, and that the content of the request is summary creation. Therefore, the prompt generation unit 274 generates a prompt that includes the identification information of the display device 700C and dialogue text data, based on the prompt template corresponding to summary creation.

[0152] The process from step S2118 to step S2123 in Figure 21 is the same as the process from step S1321 to step S1326 in Figure 13, so the explanation is omitted.

[0153] Next, the information processing device 200 generates screen information for the screen to be displayed on the display device 700A using the screen information generation unit 277 (step S2124). Specifically, the screen information generation unit 277 generates screen information for displaying a screen that includes information indicating the type of assist information to be displayed on the display devices 700B and 700C, respectively.

[0154] Next, the information processing device 200 transmits the generated screen information to the terminal device 300 via the screen information transmission unit 278 (step S2125). The terminal device 300 transmits the screen information to the display device 700A via the communication control unit 360 (step S2126).

[0155] When the display device 700A receives screen information from the communication control unit 755, the display control unit 765 causes the display based on the screen information to be displayed on the display 780 (step S2127).

[0156] Figure 22 shows an example of a prompt in the second embodiment. Prompt 195 shown in Figure 22 is an example of a prompt generated in step S2106 of Figure 21.

[0157] Prompt 195 contains text data 141, 142, and 196.

[0158] Text data 196 is text data requesting a decision on the combination of a display device 700 in the event space and the type of assist information to be displayed on this display device 700.

[0159] Figure 23 shows an example of the display of the display device in the second embodiment. Figure 23 shows the event space in the second embodiment where the assist information is displayed.

[0160] In the example shown in Figure 23, display device 700A is displaying screen 150A, display device 700B is displaying screen 170, and display device 700C is displaying screen 190.

[0161] Screen 150A displayed on the display device 700A is an example of the screen displayed on the display device 700A in step S2127 of Figure 21. Screen 150A includes a display area 153. The display area 152 displays information indicating the type of assist information to be displayed on the display devices 700B and 700C, respectively.

[0162] In this embodiment, the display device 700A, which is associated with the content of the request "command center" and identification information, displays the type of assist information that is displayed on other display devices 700. In this embodiment, this allows event participants to understand that the information displayed on the display device 700A is response data from the large-scale language model 411, which acts as a command center that manages the progress of the event.

[0163] In this embodiment, for example, if the content of the request corresponding to the display device 700B in the response data received in step S2108 is "dialogue", the prompt generation unit 274 may generate a prompt in step S2110 that requests dialogue with the large-scale language model 411 which functions as a command center.

[0164] In this case, the display device 700A displays response data from the large-scale language model 411, which functions as a command center, and the display device 700B displays response data from the large-scale language model 411, which uses the response data displayed on the display device 700A as input data.

[0165] In this way, it becomes possible to enable dialogue between functions realized by the large-scale language model 411, and to allow event participants to view the content of the dialogue.

[0166] (Third embodiment) A third embodiment will be described below with reference to the drawings. The third embodiment differs from the first embodiment in that it displays multiple types of assist information on the display device 700. In the following description of the third embodiment, the differences from the first embodiment will be explained, and components having the same functional configuration as the first embodiment will be given the same reference numerals as those used in the description of the first embodiment, and their explanations will be omitted.

[0167] Figure 24 is a diagram illustrating the functional configuration of each device in the information processing system of the third embodiment.

[0168] In the information processing system 100A of this embodiment, the information processing device 200 has a storage unit 230 and a control unit 270A. The control unit 270A has all the parts of the control unit 270 of the first embodiment, plus a layout determination unit 281.

[0169] The layout determination unit 281 determines the screen layout of the display device 700 when displaying multiple types of assist information on one display device 700 included in the event space.

[0170] Figure 25 is a sequence diagram illustrating the operation of the information processing system in the third embodiment. Figure 25 shows the operation when an event identified by event identification information "I0001" is held in an event space identified by space ID "2" to which a space ID is associated. Conference room 2, which is the event space identified by space ID "2", includes one display device 700 identified by identification information DY4, and the screens displayed on the display device 700 are three screens identified by the identification information G1, G2, and G3 of each screen (see Figure 7). Furthermore, in the display device 700 included in conference room 2, the screen identified by identification information G1 is identified as the display destination where response data from the large-scale language model 411 is displayed when the large-scale language model 411 is functioning as a command center to assist in the progress of the event.

[0171] The process from step S2501 to step S2512 in Figure 25 is the same as the process from step 2101 to step S2112 in Figure 21, except that the destination of the assist information display is the identification information of the display device 700, and the identification information of the display device 700 is the identification information that identifies the screen displayed on the display device 700. Therefore, the explanation is omitted.

[0172] Following step S2512, the information processing device 200A determines the layout of the multiple screens to be displayed on the display device 700 using the layout determination unit 281 (step S2513). The layout determination unit 281 may determine the layout such that, for example, on the display 780 of the display device 700 identified by the identification information DY4, the display area of ​​the screen with the most recent assist information displayed is maximized.

[0173] The process from step S2514 to step S2520 in Figure 25 is the same as the process from step S2113 to step S2119 in Figure 21, so the explanation is omitted.

[0174] Following step S2520, the information processing device 200A determines the layout of the multiple screens to be displayed on the display device 700 using the layout determination unit 281 (step S2521). The processes from steps S2512 to S2529 in Figure 25 are the same as the processes from steps S2114 to S2127 in Figure 21, so their explanation is omitted.

[0175] Next, an example of the display of the display device of this embodiment will be described with reference to Figure 26. Figure 26 is a diagram showing an example of the display of the display device of the third embodiment.

[0176] The screen 260A shown in Figure 26(A) is an example of a screen displayed on the display device 700 in step S2517 of Figure 25. Screen 260A includes screen 150B and screen 170A. Screen 150B is the screen identified by identification information G1 in the spatial information identified by spatial ID "2", and screen 170A is the screen identified by identification information G2 in the spatial information identified by spatial ID "2".

[0177] In screen 260A, screen 150B displays information indicating that a summary is being created by the large-scale language model 411, and screen 170A displays the assistance information contained in the response data received in step S2512 of Figure 25.

[0178] Screen 260B, shown in Figure 26(B), is an example of a screen displayed on the display device 700 in step S2529 of Figure 25. Screen 260B includes screens 150B, 170A, and 190A. Screen 190A is a screen identified by identification information G3 in the spatial information identified by spatial ID "2".

[0179] In screen 260B, screen 150B displays information indicating the type of assistance information displayed on the other two screens, and screen 190A displays the assistance information included in the response data received in step S2520 of Figure 25.

[0180] Thus, according to this embodiment, different types of assist information can be displayed on a single display device 700. Therefore, according to this embodiment, even if there is only one display device 700 installed in the event space, multiple types of assist information can be displayed.

[0181] (Fourth embodiment) The fourth embodiment will be described below with reference to the drawings. The fourth embodiment differs from the first embodiment in that the type of assist information displayed is determined according to the flow of the event. In the following description of the fourth embodiment, the differences from the first embodiment will be explained, and components having the same functional configuration as the first embodiment will be given the same reference numerals as those used in the description of the first embodiment, and their explanation will be omitted.

[0182] Figure 27 is a sequence diagram illustrating the operation of the information processing system according to the fourth embodiment. Figure 27 illustrates the operation when setting assist information to be displayed on the display device 700 according to the flow of event progression.

[0183] The information processing device 200 generates screen information for a settings screen, which sets the flow of events and the type of assistance information, using the screen information generation unit 277, and transmits a screen display instruction to the terminal device 300 using the screen information transmission unit 278 (step S2701).

[0184] The terminal device 300 displays a settings screen via the display control unit 350, and accepts the selection of the event progression flow and the type of assist information on the settings screen (step S2702). Subsequently, the terminal device 300 transmits the selection result to the information processing device 200 via the communication control unit 360 (step S2703).

[0185] When the information processing device 200 receives a selection result from the terminal device 300 via the input receiving unit 271, the event storage unit 240 sets the type of assist information selected for the event (step S2704). In other words, the information processing device 200 associates information indicating the type of assist information selected with the event information targeted for setting in the event storage unit 240.

[0186] Next, the information processing device 200 generates screen information for displaying the event start screen using the screen information generation unit 277 (step S2705), and transmits the generated screen information to the terminal device 300 using the screen information transmission unit 278 (step S2706). The terminal device 300 transmits the screen information to the display device 700 using the communication control unit 360 (step S2707). The display device 700 causes the display 780 to display a screen based on the screen information using the display control unit 765 (step S2708).

[0187] Next, the information processing system 100 performs the processing from step S2709 to step S2720 each time a participant in the event space speaks, or at predetermined intervals.

[0188] The process from steps S2709 to S2720 in Figure 27 is the same as the process from steps S1301 to S1312 in Figure 13, except that the prompt generated in step S2714 includes the type of assist information set in step S2702 and the content of the corresponding request, so the explanation is omitted.

[0189] The following describes the settings screen displayed on the terminal device 300 in step S2702, with reference to Figure 28.

[0190] Figure 28 is the first diagram showing an example of the display of the terminal device in the fourth embodiment. The screen 290 shown in Figure 28 is an example of a settings screen displayed on the terminal device 300 in step S2702 of Figure 27. Note that screen 290 can also be displayed on the display device 700.

[0191] Screen 290 includes display areas 291 and 292 and operation buttons 293. Display area 291 displays the name of the event. Display area 292 displays information that associates the flow of the event with the type of assistance information corresponding to the flow of the event.

[0192] In the example shown in Figure 28, an event named "Customer Experience Workshop" is selected, and the flow of this event is set to icebreaker, discussion to deepen understanding, proposal creation, and presentation. In this embodiment, information indicating the flow of the event is pre-set as part of the event information and is stored in the event storage unit 240 as part of the event information.

[0193] Furthermore, the information indicating the flow of the event can be information that associates the time period from the start time to the end time of the event with the progress of each event. For example, in this embodiment, the progress of the event during a predetermined time period from the start time can be designated as "icebreaker," and the progress of the event during the next time period can be designated as "discussion to deepen understanding," etc.

[0194] Furthermore, in the example shown in Figure 28, the type of assist information displayed on the display device 700 in the event space is associated with each stage of the event's progress. Specifically, when the event's progress is "icebreaker," the type of assist information displayed on the display device 700 is "dialogue." Also, when the event's progress is "icebreaker," the type of assist information displayed on the display device 700 is "information retrieval."

[0195] Operation button 293 is used to set the information processing device 200 based on the content displayed in the display area 292. When operation button 293 is selected on screen 290, information displayed in the display area 292, which associates the flow of event progression with the type of assistance information, is sent to the information processing device 200.

[0196] In addition, information showing a typical flow of an event may be pre-stored in the memory unit 230 as information indicating the flow of the event's progress.

[0197] Furthermore, the information processing device 200 in this embodiment can input the results of events included in the event information stored in the event storage unit 240 into the large-scale language model 411, thereby acquiring information indicating the relationships between past events and planned events, and displaying it on the display device 700.

[0198] Figure 29 is a second diagram showing an example of the display of the terminal device in the fourth embodiment. The screen 295 shown in Figure 29 can also be displayed on the display device 700.

[0199] Screen 295 includes display areas 296 and 297. Display area 296 displays a list of event results included in the event information of relevant past events. Display area 297 displays advice information based on the event results.

[0200] In this embodiment, screen 295 can be displayed on the display device 700 or terminal device 300 before the event or after the event has ended.

[0201] By displaying a screen like this, event participants can view information about upcoming events, as well as past event results related to completed events.

[0202] (Fifth embodiment) The fifth embodiment will be described below with reference to the drawings. The fifth embodiment differs from the first embodiment in that the emotion information estimated using the emotion estimation device is displayed as part of the assistance information, and the information processing system is linked with an external system. In the following description of the fifth embodiment, the differences from the first embodiment will be explained, and components having the same functional configuration as the first embodiment will be given the same reference numerals as those used in the description of the first embodiment, and their explanation will be omitted.

[0203] Figure 30 shows an example of the system configuration of the information processing system of the fifth embodiment. The information processing system 100B of this embodiment includes an information processing device 200B, a terminal device 300, a sound collection device 600, and display devices 700A, 700B, 700C, and 700D. In the information processing system 100B, the information processing device 200B is connected to a server device 400, an emotion estimation device 800, an external system 900, and a communication network N1.

[0204] The emotion estimation device 800 collects biometric information from each event participant and estimates the emotions of the participant from whom the biometric information was obtained based on the collected biometric information. Specifically, the emotion estimation device 800 is installed in the event space together with the terminal device 300, the sound collection device 600, and the display devices 700A, 700B, 700C, and 700D, and collects biometric information from people in the event space, analyzes the biometric information, and estimates the emotions of people in the event space. The emotion estimation device 800 can collect biometric information while in contact with the human body or without contact with the human body. Multiple emotion estimation devices 800 can be installed in the event space. In that case, the emotion estimation device 800 only needs to collect biometric information from the participant closest to it.

[0205] The biometric information collected by the emotion estimation device 800 includes, for example, information indicating the participant's heart rate, pulse rate, blood pressure, body temperature, and body movement. Furthermore, the emotions of a person estimated by the emotion estimation device 800 include, for example, immersion level (degree of concentration), drowsiness level (boredom, laziness), activity level (tension, energy), and exhaustion level (degree of fatigue). In the following explanation, the emotions of a participant estimated based on biometric information will be referred to as emotional information.

[0206] The emotion estimation device 800 can acquire both biometric information and emotion information and transmit them to the terminal device 300, or it can transmit only biometric information to the terminal device 300. The type of information transmitted from the emotion estimation device 800 to the terminal device 300 can be set in the terminal device 300. The terminal device 300 transmits the biometric information and emotion information acquired from the emotion estimation device 800 to the information processing device 200B.

[0207] In this embodiment, the emotion estimation device 800 is installed within the event space, but it is not limited to this. The emotion estimation device 800 can also be, for example, a wearable terminal worn by participants within the event space. In this case, the emotion estimation device 800 communicates with the terminal device 300 via wireless communication such as Bluetooth® and transmits biometric information and emotion information to the terminal device 300. Furthermore, if the emotion estimation device 800 has a sound collection function, it can be used as a sound collection device instead of the sound collection device 600 installed within the event space.

[0208] Furthermore, in this embodiment, the emotion estimation device 800 transmits biometric information and emotion information to the terminal device 300, but is not limited to this. The emotion estimation device 800 can also transmit biometric information and emotion information to the information processing device 200B or the server device 400 via the communication network N1.

[0209] In this embodiment, the external system 900 stores the dialogue text data accumulated in the information processing device 200B. The external system 900 is a system that utilizes data acquired by the information processing system 100B. The data acquired by the information processing system 100B includes dialogue text data and assistance information.

[0210] The external system 900 in this embodiment can be, for example, a business management system that stores summary information and minutes of meetings, and manages tasks extracted during meetings. The external system 900 can also be a customer management system that stores audio data and text data indicating the content spoken during business negotiations and meetings as customer data for each customer, and proposes countermeasures for each customer. Furthermore, the external system 900 can be a display support system that displays summary information and minutes of meetings on a display device such as an electronic whiteboard in a chronological order or according to the content of the data. Finally, the external system 900 can be a summary management system that manages meeting summary information for each user of the external system 900.

[0211] Next, with reference to Figure 31, the functional configuration of each device in the information processing system 100B of this embodiment will be described. Figure 31 is a diagram illustrating the functional configuration of each device in the information processing system of the fifth embodiment.

[0212] The information processing device 200B of this embodiment includes a storage unit 230A and a control unit 270B. The storage unit 230A includes an event storage unit 240, a spatial storage unit 250A, a template storage unit 255A, and a summary management storage unit 265. The summary management storage unit 265 stores summary information to be output to the external system 900 and summary information acquired from the external system 900. Details of the spatial storage unit 250A, the template storage unit 255A, and the summary management storage unit 265 will be described later.

[0213] The control unit 270B includes, in addition to the components of the control unit 270, an information acquisition unit 282, a storage control unit 283, and an output unit 284. The information acquisition unit 282 acquires various types of information. Specifically, for example, the information acquisition unit 282 acquires biometric information collected by the emotion estimation device 800 and emotion information indicating emotions estimated by the emotion estimation device 800 from the emotion estimation device 800. The information acquisition unit 282 also acquires various types of information from the external system 900.

[0214] When summary information is generated in an event, the storage control unit 283 associates the generated summary information with the event identification information and the spatial ID and stores it in the summary management storage unit 265. Furthermore, if the generated summary information is generated in accordance with the specifications of the external system 900, the storage control unit 283 also associates the summary information with the identification information that identifies the corresponding external system 900 and stores it in the summary management storage unit 265.

[0215] Furthermore, when the information acquisition unit 282 acquires summary information from the external system 900, the storage control unit 283 associates the acquired summary information with identification information that identifies the external system 900 from which the summary information was acquired and stores it in the summary management storage unit 265.

[0216] The output unit 284 outputs various types of information from the information processing device 200. Specifically, the output unit 284 outputs the summary information stored in the summary management storage unit 265 to the external system 900. Furthermore, if the summary information stored in the summary management storage unit 265 has not been converted to a format (template) corresponding to the external system 900, the output unit 284 converts the summary information to a format corresponding to the external system 900 and outputs the converted summary information to the external system 900. In this case, it is assumed that the information processing device 200B has already acquired information indicating the format of the summary information corresponding to the external system 900.

[0217] Furthermore, in the information processing device 200B, the storage unit 230A can store dialogue text data and assist information, which is response data output from the server device 400 in relation to the dialogue text data, in association with each other. The output unit 284 can also output the dialogue text data and assist information stored in the storage unit 230A to the external system 900.

[0218] Next, with reference to Figures 32 to 35, the spatial storage unit 250A, template storage unit 255A, and summary management storage unit 265 of the information processing device 200B of this embodiment will be described.

[0219] Figure 32 shows an example of a spatial memory unit of the fifth embodiment. In the spatial memory unit 250A of this embodiment, the space identified by spatial ID "1", "Conference Room 1", has four displays and a camera, and the identification information of the four displays is "DY1", "DY2", "DY3", and "DY4", respectively. Furthermore, the display with identification information "DY1" is associated with "Dialogue", which is the content of the request included in the prompt, and the display with identification information "DY2" is associated with "Information Retrieval", which is the content of the request included in the prompt.

[0220] Furthermore, in the spatial memory unit 250A, the display for identification information "DY3" is associated with the content of the request included in the prompt, "summary generation tailored to the external system 900," and the display for identification information "DY2" is associated with the content of the request included in the prompt, "emotion estimation" and "emotion-based suggestions."

[0221] In this embodiment, the information acquisition unit 282 of the information processing device 200B can select either "emotion estimation" or "emotion-based suggestion," which are the content of the requests included in the prompt, depending on the information it acquires from the emotion estimation device 800. For example, if the information acquired by the information acquisition unit 282 from the emotion estimation device 800 is emotion information estimated by the emotion estimation device 800, then "emotion-based suggestion" is selected as the content of the request. If the information acquired by the information acquisition unit 282 from the emotion estimation device 800 is biometric information, then either "emotion estimation" or both "emotion estimation" and "emotion-based suggestion" are selected as the content of the request.

[0222] Figure 33 shows an example of a template storage unit of the fifth embodiment. The template storage unit 255A of this embodiment includes text data 3-1, which is a prompt template corresponding to the request content "summary generation tailored to external system 900", text data 5, which is a prompt template corresponding to the request content "emotion estimation", and text data 6, which is a prompt template corresponding to "emotion-based suggestion".

[0223] The text data 3-1 of this embodiment includes information that identifies the external system 900 to which the summary information will be output. Specifically, for example, the text data 3-1 may read: "The summary information will be output to the 'identification information that identifies the external system'. The specifications of the 'identification information that identifies the external system' are... Please generate the summary information according to the specifications of the 'identification information that identifies the external system'."

[0224] The text data 5 in this embodiment includes content requesting "emotion estimation." Specifically, the text data 5 may read, "Estimate the emotions of each event participant and generate an icon image for each participant. Generate an icon image showing the average emotions of all event participants." The prompt generated based on the text data 5 also includes biometric information obtained from the emotion estimation device 800. While the emotions of event participants are estimated based on biometric information, this is not the only way. Instead of biometric information obtained from the emotion estimation device 800, the emotions of event participants can also be estimated based on the tone of voice indicated by the audio data collected by the sound collection device 600, the speaker's speaking speed, etc. In other words, in this embodiment, audio data can also be treated as part of the biometric information.

[0225] The text data 6 in this embodiment includes content requesting "suggestions based on emotions." Specifically, the text data 6 may read, "Please suggest improvements to improve the situation based on the emotions of the event participants." By requesting suggestions for improvements to improve the situation in this way, if the estimated emotions are negative, it is possible to help to change those negative emotions into positive ones.

[0226] Figure 34 shows an example of a summary management storage unit of the fifth embodiment. The summary management information stored in the summary management storage unit 265 of this embodiment includes, as information items, event identification information, spatial ID, summary information, external system, etc. In the summary management information, the value of the item "summary information" indicates summary information of dialogue text data generated by the information processing device 200B using the server device 400, or summary information obtained from the external system 900. Note that the summary information generated by the information processing device 200B using the server device 400 can also be summary information based on dialogue text data and assist information.

[0227] The item "External System" is further associated with the items "Import Source" and "Export Destination". The value of the item "Import Source" indicates identification information that identifies the external system 900 from which the summary information is obtained. The value of the item "Export Destination" indicates identification information that identifies the external system 900 to which the summary information is output.

[0228] In the example in Figure 34, it can be seen that the summary information generated from the dialogue text data and assistance information of the event identified by event identification information "I0001" and spatial ID "1" is exported to the business management system, which is one of the external systems 900.

[0229] Furthermore, in the example in Figure 34, it can be seen that summary information associated with the event identified by event identification information "I0002" and spatial ID "2" is imported from the customer management system, which is one of the external systems 900. If summary information imported from the external system 900 exists in the summary management storage unit 265, the information processing device 200B can display the summary information imported from the external system 900 on the display device when this event starts. In the example in Figure 34, when the event identified by event identification information "I0002" and spatial ID "2" is held, the information processing device 200B displays the summary information associated with event identification information "I0002" and spatial ID "2" on the display device 700.

[0230] In Figure 34, the information imported from the external system 900 is shown as summary information, but this is not the only option. The information imported from the external system 900 can be arbitrarily set by the user of the terminal device 300. Furthermore, the summary information imported from the external system 900 can also be summary information previously exported from the information processing device 200B to the external system 900.

[0231] Thus, according to this embodiment, the summary information for each event generated in the information processing device 200B can be utilized in the external system 900. Furthermore, in this embodiment, by displaying the information acquired (imported) from the external system 900 on the display device 700 at the start of an event, it is possible to review the contents of the previous event at the start of the event.

[0232] Next, with reference to Figure 35, a modified example of the spatial storage unit 250A of this embodiment will be described. Figure 35 shows another example of the spatial storage unit of the fifth embodiment.

[0233] In the spatial storage unit 250B shown in Figure 35, a session ID is used instead of identification information to identify the display device 700. In this embodiment, the session ID is identification information that identifies communication from the information processing device 200B to the server device 400. In other words, the session ID is identification information that associates the prompt output from the information processing device 200B with the response data received from the server device 400.

[0234] In the spatial storage unit 250B, the value of the item "Spatial Information" contains identification information that identifies the terminal device 300, instead of identification information that identifies the display device 700. In addition, in the spatial storage unit 250B, the value of the item "Request Content" associates the request content with the session ID.

[0235] When spatial information is stored, such as in the spatial memory unit 250B, the information processing device 200B transmits screen information associated with the session ID to the terminal device 300.

[0236] When the terminal device 300 receives screen information associated with a session ID, the display control unit 350 launches a web browser to display screen information for each session ID, and displays a window containing the screen information associated with the session ID on each of the display devices 700 connected to the terminal device.

[0237] Next, with reference to Figures 36 and 37, an example of the display of the display device 700D of this embodiment will be described. Figure 36 is a diagram showing an example of the display of the display device of the fifth embodiment.

[0238] The screens 390 and 390B shown in Figures 36(A) and (B) are examples of screens displayed on the display device 700D by the same process as the process from step S1301 to step S1312 in Figure 13.

[0239] In other words, screens 390 and 390B are examples of displays when the information processing device 200B outputs a prompt containing text data 5 and text data 6 from the template storage unit 255A to the server device 400, and obtains information about the participant's emotions as assist information. In the following description, the prompt containing text data 5 and text data 6 from the template storage unit 255A will be called the fourth prompt, and the assist information displayed on the display device 700D will be called the fourth assist information. The fourth prompt is a prompt that includes a request for either emotion estimation, a suggestion based on emotion estimation, or both.

[0240] Screen 390 includes a display area 391, an icon image 392, and a display area 393. Display area 391 displays the average emotions of the event participants. Icon image 392 is an icon image corresponding to the average emotions of the event participants. Display area 393 displays suggestions for improving the situation. In addition to the example shown in Figure 39(A), display area 393 may also display suggestions such as, for example, "Your concentration seems to be decreasing, let's refresh."

[0241] Screen 390A includes display areas 394, 395, and 396. Display area 394 displays icon images representing the emotions of each event participant. Display area 395 displays icon images for each emotion. Display area 396 displays waveforms showing the changes in the emotions of each event participant. Changes in emotions include, for example, changes in the degree of tension or the intensity of drowsiness.

[0242] Figure 37 shows the event space where the assist information is displayed in the fifth embodiment.

[0243] In Conference Room 1, which is an event space identified by spatial ID "1", when an event is held, the following screens are displayed on the display devices 700A, 700B, 700C, and 700D installed in Conference Room 1: screen 150 containing the first assist information, screen 170 containing the second assist information, screen 190 containing the third assist information, and screen 390 containing the fourth assist information.

[0244] In this embodiment, by estimating the emotions of event participants and displaying them on the display device 700 in the event space, participants can understand the emotions of other participants, which can contribute to the liveliness of the event. Furthermore, in this embodiment, suggestions for improving the event situation can be displayed on the display device 700 based on the emotions of the event participants, which can support the smooth progress of the event.

[0245] Furthermore, in this embodiment, for example, multiple patterns of response data with different content can be displayed on the display device 700 as assist information in response to the participant's utterance shown in Figure 37. In this way, the participant can be given the experience of having a conversation with multiple third parties with different personalities.

[0246] Specifically, for example, the information processing device 200B includes in its prompt requesting the generation of first assist information to be displayed on the display device 700A content that requests the generation of response data by a first personality, and in its prompt requesting the generation of second assist information to be displayed on the display device 700B content that requests the generation of response data by a second personality of a different type from the first personality. In this way, the first assist information displayed on the display device 700A and the second assist information displayed on the display device 700B can be made to appear as if they were responded by different people.

[0247] Furthermore, in the example shown in Figure 37, display devices 700A and 700B are installed on two corners of the room that serves as the event space. In this embodiment, by arranging the display devices 700 in this way, participant P3, who is located in the corner of the room, can more clearly experience a sense of immersion as if they were conversing with multiple third parties.

[0248] In this embodiment, biometric information acquired from the emotion estimation device 800 is transmitted to the server device 400, and emotions are estimated using the server device 400. However, the embodiment is not limited to this. The information processing device 200B in this embodiment may also have an estimation processing unit that performs estimation processing to estimate emotions based on biometric information. In that case, the information processing device 200B, using the estimation processing unit, estimates the emotions of the participant from whom the biometric information was acquired, based on the biometric information acquired by the information acquisition unit 282 from the emotion estimation device 800.

[0249] Furthermore, in this embodiment, the estimation of emotions based on biometric information can also be performed by an external server device of the information processing system 100B. In that case, the information processing device 200B transmits the biometric information acquired from the emotion estimation device 800 to the external server via the output unit 284, and the information acquisition unit 282 acquires the emotion information estimated based on the biometric information from the external server.

[0250] Next, with reference to Figure 38, the operation of the information processing device 200B and the terminal device 300 will be further explained. Figure 38 is a sequence diagram illustrating the operation of the information processing system according to the fifth embodiment. Figure 38 shows, for example, the process from receiving the operation to select an event in step S1005 of Figure 10 to displaying a screen containing assist information on the terminal device 300 in step S1312 of Figure 13.

[0251] First, before proceeding with the processing, the functions of the terminal device 300 will be described. The terminal device 300 has a web browser 351. The web browser 351 is a general-purpose web browser and includes a browser engine 352, a scripting engine 353, and a network engine 354.

[0252] The browser engine 352 interprets the HTML (HyperText Markup Language) and CSS (Cascading Style Sheets) data that make up the web page and displays the web page. The script engine 353 executes the scripts that make up the web page (for example, JavaScript®). The network engine 354 sends HTTP requests and receives HTTP responses.

[0253] When the terminal device 300 receives an operation to select an event on the event list screen displayed in step S1005 of Figure 10, and receives a display instruction corresponding to the selection (step S3801), the browser engine 352 inputs the URL associated with the display instruction, which is the destination URL for the HTTP request to the information processing device 200B, to the network engine 354 (step S3802). The network engine 354 sends an HTTP request to the said URL (step S3803). The HTTP request sent to the information processing device 200B in step S3803 is an example of a first HTTP request.

[0254] The output unit 284 of the information processing device 200B generates an HTTP response containing Web content data (HTML data, CSS data, and scripts (hereinafter referred to as "JS")) in response to the HTTP request (step S3804). Subsequently, the output unit 284 transmits the HTTP response to the terminal device 300 (step S3805).

[0255] The Web content data in question is an example of Web content data for displaying a start screen (first Web page) which serves as a waiting screen until audio data input from event participants is received, and a screen containing assistance information (second Web page), and includes a script that at least executes the process of sending audio data input from the terminal device 300 to the information processing device 200B, and the process of displaying the second Web page containing assistance information.

[0256] Next, when the network engine 354 of the terminal device 300 receives the HTTP response sent in step S3805, it inputs the HTML data, CSS data, and JS contained in the HTTP response into the browser engine 352 (step S3806). The browser engine 352 inputs the JS input from the network engine 354 into the script engine 353 (step S3807). The script engine 353 loads the JS (step S3808) and requests the browser engine 352 to update the screen (step S3809). The screen update includes displaying a new screen.

[0257] Here, the HTTP response generated in step S3804 may contain the filename of the JS file rather than the actual JS file itself. In this case, in step S3808, the script engine 353 accesses the file based on the filename and downloads the JS file. This method involves loading the JS file as an external file.

[0258] Web content data for displaying a first Web page (start screen) and a second Web page (a screen including assist information), the script included in the Web content data at least executing a process of transmitting voice data input during display of the first Web page to the information processing apparatus 200B and a process of displaying the second Web page including the assist information, includes not only JS directly described in HTML but also JS that is downloaded upon receiving access based on the corresponding file name by a method of loading JS as an external file.

[0259] Subsequently, the browser engine 352 transmits an instruction to display the Web page as the start screen to the display device 700 based on the HTML data and the CSS data (step S3810). The display device 700 displays the start screen as the Web page (step S3811).

[0260] When voice data is input by the user (step S3812), the browser engine 352 notifies the script engine 353 of the input of the voice data (step S3813). In response to the notification from the browser engine 352, the script engine 353 executes JS (step S3814), and inputs the voice data to the network engine 354 (step S3815). The JS executed at this time is JS that executes a process of transmitting an HTTP request including information input by the user such as voice data to the information processing apparatus. The network engine 354 transmits an HTTP request corresponding to the reception of the voice data to the information processing apparatus 200B (step S3816). The HTTP request is an HTTP request transmitted by executing JS at the terminal device 300, and is an example of an HTTP request including a notification indicating that voice data input during display of the start screen displayed on the terminal device 300 based on the Web content data has been input.

[0261] Upon receiving the HTPT request, the information processing apparatus 200B performs steps S1303 to S1316 in FIG. 13 (step S3817). Subsequently, the information processing apparatus 200B generates an HTTP response including JSON (JavaScript (registered trademark) Object Notation) in which the assist information is described, so that the terminal apparatus 300 executes JS to display a screen including the assist information (step S3818). Subsequently, the information processing apparatus 200B transmits the HTTP response to the terminal apparatus 300 (step S3819).

[0262] Upon receiving the HTTP response, the network engine 354 of the terminal device 300 inputs the assist information included in the HTTP response to the script engine 353 (step S3820). The script engine 353 executes JS (step S3821), and requests the browser engine 352 to update the display content of the web page (step S3822). The JS executed at this time is JS that executes processing for updating the screen. Specifically, it is JS that executes processing for updating the screen based on data such as JSON and images included in the HTTP response. Based on the HTML data and CSS data acquired in step S3806, the browser engine 352 transmits a display instruction for a screen including the assist information, which is the updated screen, to the terminal device 300 (step S3823).

[0263] In other words, the browser engine 352 transmits an update instruction for the screen displayed on the display device 700 to the display device 700 based on the HTML data and CSS data acquired in step S3806. The display device 700 updates the start screen to a screen including the assist information in response to the screen update instruction (step S3824).

[0264] As described above, in this embodiment, the start screen and the screen containing assist information, and the execution of processing in response to operations on each of these screens, are realized by a single-page application. Specifically, when displaying the start screen, the terminal device 300 receives web content data that displays the start screen and the screen containing assist information, rather than web content data that is solely for displaying the start screen, and includes JavaScript that executes processing such as screen transitions in response to operations on the start screen. Therefore, the screen transition from the start screen to the screen containing assist information is executed by JavaScript, and the terminal device 300 does not need to download web content data for the screen containing assist information. As a result, it is possible to solve technical problems such as improving the display speed of the screen containing assist information and reducing the communication load during screen transitions.

[0265] (Sixth Embodiment) A sixth embodiment will be described below with reference to the drawings. The sixth embodiment differs from the first embodiment in that the terminal device 300 has the functions of the large-scale language model 411 and the control unit 270. In the following description of the sixth embodiment, the differences from the first embodiment will be explained, and components having the same functional configuration as the first embodiment will be given the same reference numerals as those used in the description of the first embodiment, and their explanation will be omitted.

[0266] Figure 39 is a diagram illustrating the functions of each device in the information processing system of the sixth embodiment.

[0267] The information processing system 100B of this embodiment includes an information processing device 200C, a terminal device 300A, a sound collection device 600, and a display device 700. In the information processing system 100C, the information processing device 200C and the terminal device 300A are connected via a communication network N1.

[0268] The information processing device 200C includes a storage unit 230 and a communication control unit 270C-1. The communication control unit 270C-1 controls communication between the terminal device 300A and the information processing device 200C.

[0269] The terminal device 300A includes a large-scale language model storage unit 410, a control unit 270C-2, and a communication control unit 360. The large-scale language model storage unit 410 stores a large-scale language model 411. The control unit 270C-2 includes an input receiving unit 271, a person identification unit 272, a speech recognition unit 273, a prompt generation unit 274, a prompt output unit 275A, a response output unit 276A, a screen information generation unit 277, a screen information transmission unit 278, an event setting unit 279, and a mode identification unit 280.

[0270] The prompt output unit 275A outputs a prompt to the large-scale language model 411 stored in the large-scale language model memory unit 410 of the machine. The response output unit 276A acquires the response data output from the large-scale language model 411.

[0271] In this embodiment, by providing the functions of the large-scale language model 411 and the control unit 270C-2 in the terminal device 300A, assist information can be displayed on the display device 700 without communicating with the server device 400.

[0272] In Figure 39, the terminal device 300A is shown to implement the functions of the control unit 270C-2, but this is not limited to this configuration. The control unit 270C-2 can be implemented by the terminal device 300A and the information processing device 200C. Also, in the example in Figure 39, the large-scale language model 411 is assumed to be provided by the terminal device 300A, but this is not limited to this configuration. The large-scale language model 411 can also be provided by the information processing device 200C.

[0273] (Seventh Embodiment) The seventh embodiment will be described below with reference to the drawings. The seventh embodiment differs from the first embodiment in that it displays assist information based on processing procedure information that defines the process up to the output of assist information. In the following description of the seventh embodiment, the differences from the first embodiment will be explained, and components having the same functional configuration as the first embodiment will be given the same reference numerals as those used in the description of the first embodiment, and their explanation will be omitted.

[0274] Figure 40 shows an example of the system configuration of an information processing system according to the seventh embodiment. The information processing system 100D of this embodiment includes an information processing device 200D, a terminal device 300, a sound collection device 600, and display devices 700A, 700B, and 700C. In the information processing system 100D, the information processing device 200D and the terminal device 300 are connected via a communication network N1. The information processing device 200D is also connected to a server device 400, a display management device 850, and a processing procedure management device 950 via the communication network N1. The processing procedure management device 950 and the display management device 850 are, for example, examples of computers and can have the same hardware configuration as the information processing device 200D.

[0275] In this embodiment, the display management device 850 generates screen information to display the instructed screen on the display device 700 in response to instructions from the information processing device 200D, and transmits it to the terminal device 300.

[0276] The processing procedure management device 950 of this embodiment manages procedure information that defines the work content for each event. In this embodiment, "work" refers to causing a computer, such as the information processing device 200D or the display management device 850, to execute a series of processes. Furthermore, the procedure information of this embodiment is information that defines the order of the processes to be executed by the information processing device 200D or the display management device 850, and the content of those processes.

[0277] In this embodiment, when an event to be held in the event space is identified, the information processing device 200D acquires procedure information corresponding to the identified event from the procedure information managed by the processing procedure management device 950. Then, based on the procedure information, the information processing device 200D executes the processing specified in the procedure information in the specified order.

[0278] Furthermore, if the processing content specified in the procedure information includes a request to the display management device 850 to generate screen information, the information processing device 200D requests the display management device 850 to generate screen information.

[0279] In the present embodiment, as described above, by causing the information processing apparatus 200D to cooperate with the display management apparatus 850 and the processing procedure management apparatus 950, the processing load on the information processing apparatus 200D can be reduced. Therefore, in the present embodiment, even when the number of display apparatuses 700 arranged in the event space is increased, the delay time from when audio data is input until a screen is displayed on each display apparatus 700 can be shortened.

[0280] Note that in the example of FIG. 40, the information processing apparatus 200D, the display management apparatus 850, and the processing procedure management apparatus 950 are each implemented by separate computers, but the present invention is not limited thereto. The information processing apparatus 200D may include both or one of the display management apparatus 850 and the processing procedure management apparatus 950.

[0281] Next, with reference to FIG. 41, the functional configuration of each apparatus included in the information processing system 100D will be described. FIG. 41 is a diagram for explaining functions of each apparatus included in the information processing system according to the seventh embodiment.

[0282] The information processing apparatus 200D of the present embodiment includes a storage unit 230B and a control unit 270D. The storage unit 230B includes an event storage unit 240A, a space storage unit 250C, and a template storage unit 255B. The control unit 270D includes an input reception unit 271, a person identification unit 272, a voice recognition unit 273, a prompt generation unit 274, a prompt transmission unit 275, a response reception unit 276, a request generation unit 277A, a request transmission unit 278A, an event setting unit 279, a mode identification unit 280, and a procedure control unit 286.

[0283] The request generation unit 277A generates a screen information generation request to be transmitted to the display management apparatus 850. The screen information generation request includes, for example, the response data received by the response reception unit 276. The request transmission unit 278A transmits the screen information generation request generated by the request generation unit 277A to the display management apparatus 850.

[0284] The procedure control unit 286 identifies procedure information associated with events taking place within the event space and acquires the identified procedure information from the processing procedure management device 950. The procedure control unit 286 also causes each part of the control unit 270D to execute processing according to the procedure information acquired from the processing procedure management device 950.

[0285] The display management device 850 in this embodiment includes an input receiving unit 851, a screen information generation unit 852, and a screen information transmission unit 853. The input receiving unit 851 receives input to the display management device 850. Specifically, the input receiving unit 851 receives a request to generate screen information to the display management device 850. The display management device 850 generates screen information in response to the screen information generation request. The screen information transmission unit 853 transmits the screen information generated by the screen information generation unit 852 to a terminal device 300, etc. As an example, the display management device 850 provides an online whiteboard tool. An online whiteboard tool is a whiteboard that can be edited online.

[0286] The processing procedure management device 950 includes a procedure storage unit 951, an input receiving unit 952, a procedure acquisition unit 953, and a procedure transmission unit 954. The procedure storage unit 951 stores procedure information. The input receiving unit 952 receives input to the processing procedure management device 950. Specifically, the input receiving unit 952 receives requests to acquire procedure information. The procedure acquisition unit 953 acquires the procedure information stored in the procedure storage unit 951. The procedure transmission unit 954 transmits the procedure information acquired by the procedure acquisition unit 953 to the information processing device 200D.

[0287] Next, the storage units of the information processing device 200D will be described. Figure 42 shows an example of an event storage unit in the seventh embodiment. The event information stored in the event storage unit 240A of this embodiment includes, as information items, event identification information, spatial ID, event name, event type, duration, number of participants, event content, purpose of the event, result of the event, and procedure ID. The value of the item "procedure ID" indicates identification information for identifying the procedure information.

[0288] Figure 43 shows an example of a spatial storage unit according to the seventh embodiment. The spatial information stored in the spatial storage unit 250C of this embodiment includes the spatial ID, name, and device as information items. In this embodiment, processing is executed according to the procedure information, so the spatial storage unit 250C can execute the processing of the control unit 270D if a space and a device placed in that space are associated.

[0289] Figure 44 shows an example of a template storage unit of the seventh embodiment. The template storage unit 255B of this embodiment includes text data 7, which is a prompt template corresponding to the request content "summary generation for each utterance," and text data 8, which is a prompt template corresponding to the request content "grouping summaries for each utterance."

[0290] The text data 7 of this embodiment may be, for example, "Generate a summary for each utterance and display each summary as a sticky note." The text data 8 of this embodiment may be, for example, "Group summaries that are similar in content and display a sticky note for each group."

[0291] Figure 45 shows an example of a procedure storage unit in the seventh embodiment of the processing procedure management device. The procedure information stored in the procedure storage unit 951 includes information items such as event identification information, spatial ID, procedure ID, display identification information, process 1, process 2, process 3, process 4, process 5, etc., and the item "procedure ID" is associated with the other items. In the example in Figure 45, the processes associated with the procedure ID are processes 1 to 5, but the number of processes can be arbitrarily changed according to the work defined in the procedure information. Furthermore, the procedure information stored in the procedure storage unit 951 can be created and edited by the user.

[0292] The value of the "Procedure ID" field is identification information used to identify the procedure. The values ​​of the "Process 1" to "Process 5" fields indicate the execution order and content of the processes.

[0293] In the example shown in Figure 45, the procedure information identified by procedure ID "11" specifies the procedure for causing the display management device 850 to generate screen information for the display device 700A, identified by spatial ID "1" and display identification information "DY1", in the event identified by event identification information "I0001".

[0294] Specifically, the procedure information identified by procedure ID "11" specifies the process of identifying the speaker who uttered the audio data, speech recognition processing of the audio data, the process of generating a prompt based on the prompt template corresponding to the request content "summary generation for each utterance", the process of acquiring response data from the server device 400, and the process of requesting the display management device 850 to generate screen information.

[0295] Similarly, the procedure information identified by procedure ID "12" specifies the procedure for causing the display management device 850 to generate screen information for the display device 700B, identified by spatial ID "1" and display identification information "DY2", in the event identified by event identification information "I0001". Session identification information may be used instead of display identification information.

[0296] The procedure information identified by procedure ID "12" specifies the process of identifying the speaker who uttered the audio data, the speech recognition process for the audio data, the process of generating a prompt based on a prompt template corresponding to the request content "group summaries for each utterance", the process of acquiring response data from the server device 400, and the process of requesting the display management device 850 to generate screen information.

[0297] Next, the operation of the information processing system 100D of this embodiment will be described with reference to Figure 46. Figure 46 is a sequence diagram illustrating the operation of the information processing system of the seventh embodiment. Note that the operation of the information processing system 100D until the processing shown in Figure 46 is started is the same as in the first embodiment. Therefore, when the operation shown in Figure 46 is disclosed, the event to be started has been selected in the information processing system 100D.

[0298] In the information processing system 100D of this embodiment, the sound collection device 600 collects voice data from event participants, and the communication control unit 610 transmits the voice data to the terminal device 300 (step S4601). The terminal device 300 transmits the voice data received from the sound collection device 600 to the information processing device 200D via the communication control unit 360 (step S4602).

[0299] When the input receiving unit 271 of the information processing device 200D receives audio data, the procedure control unit 286 refers to the event storage unit 240A and identifies the procedure ID of the procedure information corresponding to the selected event (step S4603).

[0300] Next, the procedure control unit 286 of the information processing device 200D sends a request to the processing procedure management device 950 to acquire procedure information including the procedure ID (step S4604), and acquires the procedure information identified by the procedure ID from the processing procedure management device 950 (step S4605).

[0301] Next, the procedure control unit 286 of the information processing device 200D uses the various parts of the control unit 270D to start executing the processing specified in the order specified in the acquired procedure information (step S4606).

[0302] In the example shown in Figure 46, it is assumed that in step S4603, procedure IDs "11", "12", and "13" are identified, and in step S4605, the procedure information identified by procedure IDs "11", "12", and "13" is obtained. Therefore, in step S4606, when the execution of the process defined in the procedure information is started, the procedure control unit 286 causes the process defined for each of the procedure IDs "11", "12", and "13" to be executed.

[0303] The example in Figure 46 shows an instance where the process specified in the procedure information identified by procedure ID "11" is executed. In this embodiment, the processes specified in the procedure information identified by procedure IDs "12" and "13" are executed in the same manner.

[0304] The process from step S4607 to step S4612 is the same as the process from step S1303 to step S1308 in Figure 13, so the explanation will be omitted. In the example in Figure 46, the first prompt is generated, which includes the content of the request "Generate a summary for each utterance" and the text data 7 corresponding to it, which is "Generate a summary for each utterance and display each summary as a sticky note."

[0305] In step S4612, when the information processing device 200D receives the first response data from the server device 400 via the response receiving unit 276, the request generation unit 277A generates a screen information generation request for the screen including the first response data (step S4613). Subsequently, the information processing device 200D transmits the screen information generation request to the display management device 850 via the request transmission unit 278A (step S4614).

[0306] When the display management device 850 receives a request to generate screen information from the input reception unit 851, the screen information generation unit 852 generates screen information based on the generation request, and the screen information transmission unit 853 transmits the screen information and a screen display instruction based on the screen information to the terminal device 300 (step S4615).

[0307] The processes in steps S4616 and S4617 in Figure 46 are the same as those in steps S1311 and S1312 in Figure 13, so their explanation is omitted. In the above description of the sequence, an example was shown in which audio data is sent from the terminal device 300 to the information processing device 200D in S4602, but this is not the only example. If a screen provided by the display management device 850 is displayed on the terminal device 300, audio data can also be sent from the terminal device 300 to the display management device 850 in S4602. In response, a request is sent from the display management device 850 to the information processing device 200D, and the process in S4603 is then carried out. In other words, the functions of the information processing device 200D are called from the display management device 850.

[0308] Next, with reference to Figures 47 and 48, an example of the display of the display device 700 of this embodiment will be described. Figure 47 is the first diagram illustrating an example of the display of the display device of the seventh embodiment.

[0309] The screen 150A shown in Figure 47 is an example of a screen displayed on the display device 700A in step S4617 of Figure 46. Screen 150A includes display areas 154 to 159. Each of the display areas 154 to 159 displays text data summarizing the content of each speaker's utterance. In other words, display areas 154 to 159 are displayed on the display device 700A as sticky notes summarizing the content of each utterance. These sticky notes can be moved to any coordinate within screen 150A according to user operation. A sticky note, in this context, is a display rectangle on which text summarizing the content of each utterance is displayed. The text data showing the summary of each utterance displayed in display areas 154 to 159 is an example of the first assist information. Screen 150A is an example of a screen displayed on the display device 700A by executing the process specified in the procedure information identified by procedure ID "11".

[0310] Figure 48 is a second diagram illustrating an example of the display of the seventh embodiment of the display device. Screen 170B is an example of a screen displayed on the display device 700B upon execution of the process specified in the procedure information identified by procedure ID "12".

[0311] Screen 170B is an example of a screen displayed on the display device 700B when the information processing device 200D sends a prompt to the server device 400 as the first prompt, which includes the request content "Group the summaries for each utterance," the corresponding text data 8 which is the text data "Group summaries with similar content together, and display sticky notes for each group," and text data indicating the summary for each utterance.

[0312] Screen 170B includes display area 173. Display area 173 includes display areas 174, 175, 176, 177, 178, and 179. Display areas 174 to 179 show the names of the groups. In the example in Figure 48, the group name "Phase" is displayed in display area 174, the group name "Awareness" is displayed in display area 175, and the group name "Understanding" is displayed in display area 176. Also in the example in Figure 48, the group name "Consideration" is displayed in display area 177, the group name "Negotiation" is displayed in display area 178, and the group name "Purchase / Recommendation" is displayed in display area 179.

[0313] Furthermore, the display area 173 displays text data showing summaries of each utterance, categorized by group. In other words, in the display area 173, sticky notes containing text data showing summaries of each utterance are arranged in association with the group names.

[0314] Specifically, in display area 173, the group name "Phase" displayed in display area 174 is associated with display area 154, the group name "Awareness" displayed in display area 175 is associated with display area 155, and the group name "Understanding" displayed in display area 176 is associated with display area 157. In addition, in display area 173, the group name "Consideration" displayed in display area 177 is associated with display area 156, the group name "Negotiation" displayed in display area 178 is associated with display area 159, and the group name "Purchase / Recommendation" displayed in display area 179 is associated with display area 158. In other words, in display area 173, sticky notes displaying text data corresponding to the group are placed for each group name.

[0315] In this embodiment, event participants are no longer required to manually take notes simultaneously with the discussion and then paste or categorize them on a display device such as a whiteboard 700. Therefore, this embodiment prevents participants from being preoccupied with tasks such as categorizing spoken content and structuring the discussion, which can hinder their ability to concentrate on the truly important creative ideas and decision-making. In particular, workshops and meetings dealing with complex issues require facilitators to summarize, cluster, and structure the discussion as they proceed, demanding advanced skills and concentration. Furthermore, in online meetings and hybrid meetings, it is possible to efficiently automate the structuring of discussion content in real time and aggregate participants' opinions.

[0316] Furthermore, each function of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" in this specification includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0317] Furthermore, the apparatus described in each embodiment represents only one of several computing environments for carrying out the embodiments disclosed herein.

[0318] In one embodiment, the information processing device 200 includes a plurality of computing devices, such as a server cluster. The plurality of computing devices are configured to communicate with each other via any type of communication link, including a network or shared memory, and perform the processing disclosed herein. Similarly, the information processing device 200 may include a plurality of computing devices configured to communicate with each other.

[0319] Furthermore, the information processing device 200 can be configured to share the disclosed processing steps in various combinations. For example, a process performed by a predetermined unit may be performed by the information processing device 200. Similarly, the functions of a predetermined unit may be performed by the information processing device 200. Also, each element of the information processing device 200 may be combined into a single server device or divided among multiple devices.

[0320] The information processing device 200 may be any device equipped with communication capabilities. For example, the information processing device 200 may be an output device such as a PJ (Projector), IWB (Interactive White Board: an electronic whiteboard with the ability to communicate with each other), or digital signage; a HUD (Head Up Display) device; industrial machinery; imaging devices; sound collection devices; medical equipment; networked home appliances; automobiles (Connected Cars); notebook PCs (Personal Computers); mobile phones; smartphones; tablet terminals; game consoles; PDAs (Personal Digital Assistants); digital cameras; wearable PCs; or desktop PCs.

[0321] Examples of the present invention are as follows: <1> An information processing system including an information processing device, a terminal device, and a sound collection device and display device included in space, The aforementioned information processing device is An input receiving unit that receives audio data, including speech from a person in the space, transmitted from the terminal device, A transmission unit transmits a prompt to a large-scale language model, which includes event information relating to the content of an event in the space and the content of the audio data, and which requests the output of assist information that assists the progress of the event based on the audio data. A response receiving unit that receives response data transmitted from the aforementioned large-scale language model, A screen information generation unit generates screen information for a screen that includes assist information to assist in the progress of the event, based on the response data. The system includes a screen information transmission unit that transmits the screen information generated by the screen information generation unit to a display device included in the space, The aforementioned terminal device is The communication control unit receives audio data collected by the sound collection device, transmits the audio data to the information processing device, and, upon receiving a screen display instruction based on the screen information transmitted from the information processing device, transmits the screen information and the display instruction to a display device included in the space. The display device included in the space is An information processing system having a display control unit that displays the screen including the aforementioned assist information. <2> The event information is input by the user in the terminal device and stored in the event storage unit. The terminal device displays a screen on its display unit for setting event information for an event to be held, based on the event information stored in the event storage unit. <1> The information processing system described. <3> The screen for setting the aforementioned event information is: This is the screen for selecting the type of event to be held. <2> The information processing system described. <4> The aforementioned event information includes at least one of the following: event name, event purpose, event content, event description, number of participants, event duration, and event location. <1> ~ <3> An information processing system as described in any one of the following items. <5> The aforementioned transmitting unit Spatial information indicating the characteristics of the space where the event is held is transmitted to the large-scale language model. <1> ~ <4> An information processing system as described in any one of the following items. <6> The aforementioned assist information is, The information includes at least one of the following: information visually summarizing the utterances made at the event; information resulting from a search for information related to the utterances made at the event; information for facilitating the event; and information responding to the utterances made by the person at the event. <1> ~ <5> An information processing system as described in any one of the following items. <7> The aforementioned space is where the first display device and the second display device are installed. The aforementioned transmitting unit A first prompt is sent to the large-scale language model requesting the output of first assist information that assists the progress of the event based on the audio data, and a second prompt is sent to the large-scale language model requesting the output of second assist information that assists the progress of the event based on the audio data. The response receiving unit is, Having received the first response data for the first prompt and the second response data for the second prompt, The aforementioned screen information generation unit, First display screen information based on the first response data and second display screen information based on the second response data are generated. The aforementioned screen information transmission unit, The first display screen information is transmitted in association with identification information that identifies the first display device, and the second display screen information is transmitted in association with identification information that identifies the second display device. The first display device is The system displays the first assist information that assists in the progress of the event based on the aforementioned audio data. The second display device is, Displaying the second assist information that assists in the progress of the event based on the aforementioned audio data, <1> ~ <6> An information processing system as described in any one of the following items. <8> The aforementioned transmitting unit A transmission unit that transmits event information relating to the content of an event in the aforementioned space and the content of the audio data to the large-scale language model, Based on the aforementioned audio data, a prompt is sent to the large-scale language model instructing it to select a mode appropriate for assisting the event from among several modes. The response receiving unit further includes a mode identifying unit that identifies a mode based on the response data for the prompt received by the response receiving unit, The transmitting unit sends a prompt to the large language model requesting the output of the assist information corresponding to the identified mode 1. <1> ~ <7> An information processing system as described in any one of the following items. <9> An input receiving unit that receives audio data, including speech from a person in the space, transmitted from a terminal device, A transmission unit transmits a prompt to a large-scale language model, which includes event information relating to the content of an event in the space and the content of the audio data, and which requests the output of assist information that assists the progress of the event based on the audio data. A response receiving unit that receives response data transmitted from the aforementioned large-scale language model, A screen information generation unit generates screen information for a screen that includes assist information to assist in the progress of the event, based on the response data. An information processing apparatus comprising: a screen information transmission unit that transmits the screen information generated by the screen information generation unit to the terminal device. <10> It accepts input of audio data, including speech from a person in the space, transmitted from a terminal device. A prompt including event information relating to the content of an event in the space and the content of the audio data, the prompt being sent to a large-scale language model, which requests the output of assist information that assists the progress of the event based on the audio data, Receiving response data transmitted from the aforementioned large-scale language model, Based on the response data, screen information for a screen including assist information to assist in the progress of the event is generated. A program that causes an information processing device to execute a process that transmits the generated screen information to the terminal device. <11> An information processing method using an information processing system that includes an information processing device, a terminal device, a sound collection device and a display device included in space, The aforementioned information processing device The terminal device receives input of voice data, including speech from a person in the space, A prompt including event information relating to the content of an event in the space and the content of the audio data, the prompt being sent to a large-scale language model, which requests the output of assist information that assists the progress of the event based on the audio data, Receiving response data transmitted from the aforementioned large-scale language model, Based on the response data, screen information for a screen including assist information to assist in the progress of the event is generated. The generated screen information is transmitted to the terminal device. The aforementioned terminal device The sound collection device receives the audio data collected by the sound collection device, transmits the audio data to the information processing device, and upon receiving a screen display instruction based on the screen information transmitted from the information processing device, transmits the screen information and the display instruction to the display device included in the space. The display device included in the space An information processing method for displaying the screen containing the aforementioned assist information. <12> An information processing system including an information processing device, a sound collection device and a display device included in space, The aforementioned information processing device is An input receiving unit that receives audio data, including speech from a person in the space, transmitted from the sound collection device, A transmission unit transmits a prompt to a large-scale language model, which includes event information relating to the content of an event in the space and the content of the audio data, and which requests the output of assist information that assists the progress of the event based on the audio data. A response receiving unit that receives response data transmitted from the aforementioned large-scale language model, A screen information generation unit generates screen information for a screen that includes assist information to assist in the progress of the event, based on the response data. The system includes a screen information transmission unit that transmits the screen information generated by the screen information generation unit to the display device, The display device included in the space is An information processing system having a display control unit that displays the screen including the assist information based on the screen information. <13> An input receiving unit that accepts audio data, including speech from people in a space, A large-scale language model memory unit where a large-scale language model is stored, A prompt output unit outputs a prompt to the large-scale language model, which includes event information relating to the content of an event in the space and the content of the audio data, and which requests the output of assist information that assists the progress of the event based on the audio data. A response acquisition unit that receives response data output from the aforementioned large-scale language model, A screen information generation unit generates screen information for a screen that includes assist information to assist in the progress of the event, based on the response data. A terminal device comprising: a screen information transmission unit that transmits the screen information generated by the screen information generation unit to a display device. <14> An input receiving unit that accepts audio data, including speech from people in a space, A transmission unit transmits a prompt to a large-scale language model, which includes event information relating to the content of an event in the space and the content of the audio data, and which requests the output of assist information that assists the progress of the event based on the audio data. A response receiving unit that receives response data transmitted from the aforementioned large-scale language model, A screen information generation unit generates screen information for a screen that includes assist information to assist in the progress of the event, based on the response data. The system includes a screen information transmission unit that transmits the screen information generated by the screen information generation unit to a terminal device, The aforementioned screen information transmission unit, Web content data for displaying a first web page that awaits input of the aforementioned voice data and a second web page containing the aforementioned assistance information, the web content data includes a script that performs at least the following processes: a process that causes the information processing device to transmit the data entered on the first web page, and a process that causes the second web page containing the aforementioned assistance information, and the web content data is transmitted to the terminal device. When the input receiving unit receives an HTTP request transmitted by the execution of the script on the terminal device, which includes a notification indicating that the audio data was input while the first web page displayed on the terminal device based on the web content data was being displayed, the transmitting unit sends a prompt to the large language model, which includes event information relating to the content of an event in the space and the content of the audio data, and which requests the output of assist information to assist in the progress of the event based on the audio data, and the response receiving unit receives the response data transmitted from the large language model as assist information, The screen information transmission unit is an information processing device that transmits the assist information to the terminal device, including it in an HTTP response which is a response to the HTTP request, in order for the terminal device to display the second web page by executing the script.

[0322] Although the present invention has been described above based on various embodiments, the present invention is not limited to the requirements shown in the above embodiments. These points can be modified as long as they do not impair the spirit of the present invention, and can be appropriately determined according to their application. [Explanation of Symbols]

[0323] 100, 100A Information Processing System 200, 200A Information Processing Equipment 230 Storage section 240 Event Memory Unit 250 Spatial memory section 255 Template memory section 270, 270A Control Unit 271 Input Reception Section 272 Person Identification Department 273 Voice Recognition Unit 274 Prompt generation unit 275 Prompt transmission unit 276 Response receiving unit 277 Screen information generation section 278 Screen Information Transmission Unit 279 Event Settings Section 280 Mode Specific Section 281 Layout Determination Section 300 terminal devices 400 Server Devices 410 Large-scale language model memory 411 Large-scale language models 600 Sound collection device 700 Display device [Prior art documents] [Patent Documents]

[0324] [Patent Document 1] Japanese Patent Publication No. 2020-154847

Claims

1. An information processing system including an information processing device, a terminal device, and a sound collection device and display device included in space, The aforementioned information processing device is An input receiving unit that receives audio data, including speech from a person in the space, transmitted from the terminal device, A transmission unit transmits a prompt to a large-scale language model, which includes event information relating to the content of an event in the space and the content of the audio data, and which requests the output of assist information that assists the progress of the event based on the audio data. A response receiving unit that receives response data transmitted from the aforementioned large-scale language model, A screen information generation unit generates screen information for a screen that includes assist information to assist in the progress of the event, based on the response data. The system includes a screen information transmission unit that transmits the screen information generated by the screen information generation unit to the terminal device, The aforementioned terminal device is The communication control unit receives audio data collected by the sound collection device, transmits the audio data to the information processing device, and, upon receiving a screen display instruction based on the screen information transmitted from the information processing device, transmits the screen information and the display instruction to a display device included in the space. The display device included in the space is An information processing system having a display control unit that displays the screen including the aforementioned assist information.

2. The event information is input by the user in the terminal device and stored in the event storage unit. The information processing system according to claim 1, wherein the terminal device displays a screen on the display unit of the terminal device for setting event information for an event to be held, from the events whose event information is stored in the event storage unit.

3. The screen for setting the aforementioned event information is: The information processing system according to claim 2, which is a screen for selecting the type of event to be held.

4. The information processing system according to claim 1, wherein the event information includes at least one of the following: the name of the event, the purpose of the event, the content of the event, a description of the event, the number of participants in the event, the duration of the event, and the location where the event is held.

5. The aforementioned transmitting unit The information processing system according to claim 1, which transmits spatial information indicating the characteristics of the space where the event is held to the large-scale language model.

6. The aforementioned assist information is, The information processing system according to claim 1, which includes at least one of the following: information visually summarizing the utterances in the event; information as a result of searching for information related to the utterances in the event; information for facilitating the event; and information responding to the utterances of the person in the event.

7. The aforementioned space is where the first display device and the second display device are installed. The aforementioned transmitting unit A first prompt is sent to the large-scale language model requesting the output of first assist information that assists the progress of the event based on the audio data, and a second prompt is sent to the large-scale language model requesting the output of second assist information that assists the progress of the event based on the audio data. The response receiving unit is, Having received the first response data for the first prompt and the second response data for the second prompt, The aforementioned screen information generation unit, First display screen information based on the first response data and second display screen information based on the second response data are generated. The aforementioned screen information transmission unit, The first display screen information is transmitted in association with identification information that identifies the first display device, and the second display screen information is transmitted in association with identification information that identifies the second display device. The first display device is The system displays the first assist information that assists in the progress of the event based on the aforementioned audio data. The second display device is, The information processing system according to claim 1, which displays the second assist information that assists in the progress of the event based on the audio data.

8. The aforementioned transmitting unit A transmission unit that transmits event information relating to the content of an event in the aforementioned space and the content of the audio data to the large-scale language model, Based on the aforementioned audio data, a prompt is sent to the large-scale language model instructing it to select a mode appropriate for assisting the event from among several modes. The response receiving unit further includes a mode identifying unit that identifies one mode based on the response data for the prompt received by the response receiving unit, The information processing system according to claim 1, wherein the transmitting unit transmits a prompt to the large-scale language model requesting the output of the assist information corresponding to the specified mode 1.

9. An input receiving unit that receives audio data, including speech from a person in the space, transmitted from a terminal device, A transmission unit transmits a prompt to a large-scale language model, which includes event information relating to the content of an event in the space and the content of the audio data, and which requests the output of assist information that assists the progress of the event based on the audio data. A response receiving unit that receives response data transmitted from the aforementioned large-scale language model, A screen information generation unit generates screen information for a screen that includes assist information to assist in the progress of the event, based on the response data. An information processing apparatus comprising: a screen information transmission unit that transmits the screen information generated by the screen information generation unit to the terminal device.

10. It accepts input of audio data, including speech from a person in the space, transmitted from a terminal device. A prompt including event information relating to the content of an event in the space and the content of the audio data, the prompt being sent to a large-scale language model, which requests the output of assist information that assists the progress of the event based on the audio data, Receiving response data transmitted from the aforementioned large-scale language model, Based on the response data, screen information for a screen including assist information to assist in the progress of the event is generated. A program that causes an information processing device to execute a process that transmits the generated screen information to the terminal device.

11. An information processing method using an information processing system that includes an information processing device, a terminal device, a sound collection device and a display device included in space, The aforementioned information processing device The terminal device receives input of voice data, including speech from a person in the space, A prompt including event information relating to the content of an event in the space and the content of the audio data, the prompt being sent to a large-scale language model, which requests the output of assist information that assists the progress of the event based on the audio data, Receiving response data transmitted from the aforementioned large-scale language model, Based on the response data, screen information for a screen including assist information to assist in the progress of the event is generated. The generated screen information is transmitted to the display device. The aforementioned terminal device The sound collection device receives the audio data collected by the sound collection device, transmits the audio data to the information processing device, and upon receiving a screen display instruction based on the screen information transmitted from the information processing device, transmits the screen information and the display instruction to the display device included in the space. The display device included in the space An information processing method for displaying the screen containing the aforementioned assist information.

12. An information processing system including an information processing device, a sound collection device and a display device included in space, The aforementioned information processing device is An input receiving unit that receives audio data, including speech from a person in the space, transmitted from the sound collection device, A transmission unit transmits a prompt to a large-scale language model, which includes event information relating to the content of an event in the space and the content of the audio data, and which requests the output of assist information that assists the progress of the event based on the audio data. A response receiving unit that receives response data transmitted from the aforementioned large-scale language model, A screen information generation unit generates screen information for a screen that includes assist information to assist in the progress of the event, based on the response data. The system includes a screen information transmission unit that transmits the screen information generated by the screen information generation unit to a display device included in the space, The display device included in the space is An information processing system having a display control unit that displays the screen including the assist information based on the screen information.

13. An input receiving unit that accepts audio data, including speech from people in a space, A large-scale language model memory unit where a large-scale language model is stored, A prompt output unit outputs a prompt to the large-scale language model, which includes event information relating to the content of an event in the space and the content of the audio data, and which requests the output of assist information that assists the progress of the event based on the audio data. A response acquisition unit that receives response data output from the aforementioned large-scale language model, A screen information generation unit generates screen information for a screen that includes assist information to assist in the progress of the event, based on the response data. A terminal device comprising: a screen information transmission unit that transmits the screen information generated by the screen information generation unit to a display device.

14. An input receiving unit that accepts audio data, including speech from people in a space, A transmission unit transmits a prompt to a large-scale language model, which includes event information relating to the content of an event in the space and the content of the audio data, and which requests the output of assist information that assists the progress of the event based on the audio data. A response receiving unit that receives response data transmitted from the aforementioned large-scale language model, A screen information generation unit generates screen information for a screen that includes assist information to assist in the progress of the event, based on the response data. The system includes a screen information transmission unit that transmits the screen information generated by the screen information generation unit to a terminal device, The aforementioned screen information transmission unit, Web content data for displaying a first web page waiting for input of the aforementioned voice data and a second web page containing the aforementioned assistance information, the web content data includes a script that performs at least the following processes: a process of causing the data entered on the first web page to be transmitted to the information processing device, and a process of displaying the second web page containing the aforementioned assistance information, and the web content data is transmitted to the terminal device. When the input receiving unit receives an HTTP request transmitted by the execution of the script on the terminal device, which includes a notification indicating that the audio data was input while the first web page displayed on the terminal device based on the web content data was being displayed, the transmitting unit sends a prompt to the large language model, which includes event information relating to the content of an event in the space and the content of the audio data, and requests the output of assist information to assist the progress of the event based on the audio data, and the response receiving unit receives the response data transmitted from the large language model as assist information, The screen information transmission unit is an information processing device that transmits the assist information to the terminal device, including it in an HTTP response which is a response to the HTTP request, in order for the terminal device to display the second web page by executing the script.

Images