Information processing device, method, system and program

Abstract

To remove restriction of support of a conference.SOLUTION: An information processing device includes: means for acquiring text information corresponding to voice obtained by sound collection by a microphone; means for acquiring speaker information on a speaker associated with voice; means for storing the speaker information in association with the text information corresponding to voice; and means for generating a summary of the text information corresponding to voice associated with the speaker information for each speaker.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, a method, a system, and a program. [Background technology]

[0002] Conventionally, various technical ideas have been proposed for supporting conferences.

[0003] For example, Patent Document 1 discloses a technical idea of ​​acquiring audio data from a meeting in which multiple speakers make statements, identifying the speaker of each statement based on the audio data, applying voice recognition processing to the audio data to generate text data, detecting statements by a specified speaker or statements containing specified words and phrases from the text data, and generating minutes of the meeting based on the text data of the detected statements.

[0004] For example, Patent Document 2 discloses a technical idea of ​​determining whether or not factors affecting the progress of a conference satisfy certain conditions, and if the certain conditions are satisfied, having a microphone / speaker device selected from a plurality of microphone / speaker devices according to the factors affecting the progress of the conference notify specific information related to the certain conditions. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2022-118302 A [Patent Document 2] Patent Publication No. 2022-138245 Summary of the Invention [Problem to be solved by the invention]

[0006] However, conventional minutes, although the amount of information may vary, are often in a format that basically lists the contents of remarks in chronological order, and there is room for improvement in terms of quickly grasping the state of discussion in a meeting. The technical idea of ​​Patent Document 1 is ingenious in the summary format, but it is difficult to apply it to meetings other than those dealing with specific topics.

[0007] The technical idea of ​​Patent Document 2 is premised on each of the conference participants being equipped with a wearable microphone-speaker device, which places significant restrictions on conducting the conference.

[0008] As described above, conventionally, there are various limitations to meeting support.

[0009] The purpose of this disclosure is to remove the constraints of conferencing support. [Means for solving the problem]

[0010] An information processing device of one embodiment of the present disclosure includes means for acquiring text information corresponding to speech obtained by sound collection by a microphone, means for acquiring speaker information regarding a speaker associated with the speech, means for storing the speaker information in association with the text information corresponding to the speech, and means for generating, for each speaker, a summary of the text information corresponding to the speech associated with the speaker information of the speaker.

[0011] Another aspect of an information processing device of the present disclosure includes a means for acquiring conference information, the conference information including at least information regarding the content of the conference, a means for acquiring text information corresponding to audio obtained by collection by a microphone during the conference, a means for acquiring speaker information regarding a speaker associated with the audio, a means for making a determination regarding at least one of the status of the conference participants or the progress of the conference based on the conference information, the text information, and the speaker information, a means for generating UI (User Interface) information for presenting an alert or suggestion to some or all of the conference participants depending on the result of the determination, and a means for outputting the UI information. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing a configuration of an information processing system according to an embodiment of the present invention; [Diagram 2] FIG. 1 is a block diagram showing a configuration of a display device according to an embodiment of the present invention. [Diagram 3] 1 is a block diagram showing a configuration of a microphone device according to an embodiment of the present invention. [Figure 4] FIG. 1 is an explanatory diagram of one aspect of the present embodiment. [Diagram 5] FIG. 2 is a diagram showing a data structure of a speaker database according to the present embodiment. [Figure 6] 4 is a diagram showing a data structure of a comment log database according to the present embodiment. FIG. [Figure 7] 4 is a flowchart of microphone processing according to the present embodiment. [Figure 8] FIG. 2 is a diagram for explaining sound collection by a microphone. [Figure 9] FIG. 13 is a diagram for explaining the direction of a speaker. [Figure 10] 10 is a flowchart of a comment log collection process according to the present embodiment. [Figure 11] 11 is a flowchart of a summarization process according to the present embodiment. [Figure 12] 11A and 11B are diagrams illustrating an example of a screen displayed in the summarization process of the present embodiment. [Figure 13] FIG. 13 is an explanatory diagram of one aspect of Modification 1. [Figure 14] 13 is a flowchart of a facilitation process in the first modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings for explaining the embodiment, the same components are generally designated by the same reference numerals, and the repeated description will be omitted.

[0014] In the following explanation, a coordinate system based on the position and orientation of the microphone device (microphone coordinate system) may be used. The microphone coordinate system has the position of the microphone device (for example, the center of gravity of the microphone device) as its origin, and the x-axis and y-axis intersect at the origin. In the microphone coordinate system, when the x+ direction is defined as the front of the microphone device, the x-direction is defined as the rear of the microphone device, the y+ direction as the leftward direction of the microphone device, and the y- direction as the rightward direction of the microphone device. Furthermore, a direction in a particular coordinate system refers to a direction relative to the origin of the coordinate system.

[0015] (1) Information Processing System Configuration The configuration of the information processing system will be described below. Fig. 1 is a block diagram showing the configuration of the information processing system according to the present embodiment.

[0016] As shown in FIG. 1, the information processing system 1 includes a display device 10 and a microphone device 30. A plurality of users may be involved in the information processing system 1. At least one of the users may be hearing-impaired, but not all of the users may be hearing-impaired (that is, all of the users may have sufficient hearing for conversation).

[0017] The display device 10 and the microphone device 30 may be connected via a communication cable or a wireless channel (eg, a Wi-Fi channel or a Bluetooth® channel).

[0018] The display device 10 includes one or more displays (one example of a "display unit"). The display device 10 generates an image signal based on information received from an external device, or receives an image signal from an external device, and displays an image corresponding to the image signal on the display. The display device 10 is, for example, a tablet terminal, a personal computer, a smartphone, a monitor device, or a display device for a conference. A user of the display device 10 may be a participant of the conference (which may include a facilitator) or a person not participating in the conference. In addition, multiple users can use the display device 10 at the same time.

[0019] The microphone device 30 may be configured to be installable independently of the display device 10. In other words, the position and orientation of the microphone device 30 can be determined independently of the position and orientation of the display device 10.

[0020] (1-1) Display device configuration The configuration of the display device will now be described with reference to Fig. 2, which is a block diagram showing the configuration of the display device of this embodiment.

[0021] As shown in FIG. 2, the display device 10 includes a storage device 11, a processor 12, an input / output interface 13, a communication interface 14, and a display unit 15.

[0022] The storage device 11 is configured to store programs and data. The storage device 11 is, for example, a combination of a Read Only Memory (ROM), a Random Access Memory (RAM), and a storage (for example, a flash memory or a hard disk).

[0023] The programs include, for example, the following programs: ·OS (Operating System) programs - Programs for applications that perform information processing (e.g., web browsers, applications with functions to support meetings, etc.)

[0024] The data includes, for example, the following data: Databases referenced in information processing Data obtained by performing information processing (i.e., the results of performing information processing)

[0025] The processor 12 is a computer that realizes the functions of the display device 10 by running a program stored in the storage device 11. The processor 12 is, for example, at least one of the following: ·CPU(Central Processing Unit) ·GPU(Graphic Processing Unit) ·ASIC(Application Specific Integrated Circuit) ·FPGA(Field Programmable Gate Array)

[0026] The input / output interface 13 is configured to receive information (e.g., a user's instruction, an audio signal) from an input device connected to the display device 10, and to output information (e.g., an image signal) to an output device connected to the display device 10.

[0027] The input device is, for example, a microphone device 30, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, an external display different from the display unit 15, a speaker, or a combination thereof.

[0028] The communication interface 14 is configured to control communications between the display device 10 and external devices (eg, the microphone device 30).

[0029] The display unit 15 is configured to display an image (a still image or a moving image). The display unit 15 is, for example, a liquid crystal display or an organic EL display.

[0030] (1-2) Microphone device configuration The configuration of the microphone device will now be described with reference to Fig. 3, which is a block diagram showing the configuration of the microphone device of this embodiment.

[0031] As shown in FIG. 3, the microphone device 30 includes a storage device 31, a processor 32, an input / output interface 33, a communication interface 34, and a microphone 51.

[0032] The storage device 31 is configured to store programs and data, and is, for example, a combination of a ROM, a RAM, and a storage (for example, a flash memory or a hard disk).

[0033] The programs include, for example, the following programs: -OS programs - Programs for applications that perform information processing (for example, applications with a voice processing function, etc.)

[0034] The data includes, for example, the following data: Databases referenced in information processing Results of information processing

[0035] The processor 32 is a computer that realizes the functions of the microphone device 30 by starting a program stored in the storage device 31. The processor 32 is, for example, at least one of the following: ·CPU GPU ·ASIC FPGA

[0036] The input / output interface 33 is configured to obtain information (eg, a user's instruction) from an input device connected to the microphone device 30, and to output information (eg, an audio signal) to an output device connected to the microphone device 30.

[0037] The input device is, for example, a physical button, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display, a speaker, or a combination thereof.

[0038] The communication interface 34 is configured to control communications between the microphone device 30 and an external device (eg, the display device 10).

[0039] The microphone 51 collects, for example, sounds around the microphone device 30. The sounds collected by the microphone 51 include, for example, at least one of the following sounds. -Speech by a person -Environmental sound around microphone device 30

[0040] The microphone device 30 includes at least one microphone 51. In the following description, the microphone device 30 includes a plurality of microphones 51. In other words, the microphone device 30 is a multi-microphone device. The microphone device 30 uses the microphone 51 to receive (collect) a sound emitted by a speaker, thereby generating a voice signal (an example of "voice information"). The microphone device 30 also estimates the direction from which the sound comes in the microphone coordinate system (i.e., the direction of the speaker) based on the collected voice signal. The "direction of the speaker" refers to the direction of the speaker relative to the microphone device 30. The microphone device 30 also performs a beamforming process (described later) on the collected voice signal according to the estimated direction of the speaker.

[0041] The microphone device 30 may have a mark on the surface of the housing indicating the reference direction of the microphone device 30 (for example, forward (i.e., the x+ direction), but it may be another specified direction). This allows the user to easily recognize the orientation of the microphone device 30 from visual information. Note that the means for recognizing the orientation of the microphone device 30 is not limited to this. The mark may be integrated with the housing of the microphone device 30.

[0042] The microphone device 30 may be equipped with an Inertial Measurement Unit (IMU) to detect the movement and state of the microphone device 30.

[0043] (2) One aspect of the embodiment An embodiment of the present invention will now be described with reference to Fig. 4, which is an explanatory diagram of an embodiment of the present invention.

[0044] 4, the microphone device 30 is installed in, for example, a conference room and detects voices emitted by participants during a conference. The microphone device 30 estimates the direction from which the voice comes to the multiple microphones 51 and identifies the speaker based on the estimation result. The microphone device 30 transmits to the display device 10 a voice signal obtained by detecting the voice and speaker information related to the speaker of the voice.

[0045] The display device 10 obtains text information that is a speech recognition result of the speech signal received from the microphone device 30. The display device 10 extracts text information corresponding to the speech associated with each speaker from the obtainable text information. The display device 10 generates a model input based on the text information for each speaker, and provides the model input to the trained model LM1 to generate a summary of the text information for each speaker.

[0046] The trained model LM1 is trained to generate a response sentence for a given model input (e.g., a directive sentence). The trained model LM1 may be, for example, a large-scale language model. The trained model LM1 may be constructed in an information processing device (e.g., a server providing an external service) different from the display device 10. Alternatively, the trained model LM1 may be constructed inside the display device 10. The trained model LM1 of this embodiment generates a summary of the text information for each speaker in response to a model input based on the text information for each speaker, and outputs the generated result.

[0047] Based on the acquired summary, the display device 10 generates UI (User Interface) information regarding a screen for displaying the summary, and displays (an example of "output") on the display unit 15. This allows the user of the display device 10 to easily understand from the screen of the display unit 15 what each participant in the conference has said, or what stance each participant has on the agenda of the conference, and the like.

[0048] (3) Database The database of this embodiment will be described.

[0049] (3-1) Speaker Database The speaker database of this embodiment will be described below. Fig. 5 is a diagram showing the data structure of the speaker database of this embodiment. The speaker database can be stored in the storage device 31.

[0050] The speaker database stores speaker information. The speaker information is information about speakers around the microphone device 30 who are identified by the microphone device 30.

[0051] 5, the speaker database includes an "ID" field, a "direction" field, and a "name" field. Each field is associated with the others.

[0052] The "ID" field stores a speaker ID. The speaker ID is information for identifying a speaker. When the microphone device 30 detects a new speaker, it issues a new speaker ID and assigns the speaker ID to the speaker.

[0053] The "direction" field stores speaker direction information. The speaker direction information is information about the direction of the speaker. As an example, the speaker direction is expressed as an angle from an axis that has a reference direction (in this embodiment, the front (x+ direction) of the microphone device 30) as 0 degrees in the microphone coordinate system.

[0054] The "Name" field stores speaker name information. Speaker name information is information about the name of a speaker. The microphone device 30 may determine the speaker name information automatically or may set it according to a user instruction. The microphone device 30 may assign some initial speaker name to a newly detected speaker according to a predetermined rule or randomly.

[0055] In addition, the speaker information stored in the speaker database may include at least one of the following information: Recognized language information Translation language information Speaker distance information

[0056] The recognition language information is information about the language used by the speaker. Based on the recognition language information of the speaker, a speech recognition engine to be applied to the speech generated by the speaker is selected. The recognition language information may be set by a user operation or automatically based on the language recognition result by a speech recognition model.

[0057] The translation language information is information about a target language when machine translation is applied to a speech recognition result (text) for a speech uttered by a speaker. A machine translation engine to be applied to the speech recognition result for the speech uttered by the speaker is selected based on the translation language information of the speaker. The translation language information may be set collectively for all speakers, not for individual speakers, or may be set for each display device 10.

[0058] The speaker distance information is information about the distance from the microphone device 30 to the speaker. The speaker direction information and the speaker distance information can also be expressed as speaker position information. The speaker position information is information about the relative position of the speaker with respect to the microphone device 30 (i.e., the coordinates of the speaker in the coordinate system of the microphone device 30).

[0059] (3-2) Speech log database The comment log database of this embodiment will be described below. Fig. 6 is a diagram showing the data structure of the comment log database of this embodiment. The comment log database can be stored in the storage device 11.

[0060] The speech log database stores speech log information. The speech log information is information related to a log of speech contents by speakers. The speech log information associates speakers with speech contents.

[0061] 6, the speech log database includes a "date and time" field, a "speaker ID" field, and a "speech" field. Each field is associated with the others.

[0062] The "date and time" field stores date and time information. The date and time information is information about the date and time when the voice of the utterance was detected (that is, when the voice was detected by collecting the voice through the microphone 51).

[0063] The "speaker ID" field stores a speaker ID. The speaker ID is information for identifying the speaker whose utterance was detected. The speaker ID may have a one-to-one correspondence with the speaker ID in the speaker database (FIG. 5).

[0064] The "utterance" field stores utterance information. The utterance information is text information corresponding to the voice of the utterance (for example, the result of voice recognition).

[0065] (4) Information processing The information processing of this embodiment will be described.

[0066] (4-1) Microphone processing The microphone processing of this embodiment will be described. Fig. 7 is a flowchart of the microphone processing of this embodiment. Fig. 8 is a diagram for explaining sound collection by a microphone. Fig. 9 is a diagram for explaining the direction of a speaker.

[0067] The microphone processing of this embodiment is started, for example, in response to a user instruction after the display device 10 and the microphone device 30 are turned on and the initial settings are completed. However, the start timing of the microphone processing of this embodiment is not limited to this. The microphone processing of this embodiment may be executed repeatedly, for example, at a predetermined cycle.

[0068] The microphone device 30 performs voice detection via the microphone 51 (S130). Specifically, it is assumed that the microphone device 30 includes five microphones 51-1, 51-2, 51-3, 51-4, and 51-5. The microphones 51-1 to 51-5 each collect a speech sound emitted by a speaker. The microphones 51-1 to 51-5 collect the speech sound that arrives via the multiple paths shown in Fig. 8. The microphones 51-1 to 51-5 convert the collected speech sound into an audio signal.

[0069] The processor included in the microphone device 30 acquires, from the microphones 51-1 to 51-5, audio signals including speech sounds emitted from at least one of the speakers PR1, PR2, and PR3. The audio signals acquired from the microphones 51-1 to 51-5 include spatial information (e.g., delay and phase change) based on the path along which the speech sounds have traveled.

[0070] After step S130, the microphone device 30 performs estimation of the direction of arrival (S131). An arrival direction estimation model is stored in the storage device 31. In the arrival direction estimation model, information for identifying the correlation between spatial information included in the voice signal and the arrival direction of the speech sound is described.

[0071] The direction of arrival estimation method used in the direction of arrival estimation model may be any existing method, such as MUSIC (Multiple Signal Classification) using eigenvalue expansion of an input correlation matrix, a minimum norm method, or ESPRIT (Estimation of Signal Parameters via Rotational Invariance Techniques).

[0072] The microphone device 30 estimates the direction of a speaker based on the direction of arrival of the speech sound collected by the microphones 51-1 to 51-5 by inputting the voice signals received from the microphones 51-1 to 51-5 to the arrival direction estimation model. At this time, the microphone device 30 expresses the arrival direction of the speech sound by an angle from an axis having a reference direction (in this embodiment, the front (x+ direction) of the microphone device 30) set at 0 degrees in a microphone coordinate system, for example. In the example shown in FIG. 9, the microphone device 30 estimates the arrival direction of the speech sound emitted from the speaker PR1 as a direction shifted by an angle A1 to the right from the x-axis. The microphone device 30 estimates the arrival direction of the speech sound emitted from the speaker PR2 as a direction shifted by an angle A2 to the left from the x-axis. The microphone device 30 estimates the arrival direction of the speech sound emitted by the speaker PR3 to be a direction shifted to the left from the x-axis by an angle A3.

[0073] After step S131, the microphone device 30 executes extraction of an audio signal (S132). A beamforming model is stored in the storage device 31. The beamforming model describes information for identifying the correlation between a specific direction and parameters for forming a directivity having a beam in that direction. Here, forming a directivity is a process for amplifying or attenuating a sound from a specific arrival direction.

[0074] The microphone device 30 inputs the estimated arrival direction to the beamforming model, and calculates parameters for forming a directivity having a beam in the arrival direction.

[0075] In the example shown in Fig. 9, the microphone device 30 inputs the calculated angle A1 into the beamforming model and calculates parameters for forming a directivity having a beam in a direction shifted by angle A1 to the right from the x-axis. The microphone device 30 inputs the calculated angle A2 into the beamforming model and calculates parameters for forming a directivity having a beam in a direction shifted by angle A2 to the left from the x-axis. The microphone device 30 inputs the calculated angle A3 into the beamforming model and calculates parameters for forming a directivity having a beam in a direction shifted by angle A3 to the left from the x-axis.

[0076] The microphone device 30 amplifies or attenuates the audio signals acquired from the microphones 51-1 to 51-5 using parameters calculated for the angle A1. The microphone device 30 synthesizes the amplified or attenuated audio signals to extract, from the acquired audio signals, an audio signal regarding a speech sound coming from a speaker in a direction corresponding to the angle A1.

[0077] The microphone device 30 amplifies or attenuates the audio signals acquired from the microphones 51-1 to 51-5 using parameters calculated for the angle A2. The microphone device 30 synthesizes the amplified or attenuated audio signals to extract, from the acquired audio signals, an audio signal regarding a speech sound coming from a speaker in a direction corresponding to the angle A2.

[0078] The microphone device 30 amplifies or attenuates the audio signals acquired from the microphones 51-1 to 51-5 using parameters calculated for the angle A3. The microphone device 30 synthesizes the amplified or attenuated audio signals to extract, from the acquired audio signals, an audio signal regarding a speech sound coming from a speaker in a direction corresponding to the angle A3.

[0079] After step S132, the microphone device 30 performs speaker identification (S133). Specifically, the microphone device 30 identifies a speaker present around the microphone device 30 based on the arrival direction information acquired in step S132, i.e., the estimation result of the direction of the speaker relative to the microphone device 30 (hereinafter referred to as the "target speaker direction").

[0080] As an example, the microphone device 30 determines whether the speaker corresponding to the target speaker direction is the same as an identified speaker, and assigns a new speaker ID (FIG. 5) if the speaker corresponding to the target speaker direction is not an identified speaker. The microphone device 30 adds a record representing speaker information including the new speaker ID to the speaker database (FIG. 5). Specifically, the microphone device 30 compares the target speaker direction with speaker direction information for identified speakers (FIG. 5). Then, when the microphone device 30 determines that the target speaker direction matches any of the speaker direction information for identified speakers, it treats the speaker corresponding to the target speaker direction as a (identified) speaker having the matching speaker direction information. On the other hand, when the microphone device 30 determines that the target speaker direction does not match any of the speaker direction information for the identified speakers, it detects that a new speaker exists in the target speaker direction and assigns a new speaker ID to the new speaker. Here, the matching of the target speaker direction with the speaker direction information includes at least the target speaker direction matching the direction indicated by the speaker direction information, and further includes the difference or ratio of the target speaker direction to the direction indicated by the speaker direction information being within an allowable range.

[0081] The microphone device 30 outputs (transmits) the voice signal extracted in step S132 to the display device 10 together with the speaker information (eg, speaker ID) identified in step S133 (S135).

[0082] In other words, the microphone device 30 extracts (emphasizes) the voice signal only from the direction specified as the speaker, suppresses (erases) the voice signal from the other directions, and transmits the voice signal to the display device 10. This makes it possible to efficiently provide the display device 10 with a voice signal having a high priority as a voice recognition target while suppressing the data amount of the voice signal to be transmitted.

[0083] When multiple speakers are identified, the microphone device 30 may buffer the voice signals of each speaker and transmit them in turn to the display device 10. This allows the speech recognition process to be performed individually on the speech content of each speaker even when multiple people speak at the same time.

[0084] (4-2) Speech log collection process The comment log collection process of this embodiment will be described with reference to the flowchart of FIG 10.

[0085] The speech log collection process of this embodiment can be started together with the microphone process of this embodiment, for example.

[0086] As shown in FIG. 10, the display device 10 acquires a voice signal and speaker information (S110). Specifically, the display device 10 acquires the voice signal and the speaker information output by the microphone device 30 in step S134.

[0087] After step S110, the display device 10 performs voice recognition (S111). As an example, a voice recognition model is stored in the storage device 11. The voice recognition model describes information for identifying a correlation between a voice signal and a text for the voice signal. The voice recognition model is, for example, a trained model generated by machine learning. As another example, the voice recognition model may be stored in an external device (e.g., a cloud server) accessible by the display device 10 via a network (e.g., the Internet) instead of the storage device 11.

[0088] The display device 10 inputs the voice signal acquired in step S110 to the voice recognition model to obtain text information corresponding to the input voice signal. The display device 10 may select a voice recognition engine based on recognition language information of a speaker corresponding to the voice signal.

[0089] In the speech recognition (S111), the display device 10 may optionally execute a machine translation process in addition to the speech recognition process. Specifically, when translation language information is set for the speaker corresponding to the voice signal acquired in step S110, the display device 10 performs machine translation processing in addition to speech recognition processing on the voice signal. In this way, the display device 10 obtains text information in the language specified by the translation language information. The display device 10 may select a machine translation engine based on the translation language information of the speaker corresponding to the voice signal. Even when this option is adopted, the display device 10 may omit the machine translation processing and perform only the speech recognition processing when translation language information is not set for the speaker of the voice corresponding to the voice signal acquired in step S110.

[0090] After step S111, the display device 10 updates the message log database (S112). Specifically, the display device 10 stores the text information acquired in step S111 in the utterance log database (FIG. 6) in association with the speaker information acquired in step S110.

[0091] (4-3) Summarization The summarization process of this embodiment will now be described with reference to Fig. 11, which is a flowchart of the summarization process of this embodiment. Fig. 12 is a diagram showing an example of a screen displayed in the summarization process of this embodiment.

[0092] The summarization process of this embodiment may be started, for example, in response to the display device 10 receiving a user instruction requesting presentation of a summary.

[0093] As shown in FIG. 11, the display device 10 executes determination of a reference date and time (S210). Specifically, the display device 10 determines a date and time that serves as a reference for creating a summary (hereinafter, referred to as a "reference date and time").

[0094] As a first example of determining the reference date and time (S210), the display device 10 determines the current time (that is, the time when this step is executed) as the reference date and time.

[0095] As a second example of determining the reference date and time (S210), the display device 10 determines the date and time designated by the user as the reference date and time.

[0096] After step S210, the display device 10 acquires the message log information (S211). Specifically, the display device 10 refers to the speech log database (FIG. 6) and acquires speech log information including date and time information corresponding to the date and time from a predetermined starting point to the reference date and time determined in step S210. Here, the predetermined starting point may be the date and time when the conference started, may be a date and time a predetermined time before the reference date and time, or may be a date and time designated by the user.

[0097] After step S211, the display device 10 executes generation of model inputs (S212). Specifically, the display device 10 generates a model input for the trained model LM1 based on the speaker ID and the utterance information included in the utterance log information acquired in step S211. As an example, the model input may include an instruction to summarize the utterance information by speaker.

[0098] After step S212, the display device 10 performs summary generation (S213). Specifically, the display device 10 generates a summary by providing the model input generated in step S212 to the trained model LM1. The summary may include text that represents an outline of each speaker's utterance. In the following description, "generating a summary" by the display device 10 or another information processing device may include obtaining a summary generated by a trained model LM1 constructed outside the display device 10 or another information processing device.

[0099] After step S213, the display device 10 generates UI information (S214). Specifically, the display device 10 generates UI information based on the summary obtained in step S213. The UI information may include at least one of the following elements, for example. Text contained in the abstract Speaker's name Speaker icon Reference date and time

[0100] After step S214, the display device 10 executes output of UI information (S215). Specifically, the display device 10 outputs the UI information generated in step S214. As an example, the display device 10 displays a screen corresponding to the UI information on the display unit 15. An example of the screen is shown in FIG.

[0101] The screen of FIG. 12 includes objects J20, J21, J21a, and J22a to J22c. Object J20 accepts user instructions to switch the screen mode. In the example of Figure 12, "Summary" mode is selected, and a screen for confirming the summary is displayed.

[0102] Object J21 represents a time axis. Object J21 may be configured to accept the specification (which may include modification) of a reference date and time. Object J21a represents a reference date and time and is placed on object J21.

[0103] Objects J22a to J22c (hereinafter, the same objects are referred to as "object J22") each display a summary corresponding to one speaker. The number of objects J22 may be determined according to the number of speakers. As a first example, the number of objects J22 is determined to match the number of speakers. That is, the objects J22 have a one-to-one correspondence with the speakers. As a second example, an upper limit is set for the number of objects J22. In this case, the number of objects J22 is determined to match the number of speakers until the number of speakers reaches the upper limit, and is determined to match the upper limit when the number of speakers reaches the upper limit.

[0104] The object J22a includes objects J22a1 to J22a3. Note that the objects J22b and J22c also include similar objects, but the description thereof will be omitted.

[0105] Object J22a1 represents a speaker icon. Object J22a2 represents the name of the speaker.

[0106] Object J22a3 represents a summary of what the speaker said. The summary may include the following information: - A sentence that expresses what the speaker said in fewer characters · A sentence expressing the speaker's stance on the topic of the meeting Here, the speaker's stance can be inferred by the trained model LM1 depending on the model input.

[0107] (5) Summary As described above, the display device 10 of this embodiment acquires text information corresponding to the voice obtained by collecting the sound using the microphone 51, and acquires speaker information related to the speaker associated with the voice. The display device 10 stores the speaker information in association with the text information corresponding to the voice. For each speaker, the display device 10 generates a summary of the text information corresponding to the voice associated with the speaker information of the speaker. This makes it possible to provide information that allows the outline of the remarks (assertions and opinions) made by each participant in the conference to be easily grasped. By using such information, the progress of the conference and the utilization of information can be facilitated, and productivity can be improved.

[0108] The display device 10 may generate UI information related to a screen for displaying the summary and output the UI information, thereby allowing the user to easily visually grasp the outline of the remarks (assertions and opinions) of each participant in the conference.

[0109] The display device 10 may generate summaries for each speaker by providing model inputs to the large-scale language model that include text information corresponding to the speech associated with that speaker, thereby allowing for use of relevant summaries without requiring the development of logic for generating summaries.

[0110] The screen displaying the summaries may include objects that correspond one-to-one with speakers, and each object may display a summary corresponding to one corresponding speaker. This allows for one-to-one correspondence between each participant in the conference and a summary, making it possible to display a screen that allows the user to more easily grasp the outline of what each participant said.

[0111] The summary may include a sentence that expresses the utterance of the corresponding speaker in a smaller number of characters, thereby enabling the contents of the utterances of each participant in the conference to be displayed compactly and reducing the time required for confirmation.

[0112] The summary may include a sentence expressing the stance of the speaker on the topic of the meeting, which is estimated from the speaker's statement corresponding to the summary. This makes it possible to display a screen that allows the user to easily grasp the overall situation of the discussion.

[0113] The display device 10 may generate a summary based on text information corresponding to the voice collected by the microphone 51 up to the reference date and time. This makes it possible to display a screen that allows the user to easily grasp the status of the discussion up to the reference date and time.

[0114] (6) Variations A modification of this embodiment will now be described.

[0115] (6-1) Variation 1 The following describes modified example 1. Modified example 1 is an example in which an alert or a suggestion is presented to some or all of the participants in a conference depending on the result of a determination regarding at least one of the status of the participants in the conference or the progress of the conference. In other words, modified example 1 is an example in which the conference is facilitated.

[0116] (6-1-1) One aspect of Modification 1 A description will be given of one aspect of Modification 1. Fig. 13 is an explanatory diagram of one aspect of Modification 1.

[0117] 13, the microphone device 30 is installed in, for example, a conference room and detects voices emitted by participants during a conference. The microphone device 30 estimates the direction from which the voice comes to the multiple microphones 51 and identifies the speaker based on the estimation result. The microphone device 30 transmits to the display device 10 a voice signal obtained by detecting the voice and speaker information related to the speaker of the voice.

[0118] The display device 10 obtains text information that is a speech recognition result of a speech signal received from the microphone device 30. The display device 10 also obtains conference information related to the conference. The conference information includes at least information related to the contents of the conference. The display device 10 generates a model input based on the conference information, the text information, and the speaker information, and provides the model input to the trained model LM2 to make a determination regarding at least one of the states of the conference participants or the progress state of the conference.

[0119] The trained model LM2 is trained to generate a response sentence for a given model input (e.g., a directive sentence). The trained model LM2 may be, for example, a large-scale language model. The trained model LM2 may be constructed in an information processing device (e.g., a server providing an external service) different from the display device 10. The trained model LM2 of the first modified example may be the same as the trained model LM1 of the present embodiment. The trained model LM2 makes a determination regarding at least one of the states of the conference participants or the progress state of the conference according to a model input based on the conference information, the text information, and the speaker information (i.e., generates a response sentence corresponding to the determination result), and outputs the determination result.

[0120] The display device 10 generates UI information for presenting an alert or a proposal to some or all of the participants of the conference according to the acquired determination result, and outputs the UI information. For example, the display device 10 may display a screen displaying the alert or proposal on the display unit 15, may output a sound representing the alert or proposal from a speaker, or may post a message representing the alert or proposal to a chat tool. This can encourage behavioral change in the participants to whom the alert or proposal is presented, and contribute to improving the conference (for example, smooth progress of the conference).

[0121] (6-1-2) Information processing The information processing of the first modification will be described.

[0122] (6-1-2-1) Facilitation process A description will now be given of the facilitation process of the modified example 1. Fig. 14 is a flowchart of the facilitation process of the modified example 1.

[0123] The facilitation process of the first modification may be started, for example, in response to the start of a conference. The start of a conference may be detected based on a schedule of the conference, or may be detected in response to acquisition of a user instruction, or acquisition of a voice signal and speaker information. In addition, this facilitation process may be repeatedly executed during the conference.

[0124] As shown in FIG. 14, the display device 10 acquires meeting information (S310). Specifically, the display device 10 obtains meeting information including at least one of the following information: Information about the contents of the meeting Information about the participants of the meeting (which may include information about the participants' roles (roles in the meeting, roles in the organization (e.g., company) to which the participants belong, or a combination of both)) Information about the agenda that constitutes the meeting Information about meeting schedules Information about the location of the meeting Information about the meeting format

[0125] Here, the contents of the meeting include the purpose of the meeting, the attributes of the participants of the meeting (internal or external), the number of participants in the meeting, or a combination thereof.

[0126] As a first example of acquiring conference information (S310), the display device 10 acquires conference information in response to an input from a user. For example, the display device 10 may present a form for entering the conference information and accept the input from the user. Alternatively, the display device 10 may acquire the conference information by having a trained model analyze an explanatory document or explanatory voice created by the user in a free format. The input from the user may be made before the conference or during the conference.

[0127] As a second example of acquiring the conference information (S310), the display device 10 acquires the conference information by referring to external information. For example, the display device 10 may acquire the conference information based on information of a user's schedule, information of a conference room's schedule, or the like.

[0128] As a third example of acquiring the conference information (S310), the display device 10 acquires the conference information based on the voice signal and speaker information acquired during the conference.

[0129] The fourth example of acquiring conference information (S310) is a combination of multiple of the above first to third examples.

[0130] After step S310, the display device 10 acquires the message log information (S311). Specifically, the display device 10 refers to the speech log database (FIG. 6) and acquires speech log information including date and time information corresponding to a period from a predetermined starting point to the current time (i.e., the time when this step is executed). Here, the predetermined starting point may be the date and time when the conference started, may be a date and time a predetermined time before the current time, or may be a date and time designated by the user.

[0131] As an option for acquiring the conference information (S310), the display device 10 may acquire further information by combining the acquired information. Specifically, the display device 10 may acquire information on the time allocated to each agenda constituting the conference by allocating time to each agenda based on information on the agenda constituting the conference and information on the schedule of the conference. A trained model may be used for allocating time. Alternatively, the information on the time allocated to each agenda constituting the conference may be included in the information on the schedule of the conference.

[0132] In acquiring the utterance log information (S311), the display device 10 may optionally identify which of the conference participants the speaker ID included in the acquired utterance log information corresponds to. The display device 10 may identify the relationship between the conference participants and the speaker ID in response to a user instruction, or may identify the relationship between the conference participants and the speaker ID based on the features of the voice.

[0133] After step S311, the display device 10 executes generation of model inputs (S312). Specifically, the display device 10 generates a model input for the trained model LM2 based on the conference information acquired in step S310 and the speaker ID and utterance information included in the utterance log information acquired in step S311. As an example, the model input may include a type of judgment regarding the state of the conference participants or the state of the progress of the conference, and an instruction statement requesting execution of the judgment based on the conference information, the speaker ID, and the utterance information.

[0134] Determination regarding the status of a conference participant or the status of the conference progress may include, for example, any of the following: -Judge whether the comments made are in line with the purpose of the meeting -Is there a participant who is not speaking enough (one example of determining whether there is a quantitative bias in participants' comments) and who these participants are? -Is there a participant who speaks excessively (one example of determining whether there is a quantitative bias in participants' speech) and who these participants are? Determine if any participants are falling behind in the discussion and who they are -Determine whether there are any points of view that have not been discussed sufficiently and what those points of view are -Determine if an ongoing agenda item has exceeded its allotted time ·Judge whether the content of the meeting discussion is negative Deciding whether to close the meeting Determine if there are any action items that have arisen in the meeting and what those action items are

[0135] Here, participants who are unable to keep up with the discussion may be judged in the same way as participants who speak too little, or may be judged based on information about the participant's background, such as the participant's past conference participation records and the role of the participant in the organization to which the participant belongs, in addition to the amount of speech. This makes it possible to distinguish between participants who are unable to speak due to a lack of underlying knowledge and those who do not speak even though they have no lack of knowledge.

[0136] Here, the determination as to whether the conference should be closed may be substituted by a determination as to whether at least one of the following situations exists. Have all the topics been discussed? -Did the meeting end time reach a certain time before the end?

[0137] After step S312, the display device 10 performs a determination (S313). Specifically, the display device 10 performs the judgment by providing the model input generated in step S312 to the trained model LM2. The result of the judgment may include text representing the result of inference by the trained model LM2 for the requested judgment. In the following description, "making a judgment" by the display device 10 or another information processing device may include obtaining the result of a judgment made by the trained model LM2 constructed outside the display device 10 or another information processing device.

[0138] After step S313, the display device 10 generates UI information (S314). Specifically, the display device 10 generates UI information according to the result of the determination obtained in step S313. The UI information may include at least one of the following elements, for example. The content of the alert (specifically, text, audio, image, or a combination of these) -Who will receive the alert (specifically, all participants, participants with a specific role, participants who are determined to be in a specific state (e.g., excessive or insufficient speech volume)) The content of your proposal (specifically, text, audio, image, or a combination of these) -To whom the proposal is presented (specifically, all participants, participants with a specific role, participants who are judged to be in a specific state (e.g., those who speak too much or too little))

[0139] As a first example of generating UI information (S314), the display device 10 may generate an alert such as "The conversation is going off track" for all participants depending on the result of the determination that the remarks are not in line with the purpose of the conference, or may generate a suggestion such as "Let's go back to the topic of XX that Mr. OO mentioned earlier" for all participants. Alternatively, the display device 10 may generate a suggestion showing multiple candidate points for all participants.

[0140] As a second example of generating UI information (S314), depending on the result of the determination that Mr. OO has spoken too little, the display device 10 may generate an alert such as "Mr. OO is not speaking much" (an example of "information indicating a quantitative bias in participants' speech") for all participants, for the facilitator, or for the relevant person (Mr. OO in this example).

[0141] As a third example of generating UI information (S314), depending on the result of determining that Mr. OO has spoken excessively, the display device 10 may generate an alert such as "Some participants are speaking more than others" (an example of "information indicating a quantitative bias in participants' speech") for all participants, the facilitator, or the relevant person (Mr. OO in this example).

[0142] As a fourth example of generating UI information (S314), depending on the result of the determination that Mr. OO may not be able to keep up with the discussion, the display device 10 may generate an alert such as "Mr. OO may not be able to keep up with the discussion" for all participants or the facilitator, or may generate a suggestion for the relevant person (Mr. OO in this example) including information to supplement the knowledge, such as "This is the knowledge that is the premise for the discussion so far."

[0143] As a fifth example of generating UI information (S314), depending on the result of determining that there is a viewpoint that has not been discussed enough, the display device 10 may generate an alert for all participants such as, for example, "The viewpoint of OO is not currently being discussed." (an example of "information indicating a viewpoint that has not been discussed enough"), or may generate a suggestion for all participants such as, "From the viewpoint of OO, something like XX is also possible. What do you think?" (an example of "information indicating a viewpoint that has not been discussed enough").

[0144] As a sixth example of generating UI information (S314), the display device 10 may generate an alert such as “The scheduled time for the OO agenda item has been exceeded” for all participants depending on the result of determining that the time allotted for the ongoing agenda item has been exceeded, or may generate a suggestion such as “To keep to the time, how about focusing on the topic of OO this time and postpone the rest to next time?” for all participants.

[0145] As a seventh example of generating UI information (S314), the display device 10 may generate an alert such as "Negative comments are increasing" for all participants or for the facilitator depending on the result of determining that the content of the discussion in the meeting is negative, or may generate a suggestion such as "How about leaving the topic of OO for now and talking about XX first" for all participants or for the facilitator.

[0146] As an eighth example of generating UI information (S314), depending on the result of determining that the meeting should be closed, the display device 10 may generate an alert for all participants such as "It's time to decide the next meeting date," or may generate a suggestion for all participants such as "After checking the calendar, the next suggested date is OO."

[0147] As a ninth example of generating UI information (S314), the display device 10 may generate an alert such as "No action has been created yet" for all participants in response to a determination that no action item has occurred in the meeting, or may generate a suggestion such as "Action items for next time are OO" for all participants in response to a determination that the action item that has occurred in the meeting is OO.

[0148] After step S314, the display device 10 executes output of UI information (S315). Specifically, the display device 10 outputs the UI information generated in step S314.

[0149] As a first example of outputting UI information (S315), the display device 10 transmits information on the content of the alert to a destination corresponding to the presentation destination of the alert (for example, a terminal used by the participant to whom the alert will be presented). Alternatively, the display device 10 may post a message according to the content of the alert in a chat tool, addressed to the participant to whom the alert will be presented.

[0150] As a second example of outputting UI information (S315), the display device 10 transmits information on the content of the proposal to a destination corresponding to the proposal recipient (for example, a terminal used by the participant to whom the proposal is to be presented). Alternatively, the display device 10 may post a message according to the content of the proposal in a chat tool, addressed to the participant to whom the proposal is to be presented.

[0151] As a third example of outputting UI information (S315), the display device 10 displays information on the content of the alert or suggestion on the display unit 15 or outputs it from the speaker.

[0152] The fourth example of the output of UI information (S315) is a combination of multiple of the above first to third examples.

[0153] (6-1-3) Summary As described above, the display device 10 of the first modification acquires conference information that is information related to the conference and includes at least information related to the contents of the conference, acquires text information corresponding to the voice obtained by collecting sound by the microphone 51 during the conference, and acquires speaker information related to the speaker associated with the voice. The display device 10 determines at least one of the states of the conference participants or the progress state of the conference based on the conference information, the text information, and the speaker information, generates UI information for presenting an alert or a proposal to some or all of the conference participants according to the result of the determination, and outputs the UI information. This encourages behavioral change in the participants to whom the alert or proposal is presented, contributes to improvement of the conference (for example, smooth progress of the conference), and can improve productivity.

[0154] The display device 10 may determine whether any remarks are in line with the purpose of the meeting, and may generate UI information for presenting an alert or a suggestion to all participants in the meeting depending on the result of the determination that any remarks are not in line with the purpose of the meeting. This can encourage the discussion to be corrected in line with the purpose of the meeting.

[0155] The conference information may include information about the conference participants. The display device 10 may determine whether there is a quantitative bias in the participants' comments, and generate UI information for presenting information indicating the quantitative bias in the participants' comments to all the conference participants according to the result of the determination that there is a quantitative bias in the participants' comments. This makes it easier to elicit diverse opinions and ensures that each participant can easily speak.

[0156] The conference information may include information about the conference participants. The display device 10 may determine whether any of the participants are unable to keep up with the conference discussion, and may generate UI information for presenting information that supplements the knowledge of the participant depending on the result of the determination that any of the participants are unable to keep up with the conference discussion. This can correct the asymmetry of information between the participants. As a result, it can be made easier to elicit diverse opinions and ensure ease of speaking for each participant.

[0157] The display device 10 may determine whether there is a viewpoint that has not been discussed enough, and depending on the result of the determination that there is a viewpoint that has not been discussed enough, generate UI information for presenting information indicating the viewpoint that has not been discussed enough to all the participants of the meeting. This can encourage the implementation of a multifaceted discussion.

[0158] The conference information may include information about the agenda constituting the conference and information about the schedule of the conference. The display device 10 may determine whether the time allotted to the ongoing agenda has been exceeded, and may generate UI information for presenting an alert or a suggestion to all the participants of the conference according to the result of the determination that the time allotted to the ongoing agenda has been exceeded. This can assist in time management of the conference.

[0159] The display device 10 may determine whether the contents of the discussion in the meeting are negative, and may generate UI information for presenting an alert or a suggestion to all the participants of the meeting or to the facilitator of the meeting according to the result of the determination that the contents of the discussion in the meeting are negative. This can encourage the discussion to be corrected so that constructive comments are made.

[0160] The display device 10 may determine whether any action items have occurred in the meeting, and generate UI information for presenting an alert or a suggestion to all participants in the meeting according to the result of the determination of whether any action items have occurred in the meeting. This allows matters decided in the meeting to be more reliably incorporated as action items.

[0161] The meeting information may include at least one of information about the agenda constituting the meeting and schedule information of the meeting. The display device 10 may determine whether the meeting should be closed, and generate UI information for presenting an alert or a suggestion to all the participants of the meeting according to the result of the determination that the meeting should be closed. This allows the end time of the meeting to be kept and the next meeting date to be smoothly determined.

[0162] The display device 10 may perform the determination by providing a model input based on the type of determination, the conference information, the text information, and the speaker information to a large-scale language model. This allows the display device 10 to use a valid determination result without creating logic for various determinations.

[0163] (7) Other modifications The storage device 11 may be connected to the display device 10 via a network NW. The display unit 15 may be externally attached to the display device 10. The storage device 31 may be connected to the microphone device 30 via the network NW.

[0164] In the above description, an example in which each step in each process is executed in a specific order is shown, but the execution order of each step is not limited to the described example as long as there is no dependency. In addition, each step of the above information processing can be executed by any of the display device 10, the microphone device 30, and another device. For example, the utterance log collection process of this embodiment, the summarization process of this embodiment, or the facilitation process of the first modified example may be executed mainly by a cloud server. For example, a process described as being performed by any device may be executed by another device, or a process described as being performed by the interaction of multiple devices may be executed by a single device.

[0165] In the present embodiment, an example has been shown in which a summary according to a reference date and time is generated, and UI information based on the summary is generated. The reference date and time may be variable. For example, the display device 10 may detect a change in the reference date and time in response to a user instruction or in response to the passage of time. When a change in the reference date and time is detected, the display device 10 may reacquire a summary based on text information corresponding to the voice obtained by collecting sound by the microphone 51 up to the changed reference date and time, regenerate UI information based on the reacquired summary, and output the regenerated UI information. This makes it possible to display a screen that allows the user to easily grasp the latest discussion situation or the discussion situation at a time point desired by the user.

[0166] In this embodiment, an example has been shown in which an object (e.g., object J22) is displayed to display a summary corresponding to a speaker. Such an object may be placed at a position according to the opinion of the corresponding speaker. As an example, the closer the opinion of a speaker corresponding to an object is to the opinion of a speaker corresponding to another object, the closer the two objects may be placed. Alternatively, such objects may be displayed in the form of a tree structure according to the content of the opinion. This makes it easier to intuitively grasp the tendency of opinions between speakers.

[0167] In the present embodiment, an example of summarizing based on speech log information from the same conference has been shown. However, summarizing may also be based on speech log information from the same person in other conferences held in the past. This allows for more accurate summarization.

[0168] In this embodiment, an example of summarizing based on utterance log information has been shown. However, summarizing may also be performed based on information on the speaker's attributes (for example, the role (e.g., position) of the speaker in the organization to which the speaker belongs). This allows for more accurate summarization.

[0169] A controller (information processing device) may be interposed between the display device 10 and the microphone device 30. The controller may execute part of the processing (e.g., utterance log collection processing) that has been described as being executed by the display device 10, or may execute part of the processing that has been described as being executed by the microphone device 30. The controller may be, for example, a smartphone, a tablet terminal, a personal computer, or a server computer.

[0170] In the above description, the display device 10 is an electronic device that can easily share display contents with multiple users, such as a tablet terminal, a personal computer, a smartphone, a monitor, or a display device for a conference. However, the display device 10 may be configured to be wearable on a human head. For example, the display device 10 may be a glass-type display device, a head-mounted display, a wearable device, or smart glasses. The display device 10 may be an optical see-through glass-type display device, but the form of the display device 10 is not limited thereto. For example, the display device 10 may be a video see-through glass-type display device. That is, the display device 10 may be equipped with a camera. The display device 10 may display a composite image obtained by combining a text image generated based on voice recognition with an image captured by a camera on the display unit. The captured image is an image captured in the front direction of the user and may include an image of the speaker. In addition, the display device 10 may perform AR (Augmented Reality) display by combining a text image generated based on voice recognition with an image captured by a camera in, for example, a smartphone, a personal computer, or a tablet terminal.

[0171] The display unit of the display device 10 may be realized in any manner as long as it can present an image to the user. The display unit may be realized, for example, by the following methods. - HOE (Holographic optical element) or DOE (Diffractive optical element) using optical elements (for example, light guide plates) LCD display Retina projection display LED (Light Emitting Diode) display - Organic EL (Electro Luminescence) display Laser display A display that uses optical elements (such as lenses, mirrors, diffraction gratings, liquid crystals, MEMS mirrors, and HOEs) to guide light emitted from a light emitter. In particular, when a retinal projection display is used, even people with weak eyesight can easily observe the image, and therefore people who suffer from both hearing loss and weak eyesight can more easily recognize the direction from which speech sounds are coming.

[0172] In the voice extraction process by the microphone device 30, as long as a voice signal corresponding to a specific speaker can be extracted, the method for achieving this is not important. The microphone device 30 may extract a voice signal by, for example, the following method. Frost Beamformer Adaptive filter beamforming (for example, the generalized sidelobe canceller) - Audio extraction methods other than beamforming (e.g. frequency filters or machine learning)

[0173] In the above description, it is assumed that the microphone device 30 includes multiple microphones 51. However, for example, when speaker identification is performed based on a voiceprint instead of estimating the direction of a speaker, the microphone device 30 including only a single microphone 51 can be used. In other words, the microphone device 30 may acquire information indicating a speaker identified by voiceprint recognition as speaker information.

[0174] Although the embodiment of the present invention has been described in detail above, the scope of the present invention is not limited to the above embodiment. Furthermore, the above embodiment can be improved or modified in various ways without departing from the spirit of the present invention. Furthermore, the above embodiment and the modified examples can be combined. [Explanation of symbols]

[0175] 1: Information processing system 10: Display device 11:Storage device 12: Processor 13: Input / Output Interface 14: Communication interface 15: Display section 30: Microphone device 31:Storage device 32 : Processor 33: Input / Output Interface 34: Communication interface 51: Microphone

Claims

1. A means for acquiring text information corresponding to a voice obtained by collecting sound using a microphone; means for obtaining speaker information relating to a speaker associated with the speech; a means for storing the speaker information in association with text information corresponding to the voice; A means for generating, for each speaker, a summary of text information corresponding to a speech associated with speaker information of the speaker; An information processing device comprising:

2. and generating the summary by providing, for each of the speakers, a model input to a large-scale language model, the model input including text information corresponding to a speech associated with the speaker, to generate the summary. The information processing device according to claim 1 .

3. means for generating UI (User Interface) information relating to a screen for displaying the summary; A means for outputting the UI information; Further comprising: The information processing device according to claim 1 .

4. the screen displaying the summaries includes objects in one-to-one correspondence with the speakers, and each of the objects displays a summary corresponding to a corresponding speaker; The information processing device according to claim 3 .

5. The summary includes a sentence that expresses the utterance of the corresponding speaker in a smaller number of characters. The information processing device according to claim 1 .

6. The summary includes a sentence expressing a stance of a speaker on the agenda of the meeting, the stance being estimated from a statement of the speaker corresponding to the summary. The information processing device according to claim 1 .

7. the means for generating a summary generates the summary based on text information corresponding to the voice obtained by collecting the voice using the microphone up to a reference date and time. The information processing device according to claim 1 .

8. means for accepting a user's designation regarding the reference date and time; The information processing device according to claim 7.

9. The method further includes detecting a change in the reference date and time. when a change in the reference date and time is detected, the summary generating means reacquires the summary based on text information corresponding to the voice obtained by collecting the voice through the microphone up to the changed reference date and time. The information processing device according to claim 7.

10. A means for acquiring conference information including at least information regarding the contents of the conference; means for acquiring text information corresponding to a voice obtained by collecting sound using a microphone during the conference; means for obtaining speaker information relating to a speaker associated with the speech; a means for determining at least one of a state of a participant of the conference or a progress state of the conference based on the conference information, the text information, and the speaker information; a means for generating UI (User Interface) information for presenting an alert or a suggestion to some or all of the participants of the conference according to a result of the determination; A means for outputting the UI information; An information processing device comprising:

11. The determining means determines whether a statement is made that is consistent with the purpose of the meeting, the means for generating UI information generates UI information for presenting an alert or a suggestion to all participants of the conference according to a result of the determination that the statement is not in line with the purpose of the conference. The information processing device according to claim 10.

12. the conference information includes information regarding participants of the conference; The determining means determines whether there is a quantitative bias in the comments of the participants, the means for generating UI information generates UI information for presenting information indicating the quantitative bias of the comments of the participant to all the participants of the conference according to the result of the determination that there is a quantitative bias in the comments of the participant. The information processing device according to claim 10.

13. the conference information includes information regarding participants of the conference; The determining means determines whether any participant is unable to keep up with the discussion in the conference. The means for generating UI information generates UI information for presenting information supplementing the knowledge of a participant in response to a result of determining that the participant is unable to keep up with the discussion in the conference. The information processing device according to claim 12.

14. The determining means determines whether there is a viewpoint that is not fully discussed, The means for generating UI information generates UI information for presenting information indicating the viewpoint of insufficient discussion to all participants of the conference according to a result of the determination that there is the viewpoint of insufficient discussion. The information processing device according to claim 10.

15. The meeting information includes information regarding an agenda constituting the meeting and information regarding a schedule of the meeting; The determining means determines whether the time allotted for the ongoing agenda has been exceeded; The means for generating UI information generates UI information for presenting an alert or a suggestion to all participants of the meeting in response to the result of determining that the time allotted to the ongoing agenda has been exceeded. The information processing device according to claim 10.

16. The determining means determines whether the content of the discussion in the meeting is negative, The means for generating UI information generates UI information for presenting an alert or a suggestion to all participants of the conference or a facilitator of the conference according to the result of the determination that the content of the discussion in the conference is negative. The information processing device according to claim 10.

17. The determining means determines whether an action item has occurred in the meeting. the means for generating UI information generates UI information for presenting an alert or a suggestion to all participants of the conference according to a result of determining whether an action item has occurred in the conference. The information processing device according to claim 10.

18. The meeting information includes at least one of information regarding an agenda constituting the meeting or schedule information of the meeting; The determining means determines whether the meeting should be closed; the means for generating UI information generates UI information for presenting an alert or a suggestion to all of the participants of the conference in response to a result of the determination that the conference should be closed. The information processing device according to claim 10.

19. the determining means performs the determination by providing a model input based on the type of determination, the conference information, the text information, and the speaker information to a large-scale language model. The information processing device according to claim 10.

20. The computer acquiring text information corresponding to a voice obtained by collecting sound using a microphone; obtaining speaker information regarding a speaker associated with the audio; storing the speaker information in association with text information corresponding to the voice; generating a summary of text information corresponding to a speech associated with speaker information of each speaker; How to do it.

21. The computer acquiring conference information including at least information regarding the contents of the conference; acquiring text information corresponding to a voice collected by a microphone during the conference; obtaining speaker information regarding a speaker associated with the audio; determining at least one of a state of a participant of the conference or a progress state of the conference based on the conference information, the text information, and the speaker information; generating UI (User Interface) information for presenting an alert or a suggestion to some or all of the participants of the conference according to a result of the determination; outputting the UI information; How to do it.

22. A program for causing a computer to realize each of the means of the information processing device according to any one of claims 1 to 19.

23. A system including a plurality of devices, A means for acquiring text information corresponding to a voice obtained by collecting sound using a microphone; means for obtaining speaker information relating to a speaker associated with the speech; a means for storing the speaker information in association with text information corresponding to the voice; A means for generating a summary of text information corresponding to a speech associated with speaker information of each speaker; A system comprising:

24. A system including a plurality of devices, A means for acquiring conference information including at least information regarding the contents of the conference; means for acquiring text information corresponding to a voice obtained by collecting sound using a microphone during the conference; means for obtaining speaker information relating to a speaker associated with the speech; a means for determining at least one of a state of a participant of the conference or a progress state of the conference based on the conference information, the text information, and the speaker information; a means for generating UI (User Interface) information for presenting an alert or a suggestion to some or all of the participants of the conference according to a result of the determination; A means for outputting the UI information; A system comprising:

Images