Summarization system, information processing apparatus, summarization method, and program

Abstract

To summarize utterances of a plurality of participants participating in communication in accordance with the progress of communication, so that points regarded as important in the progress of communication can be easily understood.SOLUTION: A summarization system summarizes utterances of a plurality of participants participating in communication, and the summarization system has: a specification unit that specifies a predetermined participant on the basis of a display image that displays the plurality of participants; a determination unit that determines weighting for the plurality of participants so that weighting for the predetermined participant is made larger than weighting for the other participants; a summarization control unit that controls a summarization unit to summarize the details of the utterances of the plurality of participants in accordance with the progress of communication on the basis the weighting for the plurality of participants; and a presentation unit that presents a summary sentence obtained through the summarization performed by the summarization unit.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a summarization system, an information processing device, a summarization method, and a program. [Background technology]

[0002] There is a summarization system that generates a summary based on speech data. For example, a minutes-of-meetings generating device is known that acquires speech data of meeting attendees, and generates minutes by determining whether or not to include the speech data in the minutes based on the minutes levels selected and input by the meeting attendees for the speech data (see, for example, Patent Document 1). Summary of the Invention [Problem to be solved by the invention]

[0003] Conventional summarization systems have a problem in that they are unable to summarize the speech of multiple participants in a communication such as a conference in an easy-to-understand manner, in a way that highlights the key points in the progress of the communication, and in line with the progress of the communication.

[0004] One embodiment of the present invention has been made in consideration of the above-mentioned problems, and makes it possible to summarize the speech of multiple participants in a communication in an easy-to-understand manner, in accordance with the progress of the communication, and in line with the progress of the communication. [Means for solving the problem]

[0005] In order to solve the above problems, a summarization system according to one embodiment of the present invention is a summarization system that summarizes speech from multiple participants participating in a communication, and includes an identification unit that identifies a specific participant based on a display image displaying the multiple participants, a determination unit that determines weightings for the multiple participants so that the weighting of the specific participant is greater than the weightings of other participants, a summary control unit that controls the summarization unit to summarize the speech content of the multiple participants in accordance with the progress of the communication based on the weightings of the multiple participants, and a provision unit that provides a summary sentence summarized by the summarization unit. Effect of the Invention

[0006] According to one embodiment of the present invention, it is possible to easily summarize the utterances of multiple participants in a communication, in accordance with the progress of the communication, in a manner that is important in the progress of the communication. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of a summarization system according to an embodiment. [Diagram 2] FIG. 13 is a diagram illustrating another example of a system configuration of the summary system according to an embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer according to an embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of a conference device according to an embodiment. [Diagram 5] FIG. 1 is a diagram illustrating an example of a functional configuration of a summary system according to an embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of a functional configuration of a conference device according to an embodiment. [Figure 7] 4 is a flowchart showing an example of processing of the summarization system according to the first embodiment. [Figure 8] FIG. 2 is a diagram showing an image of a conference video according to the first embodiment. [Figure 9]FIG. 4 is a diagram for explaining weighting information according to the first embodiment. [Figure 10] FIG. 4 is a diagram showing an example of a prompt according to the first embodiment. [Figure 11] 3A to 3C are diagrams showing an example of a speech content and a summary according to the first embodiment. [Figure 12] 13 is a flowchart showing an example of processing of a summarization system according to the second embodiment. [Figure 13] FIG. 11 is a diagram showing an image of a conference video according to the second embodiment. [Figure 14] FIG. 11 is a diagram showing an image of weighting information according to the second embodiment. [Figure 15] FIG. 11 is a diagram illustrating an example of a prompt according to the second embodiment. [Figure 16] 13A to 13C are diagrams showing an example of a speech content and a summary according to the second embodiment. [Figure 17] 13 is a flowchart showing an example of processing of a summarization system according to the third embodiment. [Figure 18] FIG. 13 is a diagram showing an image of a conference video according to the third embodiment. [Figure 19] FIG. 1 is a diagram showing an example of a display screen of a terminal device according to an embodiment. [Figure 20] FIG. 2 is a diagram showing an example of a display screen of the terminal device according to an embodiment; [Figure 21] FIG. 2 is a diagram illustrating a data structure of a summarization system according to an embodiment. [Figure 22] 1 is a flowchart illustrating an example of a real-time summarization process according to an embodiment. [Figure 23] 11 is a flowchart illustrating an example of a determination process according to an embodiment. [Figure 24] FIG. 11 is a diagram for explaining an example of a determination process according to an embodiment. [Diagram 25] 11 is a flowchart illustrating another example of a determination process according to an embodiment. [Figure 26] FIG. 11 is a diagram for explaining another example of the determination process according to an embodiment. [Figure 27] 11 is a flowchart illustrating an example of a process for creating a conference video according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present invention (the present embodiment) will be described in detail with reference to the drawings.

[0009] <System configuration> Fig. 1 is a diagram showing an example of a system configuration of a summarization system according to an embodiment of the present invention. In the example of Fig. 1, the summarization system 1 includes a summarization server 10, a terminal device 100, a conference device 110, and the like.

[0010] The summarization system 1 is a system that summarizes in real time the speech of multiple participants A to D participating in communication such as a conference, and displays the most recent summary sentence on the terminal device 100 used by the participants. In the following description, the term "participant" is used to refer to any of the participants A to D. The example in FIG. 1 shows a configuration example of the summarization system 1 when the communication in which multiple participants participate is a general conference held face-to-face by the participants.

[0011] The conference device 110 is a device equipped with a camera for capturing images of the conference, a microphone for capturing audio of the conference, a speaker for outputting the audio of the conference, and the like, and is also called a web conference device.

[0012] Preferably, the conference device 110 includes a microphone array (microphone array) configured by arranging multiple microphones, and has a function of detecting the direction of a speaker using the microphone array. The conference device 110 also has one or more cameras that capture images of users A to D around the conference device 110. Furthermore, the conference device 110 has a function of identifying a speaker (or the direction of a speaker) based on the voices of the multiple participants captured by the microphone array and images of the multiple participants captured by one or more cameras.

[0013] Furthermore, the conference device 110 outputs, for example, video of multiple participants, speech of multiple participants, and information for identifying the speaker to the terminal device 100. Also, the conference device 110 can use a speaker to convert audio data of the conference audio output from the terminal device 100 into audio and output the audio.

[0014] The terminal device 100 is an information terminal used by a participant (user), such as a PC (Personal Computer), a tablet terminal, or a smartphone. The terminal device 100 is communicatively connected to a conference device 110 via, for example, a USB (Universal Serial Bus) cable 102. The terminal device 100 is also communicatively connected to a summary server 10 via a communication network 2, such as the Internet and a LAN (Local Area Network).

[0015] For example, the terminal device 100 executes an application program (hereinafter, referred to as an app) for the summarization system 1 to display a conference video including images of multiple participants captured by the conference device 110, and transmits the speech sounds (audio data) of the multiple participants and information for identifying the speakers to the summary server 10. The terminal device 100 can also display a summary of the speech of each participant summarized in real time by the summary server 10.

[0016] The summary server 10 is, for example, an information processing device having a computer configuration, or a system including multiple computers. The summary server 10 provides a summary service that summarizes the speech of multiple participants acquired from the terminal devices 100 in accordance with the progress of the conference and displays the summary on the terminal devices 100.

[0017] (Summary process overview) For example, assume that speeches made in a general conference held face-to-face by multiple participants A to D are to be summarized. In this case, the participants connect the terminal device 100 and the conference device 110 with a USB cable 102 or the like, and turn on the power of the terminal device 100 and the conference device 110. The participants also start up an application for the summarization system 1 that is installed in the terminal device 100.

[0018] The terminal device 100 executes an application corresponding to the summarization system 1 to transmit to the summarization server 10 the speech sounds (speech data) of the multiple participants acquired from the conference device 110 and information for identifying the speakers.

[0019] The summary server 10 converts the speech sounds of the multiple participants received from the terminal device 100 into multiple pieces of text data. When the multiple pieces of text data reach a predetermined summarization condition, the summary server 10 generates a summary sentence that summarizes the multiple pieces of text, and delivers the generated summary sentence to the terminal device 100.

[0020] Here, the summarization condition is, for example, when the size (e.g., number of characters) of the multiple texts converted by the summarization server 10 reaches a first threshold. As another example, the summarization condition may be when the similarity calculated from the multiple texts converted by the summarization system 1 falls below a second threshold (when the topic changes).

[0021] The terminal device 100 displays the summary delivered from the summary server 10. This allows the participants to quickly check the most recent summary. However, this method alone has the problem that it is difficult to provide a real-time summary that takes into account the role of each participant, such as the person who gave the presentation or the decision maker. There is also the problem that it is difficult to understand the points that are emphasized in the progress of the conference.

[0022] Therefore, the summarization system 1 according to the present embodiment has a function of identifying a specific participant based on a display image showing the multiple participants, and focusing on summarizing the speech of the identified specific participant. For example, the summarization system 1 determines weighting of the multiple participants so that the weight of the specific participant is greater than the weight of the other participants, and summarizes the speech content of the multiple participants in real time based on the weighting of the multiple participants.

[0023] Preferably, the summarization system 1 changes a compression ratio for summarizing the speech contents of the multiple participants according to the weighting of the multiple participants. Here, the compression ratio may be, for example, (Compression ratio) = (summary length) / (original length) It is expressed as:

[0024] As a method for identifying a specific participant, the summary system 1 may, for example, set a participant selected by a participant selection operation in a conference video showing multiple participants (one example of a display screen showing multiple participants) as the specific participant. This allows a participant to obtain a real-time summary that takes into account the role of each participant by selecting, for example, the person who gave a presentation or the decision maker from the conference video that shows multiple participants. This also allows a participant to obtain a summary that makes it difficult to understand the points that are considered important in the progress of the conference.

[0025] As another example, the summarization system 1 may set a participant who is closer to the front of the conference video output by the conference device 110 as a specified participant. For example, the conference device 110 has a function of capturing images of multiple participants with a 360° camera capable of capturing images of participants in a 360° range around the conference device 110, and setting the front of the conference video from an application executed by the terminal device 100. In this case, the summarization system 1 may set a participant who is closer to the front of the conference video than other participants as a specified participant. Note that if the conference device 110 does not have a function of setting the front, a participant may set the front of the conference video by changing the orientation of the conference device 110.

[0026] This allows participants to obtain a real-time summary that takes into account the role of each participant by, for example, setting the front of a video of a conference in which multiple participants are filmed to face the person giving the presentation or the decision maker, etc. This also allows participants to obtain a summary that does not make it difficult to understand the important points in the progress of the conference.

[0027] Note that the system configuration of the summarization system 1 shown in Fig. 1 is just an example. For example, as shown in Fig. 2, the summarization system 1 may summarize, in real time, the contents of speeches of multiple participants A to E in a web conference held between a terminal device 100 at a base 3 and a terminal device 101 at another base 4 via a conference server 20.

[0028] In this case, the terminal device 100 transmits the speech voices of participants A to D acquired by the conference device 110 and the speech voice of participant E acquired in the web conference to the summary server 10. Furthermore, the terminal device 100 displays the conference video of its own location 3 captured by the conference device 110 and the conference video of the other location 4 captured by the terminal device 101 at the other location 4. This allows the participants to select a desired participant from participants A to E included in the conference video of its own location 3 and the conference video of the other location 4.

[0029] As described above, according to this embodiment, it is possible to easily summarize in real time the utterances of multiple participants in a communication, focusing on points that are important in the progress of the communication.

[0030] <Hardware configuration> (Summary server and terminal device hardware configuration) The summary server 10 and the terminal device 100 have, for example, the hardware configuration of a computer 300 as shown in Figure 3. Alternatively, the summary server 10 is made up of a plurality of computers 300.

[0031] Fig. 3 is a diagram showing an example of a hardware configuration of a computer according to an embodiment. For example, as shown in Fig. 3, a computer 300 includes a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, a HD (Hard Disk) 304, a HDD (Hard Disk Drive) controller 305, a display 306, an external device connection I / F (Interface) 307, a network I / F 308, a keyboard 309, a pointing device 310, a DVD-RW (Digital Versatile Disk Rewritable) drive 312, a media I / F 314, and a bus line 315.

[0032] Of these, the CPU 301 controls the overall operation of the computer 300. The ROM 302 stores programs used to start up the computer 300, such as an IPL (Initial Program Loader). The RAM 303 is used, for example, as a work area for the CPU 301. The HD 304 stores programs such as an OS (Operating System), applications, device drivers, and various data. The HDD controller 305 controls the reading and writing of various data from and to the HD 304, for example, under the control of the CPU 301. The HD 304 and the HDD controller 305 are examples of storage devices included in the computer 300.

[0033] The display 306 displays various types of information such as a cursor, a menu, a window, characters, or an image. The display 306 may be provided outside the computer 300. The external device connection I / F 307 is an interface for connecting various external devices such as the conference device 110 to the computer 300. The network I / F 308 is an interface for connecting the computer 300 to the communication network 2 and communicating with other devices.

[0034] The keyboard 309 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 310 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. Note that the keyboard 309 and the pointing device 310 may be provided outside the computer 300.

[0035] The DVD-RW drive 312 controls reading and writing of various data from and to a DVD-RW 311, which is an example of a removable recording medium. The DVD-RW 311 is not limited to a DVD-RW, and may be another removable recording medium. The media I / F 314 controls reading and writing (storing) of data from and to a medium 313 such as a flash memory. The bus line 315 includes an address bus, a data bus, various control signals, and the like for electrically connecting the above components.

[0036] 3 is an example of the hardware configuration of computer 300. For example, summary server 10 is not limited to a physical machine, and may be realized by a program running on a virtual machine on the cloud.

[0037] (Hardware configuration of the conferencing device) 4 is a diagram showing an example of a hardware configuration of a conference device according to an embodiment. The conference device 110 includes, for example, a CPU 401, a ROM 402, a RAM 403, a solid state drive (SSD) 404, a network I / F 405, an external device connection I / F 406, an operation unit 408, a sound processing unit 410, a microphone array 411, a speaker 412, a video processing unit 413, one or more cameras 414, and a bus 416.

[0038] The CPU 401 is an arithmetic unit (processor) that executes a predetermined program to control various functions of the conference device 110. The ROM 402 is a non-volatile memory that stores, for example, a program used to start up the CPU 401. The RAM 403 is a volatile memory that is used, for example, as a work area for the CPU 401. The SSD 404 is an example of a storage device that stores, for example, programs, data, setting information, and the like for the conference device 110.

[0039] The network I / F 405 is a communication interface for connecting the conference device 110 to, for example, the communication network 2. The external device connection I / F 406 is an interface for connecting an external device such as the terminal device 100 to the conference device 110. The display I / F 407 is an interface for connecting a display device or the like to the conference device 110. The operation unit 408 is an input device that accepts user operations, such as, for example, an operation button, a switch, or a touch panel.

[0040] The sound processing unit 410 is a device that performs various sound processing such as beamforming that controls directivity using a microphone array 411. The sound processing unit 410 also performs sound processing that outputs various sounds such as conference voice using a speaker 412.

[0041] The video processing unit 413 is a device that acquires images (video images) of the periphery of the conference device 110 from one or more cameras 414, and performs image processing such as image synthesis, image quality correction, and distortion correction on the acquired images. The one or more cameras 414 are photographing devices that capture images of the periphery of the conference device 110. The bus 416 is commonly connected to each of the above components, and transmits, for example, address signals, data signals, and various control signals.

[0042] <Functional configuration> Next, a description will be given of the functional configuration of the summarization system 1. Fig. 5 is a diagram showing an example of the functional configuration of the summarization system according to an embodiment.

[0043] (Functional configuration of terminal device) The terminal device 100 realizes, for example, each functional configuration as shown in Fig. 5 by the CPU 301 executing a predetermined program stored in a storage medium such as the HD 304. In the example of Fig. 5, the terminal device 100 has each functional configuration such as a first communication unit 501, a display control unit 502, an operation reception unit 503, an identification unit 504, a determination unit 505, a data transmission unit 506, and a second communication unit 507. Note that at least a part of each of the above functional configurations may be realized by hardware.

[0044] The first communication unit 501 executes a first communication process for transmitting and receiving data to and from the conference device 110 using the external device connection I / F 307, for example.

[0045] The display control unit 502 executes a display control process for displaying various display screens on a display unit such as the display 306. For example, the display control unit 502 controls the display of an application screen, a conference video, a summary display screen, etc. The operation reception unit 503 executes an operation reception process for receiving operations on the terminal device 100. For example, the operation reception unit 503 receives operations from participants on the display screen displayed on the display unit by the display control unit 502.

[0046] The identification unit 504 executes an identification process for identifying a predetermined participant based on a display image displaying the multiple participants. As an example, the identification unit 504 determines a participant selected by a selection operation in a conference video displaying multiple participants as the predetermined participant. As another example, the identification unit 504 may determine a participant close to the center (front) of a conference image captured by the conference device 110 (an example of a display image displaying multiple participants) as the predetermined participant.

[0047] The determining unit 505 executes a determination process for determining weightings of multiple participants so that the weight of a specific participant identified by the identifying unit 504 is greater than the weights of other participants. For example, the determining unit 505 may set the weight of the participant identified by the identifying unit 504 to "1" from a display screen displaying multiple participants, and set the weights of the other participants to values ​​smaller than "1" (e.g., "0.3"). Alternatively, the determining unit 505 may weight multiple participants so that the weighting decreases as the participant moves away from the center (front) of the display screen displaying the multiple participants.

[0048] The data transmitting unit 506 executes a data transmitting process of acquiring the speech of multiple participants participating in the conference and transmitting the acquired speech (speech data) to the summary server 10. For example, the data transmitting unit 506 acquires the speech (speech data) of multiple participants output by the conference device 110. Note that the speech of multiple participants transmitted by the data transmitting unit 506 includes, for example, information indicating the speaker identified by the conference device 110 or the direction of the speaker (hereinafter referred to as participant identification information), and weighting information of the multiple participants determined by the determination unit 505.

[0049] The second communication unit 507 uses, for example, the network I / F 308 to connect the terminal device 100 to the communication network 2 and executes communication processing for communicating with the summary server 10, the conference server, other information processing devices, and the like.

[0050] (Summary server functional configuration) Summary server 10 realizes, for example, each functional configuration as shown in Fig. 5 by one or more computers 300 executing a predetermined program stored in a storage medium such as HD 304. In the example of Fig. 5, summary server 10 realizes each functional configuration such as a communication unit 511, an acquisition unit 512, a conversion unit 513, a judgment unit 514, a summary control unit 515, a provision unit 516, and a DB (database) 517. Note that at least a part of each of the above functional configurations may be realized by hardware.

[0051] The communication unit 511 connects the summary server 10 to the communication network 2 using, for example, the network I / F 308 and executes communication processing for communicating with other devices such as the terminal device 100 .

[0052] The acquisition unit 512 executes an acquisition process to acquire speech sounds (speech data) of multiple participants participating in a conference (communication). For example, the acquisition unit 512 acquires the speech sounds of the multiple participants that the data transmission unit 506 of the terminal device 100 transmits to the summary server 10. Note that the speech sounds of the multiple participants acquired by the acquisition unit 512 include, for example, identification information of the participants and weighting information of the multiple participants.

[0053] The conversion unit 513 executes a conversion process for converting the speech sounds of the multiple participants acquired by the acquisition unit 512 into multiple text data. The process for converting speech sounds into text is also called, for example, transcription or text conversion. The conversion unit 513 may be realized by, for example, a transcription server that converts the speech sounds of the multiple participants into multiple text data.

[0054] The determination unit 514 executes a determination process to determine whether the plurality of text data converted by the conversion unit 513 satisfy a predetermined summarization implementation condition. As one example, the determination unit 514 determines whether the size of the plurality of text data converted by the conversion unit 513 reaches a first threshold (one example of the summarization implementation condition). As another example, the determination unit 514 determines whether the similarity calculated from the plurality of text data converted by the conversion unit 513 falls below a second threshold (another example of the summarization implementation condition). Note that the determination unit 514 may be realized by a summarization trigger server or the like that determines whether the plurality of texts satisfy a predetermined summarization implementation condition.

[0055] The summary control unit 515 executes a summary control process for controlling the summarization unit 520 to summarize the speech contents of the multiple participants in real time based on the weighting of the multiple participants. In this embodiment, the summarization unit 520 can use, for example, chatGPT (Generative Pretrained Transformer), which is a known generation AI (Artificial Intelligence). The summarization unit 520 may be located outside or inside the summarization system 1.

[0056] For example, when the judgment unit 514 determines that a specified summarization condition is satisfied, the summary control unit 515 creates a prompt for summarizing text data indicating the speech content of multiple participants with a specified weighting, and transmits the created prompt to the summarization unit 520.

[0057] Based on the prompt received from the summary control unit 515, the summary unit 520 generates a summary sentence by summarizing the text data indicating the speech contents of the multiple participants with a specified weighting, and transmits the summary sentence to the summary control unit 515.

[0058] The providing unit 516 provides the summary generated by the summary control unit 515 using the summarizing unit 520 to the terminal device 100. For example, the providing unit 516 provides the terminal device 100 with the summary that the summary control unit 515 has generated using the summarizing unit 520 to summarize the progress of the conference.

[0059] DB517 is a database that stores, for example, converted text converted by conversion unit 513, a summary generated by summary control unit 515 using summary unit 520, and speech of multiple participants acquired by acquisition unit 512.

[0060] The functional configuration of summarization system 1 shown in Fig. 5 is an example. For example, among the functional components of terminal device 100 in Fig. 5, identification unit 504, determination unit 505, etc. may be included in summary server 10. Furthermore, each functional component of summary server 10 may be included in terminal device 100, or may be distributed among multiple servers. In short, each functional component of summarization system 1 shown in Fig. 5 may be included in any of the devices included in summarization system 1.

[0061] (Functional configuration of the conference device) 6 is a diagram showing an example of the functional configuration of a conference device according to an embodiment. The conference device 110 includes, for example, a communication unit 601, an audio acquisition unit 602, a direction detection unit 603, a video acquisition unit 604, an image detection unit 605, an identification unit 606, a video creation unit 607, and an audio output unit 608.

[0062] The communication unit 601 is realized by, for example, a program executed by the CPU 401, the external connection I / F 406, etc., and executes communication processing for transmitting and receiving data to and from the terminal device 100.

[0063] The voice acquisition unit 602 is realized by, for example, a program executed by the CPU 401, the microphone array 411, the sound processing unit 410, etc., and executes a voice acquisition process for acquiring voice around the conference device 110. In addition, the voice acquisition unit 602 also performs, for example, beamforming by the microphone array 411, sound quality adjustment of the acquired voice, or volume adjustment of the acquired voice.

[0064] The direction detection unit 603 is realized, for example, by a program executed by the CPU 401, the sound processing unit 410, etc., and executes a direction detection process to detect the direction of sound based on the conference sound acquired by the sound acquisition unit 602 with the microphone array 411. For example, the direction detection unit 603 analyzes the sound data acquired by the multiple microphones of the microphone array 411, and estimates the direction of the sound source.

[0065] The video acquisition unit 604 is realized by, for example, a program executed by the CPU 401, one or more cameras 414, and the video processing unit 413. The video acquisition unit 604 executes a video acquisition process to acquire a video captured around the conference device 110. The video acquisition unit 604 also performs image quality correction, distortion correction, and the like on the acquired video.

[0066] The image detection unit 605 is realized by, for example, a program executed by the CPU 401, the video processing unit 413, etc., and the video acquisition unit 604 executes image detection processing to detect images of people from video of the conference captured by one or more cameras 414. For example, the image detection unit 605 may detect images of people by estimating an area in which a person is shown from the input video using a machine learning model or the like that has been trained in advance to estimate an area in which a person is shown.

[0067] Here, machine learning is a technology for making a computer acquire human-like learning capabilities, in which the computer autonomously generates algorithms required for judgments such as data identification from learning data that is previously loaded, and applies these to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods; any learning method for machine learning is acceptable.

[0068] The identification unit 606 is realized, for example, by a program executed by the CPU 401. The identification unit 606 executes an identification process to identify a speaker or the direction of the speaker based on the direction of a sound detected by the direction detection unit 603 and an image of a person detected by the image detection unit 605. For example, the identification unit 606 generates identification information of a participant based on the direction of the speaker relative to the conference device 110, and adds the identification information of the participant to the speech sounds (audio data) of the multiple participants.

[0069] The video creation unit 607 is realized, for example, by a program executed by the CPU 401, the video processing unit 413, etc., and executes a video creation process to create a conference video displaying multiple participants based on the video of the periphery of the conference device 110 acquired by the video acquisition unit 604. In addition, the video creation unit 607 outputs the created conference video to the terminal device 100, etc.

[0070] The audio output unit 608 is realized by, for example, a program executed by the CPU 401, a sound processing unit 410, and a speaker 412. The audio output unit 608 executes audio output processing to output audio (for example, conference audio of another site 4) based on audio data input from, for example, the terminal device 100.

[0071] <Processing flow> Next, the flow of processing of the summarization method executed by the summarization system 1 will be described.

[0072] [First embodiment] (Summary system processing) Fig. 7 is a flowchart showing an example of processing of the summarization system according to the first embodiment. This processing shows an example of processing executed by the summarization system 1 described in Fig. 1, for example. It is assumed that at the start of the processing shown in Fig. 7, the conference device 110 is connected to the terminal device 100, and the terminal device 100 and the conference device 110 are powered on. It is also assumed that the terminal device 100 is executing an application for the summarization system 1.

[0073] In step S701, the display control unit 502 of the terminal device 100 displays, for example, a conference video 800 as shown in FIG.

[0074] Fig. 8 is a diagram showing an image of a conference video according to the first embodiment. In the example of Fig. 8, the conference video 800 includes an entire display area 801 that displays the entirety of multiple participants participating in the conference, and a close-up display area 802 that displays close-up images of a predetermined number of participants. Note that the conference video 800 is an example of a display image that displays multiple participants.

[0075] In the whole display area 801, for example, an image of the entire conference room where the conference is taking place is displayed by one or more cameras 414 provided in the conference device 110. In addition, in the close-up display area 802, a predetermined number of participants who have recently spoken are displayed in close-up.

[0076] 8, three display frames 803-1, 803-2, and 803-3 are displayed in the close-up display area 802. In this case, for example, when users A, B, and C speak in this order, an image of user A, an image of user B, and an image of user C are displayed in close-up (enlarged) in the close-up display area 802. Note that the number of display frames displayed in the close-up display area 802 may be two, or may be four or more.

[0077] In step S702, the operation reception unit 503 of the terminal device 100 determines whether or not a selection operation has been received for the conference video 800. If a selection operation has been received, the operation reception unit 503 shifts the process to step S703. On the other hand, if a selection operation has not been received, the operation reception unit 503 executes the process of step S702 again.

[0078] In step S703, the identification unit 504 of the terminal device 100 determines whether or not a close-up video has been selected. If a close-up video has been selected, the operation acceptance unit 503 transitions the process to step S704. On the other hand, if a close-up video has not been selected, the operation acceptance unit 503 returns the process to step S702.

[0079] Preferably, for example, in the conference video 800 shown in Fig. 8, when a close-up image of participant A displayed in display frame 803-1 is selected, the display control unit 502 displays, for example, a selection mark 803 on the close-up image of the selected participant A. This allows the participants to easily understand that participant A is currently set as a predetermined participant. Note that the processes of steps S702 and S703 are an example of a process of identifying a predetermined participant based on a display image displaying a plurality of participants.

[0080] When the process proceeds to step S704, the determining unit 505 of the terminal device 100 determines the weighting of the participant of the selected close-up video to be greater than the weighting of the other participants. In the example of Fig. 8, since the close-up video of participant A is selected, the determining unit 505 determines the weighting of the multiple participants A to D such that the weighting of participant A is greater than the weighting of the other participants B to D. In addition, the determining unit 505 manages the weighting of the multiple participants using, for example, weighting information 900 as shown in Fig. 9(A).

[0081] FIG. 9 is a diagram for explaining weighting information according to the first embodiment. In the example of FIG. 9(A), the weighting information 900 includes information such as "direction", "person identification symbol", "importance", and "weight" as items. "Direction" is information indicating the direction of each participant with respect to the conference device 110. "Person identification symbol" is identification information for identifying a participant. For example, the summary system 1 manages a participant who is in the direction of "1" with respect to the conference device 110 by a person identification symbol "A". Note that here, the person identification symbol "A" corresponds to participant A, and the person identification symbol "B" corresponds to participant B. Also, the person identification symbol "C" corresponds to participant C, and the person identification symbol "D" corresponds to participant D. Note that the "direction" and "person identification information" are examples of identification information of participants.

[0082] The "importance" is managed as "○" for participants with high importance and "×" for other participants. In the first embodiment, the determining unit 505 manages the importance of a specific participant (here, participant A) identified by the identifying unit 504 as "○".

[0083] "Weight" is information indicating the weighting of multiple participants determined by the determination unit 505. In the example of FIG. 9, the determination unit 505 sets the weight of participant A, whose importance is "○", to "1", and the weight of other participants B to D to "0.5". Here, values ​​such as "1" and "0.5" are merely examples, and other values ​​may be used. In the example of FIG. 9, the weights of other participants B to D are set to the same value "0.5", but since user D, who is not displayed in close-up, is considered to have had no recent utterance, the weight may be set to a value lower than "0.5", such as "0.3".

[0084] FIG. 9(B) is a diagram showing an example of the data structure of converted text converted by conversion unit 513. "Speech direction", which is information corresponding to the "direction" described in FIG. 9(A), is added to converted text 901. When this "speech direction" has the same value, summarization system 1 determines that the speech is from the same participant. Note that summarization system 1 may add "person identification information" described in FIG. 9(A) to converted text 901 instead of "speech direction" and manage it. Furthermore, summarization system 1 may further add "weight" described in FIG. 9(A) to converted text 901 and manage it.

[0085] In step S705, the summary control unit 515 of the summary server 10 controls the summarization unit 520 so as to summarize the speech contents of the multiple participants in real time with the weighting determined by the determination unit 505.

[0086] For example, the speech voices of the multiple participants received by the summary server 10 from the terminal device 100 are added with a person identification symbol (or direction) and weight information from the weighting information 900 described in Fig. 9. When a predetermined summary implementation condition is satisfied, the summary control unit 515 transmits, based on this information, a prompt as shown in Fig. 10 and text data indicating the speech contents of the multiple participants to a summary unit 520 such as chatGPT.

[0087] FIG. 10 is a diagram showing an example of a prompt according to the first embodiment. In the example of FIG. 10, the prompt 1000 includes information 1001 that specifies the weights of multiple participants. Furthermore, the letters A, B, C, and D included in the information 1001 that specifies the weights of multiple participants correspond to the above-mentioned "person identification code." Furthermore, in the prompt 1000, the conversation text corresponds to a converted text in which the contents of the speech of the multiple participants are converted into text. This prompt 1000 allows the summary control unit 515 to instruct the summarizing unit 520 to create a summary text based on the weightings of the multiple participants.

[0088] Fig. 11 is a diagram showing an image of the utterance contents and the summary sentence according to the first embodiment. The utterance contents of multiple participants A to D are converted into text data. Summary 1120 shows an example of a summary sentence that the summary control unit 515 has the summarizing unit 520 summarize based on the weighting of the multiple participants A to D. In the example of Fig. 11, the summary 1120 is a summary sentence that summarizes the utterance 1110 with the ratio of participant A being "1" and the ratio of the other participants B to D being "0.5".

[0089] In step S 706 , the providing unit 516 of the summary server 10 provides the summary generated by the summary control unit 515 using the summarizing unit 520 to the terminal device 100 .

[0090] Thus, according to the first embodiment, a participant can select, for example, the person who gave the presentation or the decision maker from the conference video 800 as shown in FIG. 8, and thereby summarize the speeches of multiple participants with a focus on the speeches of the selected participant.

[0091] [Second embodiment] (Summary system processing) Fig. 12 is a flowchart showing an example of processing of the summarization system according to the second embodiment. This processing shows another example of processing executed by the summarization system 1 described in Fig. 1, for example. Note that the basic processing content is similar to the processing of the summarization system according to the first embodiment described in Fig. 7, so detailed description of the processing content similar to that of the first embodiment will be omitted here.

[0092] In step S1201, the display control unit 502 of the terminal device 100 displays, for example, a conference video 1300 as shown in FIG.

[0093] Fig. 13 is a diagram showing an image of a conference video according to the second embodiment. In the example of Fig. 13, a conference video 1300 displays an overall display image 1301 that displays the entirety of multiple participants participating in the conference, and an operation area 1302 for setting the front (center) of the overall display image 1301. A participant can set (change) the front of the overall display image 1301 by, for example, moving the operation area 1302 or the overall display image 1301 left or right.

[0094] Preferably, the overall display image 1301 is an image of multiple participants captured by a 360-degree camera capable of capturing images in all directions.

[0095] In step S1202, the identification unit 504 of the terminal device 100 determines whether or not the front of the conference video 1300 is set. If the front of the conference video 1300 is set, the identification unit 504 shifts the process to step S1203. On the other hand, if the front of the conference video 1300 is not set, the identification unit 504 repeatedly executes the process of step S1202.

[0096] Note that the process in step S1202 is an example. For example, if the front of the conference video 1300 is not set within a predetermined time, the specifying unit 504 may set the center of the current conference video 1200 as the front.

[0097] When the process proceeds to step S1203, the identification unit 504 of the terminal device 100 identifies a participant who is closest to the set front. For example, the identification unit 504 identifies participant B who is closest to the front 1303 in the conference video 1300 of Fig. 13 as the predetermined participant. Alternatively, the identification unit 504 may identify participants A and B who are within a predetermined range from the front 1303 as the predetermined participants.

[0098] In step S1204, the determining unit 505 determines the weighting of the participant closer to the front identified by the identifying unit 504 to be greater than the weighting of the other participants. In the example of Fig. 13, the determining unit 505 determines the weighting of the multiple participants A to D such that the weighting of participants A and B closer to the front 1303 is greater than the weighting of the other participants C and D. The determining unit 505 manages the weighting of the multiple participants using, for example, weighting information 1400 as shown in Fig. 14.

[0099] Fig. 14 is a diagram showing an image of weighting information according to the second embodiment. Weighting information 1400 according to the second embodiment includes information on "front" instead of the "importance" of weighting information 900 according to the first embodiment described in Fig. 9. The "front" information is managed by marking participants close to the front with "○" and other participants with "×".

[0100] "Weight" is information indicating weighting of multiple participants determined by the determination unit 505. In the example of FIG. 14, the determination unit 505 assigns a weighting of "1" to participants A and B, whose front is "○", and a weighting of "0.3" to other participants C and D. Here, values ​​such as "1" and "0.3" are merely examples, and other values ​​may be used. Furthermore, the determination unit 505 may determine the weighting of multiple participants such that the weighting of multiple participants A to D is greater the closer they are to the front.

[0101] In step S1205, the summary control unit 515 of the summary server 10 controls the summarization unit 520 so as to summarize the speech contents of the multiple participants in real time with the weighting determined by the determination unit 505.

[0102] For example, the speech voices of the multiple participants received by the summary server 10 from the terminal device 100 are added with a person identification symbol (or direction) and weight information from the weighting information 1400 described in Fig. 14. When a predetermined summary implementation condition is satisfied, the summary control unit 515 transmits, based on this information, a prompt as shown in Fig. 15 and text data indicating the speech contents of the multiple participants to a summary unit 520 such as chatGPT.

[0103] Fig. 15 is a diagram showing an example of a prompt according to the second embodiment. In the example of Fig. 15, a prompt 1500 includes information 1501 that specifies the weights of multiple participants. In the prompt 1500, the conversational sentence corresponds to a converted text that indicates the contents of the speech of the multiple participants. This prompt 1500 enables the summary control unit 515 to instruct the summarization unit 520 to create a summary sentence based on the weightings of the multiple participants.

[0104] Fig. 16 is a diagram showing an image of the utterance contents and a summary sentence according to the second embodiment. A utterance 1610 in Fig. 16 is data obtained by converting the utterance contents of multiple participants A to D into text. A summary 1620 shows an example of a summary sentence that the summary control unit 515 has the summarizing unit 520 summarize based on the weighting of the multiple participants A to D. In the example of Fig. 16, the summary 1620 is a summary sentence that summarizes the utterance 1610 with a ratio of "1" for participants A and B and "0.3" for other participants C and D.

[0105] In step S 1206 , the providing unit 516 of the summary server 10 provides the summary generated by the summary control unit 515 using the summarizing unit 520 to the terminal device 100 .

[0106] Thus, according to the second embodiment, a participant can focus on summarizing the speech of a participant who is closer to the front by, for example, aligning the front of the conference video 1300 as shown in FIG. 13 with the person giving the presentation or the decision maker.

[0107] [Third embodiment] In the first and second embodiments, for example, an example of processing in a case where participants A to D hold a face-to-face conference in a system configuration as shown in Fig. 1 has been described. However, the present invention is not limited to this, and the summary system 1 according to the present embodiment can also be applied to a case where participants A to E hold a web conference in a system configuration as shown in Fig. 2. Note that the basic processing contents are the same as in the first embodiment, and therefore detailed explanations of the processing contents similar to those in the first embodiment will be omitted here.

[0108] (Summary system processing) Fig. 17 is a flowchart showing an example of processing of the summarization system according to the third embodiment. This processing shows an example of processing executed by the summarization system 1 described in Fig. 2. It is assumed that at the start of the processing shown in Fig. 17, the terminal device 100 and the terminal device 101 are participating in the same web conference via the conference server 20 and are transmitting and receiving video and audio of the web conference to each other.

[0109] In step S1701, the display control unit 502 of the terminal device 100 displays, for example, a conference video 1800 as shown in FIG.

[0110] Fig. 18 is a diagram showing an image of a conference video according to the third embodiment. In the example of Fig. 18, a conference video 1800 includes a conference video 1801 output from the conference device 110 and a video 1803 of another location transmitted from a terminal device 101 of another location 4, as described in Fig. 8. In the example of Fig. 18, a participant E of another location 4 is shown in the video 1803 of another location. Note that the conference video 1800 is an example of a display image displaying multiple participants.

[0111] In step S1702, the operation acceptance unit 503 of the terminal device 100 determines whether or not a selection operation has been accepted for the conference video 1800. If a selection operation has been accepted, the operation acceptance unit 503 shifts the process to step S1703. On the other hand, if a selection operation has not been accepted, the operation acceptance unit 503 executes the process of step S1702 again.

[0112] When the process proceeds to step S1703, the identifying unit 504 of the terminal device 100 determines whether or not the close-up video 1802 or the video of another location 1803 has been selected from the conference video 1800 as shown in Fig. 18. If the close-up video 1802 or the video of another location 1803 has been selected, the operation accepting unit 503 proceeds to step S1704. On the other hand, if the close-up video 1802 or the video of another location 1803 has not been selected, the operation accepting unit 503 returns the process to step S1702.

[0113] When the process proceeds to step S1704, the determining unit 505 of the terminal device 100 determines the weight of the participant of the selected close-up video or video of the other location to be greater than the weight of the other participants. In the example of Fig. 18, since the close-up video of the participant A is selected, the determining unit 505 determines the weighting of the multiple participants A to E such that the weight of the participant A is greater than the weights of the other participants B to E.

[0114] 2, the speech of participant E transmitted from the terminal device 101 at the other location 4 is output from the conference device 110 at the own location 3. Preferably, the conference device 110 adds a person identification symbol (e.g., "E") different from the participants A to D to the speech (audio data) of participant E from the other location 4, and outputs the speech (audio data) to the terminal device 100.

[0115] In step S 1705 , the summary control unit 515 of the summary server 10 controls the summarization unit 520 to summarize the speech contents of a plurality of participants in real time with the weighting determined by the determination unit 505 .

[0116] In step S 1706 , the providing unit 516 of the summary server 10 provides the summary generated by the summary control unit 515 using the summarizing unit 520 to the terminal device 100 .

[0117] In this manner, the summarization system 1 can weight the speech of multiple participants in a web conference and summarize it in real time.

[0118] <Display screen> (Display screen example 1) 19 is a diagram (1) showing an example of a display screen of a terminal device according to an embodiment. This diagram shows an example of a display screen of an app on which the display control unit 502 of the terminal device 100 displays the converted text and the summary notified from the summary server 10.

[0119] 19, the display screen 1900 includes a first display area 1910 for displaying the converted text (transcribed text) and the summary, and a second display area 1920 for displaying the decisions and action items. Note that the second display area 1920 is optional and not required.

[0120] The display control unit 502 sequentially displays the converted texts 1911a, 1911b, 1911c, and 1911d distributed from the summary server 10 in the first display area 1910 in the order in which they were received. Similarly, the display control unit 502 sequentially displays the summaries 1912a and 1912b distributed from the summary server 10 in the first display area 1010 in the order in which they were received. In this way, the display control unit 502 can display the summary 1912b in the first display area 1910 in association with the multiple converted texts 1913 used to generate the summary 1912b, as shown in FIG.

[0121] In the following description, when referring to any of the converted texts 1911a, 1911b, 1911c, and 1911d, the term "converted text 1911" will be used. When referring to any of the summaries 1912a and 1912b, the term "summary 1912" will be used.

[0122] Preferably, the first display area 1910 has an "auto scroll" button 1914 for setting whether or not to automatically scroll the converted text 1911 and the summary sentence 1912. For example, the user can disable the auto scroll function by turning off the "auto scroll button" 1914. Preferably, the summary sentence 1912 has a hide button 1915 for hiding the converted text used to generate the summary sentence. For example, the user can hide the multiple converted texts 1913 used to generate the summary sentence 1912b by selecting the hide button 1915 of the summary sentence 1912b.

[0123] Preferably, when text 1921 displayed in second display area 1920 is edited, display control unit 502 reflects the edited content in converted text 1911 or summary 1912 from which the text is copied. Furthermore, the text displayed in second display area 1920 has a jump button 1922 for jumping to converted text 1911 or summary 1912 from which the text is copied.

[0124] (Display screen example 2) Fig. 20 is a diagram (2) showing an example of a display screen of a terminal device according to an embodiment. During a conference, the terminal device 100 often displays, for example, a conference video 2001 as shown in Fig. 20. Therefore, it may be difficult to display, for example, the display screen 1900 described in Fig. 19 on a display unit such as the display 306 of the terminal device 100.

[0125] Therefore, the display control unit 502 of the terminal device 100 may, for example, superimpose a display screen 2002 displaying summaries on a conference video 2001 displayed on the display unit of the terminal device 100 as shown in Fig. 20. In the example of Fig. 20, the display control unit 502 displays a display screen 2002 displaying two summaries superimposed on the conference video 2001. Note that the number of summaries displayed on the display screen 2002 may be one, or three or more.

[0126] <Real-time summary processing> Here, an overview of the real-time summarization process according to this embodiment will be described.

[0127] (Data Structure) Fig. 21 is a diagram illustrating a data structure of a summarization system according to an embodiment. Data managed by summarization system 1 includes, for example, conversation information 2100, summary sentence 2110, and converted texts 2111a, 2111b, 2111c, etc., as shown in Fig. 21. In the following description, "converted text 2111" will be used to refer to any converted text among converted texts 2111a, 2111b, and 2111c.

[0128] 21, conversation information 2100, summary sentence 2110, and converted text 2111 each have a "conversation ID" that identifies the conversation. The "conversation ID" enables summarization system 1 to distinguish which conversation each piece of data relates to. Furthermore, summarization system 1 creates summary sentence 2110 and converted text 2111 for each "conversation ID."

[0129] 21, summary sentence 2110 and converted text 2111 have a "summary ID" that identifies the summary sentence. This "summary ID" enables summarization system 1 to distinguish which summary sentence 2110 each converted text 2111 relates to. Furthermore, if converted text 2111 is changed by a user, for example, summarization system 1 can identify summary sentence 2110 generated from the changed converted text 2111 by this "summary ID."

[0130] The summary 2110 also has an attribute called "status." The summary system 1 can use this "status" to identify whether the summary 2110 is in a state of being generated, whether the summary has been completed, whether the summary has been manually corrected, etc. The summary system 1 can also use this "status" to provide the status of the summary to an application executed by the terminal device 100.

[0131] Furthermore, the conversation information 2100, the summary sentence 2110, and the converted text 2111 include various information as shown in FIG.

[0132] (Real-time summary processing) FIG. 22 is a flowchart showing an example of real-time summarization processing according to one embodiment. This processing shows a specific example of real-time summarization processing executed by the summarization system 1 in, for example, step S705 in FIG. 7. Here, "real-time" refers to summarizing the speech contents of multiple participants in a conference in accordance with the progress of the conference. Also, a conference is an example of communication according to this embodiment.

[0133] In step S2201, the acquisition unit 512 of the summary server 10 executes an acquisition process to acquire the speech sounds (speech data) of multiple participants participating in the conference. Note that the speech sounds of the multiple participants acquired by the acquisition unit 512 are added with information such as the "person identification symbol" (participant identification information) and "weight" (participant weighting information) described in FIG.

[0134] In step S2202, the conversion unit 513 of the summary server 10 executes a conversion process for converting the speeches of the multiple participants acquired by the acquisition unit 512 into multiple pieces of text data.

[0135] Preferably, in step S2203, the providing unit 516 of the summary server 10 distributes the text data converted by the converting unit 513 (hereinafter referred to as converted text) to the terminal device 100. This enables the terminal device 100 to quickly display the converted text 1901 on, for example, the display screen 1900 of the application described in FIG.

[0136] In step S2204, the judgment unit 514 of the summary server 10 executes a judgment process to judge whether the multiple converted texts converted by the conversion unit 513 satisfy predetermined summarization implementation conditions. A specific example of the judgment process will be described later. If the predetermined summarization conditions are satisfied, the judgment unit 514 shifts the process to step S2205. On the other hand, if the predetermined summarization conditions are not satisfied, the judgment unit 514 ends the process of FIG. 22.

[0137] Proceeding to step S2205, summary control unit 515 generates a summary sentence from the converted text based on the weightings of the multiple participants. For example, summary control unit 515 creates prompt 1000 described in Fig. 10 (or prompt 1500 described in Fig. 15), transmits it to summarization unit 520 together with the multiple converted texts converted by conversion unit 513, and obtains the summary sentence generated by summarization unit 520. Note that summarization system 1 excludes the converted text used for the summary from the target for subsequent summarizations, regarding it as converted text.

[0138] In step S2206, the providing unit 516 of the summary server 10 distributes the summary generated by the summary control unit 515 using the summarizing unit 520 to the terminal device 100. This enables the terminal device 100 to display the summary on, for example, the display screen 1900 of the application described in Fig. 19 or the display screen 2002 for displaying the summary described in Fig. 20.

[0139] The summary server 10 can summarize the speech contents of a plurality of participants in a conference in accordance with the progress of the conference, for example, by repeatedly executing the process of FIG.

[0140] (Decision process 1) Fig. 23 is a diagram showing an example of a determination process according to an embodiment. This process shows an example of a determination process in which, for example, in step S2204 in Fig. 22, the determination unit 514 determines whether or not the converted text converted by the conversion unit 513 satisfies a predetermined summarization execution condition.

[0141] In step S2301, when judgment unit 514 accepts converted text, it executes the processes in and after step S2302.

[0142] In step S2302, determination unit 514 calculates the number of characters of the accepted converted text.

[0143] In step S2303, the determination unit 514 determines whether the calculated number of received characters has reached a first threshold. Here, the first threshold is a number to be summarized (number of characters to be summarized) that is set in advance. If the number of received characters has not reached the first threshold, the determination unit 514 returns the process to step S2301. On the other hand, if the number of received characters has reached the first threshold, the determination unit 514 shifts the process to step S2304.

[0144] In step S2304, determination section 514 causes summary control section 515 to execute a summary control process. For example, determination section 514 transmits the accepted converted text group to summary control section 515.

[0145] In step S2305, determination unit 514 initializes the number of received characters of the converted text.

[0146] In step S2306, judgment unit 514 judges whether to continue accepting the converted text. If accepting the converted text is to be continued, judgment unit 514 returns the process to step S2301. On the other hand, if accepting the converted text is not to be continued, judgment unit 514 ends the process of FIG.

[0147] Fig. 24 is a diagram for explaining an example of the determination process according to an embodiment. For example, it is assumed that the determination unit 514 sequentially receives converted texts 1 to 8 as shown in Fig. 24. It is also assumed that the first threshold is set to 100 characters.

[0148] 23, when judgment unit 514 receives converted text 4, the number of characters in the converted text will be 100, so it transmits converted texts 1 to 4 to summary control unit 515 as converted text group 2401. As a result, summary control unit 515 generates a summary sentence that summarizes converted text group 2401 using summarization unit 520.

[0149] Next, by executing the process of Figure 23, when judgment unit 514 receives converted text 7, the number of characters in the converted text will be 100 characters, so it transmits converted texts 5 to 7 to summary control unit 515 as converted text group 2402.

[0150] In this way, by setting the number of characters of the converted text as a threshold, it is possible to adjust the amount of conversation for which summarization is to be performed. Preferably, the first threshold (number of characters to be summarized) can be set by the participants, an administrator, or the like.

[0151] (Decision process 2) Fig. 25 is a flowchart showing another example of the determination process according to an embodiment. This process shows another example of the determination process in which the determination unit 514 of the summary server 10 determines whether or not the converted text converted by the conversion unit 513 satisfies a predetermined summarization implementation condition in step S2204 in Fig. 22.

[0152] In step S2501, when judgment unit 514 accepts converted text, it executes the processes in and after step S2502.

[0153] In step S2502, determination unit 514 generates chunk data by collecting multiple pieces of accepted converted text. This is because there are cases where a single converted text alone is meaningless, and multiple converted texts are collected into chunk data.

[0154] FIG. 26 is a diagram for explaining another example of the determination process according to an embodiment. For example, the determination unit 514 sequentially receives converted texts 1 to 8 as shown in FIG. 26. In addition, in the example of FIG. 26, the determination unit 514 treats the three converted texts (an example of a predetermined number) most recently received as one chunk of data, as an example. For example, when the determination unit 514 receives converted text 3, the determination unit 514 treats converted texts 1 to 3 as chunk data 2601, and when the determination unit 514 receives converted text 4, the determination unit 514 treats converted texts 2 to 4 as chunk data 2602. Similarly, when the determination unit 514 receives converted text 5, the determination unit 514 treats converted texts 3 to 5 as chunk data 2603, and when the determination unit 514 receives converted text 6, the determination unit 514 treats converted texts 4 to 6 as chunk data 2604. The same applies to chunk data 2605 and 2606.

[0155] In this way, as one example, determination unit 514 treats a predetermined number of most recently received converted texts as one chunk of data. As another example, determination unit 514 may treat a plurality of received converted texts as one chunk of data when the number of characters in the plurality of received converted texts reaches a predetermined number of characters.

[0156] When creating chunk data, the determination unit 514 may simply combine multiple converted texts, or may input multiple converted texts to the summarization unit 520 to be summarized.

[0157] In step S2503 in Fig. 25, the generated chunk data is vectorized and the similarity is calculated. For example, the determination unit 514 calculates the similarity between the generated chunk data and the chunk data generated previously.

[0158] For example, in FIG. 26, when the determination unit 514 receives converted text 4 and creates chunk data 2602, the determination unit 514 calculates the similarity between chunk data 2602 and chunk data 2601.

[0159] As a method of vectorizing the chunk data, for example, the chunk data is divided into words and vector values ​​are calculated from the divided words. Note that as a method of calculating the vector values, for example, a well-known natural language processing such as Bag of Words or BERT (Bidirectional Encoder Representations from Transformers) can be applied.

[0160] In step S2504 of FIG. 25, the determination unit 514 determines whether the calculated similarity falls below a second threshold. Here, the second threshold is a value set in advance for determining that the similarity between chunk data is low. If the similarity is not below the second threshold, the determination unit 514 returns the process to step S2501. On the other hand, if the similarity falls below the second threshold, the determination unit 514 shifts the process to step S2505.

[0161] Proceeding to step S2205, judgment unit 514 causes summary control unit 515 to perform summarization up to the previous chunk data. For example, in Fig. 26, when converted text 4 is received, if the similarity falls below the second threshold, judgment unit 514 transmits converted texts 1 to 3 as a converted text group to summary control unit 515. Similarly, when converted text 7 is received, if the similarity falls below the second threshold, judgment unit 514 transmits converted texts 1 to 6 as a converted text group to summary control unit 515.

[0162] In step S2506, the decision unit 514 determines that the converted text used for summarization has already been summarized, and excludes it from the targets of summarization.

[0163] In step S2507, judgment unit 514 judges whether to continue accepting the converted text. If accepting the converted text is to be continued, judgment unit 514 returns the process to step S2501. On the other hand, if accepting the converted text is not to be continued, judgment unit 514 ends the process of FIG.

[0164] 25, the determination unit 514 simply compares the calculated similarity with the second threshold, but this is just an example. For example, the moving average of the calculated similarity, or a differential value of the calculated similarity, may be compared with the threshold.

[0165] <Meeting video creation process> Here, for example, an overview of the conference video creation process executed by the conference device 110 will be described.

[0166] 27 is a flowchart showing an example of a conference video creation process according to an embodiment of the present invention, which shows an overview of the conference video creation process executed by the conference device 110 during a conference.

[0167] In step S 2701 , the direction detection unit 603 detects the direction of a sound based on the sound acquired by the sound acquisition unit 602 using the microphone array 411 .

[0168] In step S2702, the image detection unit 605 detects images of people from the video captured by the camera 414. For example, the image detection unit 605 detects images of participants attending a conference.

[0169] In step S2703, the identification unit 606 identifies the participant who has spoken based on the direction of the sound detected by the direction detection unit 603 and the image of the person detected by the image detection unit 605. For example, the identification unit 606 identifies the "direction" or "person identification symbol" of the person who has spoken. Note that the "direction" and "person identification symbol" are the same information as the "direction" and "person identification symbol" included in the weighting information 900 described in FIG. 9, for example.

[0170] In step S2704, the video creation unit 607 creates close-up videos of a predetermined number of participants who have most recently spoken from the video captured by the camera 414. For example, when the predetermined number is "3", the video creation unit 607 creates close-up videos of the participant who is currently speaking, the participant who has spoken, and the participant who spoke two times ago.

[0171] In step S2705, the video creation unit 607 creates a conference video including the video captured by the camera 414 and the created close-up video, and outputs the video to the terminal device 100, etc. This allows the terminal device 100 to display, for example, a conference video 800 as shown in FIG.

[0172] The process of step S2705 is an example. For example, the conference device 110 may output the video captured by the camera 414 and the created close-up video to the terminal device 100, and the terminal device 100 may create the conference video 800 using these videos.

[0173] In the above description, the communication in which multiple participants participate is described as a conference, but this is just one example. The communication in which multiple participants participate may be various other types of communication, such as business negotiations, lectures, seminars, information sessions, and casual conversations.

[0174] As described above, according to each embodiment of the present invention, it is possible to summarize the speech of multiple participants in a communication in an easy-to-understand manner, in accordance with the progress of the communication, and in accordance with the progress of the communication. <Supplementary Information> Each function of each embodiment described above can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above.

[0175] Additionally, the devices described in the examples are merely illustrative of one of several computing environments for implementing the embodiments disclosed herein. In one embodiment, the summary server 10 includes a plurality of computing devices, such as a server cluster, configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and to perform the processes disclosed herein.

[0176] Furthermore, the components of summary server 10 may be integrated into one server device, or may be separated into multiple devices.

[0177] <Additional Notes> This specification discloses a summarization system, an information processing device, a summarization method, and a program as described below. (Section 1) A summarization system for summarizing speeches of multiple participants participating in a communication, comprising: an identification unit that identifies a predetermined participant based on a display image that displays the plurality of participants; a determination unit that determines weightings of the plurality of participants such that a weighting of the predetermined participant is greater than a weighting of other participants; a summarization control unit that controls a summarization unit to summarize the speech contents of the plurality of participants in accordance with the progress of the communication based on the weighting of the plurality of participants; a providing unit that provides a summary text summarized by the summarizing unit; A summarization system having: (Section 2) 2. The summarization system according to claim 1, wherein the summarization control unit changes a compression rate for summarizing the speech contents of the plurality of participants in accordance with weightings of the plurality of participants. (Section 3) 3. The summary system according to claim 1, wherein the identification unit identifies a participant selected by a selection operation in a display image displaying the plurality of participants as the predetermined participant. (Section 4) 3. The summary system according to claim 1, wherein the identification unit identifies a participant who is closer to a center of a display image displaying the plurality of participants than other participants as the predetermined participant. (Section 5) the display image showing the plurality of participants includes an image of another location taken of the participants participating in the communication from the other location, When the video of the other location is selected, the identification unit identifies the participant of the other location as the predetermined participant. 3. A summarization system according to claim 1 or 2. (Section 6) The summarization system includes a conferencing device with a microphone array and one or more cameras; The conference device includes: a direction detection unit that detects a direction of a speaker based on the voice of the communication acquired by the microphone array; an image detection unit that detects an image of a person from a video of the communication captured by one or more cameras; an identification unit that identifies the speaker based on a direction of the speaker and an image of the person; having 6. The summarization system according to any one of claims 1 to 5, which outputs a video of the communication, an audio of the communication, and information for identifying the speaker. (Section 7) the conference device has an image creation unit that creates close-up images of a predetermined number of participants who have recently spoken based on the direction of the speaker and an image of the person; a display image representing the plurality of participants includes the close-up image; The identification unit identifies the participant photographed in the close-up video selected by a selection operation as the predetermined participant. 7. The summarization system according to claim 6. (Section 8) A summarization system for summarizing utterances of multiple participants in a communication, an identification unit that identifies a predetermined participant based on a display image that displays the plurality of participants; a determination unit that determines weightings of the plurality of participants such that a weighting of the predetermined participant is greater than a weighting of other participants; a summarization control unit that controls a summarization unit to summarize the speech contents of the plurality of participants in accordance with the progress of the communication based on the weighting of the plurality of participants; a providing unit that provides a summary text summarized by the summarizing unit; An information processing device having the above configuration. (Section 9) A system for summarizing the speech of multiple participants in a communication identifying a predetermined participant based on a display image showing the plurality of participants; determining weightings for the plurality of participants such that the weight of the given participant is greater than the weights of the other participants; A process of controlling a summarization unit to summarize the speech contents of the plurality of participants in accordance with the progress of the communication based on the weighting of the plurality of participants; A process of providing the summary text summarized by the summarizing unit to the participants; Execute the summary method. (Section 10) A summarization system for summarizing utterances of multiple participants in a communication, On the computer, identifying a predetermined participant based on a display image showing the plurality of participants; determining weightings for the plurality of participants such that the weight of the given participant is greater than the weights of the other participants; A process of controlling a summarization unit to summarize the speech contents of the plurality of participants in accordance with the progress of the communication based on the weighting of the plurality of participants; A process of providing the summary text summarized by the summarizing unit to the participants; A program to execute.

[0178] Although an embodiment of the present invention has been described above, the present invention is not limited to such a specific embodiment, and various modifications and applications are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0179] 1. Summary System 10 Summary Server 100 Terminal Equipment 110 Conference Devices 411 Microphone Array 414 Camera 504 Specific part 505 Decision Section 513 Conversion Unit 514 Judgment Department 515 Summary Control Section 520 Summary 603 Direction detection unit 605 Image detection unit 606 Identification Unit 800, 1300, 1800 Conference Video 1803 Videos from other locations [Prior art documents] [Patent documents]

[0180] [Patent Document 1] JP 2015-15632 A

Claims

1. A summarization system for summarizing speeches of multiple participants participating in a communication, comprising: an identification unit that identifies a predetermined participant based on a display image that displays the plurality of participants; a determination unit that determines weightings of the plurality of participants such that a weighting of the predetermined participant is greater than a weighting of other participants; a summarization control unit that controls a summarization unit to summarize the speech contents of the plurality of participants in accordance with the progress of the communication based on the weighting of the plurality of participants; a providing unit that provides a summary text summarized by the summarizing unit; A summarization system having:

2. The summarization system according to claim 1 , wherein the summarization control unit changes a compression ratio for summarizing the speech contents of the plurality of participants in accordance with the weighting of the plurality of participants.

3. The summary system according to claim 1 , wherein the specifying unit specifies a participant selected by a selection operation in a display image displaying the plurality of participants as the predetermined participant.

4. The summary system according to claim 1 , wherein the specifying unit specifies a participant that is closer to a center of a display image displaying the plurality of participants than other participants as the predetermined participant.

5. the display image showing the plurality of participants includes an image of another location taken of the participants participating in the communication from the other location, When the video of the other location is selected, the identification unit identifies the participant of the other location as the predetermined participant. The summarization system of claim 1 .

6. The summarization system includes a conferencing device with a microphone array and one or more cameras; The conference device includes: a direction detection unit that detects a direction of a speaker based on the voice of the communication acquired by the microphone array; an image detection unit that detects an image of a person from a video of the communication captured by one or more cameras; an identification unit that identifies the speaker or a direction of the speaker based on the direction of the speaker and an image of the person; having The summarization system of claim 1 , which outputs a video of the communication, an audio of the communication, and information identifying the speaker or a direction of the speaker.

7. the conference device has an image creation unit that creates close-up images of a predetermined number of participants who have recently spoken based on the direction of the speaker and an image of the person; a display image representing the plurality of participants includes the close-up image; The identification unit identifies the participant photographed in the close-up video selected by a selection operation as the predetermined participant. The summarization system of claim 6.

8. A summarization system for summarizing utterances of multiple participants in a communication, an identification unit that identifies a predetermined participant based on a display image that displays the plurality of participants; a determination unit that determines weightings of the plurality of participants such that a weighting of the predetermined participant is greater than a weighting of other participants; a summarization control unit that controls a summarization unit to summarize the speech contents of the plurality of participants in accordance with the progress of the communication based on the weighting of the plurality of participants; a providing unit that provides a summary text summarized by the summarizing unit; An information processing device having the above configuration.

9. A system for summarizing the speech of multiple participants in a communication identifying a predetermined participant based on a display image showing the plurality of participants; determining weightings for the plurality of participants such that the weight of the given participant is greater than the weights of the other participants; A process of controlling a summarization unit to summarize the speech contents of the plurality of participants in accordance with the progress of the communication based on the weighting of the plurality of participants; A process of providing the summary text summarized by the summarizing unit to the participants; Execute the summary method.

10. A summarization system for summarizing utterances of multiple participants in a communication, On the computer, identifying a predetermined participant based on a display image showing the plurality of participants; determining weightings for the plurality of participants such that the weight of the given participant is greater than the weights of the other participants; A process of controlling a summarization unit to summarize the speech contents of the plurality of participants in accordance with the progress of the communication based on the weighting of the plurality of participants; A process of providing the summary text summarized by the summarizing unit to the participants; A program to execute.

Images