Summarizing system, summarizing method, and program

The summarization system addresses the challenge of manual text selection by automatically generating and delivering summary sentences based on audio conversion and threshold conditions, enabling quick access to the latest summary.

JP2024158425APending Publication Date: 2024-11-08RICOH CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023073614
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-04-27
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing summarization systems require users to manually select text for summarization, making it difficult to quickly check the most recent summary sentence.

Method used

A summarization system that automatically generates and provides a summary sentence when predetermined conditions are met, including converting audio to text and summarizing multiple texts based on threshold conditions.

Benefits of technology

Facilitates easy access to the latest summary text by automating the summarization process, ensuring users can promptly view the most recent summary without manual selection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024158425000001_ABST
    Figure 2024158425000001_ABST
Patent Text Reader

Abstract

To provide a summarizing system for facilitating confirmation of the latest summary sentence.SOLUTION: A summarizing system includes an acquisition unit that acquires a voice sound, a translation unit that translates the voice sound to a plurality of texts, a creation unit that creates a summary sentence obtained by summarizing the plurality of texts when the texts satisfy a prescribed summarizing execution condition, and a providing unit that provides the summary sentence to a user.SELECTED DRAWING: Figure 3
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a summarization system, a summarization method, and a program. [Background technology]

[0002] There are summarization systems that generate summaries based on speech data. For example, a summary creation device is known that converts the content of each of multiple people in a conversation into text, displays the converted text in a selectable manner, and creates a summary consisting of the text selected by the user (see, for example, Patent Document 1). Summary of the Invention [Problem to be solved by the invention]

[0003] The technology disclosed in Patent Document 1 requires the user to take the time to select the text to be summarized from multiple converted texts, making it difficult to check the most recent summary sentence.

[0004] An embodiment of the present invention has been made in consideration of the above problems, and provides a summarization system that makes it easy to check the most recent summary sentence. [Means for solving the problem]

[0005] In order to solve the above problems, a summarization system according to one embodiment of the present invention has an acquisition unit that acquires speech, a conversion unit that converts the speech into a plurality of texts, a generation unit that generates a summary sentence that summarizes the plurality of texts when the plurality of texts satisfy predetermined summarization implementation conditions, and a provision unit that provides the summary sentence to a user. Effect of the Invention

[0006] According to one embodiment of the present invention, a summarization system can be provided that makes it easy to check the most recent summary sentence. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of a summary system according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer according to an embodiment. [Diagram 3] FIG. 1 is a diagram illustrating an example of a functional configuration of a summary system according to an embodiment. [Figure 4] FIG. 2 is a diagram illustrating a data structure of a summarization system according to an embodiment. [Diagram 5] FIG. 11 is a sequence diagram illustrating an example of a summarization process according to an embodiment. [Figure 6] FIG. 1 is a diagram showing an example of a display screen of a terminal device according to an embodiment. [Figure 7] FIG. 2 is a diagram showing an example of a display screen of the terminal device according to an embodiment; [Figure 8] FIG. 3 is a diagram showing an example of a display screen of the terminal device according to an embodiment. [Figure 9] 5 is a flowchart showing an example of a determination process according to the first embodiment. [Figure 10] FIG. 4 is a diagram for explaining a determination process according to the first embodiment. [Figure 11] 13 is a flowchart showing an example of a determination process according to the second embodiment. [Figure 12] FIG. 11 is a diagram for explaining a determination process according to the second embodiment. [Figure 13] FIG. 11 is a sequence diagram illustrating an example of a change process according to an embodiment. [Figure 14] FIG. 1 is a sequence diagram (1) showing an example of a re-summarization process according to an embodiment. [Figure 15] FIG. 1 is a diagram for explaining a re-summarization process according to an embodiment; [Figure 16] FIG. 13 is a diagram for explaining the re-summarization process according to an embodiment; [Figure 17] FIG. 11 is a sequence diagram (2) illustrating an example of a re-summarization process according to an embodiment. [Figure 18]FIG. 11 is a sequence diagram (3) illustrating an example of a re-summarization process according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present invention (the present embodiment) will be described in detail with reference to the drawings.

[0009] <System configuration> Fig. 1 is a diagram showing an example of a system configuration of a summarization system according to an embodiment. The summarization system 1 includes a summarization server 100 and multiple terminal devices 110a, 110b, ..., etc., which are connected to a communication network 2, such as the Internet and a LAN (Local Area network). In the following description, when referring to any of the multiple terminal devices 110a, 110b, ..., "terminal device 110" is used. Also, the number of terminal devices 110 shown in Fig. 1 is just an example, and the number of terminal devices 110 may be two or more.

[0010] The summarization system 1 is a system that summarizes, for example, a conversation between users using terminal devices 110 and provides the users with the most recent summary text. For example, the summarization system 1 summarizes audio from a web conference in which a user using terminal device 110a holds a conference with a user using another terminal device 110b via a conference server 10, and delivers the summarized text to the terminal devices 110a and 110b. Note that the conference server 10 may be a server external to the summarization system 1, or may be a server internal to the summarization system 1.

[0011] The summarization system 1 can also summarize conversations between users in general face-to-face meetings, etc. As an example, the following description will be given assuming that the summarization system 1 summarizes the audio of a web conference (hereinafter, referred to as a conference) and delivers the summarized summary to the terminal devices 110 participating in the conference.

[0012] The terminal device 110 is a general-purpose information terminal used by a user, such as a PC (Personal Computer), a tablet terminal, or a smartphone. However, the terminal device 110 is not limited to this, and may be, for example, a video conferencing device or an electronic device having a web conferencing function, such as an IWB (Interactive White Board). Here, an IWB is an electronic whiteboard having a function of a blackboard that allows mutual communication, and is also called an electronic blackboard. Here, the following description will be given assuming that the terminal device 110 is a general-purpose information terminal, as an example.

[0013] A user who wishes to participate in a web conference participates in the web conference by, for example, using a web conference application installed on the terminal device 110 or a web browser, etc., by accessing the conference address provided by the conference server 10.

[0014] The conference server 10 is, for example, an information processing device having a computer configuration, or a system including multiple computers. The conference server 10 provides a web conference service for transmitting and receiving content data including audio between multiple terminal devices 110. In this embodiment, the web conference service provided by the conference server 10 may be any web conference service.

[0015] The summary server 100 is, for example, an information processing device having a computer configuration, or a system including a plurality of information processing devices. The summary server 100 provides the summary service according to this embodiment.

[0016] 1 is an example of the system configuration of the summarization system 1. For example, in the summarization system 1, the summarization server 100 may have the functions of the conference server 10. Also, the summarization system 1 does not have to have the conference server 10.

[0017] (Summary of system processing) For example, it is assumed that a user using the terminal device 110a uses a web conference provided by the conference server 10 to hold a web conference (hereinafter, simply referred to as a conference) with another user using the terminal device 110b.

[0018] The terminal devices 110a and 110b acquire the user's voice by executing an application program (hereinafter, referred to as an app) corresponding to the summarization system 1, and transmit the acquired voice (voice data) to the summarization server 100.

[0019] The summary server 100 acquires the speech transmitted by the terminal devices 110a and 110b and converts the acquired speech into a plurality of texts (text data). In addition, when the plurality of texts meets a predetermined summarization condition, the summary server 100 generates a summary sentence that summarizes the plurality of texts and distributes the generated summary sentence to the terminal devices 110a and 110b.

[0020] Here, an example of the summarization condition is when the size (e.g., number of characters) of multiple texts converted by the summarization server 100 reaches a first threshold. Another example of the summarization condition is when the similarity calculated from multiple texts converted by the summarization system 1 falls below a second threshold (when the topic changes), etc.

[0021] The terminal devices 110a and 110b may display the summary delivered from the summary server 100, for example, on the display screen of an application, or may display the summary by superimposing it on the conference screen of the conference.

[0022] For example, in conventional technology such as that shown in Patent Document 1, the user must take the time to select the text to be summarized from multiple converted texts, making it difficult for the user to check the most recent summary sentence, for example, during a meeting.

[0023] On the other hand, the summarization system 1 according to the present embodiment automatically delivers summaries to the terminal devices 110a and 110b used by the users each time a plurality of converted texts satisfy a predetermined summarization implementation condition. Therefore, according to the present embodiment, it is possible to provide a summarization system that makes it easy to check the most recent summary.

[0024] <Hardware configuration> (Computer hardware configuration) Summary server 100 has, for example, the hardware configuration of computer 200 as shown in Fig. 2. Alternatively, summary server 100 is composed of a plurality of computers 200. Moreover, terminal device 110 has, for example, the hardware configuration of computer 200 as shown in Fig. 3.

[0025] Fig. 2 is a diagram showing an example of a hardware configuration of a computer according to an embodiment. For example, as shown in Fig. 2, the computer 200 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, a HD (Hard Disk) 204, a HDD (Hard Disk Drive) controller 205, a display 206, an external device connection I / F (Interface) 207, a network I / F 208, a keyboard 209, a pointing device 210, a DVD-RW (Digital Versatile Disk Rewritable) drive 212, a media I / F 214, and a bus line 215.

[0026] Furthermore, when the computer 200 is the terminal device 110, the computer 200 further includes a microphone 221, a speaker 222, an audio input / output I / F 223, a CMOS (Complementary Metal Oxide Semiconductor) sensor 224, an image sensor I / F 225, and the like.

[0027] Of these, the CPU 201 controls the overall operation of the computer 200. The ROM 202 stores programs used to start up the computer 200, such as an IPL (Initial Program Loader). The RAM 203 is used, for example, as a work area for the CPU 201. The HD 204 stores programs such as an OS (Operating System), applications, device drivers, and various data. The HDD controller 205 controls the reading and writing of various data from and to the HD 204, for example, under the control of the CPU 201. The HD 204 and the HDD controller 205 are examples of storage devices included in the computer 200.

[0028] The display 206 displays various types of information such as a cursor, a menu, a window, characters, or an image. The display 206 may be provided outside the computer 200. The external device connection I / F 207 is an interface for connecting various external devices to the computer 200. The network I / F 208 is an interface for connecting the computer 200 to the communication network 2 and communicating with other devices.

[0029] The keyboard 209 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 210 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The keyboard 209 and the pointing device 210 may be provided outside the computer 200.

[0030] The DVD-RW drive 212 controls reading and writing of various data from and to a DVD-RW 211, which is an example of a removable recording medium. The DVD-RW 211 is not limited to a DVD-RW, and may be another removable recording medium. The media I / F 214 controls reading and writing (storing) of data from and to a medium 213 such as a flash memory. The bus line 215 includes an address bus, a data bus, various control signals, and the like for electrically connecting the above components.

[0031] The microphone 221 is a built-in circuit that converts sound into an electrical signal. The speaker 222 is a built-in circuit that converts the electrical signal into physical vibrations to generate sound such as music or voice. The audio input / output I / F 223 is a circuit that processes input and output of audio signals between the microphone 221 and the speaker 222 under the control of the CPU 201.

[0032] The CMOS sensor 224 is a type of built-in imaging means that captures an image of a subject (e.g., a self-portrait) and obtains image data under the control of the CPU 201. Note that the computer 200 may have an imaging means such as a CCD (Charge Coupled Device) sensor instead of the CMOS sensor 224. The imaging element I / F 225 is a circuit that controls the driving of the CMOS sensor 224.

[0033] 2 is merely an example, and various modifications and applications are possible. For example, summary server 100 is not limited to a physical machine (computer 200), and may be realized by a program running on a virtual machine, etc.

[0034] <Functional configuration> Next, a description will be given of the functional configuration of the summarization system 1. Fig. 3 is a diagram showing an example of the functional configuration of the summarization system according to an embodiment.

[0035] (Functional configuration of terminal device) The terminal device 110 realizes a communication unit 311, a conference control unit 312, an audio transmission unit 313, a display control unit 314, an operation reception unit 315, and the like, by the CPU 201 executing a predetermined program stored in a storage medium such as the HD 204. At least a part of the above-mentioned functional configurations may be realized by hardware. In FIG. 3, the terminal device 110b has the same functional configuration as the terminal device 110a.

[0036] The communication unit 311 connects the terminal device 110 to the communication network 2 using the network I / F 208, for example, and executes communication processing for communicating with other devices such as the conference server 10, the summary server 100, and other terminal devices 110.

[0037] The conference control unit 312 connects to the conference server 10 using, for example, the communication unit 311, and transmits and receives content data including the conference voice. The conference control unit 312 also controls input and output of the conference voice using the audio input / output I / F 223, and output of content images (conference images, shared images, etc.) using the display 206. Note that the processing executed by the conference control unit 312 may be similar to that of a general web conference.

[0038] The voice transmitting unit 313 executes a voice transmitting process to acquire the voice of the user using the terminal device 110 and transmit it to the summary server 100. For example, the voice transmitting unit 313 acquires the voice acquired by the microphone 221 from the sound input / output I / F 223, and transmits the acquired voice (voice data) to the summary server 100. This enables the summary server 100 to summarize conversations not only in web conferences, but also in ordinary conferences and the like.

[0039] The voice transmitting unit 313 may obtain the user's voice from the conference control unit 312 and transmit the obtained voice to the summary server 100 .

[0040] The display control unit 314 executes a display control process for displaying various display screens, which will be described later, on a display unit such as the display 206. The operation acceptance unit 315 executes an operation acceptance process for accepting an operation by a user or the like. For example, the operation acceptance unit 315 accepts a user operation on a display screen displayed by the display control unit 314.

[0041] (Summary server functional configuration) Summary server 100 realizes a communication unit 301, an acquisition unit 302, a conversion unit 303, a determination unit 304, a generation unit 305, a provision unit 306, a DB (Database) 307, etc. by one or more computers 200 executing a predetermined program stored in a storage medium such as HD 204. Note that at least a part of the above functional configurations may be realized by hardware.

[0042] The summary server 100 also implements a storage unit 308 using storage devices such as the HD 204 and the HDD controller 205 .

[0043] The communication unit 301 connects the summary server 100 to the communication network 2 using, for example, the network I / F 208 and executes communication processing for communicating with other devices such as the terminal device 110 .

[0044] The acquiring unit 302 executes an acquisition process for acquiring the user's voice. For example, the acquiring unit 302 acquires the user's voice (voice data) that is transmitted by the voice transmitting unit 313 of the terminal device 110 to the summary server 100.

[0045] The conversion unit 303 executes a conversion process for converting the voice acquired by the acquisition unit 302 into a plurality of texts. The process for converting the voice into a plurality of texts may be called, for example, transcription or text conversion. The conversion unit 303 may be realized by, for example, a conversion server, a transcription server, or a text conversion server.

[0046] The determination unit 304 executes a determination process to determine whether the multiple texts converted by the conversion unit 303 satisfy a predetermined summarization implementation condition. As one example, the determination unit 304 determines whether the size of the multiple texts converted by the conversion unit 303 reaches a first threshold (one example of the summarization implementation condition). As another example, the determination unit 304 determines whether the similarity calculated from the size of the multiple texts converted by the conversion unit 303 falls below a second threshold (another example of the summarization implementation condition). Note that the determination unit 304 may be realized by a summarization trigger server or the like that determines whether the multiple texts satisfy a predetermined summarization implementation condition.

[0047] When the determination unit 304 determines that the multiple texts converted by the conversion unit 303 satisfy a predetermined summarization implementation condition, the generation unit 305 executes a generation process to generate a summary sentence summarizing the multiple texts. Note that in this embodiment, the method by which the generation unit 305 generates a summary sentence is not particularly limited. For example, the generation unit 305 may generate a summary sentence summarizing the multiple texts by using natural language processing, AI (Artificial Intelligence), or the like, and using a known cloud service or the like that summarizes sentences. Note that the generation unit 305 may be realized by a summary server or the like that summarizes sentences.

[0048] The summarization system 1 may further include a queuing server that queues a plurality of texts input to the generation unit 305. Alternatively, the generation unit 305 may have the function of a queuing server.

[0049] The providing unit 306 executes a providing process for providing the summary generated by the generating unit 305 to the user. For example, the providing unit 306 transmits the summary generated by the generating unit 305 to the terminal device 110 used by the user. Preferably, the providing unit 306 transmits, in addition to the summary generated by the generating unit 305, a plurality of texts used by the generating unit 305 to generate the summary to the terminal device 110 used by the user. Note that the providing unit 306 may be realized by, for example, a Pub / Sub server that provides a Publish / Subscribe messaging service.

[0050] The DB 307 is a database that stores, for example, converted text converted by the conversion unit 303, a summary generated by the generation unit 305, conversation information, and the like.

[0051] Fig. 4 is a diagram for explaining the data structure of a summarization system according to an embodiment. Data managed by summarization system 1 includes, for example, conversation information 400, summary sentence 410, and converted texts 411a, 411b, and 411c, as shown in Fig. 4. In the following description, "converted text 411" will be used to refer to any converted text among converted texts 411a, 411b, and 411c.

[0052] 4, conversation information 400, summary sentence 410, and converted text 411 each have a "conversation ID" that identifies the conversation. The "conversation ID" enables summarization system 1 to distinguish which conversation each piece of data relates to. Furthermore, summarization system 1 creates summary sentence 410 and converted text 411 for each "conversation ID."

[0053] 4, summary sentence 410 and converted text 411 each have a "summary ID" that identifies the summary sentence. Using this "summary ID", summarization system 1 can distinguish which summary sentence 410 each converted text 411 relates to. Furthermore, if converted text 41 is changed by a user, for example, summarization system 1 can use this "summary ID" to identify summary sentence 410 generated from the changed converted text 41.

[0054] The summary 410 also has an attribute called a "status." The "status" allows the summarization system 1 to identify whether the summary 410 is in a state of being generated, whether the summary has been completed, whether the summary has been manually corrected, etc. Using this "status," the summarization system 1 can provide the status of the summary to an application executed by the terminal device 110.

[0055] Furthermore, the conversation information 400, the summary sentence 410, and the converted text 411 include various pieces of information as shown in FIG.

[0056] The storage unit 308 stores, for example, various information (thresholds, set values, etc.), data, programs, etc. managed by the summary system 1.

[0057] In addition, in FIG. 3, the conference server 10 may be a general cloud service that provides a web conference service, and therefore a description thereof will be omitted here.

[0058] <Processing flow> Next, the process flow of the communication support method according to the present embodiment will be described.

[0059] (Summarization) 5 is a sequence diagram showing an example of a summarization process according to an embodiment. At the start of the process shown in FIG. 5, the terminal device 110a and the terminal device 110b are assumed to be participating in a conference (Web conference) in which content data including audio is transmitted and received via the conference server 10.

[0060] In steps S501 and S502, the terminal device 110a and the terminal device 110b register a subscription with the provider 306 that provides a Pub / Sub messaging service. By registering the subscription, the terminal device 110a and the terminal device 110b are able to communicate bidirectionally with the provider 306 and can receive notifications from the provider 306 without delay. However, this is not limited to the above, and the terminal device 110a and the terminal device 110b may obtain information from the provider 306 by other methods such as polling processing.

[0061] In step S503, the voice transmitting unit 313 of the terminal device 110a converts the voice acquired by the microphone 221 into voice data, and transmits the voice data to the summary server 100. In response, in step S504, the acquiring unit 302 of the summary server 100 receives the voice data transmitted by the terminal device 110a, and outputs the received voice data to the converting unit 303.

[0062] Similarly, in step S505, the voice transmitting unit 313 of the terminal device 110b converts the voice captured by the microphone 221 into voice data, and transmits the voice data to the summary server 100. In response, in step S506, the acquiring unit 302 of the summary server 100 receives the voice data transmitted by the terminal device 110b, and outputs the received voice data to the converting unit 303.

[0063] In step S507, the conversion unit 303 of the summary server 100 converts the speech data acquired by the acquisition unit 302 into text data. The conversion unit 303 also notifies the provision unit 306 of the converted text data (hereinafter referred to as converted text).

[0064] In step S509, providing unit 306 registers the converted text notified by conversion unit 303 in DB 307, for example, in a data structure as shown in FIG.

[0065] In step S510, the providing unit 306 delivers to the terminal device 110a the converted text notified by the conversion unit 303. In response to this, in step S511, the display control unit 314 of the terminal device 110a displays the delivered converted text on the display screen.

[0066] Similarly, in step S512, the providing unit 306 delivers the converted text notified by the conversion unit 303 to the terminal device 110b. In response to this, in step S513, the display control unit 314 of the terminal device 110b displays the delivered converted text on a display screen. An example of the display screen displayed by the terminal device 110 will be described later.

[0067] In step S514, providing unit 306 notifies determination unit 304 of the converted text notified by conversion unit 303. In response to this, in step S515, determination unit 304 checks the summarization execution conditions. Specifically, determination unit 304 checks whether the converted text (plurality of texts) converted by text conversion unit 507 satisfies predetermined summarization execution conditions.

[0068] In step S516, the determination unit 304 updates the information of the converted text registered in the DB 307. For example, the determination unit 304 assigns a summary ID or the like to the registered converted text.

[0069] The summarization system 1 repeatedly executes the processes of steps S503 to S516 while the conference is taking place, for example. Furthermore, if the decision unit 304 decides in step S515 that the summarization execution condition is satisfied, the summarization system 1 executes the process 520 of steps S521 to S528.

[0070] In step S521, when determination unit 504 determines that the summarization execution condition is satisfied, it notifies generation unit 305 of the multiple converted texts notified by provision unit 306 (hereinafter referred to as a converted text group). This converted text group includes the multiple converted texts notified by provision unit 306 after determination unit 504 determined the previous time that the summarization execution condition was satisfied.

[0071] In step S522, the generation unit 305 generates a summary by summarizing the converted text group (plurality of texts) notified by the determination unit 304. Furthermore, in step S523, the generation unit 305 notifies the provision unit 306 of the generated summary.

[0072] In step S524, the providing unit 306 registers the summary notified by the generating unit 305 in the DB 307, for example, in a data structure as shown in Fig. 4. For example, as shown in Fig. 4, the providing unit 306 registers the summary 410 so that the summary 410 and the converted texts 411a, 411b, and 411c used to generate the summary 410 have the same summary ID.

[0073] In step S525, the providing unit 306 delivers to the terminal device 110a the summary notified by the generation unit 305. In response to this, in step S526, the display control unit 314 of the terminal device 110a displays the delivered summary on the display screen.

[0074] Similarly, in step S527, the providing unit 306 delivers to the terminal device 110b the summary notified by the generation unit 305. In response to this, in step S528, the display control unit 314 of the terminal device 110b displays the delivered summary on the display screen.

[0075] (Display screen example 1) 6A and 6B are diagrams (1) showing examples of a display screen of a terminal device according to an embodiment. Diagram (A) shows an example of a display screen 600 of an app on which the display control unit 314 of the terminal device 110 displays the converted text and the summary notified from the summary server 100.

[0076] In the example of Fig. 6, the display screen 600 includes a first display area 610 for displaying the converted text (transcribed text) and the summary text, and a second display area 620 for displaying the decisions and action items. Note that the second display area 620 is optional and not required.

[0077] The display control unit 314 sequentially displays the converted texts 611a, 611b, 611c, and 611d distributed from the summary server 1 in the first display area 610 in the order in which they were received. Similarly, the display control unit 314 sequentially displays the summaries 612a and 612b distributed from the summary server 1 in the first display area 610 in the order in which they were received. In this way, the display control unit 314 can display the summary 612b in the first display area 610 in association with the multiple converted texts 613 used to generate the summary 612b, as shown in Fig. 6, for example.

[0078] In the following description, when referring to any of the converted texts 611a, 611b, 611c, and 611d, the term "converted text 611" is used. When referring to any of the summaries 612a and 612b, the term "summary 612" is used.

[0079] Preferably, the converted text 611 displayed in the first display area 610 has a delete icon 614 for deleting the converted text 611. The user can delete the converted text 611 corresponding to the delete icon 614 by selecting the delete icon 614. Similarly, the summary sentence 612 also has a delete icon 614 for deleting the summary sentence 612.

[0080] Preferably, the summary sentence 612 displayed in the first display area 610 has a bookmark icon 615 for displaying the summary sentence 612 in the second display area 620. A user can select a bookmark icon 615 to display a copy of the summary sentence 612 corresponding to the bookmark icon 615 in the second display area. Similarly, the converted text 611 also has a bookmark icon 615 for displaying the converted text 611 in the second display area 620.

[0081] Preferably, the first display area 610 has an “Auto Scroll” button 616 for setting whether or not to automatically scroll the converted text 611 and the summary sentence 612. For example, the user can disable the auto scroll function by turning off the “Auto Scroll Button” 616.

[0082] Preferably, the summary sentence 612 has a hide button 617 for hiding the converted text 611 used to generate the summary sentence. For example, the user can hide the multiple converted texts 613 used to generate the summary sentence 612b by selecting the hide button 617 of the summary sentence 612b, as shown in Fig. 6(B).

[0083] 6(B), when the multiple converted texts 613 used to generate the summary sentence 612b are hidden, the summary sentence 612b has a display button 618 for displaying the converted texts 613 used to generate the summary sentence 612b. By selecting the display button 618, the user can display again the converted texts 613 used to generate the summary sentence 612b, as shown in FIG. 6(A).

[0084] Preferably, when the text 621 displayed in the second display area 620 is edited, the display control unit 314 reflects the edited content in the converted text 611 or summary 612 from which the text is copied. The text displayed in the second display area 620 also has a jump button 622 for jumping to the converted text 611 or summary 612 from which the text is copied.

[0085] (Display screen example 2) 7 is a diagram (2) showing an example of a display screen of a terminal device according to an embodiment. When the terminal device 110 holds a conference (web conference) with other terminal devices 110, the terminal device 110 often displays a conference screen that displays conference video or presentation materials captured by the other terminal devices 110. For this reason, it may be difficult to display, for example, the display screen 600 described in FIG. 6 on a display unit such as the display 206 of the terminal device 110.

[0086] Therefore, the display control unit 314 of the terminal device 110 may superimpose a display screen 710 displaying the summary on a conference screen 700 displayed on the display unit of the terminal device 110, as shown in Fig. 7. In the example of Fig. 7, the display control unit 314 displays the display screen 710 displaying the two most recent summary sentences superimposed on the conference screen 700. Note that the number of summary sentences displayed on the display screen 710 may be one, or may be three or more.

[0087] 6, the summary displayed on the display screen 710 has a bookmark icon 711 for displaying the summary in the second display area 620. By selecting the bookmark icon 711, the user can display a copy of the summary corresponding to the bookmark icon 711 in the second display area 620 of the display screen 600 running in the background.

[0088] (Display screen example 3) Fig. 8 is a diagram (3) showing an example of a display screen of a terminal device in one embodiment. In the example of Fig. 8, the display control unit 314 of the terminal device 110 displays a display screen 800 displaying a summary, superimposed on the conference screen 700 displayed on the display unit of the terminal device 110. In this case, when a new summary is displayed on the display screen 800, there is a problem that the summary currently displayed on the display screen 800 disappears.

[0089] 8, a display screen 800 has a pin icon 801 for pinning the summary on the screen. By selecting the pin icon 801, the user can keep displaying (pin) a copy 802 of the summary displayed on the display screen 800 on the display unit of the terminal device 110.

[0090] (Decision making process) [First embodiment] Fig. 9 is a diagram showing an example of the judgment process according to the first embodiment. This process shows an example of the judgment process in which the judgment unit 304 of the summary server 100 judges whether or not the converted text converted by the text conversion unit 507 satisfies a predetermined summarization implementation condition in step S515 in Fig. 5.

[0091] In step S901, when the determination unit 304 receives converted text from the provision unit 306, the process proceeds to step S902 and thereafter.

[0092] In step S902, the determination unit 304 calculates the number of characters in the converted text received from the provision unit 306.

[0093] In step S903, the determination unit 304 determines whether the calculated number of received characters has reached a first threshold. Here, the first threshold is a preset number of characters to be summarized (number of characters to be summarized). If the number of received characters has not reached the first threshold, the determination unit 304 returns the process to step S901. On the other hand, if the number of received characters has reached the first threshold, the determination unit 304 transitions the process to step S904.

[0094] In step S904, the determination unit 304 causes the generation unit 305 to perform summarization. For example, the determination unit 304 notifies the generation unit 305 of the converted text group received from the provision unit 306.

[0095] In step S905, determination unit 304 initializes the number of received characters of the converted text.

[0096] In step S906, determination unit 304 determines whether to continue receiving the converted text. If reception of the converted text is to be continued, determination unit 304 returns the process to step S901. On the other hand, if reception of the converted text is not to be continued, determination unit 304 ends the process.

[0097] Fig. 10 is a diagram for explaining the determination process according to the first embodiment. For example, it is assumed that the determination unit 304 sequentially receives converted texts 1 to 8 as shown in Fig. 10 from the provision unit 306. It is also assumed that the first threshold is set to 100 characters.

[0098] 9, when converted text 4 is received, determination unit 304 will find that the number of characters in the converted text will be 100, and so will notify generation unit 305 of converted texts 1 to 4 as converted text group 1001. In this way, generation unit 305 will generate a summary that summarizes converted text group 1001.

[0099] Subsequently, when determining unit 304 receives converted text 7, the number of characters in the converted text becomes 100, and so determines converted texts 5 to 7 as converted text group 1002 and notifies generating unit 305 of converted text group 1002.

[0100] In this way, by setting the number of characters of the converted text as a threshold, it is possible to adjust the amount of conversation for which summarization is to be performed. Note that the first threshold (number of characters to be summarized) may be set by the user.

[0101] [Second embodiment] Fig. 11 is a flowchart showing an example of the judgment process according to the second embodiment. This process shows another example of the judgment process in which the judgment unit 304 of the summary server 100 judges whether or not the converted text converted by the text conversion unit 507 satisfies a predetermined summarization implementation condition in step S515 in Fig. 5.

[0102] In step S1101, when determining unit 304 accepts converted text from providing unit 306, it executes the processes in and after step S1102.

[0103] In step S1102, determination unit 304 generates chunk data by aggregating multiple converted texts received from provision unit 306. Since there are cases where a single converted text alone is meaningless, determination processing is performed by aggregating multiple converted texts as chunk data.

[0104] FIG. 12 is a diagram for explaining the determination process according to the second embodiment. For example, it is assumed that the determination unit 304 sequentially receives converted texts 1 to 8 as shown in FIG. 12 from the provision unit 306. In addition, in the example of FIG. 12, the determination unit 304 treats the three converted texts (an example of a predetermined number) received most recently as one chunk of data, as an example. For example, when the determination unit 304 receives converted text 3, it treats converted texts 1 to 3 as chunk data 1201, and when the determination unit 304 receives converted text 4, it treats converted texts 2 to 4 as chunk data 1202. Similarly, when the determination unit 304 receives converted text 5, it treats converted texts 3 to 5 as chunk data 1203, and when the determination unit 304 receives converted text 6, it treats converted texts 4 to 6 as chunk data 1204.

[0105] In this way, as one example, determination unit 304 may treat a predetermined number of most recently received converted texts as one chunk of data. As another example, when the number of characters in the received converted texts reaches a predetermined number of characters, determination unit 304 may treat the received converted texts as one chunk of data.

[0106] When creating chunk data, the determination unit 304 may simply combine multiple converted texts, or may input multiple converted texts to the generation unit 305 and summarize them.

[0107] In step S1103 in Fig. 11, the generated chunk data is vectorized and a similarity is calculated. For example, the determination unit 304 calculates a similarity between the generated chunk data and chunk data generated previously. For example, in Fig. 12, when the determination unit 304 receives converted text 4 and creates chunk data 1202, it calculates a similarity between chunk data 1202 and chunk data 1201.

[0108] As a method of vectorizing the chunk data, for example, the chunk data is divided into words and vector values ​​are calculated from the divided words. Note that as a method of calculating the vector values, for example, a well-known natural language processing such as Bag of Words or BERT (Bidirectional Encoder Representations from Transformers) can be applied.

[0109] In step S1104 of FIG. 11, the judgment unit 304 judges whether the calculated similarity falls below a second threshold. Here, the second threshold is a value set in advance for judging that the similarity between chunk data is low. If the similarity is not below the second threshold, the judgment unit 304 returns the process to step S1101. On the other hand, if the similarity falls below the second threshold, the judgment unit 304 shifts the process to step S1105.

[0110] When the process proceeds to step S1105, summarization is performed up to the previous chunk data. For example, in Fig. 12, when converted text 4 is received, if the similarity falls below the second threshold, determination unit 304 notifies generation unit 305 that converted texts 1 to 3 are a converted text group. Similarly, when converted text 7 is received, if the similarity falls below the second threshold, determination unit 304 notifies generation unit 305 that converted texts 1 to 6 are a converted text group.

[0111] In step S1106, the decision unit 304 determines that the converted text used for summarization has already been summarized, and excludes it from the targets of summarization.

[0112] In step S1107, determination unit 304 determines whether to continue receiving the converted text. If reception of the converted text is to be continued, determination unit 304 returns the process to step S1101. On the other hand, if reception of the converted text is not to be continued, determination unit 304 ends the process.

[0113] 11, the determination unit 304 simply compares the calculated similarity with the second threshold, but this is just an example. For example, the moving average of the calculated similarity, or a differential value of the calculated similarity, may be compared with the threshold.

[0114] (Change processing) For example, it is desirable that the user be able to edit the summary sentence 612 on the display screen 600 described in Fig. 6. This allows the user to easily correct the summary sentence 612 if the user feels that something is wrong with the summary sentence 612.

[0115] Furthermore, for example, it is desirable that the edited content performed on the display screen 600 displayed by the terminal device 110a, the selected content of the bookmark icon 615, and the like are also reflected on the display screen 600 displayed by the other terminal device 110b.

[0116] Fig. 13 is a sequence diagram showing an example of a change process according to an embodiment. In step S1300, the summarization system 1 executes the summarization process described in Fig. 5, and the terminal devices 110a and 110b display the display screen 600 as shown in Fig. 6.

[0117] In steps S1301 and S1302, when the user edits the summary 612, the terminal device 110b transmits the edited content to the providing unit 306 of the summary server 100. In response, in step S1303, the providing unit 306 stores the edited content received from the terminal device 110b in the DB 307.

[0118] In steps S1304 and S1305, the providing unit 306 distributes the edited results to the terminal devices 110a and 110b, so that in step S1306, the edited contents of the summary in the terminal device 110b are reflected on the display screen 600 of the terminal device 110a.

[0119] Furthermore, in steps S1311 and S13112, when the user selects the bookmark icon 615, the terminal device 110a transmits the bookmark information to the providing unit 306 of the summary server 100. As a result, in step S1313, the providing unit 306 stores in the DB 307 the bookmark information received from the terminal device 110a.

[0120] Furthermore, in steps S1314 and S1315, the providing unit 306 distributes the bookmark results to the terminal devices 110a and 110b. As a result, in step S1316, the selection of the bookmark icon 615 in the terminal device 110a is reflected in the terminal device 110b.

[0121] (Resummarization process 1) For example, it is preferable that the user can edit the converted text 611 on the display screen 600 described in Fig. 6. Furthermore, when the converted text 611 is edited, it is preferable that the summarization system 1 re-summarizes the summary sentence 612.

[0122] 14 is a sequence diagram (1) showing an example of a re-summarization process according to an embodiment of the present invention. This process shows an example of a re-summarization process executed by the summarization system 1.

[0123] In step S1400, it is assumed that summarization system 1 executes the summarization process described in Fig. 5, and terminal devices 110a, 110b display display screen 1500 as shown in Fig. 15(A). In the example of Fig. 15(A), it is assumed that converted texts 1501a, 1501b contain errors (conversion errors, etc.) 1511, 1512, and that due to these errors, summary sentence 1502 also contains errors.

[0124] In steps S1401 and S1402, when the user edits (modifies) converted texts 1501a and 1501b, terminal device 110b transmits the edited content to providing unit 306 of summary server 100. In response, in step S1403, providing unit 306 stores in DB 307 the edited content received from terminal device 110b.

[0125] In steps 1404 and S1405, providing unit 306 distributes the edited results to terminal devices 110a and 110b. As a result, in step S1406, the edited contents of the converted text in terminal device 110b are also reflected on display screen 1500 of terminal device 110a. As a result, for example, as shown in Fig. 15(B), the corrections to converted texts 1501a and 1501b are reflected on display screen 1500 displayed by terminal devices 110a and 110b.

[0126] In step S1407, providing unit 306 notifies determination unit 304 of the changes received from terminal device 110b. In response, in step S1408, determination unit 304 acquires data related to the received edited content from DB 307. For example, determination unit 304 acquires the converted text having the same summary ID as the edited converted text, and the summary sentence.

[0127] In step S1408, the judgment unit 304 checks whether the acquired summary has been manually changed. If the acquired summary has not been manually changed, the summarization system 1 executes the processes of steps S1510 and S1411. On the other hand, if the acquired summary has been manually changed, the summarization system 1 cancels the processes of steps S1410 and S1411. This prevents the summarization system 1 from overwriting a summary that has already been revised by the user.

[0128] In step S1410, the converted text group acquired in step S1408 is notified to generation unit 305. As a result, in step S1411, summarization system 1 regenerates the summary sentence and executes notification processing. For example, summarization system 1 uses the converted text group notified to generation unit 305 by determination unit 304 to reexecute the summarization processing and notification processing similar to steps S522 to S528 in Fig. 5.

[0129] Preferably, while the process of step S1411 is being performed, the terminal devices 110a and 110b display information 1523 indicating that the summary sentence 1502 is being re-summarized on the display screen 1500, as shown in FIG. 15(B).

[0130] 14, for example, the summary sentence 1502 is re-summarized as shown in Fig. 16(A). When the re-summarization is completed, the terminal devices 110a and 110b return the information 1523 indicating that re-summarization is in progress to the original display.

[0131] (Modification) In the example of Fig. 14, if there is no manual change to the summary sentence, the summarization system 1 automatically executes the processes of steps S1410 and S1411. As another example, the summarization system 1 may display an update icon 1621 indicating that the summary sentence 1502 can be re-summarized, as shown in Fig. 16(B), and execute the processes of steps S1410 and S1411 when the update icon 1621 is selected.

[0132] (Resummarization process 2) Fig. 17 is a sequence diagram (2) showing an example of re-summarization processing according to an embodiment. This processing shows another example of re-summarization processing executed by the summarization system 1. Note that the processing of steps S1400 to S1408 in Fig. 17 is similar to the processing of steps S1400 to S1408 described in Fig. 14, and therefore description thereof will be omitted here.

[0133] In step S1701, the determination unit 304 starts a summary timer that measures a first time period.

[0134] In step S1702, even if the converted text corresponding to the same summary ID has been subjected to editing processing of the converted text as shown in steps S1401 to S1408, decision section 304 does not start re-summarization processing.

[0135] In steps S1703 and S1704, when the summarization timer expires, the determination unit 304 notifies the generation unit 305 of the converted text group acquired in step S1408 or later. As a result, in step S1705, the summarization system 1 regenerates the summary sentence and executes the notification process. For example, the summarization system 1 uses the converted text group notified to the generation unit 305 by the determination unit 304 to re-execute the summarization process and notification process similar to steps S522 to S528 in Fig. 5.

[0136] 17, the summarization system 1 updates the summary sentence 1502 after a first time has elapsed since the multiple converted texts 1501a, 1501b corresponding to the summary sentence 1502 were first changed. This makes it possible to prevent the summary sentence 1502 from being frequently re-summarized while, for example, the user is revising the converted texts 1501a, 1501b.

[0137] (Resummarization process 3) Fig. 18 is a sequence diagram (3) showing an example of a re-summarization process according to an embodiment. This process shows another example of the re-summarization process executed by the summarization system 1. Note that the processes of steps S1400 to S1408 in Fig. 17 are similar to the processes of steps S1400 to S1408 described in Fig. 14, and therefore will not be described here.

[0138] In step S1801, the determination unit 304 starts a summary timer that measures a second time period.

[0139] In step S1802, when the converted text corresponding to the same summary ID is subjected to the edit processing of steps S1401 to S1408, decision section 304 executes the processing of steps S1803 and S1804.

[0140] In step S1803, the determination unit 304 stops the summary timer, and in step S1804 starts a summary timer that measures a second time period.

[0141] In steps S1805 and S1806, when the summarization timer expires, the determination unit 304 notifies the generation unit 305 of the converted text group acquired in step S1408 or later. As a result, in step S1807, the summarization system 1 regenerates the summary sentence and executes the notification process. For example, the summarization system 1 uses the converted text group notified to the generation unit 305 by the determination unit 304 to reexecute the summarization process and notification process similar to steps S522 to S528 in Fig. 5.

[0142] 18, the summarization system 1 updates the summary sentence 1502 after a second time has elapsed since the multiple converted texts 1501a, 1501b corresponding to the summary sentence 1502 were last changed. This makes it possible to prevent the summary sentence 1502 from being re-summarized during revision, even if the converted texts 1501a, 1501b are significantly revised by the user, for example.

[0143] 17 and 18, similarly to the process of FIG. 14, if there is a manual change to the summary text, the execution of the re-summarization process may be stopped.

[0144] As described above, according to each embodiment of the present invention, a summarization system that makes it easy to check the most recent summary sentence can be provided. <Supplementary Information> Each function of each embodiment described above can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above.

[0145] Additionally, the devices described in the examples are merely illustrative of one of several computing environments for implementing the embodiments disclosed herein. In one embodiment, the summary server 100 includes a plurality of computing devices, such as a server cluster, configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and to perform the processes disclosed herein.

[0146] Furthermore, as described above, the components of summary server 100 may be integrated into one server device, or may be separated into multiple devices.

[0147] <Additional Notes> This specification discloses a summarization system, a summarization method, and a program as described below. (Section 1) An acquisition unit that acquires voice; A conversion unit that converts the speech into a plurality of texts; a generation unit that generates a summary sentence summarizing the plurality of texts when the plurality of texts satisfy a predetermined summarization execution condition; a providing unit for providing the summary to a user; A summarization system having: (Section 2) 2. The summarization system according to claim 1, wherein the summarization execution condition includes that the size of the plurality of texts converted by the conversion unit reaches a first threshold value. (Section 3) 2. The summarization system according to claim 1, wherein the summarization condition includes that the similarity calculated from the plurality of texts converted by the conversion unit is below a second threshold. (Section 4) a data management unit that manages the summary and the plurality of texts in association with each other; updating the summary if the plurality of texts is changed; 2. The summarization system according to claim 1. (Section 5) the providing unit delivers the summary and the plurality of texts used to generate the summary to a terminal device used by the user; the terminal device has a display control unit that causes a display screen displaying at least the summary sentence to be displayed on a display unit; A summarization system according to any one of claims 1 to 4. (Section 6) 6. The summary system according to claim 5, wherein the display screen displays the plurality of texts used to generate the summary in an editable manner in association with the summary. (Section 7) The display screen includes: 6. The summarization system according to claim 5, wherein the summary text can be extracted, copied, or edited. (Section 8) the audio is audio from a conference in which content data is transmitted and received between a plurality of terminal devices; 6. The summary system according to claim 5, wherein the display control unit displays the display screen by superimposing it on a conference screen of the conference. (Section 9) 5. The summary system of claim 4, further comprising updating the summary sentence after a first time has elapsed since the plurality of texts was first changed or after a second time has elapsed since the plurality of texts was last changed. (Section 10) 5. The summarization system according to claim 4, further comprising: cancelling updating of the summary sentence if the summary sentence has been manually changed. (Section 11) 5. The summary system according to claim 4, wherein the summary text is updated after a predetermined operation by a user is received. (Section 12) A process of acquiring audio; converting the speech into a plurality of texts; A process of generating a summary sentence summarizing the plurality of texts when the plurality of texts satisfy a predetermined summarization execution condition; providing the summary to the user; A computer-implemented method for summarizing. (Section 13) A process of acquiring audio; converting the speech into a plurality of texts; A process of generating a summary sentence summarizing the plurality of texts when the plurality of texts satisfy a predetermined summarization execution condition; providing the summary to the user; A program that causes a computer to execute the following.

[0148] Although an embodiment of the present invention has been described above, the present invention is not limited to such a specific embodiment, and various modifications and applications are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0149] 1. Summary System 100 Summary Server 200 Computers 110, 110a, 110b Terminal equipment 302 Acquisition Department 305 Generation part 306 Provision Department 307 DB (Database) 314 Display control unit 600, 710, 800, 1500 display screen 700 Conference Screens [Prior art documents] [Patent documents]

[0150] [Patent Document 1] Patent Publication No. 2022-025665

Claims

1. An acquisition unit that acquires voice; A conversion unit that converts the speech into a plurality of texts; a generation unit that generates a summary sentence summarizing the plurality of texts when the plurality of texts satisfy a predetermined summarization execution condition; a providing unit for providing the summary to a user; A summarization system having:

2. The summarization system according to claim 1 , wherein the summarization execution condition includes that a size of the plurality of texts converted by the conversion unit reaches a first threshold value.

3. The summarization system according to claim 1 , wherein the summarization execution condition includes a condition that the similarity calculated from the plurality of texts converted by the conversion unit falls below a second threshold value.

4. a database that stores the summary sentence and the plurality of texts in association with each other; updating the summary if the plurality of texts is changed; The summarization system of claim 1 .

5. the providing unit delivers the summary and the plurality of texts used to generate the summary to a terminal device used by the user; the terminal device has a display control unit that causes a display screen displaying at least the summary sentence to be displayed on a display unit; A summarization system according to any one of claims 1 to 4.

6. The summary system according to claim 5 , wherein the display screen displays the plurality of texts used to generate the summary in an editable manner in association with the summary.

7. The display screen includes: The summarization system according to claim 5 , wherein the summarization text can be extracted, copied, or edited.

8. the audio is audio from a conference in which content data is transmitted and received between a plurality of terminal devices; The summary system according to claim 5 , wherein the display control unit displays the display screen so as to be superimposed on a conference screen of the conference.

9. 5. The summarization system of claim 4, further comprising: updating the summary sentence after a first time has elapsed since the plurality of texts was first modified or after a second time has elapsed since the plurality of texts was last modified.

10. The summarization system according to claim 4 , further comprising: a step of canceling the update of the summary sentence if the summary sentence has been manually changed.

11. 5. The summarization system according to claim 4, wherein the summary text is updated after a predetermined operation by a user is accepted.

12. A process of acquiring audio; converting the speech into a plurality of texts; A process of generating a summary sentence summarizing the plurality of texts when the plurality of texts satisfy a predetermined summarization execution condition; providing the summary to a user; A computer-implemented method for summarizing.

13. A process of acquiring audio; converting the speech into a plurality of texts; A process of generating a summary sentence summarizing the plurality of texts when the plurality of texts satisfy a predetermined summarization execution condition; providing the summary to a user; A program that causes a computer to execute the following.

Citation Information

Patent Citations

  • Summary sentence generation device, summary sentence generation method, and program

    JP2022025665A