Speech information display control device, speech information display method and program

Abstract

To appropriately adopt into a conference an idea that is not spoken in a WEB conference or a WEB seminar.SOLUTION: A speech information display control device 80 controls the display of speech information of participants in a plurality of information terminal devices used by a plurality of participants. The speech information display control device 80 comprises: a delimiter extraction section 81 which extracts delimiter information indicating a delimiter between topics from voice data of participants acquired from the plurality of information terminal devices; a conference structure display control section 83 for generating and displaying a conference structure in which the topics and the delimiter information are displayed in a time-sequential manner; a note reception section 84 which presents whether or not it is necessary to insert notes on display screens of the information terminals and receives the input of notes from note input screens displayed on the display screens of the information terminals; a selection section 85 which selects a topic at an insertion destination of a note received by the note reception section 84; and a storage section 89 which stores the note correspondingly to the selected topic.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an utterance information display control device, an utterance information display method, and a program. [Background technology]

[0002] A technology has been disclosed for a remote conference system that detects the amount of speech of each participant and detects topic breaks based on the fluctuation in the amount of speech (see, for example, Patent Document 1).A technology has been disclosed that projects the content of speech of conference participants onto a vector space and identifies their positional relationships to provide appropriate guidance to bring them closer to the purpose of the conference (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2006-081005 A [Patent Document 2] JP 2017-173874 A Summary of the Invention [Problem to be solved by the invention]

[0004] The technologies described in Patent Documents 1 and 2 support the progress of a meeting by structuring and presenting the meeting based on the contents of comments made during the meeting. However, in a meeting or seminar, even if a participant comes up with an idea, they may miss the timing to speak or may not be able to speak at all. It is desirable to appropriately incorporate such ideas that have been missed, in other words, ideas that have not been spoken, into the meeting.

[0005] The present disclosure aims to appropriately incorporate ideas that were not spoken during web conferences, web seminars, and the like. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the objective, the speech information display control device of the present disclosure is an utterance information display control device that controls the display of speech information of multiple participants on multiple information terminal devices used by the multiple participants respectively, and includes: a boundary extraction unit that extracts boundary information indicating boundary between topics from the participants' voice data acquired from the multiple information terminal devices; a meeting structure display control unit that generates and displays a meeting structure in which the topics and the boundary information are displayed in chronological order; a note acceptance unit that indicates whether or not a note needs to be inserted on a display screen of the information terminal device and accepts note input from a note input screen displayed on the display screen of the information terminal device; a selection unit that selects a topic to which the note accepted by the note acceptance unit is to be inserted; and a memory unit that associates the note with the selected topic and stores it, and the meeting structure display control unit displays the note stored in the memory unit at the position of the corresponding topic in the chronological display.

[0007] The speech information display method according to the present disclosure is a speech information display method for controlling the display of speech information of multiple participants on multiple information terminal devices used respectively by the multiple participants, and includes a boundary extraction step for extracting boundary information indicating boundary between topics from the participants' voice data acquired from the multiple information terminal devices, a meeting structure display control step for generating and displaying a meeting structure in which the topics and the boundary information are displayed in chronological order, a note acceptance step for presenting on a display screen of the information terminal device whether or not a note needs to be inserted and accepting note input from a note input screen displayed on the display screen of the information terminal device, a selection step for selecting a topic into which the note accepted by the note acceptance step is to be inserted, and a storage step for associating the note with the selected topic and storing the note in a storage unit, and the meeting structure display control step displays the note stored by the storage step at the position of the corresponding topic in the chronological display.

[0008] The program according to the present disclosure is a program for controlling the display of speech information of multiple participants on multiple information terminal devices used by the multiple participants respectively, and includes a boundary extraction step for extracting boundary information indicating boundary between topics from the voice data of the participants acquired from the multiple information terminal devices, a meeting structure display control step for generating and displaying a meeting structure in which the topics and the boundary information are displayed in chronological order, a note acceptance step for indicating whether or not a note needs to be inserted on a display screen of the information terminal device and accepting note input from a note input screen displayed on the display screen of the information terminal device, a selection step for selecting a topic into which the note accepted by the note acceptance step is to be inserted, and a storage step for associating the note with the selected topic and storing the note in a storage unit, and the meeting structure display control step displays the note stored by the storage step at the position of the corresponding topic in the chronological display. Effect of the Invention

[0009] According to the present disclosure, ideas that were not spoken during a web conference or web seminar can be appropriately incorporated into the conference. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram showing an example of the configuration of an utterance information display system according to the first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of the information terminal device according to the first embodiment. [Diagram 3] FIG. 3 is a block diagram illustrating an example of the configuration of a server device according to the first embodiment. [Figure 4] FIG. 4 is a block diagram showing an example of the configuration of the utterance information display control device according to the first embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of the progress of a Web conference. [Figure 6] FIG. 6 is a diagram showing an example of a chronological display of a meeting structure. [Figure 7] FIG. 7 is a diagram showing another example of a chronological display of a meeting structure. [Figure 8] FIG. 8 is a flowchart showing the utterance information display method according to the first embodiment. [Figure 9] FIG. 9 is a block diagram showing an example of the configuration of the utterance information display control device according to the second embodiment. [Figure 10] FIG. 10 is a flowchart showing the utterance information display method according to the second embodiment. [Figure 11] FIG. 11 is a flowchart showing the utterance information display method according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An utterance information display control device, an utterance information display method, and a program according to the present invention will be described in detail below with reference to the accompanying drawings. Note that the present invention is not limited to the following embodiments.

[0012] [First embodiment] (Speech Information Display System) 1 is a schematic diagram showing a configuration example of an utterance information display system according to a first embodiment. The utterance information display system 1 includes a plurality of information terminal devices 10 and a server device 30. The utterance information display system 1 is used, for example, in a meeting held face-to-face with a plurality of participants (hereinafter referred to as a "face-to-face meeting"), a face-to-face seminar, an online meeting (hereinafter referred to as a "WEB conference"), or a WEB seminar.

[0013] In the following explanation, the case where the device is used for a web conference will be described as an example.

[0014] (Information terminal device) 2 is a block diagram showing an example of the configuration of an information terminal device according to the first embodiment. Each information terminal device 10 starts a WEB conference application to hold a WEB conference or a WEB seminar with other information terminal devices 10 via a server device 30. The information terminal device 10 is, for example, an information terminal device such as a desktop or notebook personal computer, a smartphone, or a tablet device that is used when a participant participates in an online WEB conference or WEB seminar.

[0015] The information terminal device 10 includes a camera 11 , a microphone 12 , an audio output unit 13 , an operation unit 14 , a communication unit 18 , a display unit 19 , and a terminal control device 20 .

[0016] The camera 11 photographs the participants facing the information terminal device 10. The camera 11 photographs the participants using the information terminal device 10. The camera 11 is, for example, an in-camera provided on the display unit 19 side. The camera 11 may be an external camera connected to the information terminal device 10. The camera 11 outputs the photographed image to the photographing control unit 21 of the terminal control device 20.

[0017] The microphone 12 picks up the voices of the participants using the information terminal device 10. The microphone 12 is, for example, a built-in microphone arranged in the information terminal device 10. The microphone 12 may be an external microphone connected to the information terminal device 10. The microphone 12 outputs audio information related to the picked-up voice to the voice acquisition unit 22 of the terminal control device 20.

[0018] The audio output unit 13 is an audio output device that outputs the voices of the participants of the WEB conference or other participants. The audio output unit 13 is, for example, a built-in speaker arranged in the information terminal device 10. The audio output unit 13 may be an external speaker or headphones connected to the information terminal device 10.

[0019] The operation unit 14 is an input device capable of inputting various operations to the information terminal device 10. The operation unit 14 outputs an operation signal corresponding to the received input operation to the terminal control device 20. The operation unit 14 includes, for example, a keyboard, a mouse, a touch panel arranged over the display unit 19, buttons, switches, etc. The operation unit 14 can accept, for example, a key input operation, a click operation, or a touch operation.

[0020] In this embodiment, the operation unit 14 accepts an input operation of memos for a web conference.

[0021] A memo is a record of ideas that participants come up with during a web conference. When a participant comes up with an idea but misses the opportunity to speak or is unable to speak, the idea can be entered as a memo so that it can be brought up and discussed at an appropriate time, such as during or before the end of the conference.

[0022] The communication unit 18 is a communication unit for performing wired communication or wireless communication. The communication unit 18 performs communication by a communication method using, for example, Wi-Fi (registered trademark) or a mobile phone line, and is connected to an intranet or the Internet. The communication of the communication unit 18 is controlled by a communication control unit 26 of the terminal control device 20. The communication unit 18 performs data communication with the server device 30. More specifically, the communication unit 18 transmits to the server device 30 the voice of the participant picked up by the microphone 12, the video and video of the participant captured by the camera 11, and the memo received via the operation unit 14. The communication unit 18 receives the video of the shared screen material and the video and video of the other participants from the server device 30.

[0023] The display unit 19 displays various types of video including characters and images. The display unit 19 is, for example, a display including a liquid crystal display, an organic EL (Electro-Luminescence), etc. The display unit 19 displays a WEB conference screen including the video of the participant and other participants, the shared materials, and a list of notes.

[0024] (Terminal Control Device) The terminal control device 20 controls each part of the information terminal device 10. The terminal control device 20 has, for example, a control device such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), and a storage device such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The terminal control device 20 executes a program that controls the operation of the information terminal device 10 according to the present disclosure. The terminal control device 20 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The terminal control device 20 loads a stored program into a memory and executes instructions included in the program. The terminal control device 20 includes an internal memory such as the above-mentioned RAM, and the internal memory is used for temporary storage of data in the terminal control device 20. The terminal control device 20 may be realized by a combination of hardware and software.

[0025] The terminal control device 20 has functional blocks realized by circuits and executed programs, including a shooting control unit 21, an audio acquisition unit 22, an audio output control unit 23, an operation control unit 24, a memory unit 25, a communication control unit 26, a display control unit 27, and an application control unit 28.

[0026] The photographing control unit 21 controls the camera 11 to shoot an image. The photographing control unit 21 acquires image data of the image shot by the camera 11. More specifically, the photographing control unit 21 acquires image data of the participants using the information terminal device 10, which is shot by the camera 11. The image data acquired by shooting with the camera 11 is video data of a moving image.

[0027] The voice acquiring unit 22 acquires voice information related to the voice detected by the microphone 12. The voice acquiring unit 22 acquires voice information related to the speech of the participant using the information terminal device .

[0028] The audio output control unit 23 controls the output of audio in the audio output unit 13. For example, the audio output control unit 23 controls the audio output unit 13 to output the audio of the participant and other participants.

[0029] The operation control unit 24 acquires an operation signal corresponding to an operation on the operation unit 14 from the operation unit 14. The operation control unit 24 outputs a control signal corresponding to the acquired operation signal to each unit.

[0030] The storage unit 25 stores information such as the contents of calculations and programs of the terminal control device 20. The storage unit 25 is composed of, for example, a RAM, a main storage device such as a ROM, and a storage device such as an SSD (Solid State Drive).

[0031] The communication control unit 26 controls the communication unit 18 to control data communication between the information terminal device 10 and an external device. More specifically, the communication control unit 26 controls, for example, the communication unit 18 to control communication between the information terminal device 10 and the server device 30. The communication control unit 26 controls, for example, the communication unit 18 to transmit and receive information about a WEB conference with another information terminal device 10 to and from the server device 30. More specifically, the communication control unit 26 controls the communication unit 18 to transmit to the server device 30 the voice of the participant picked up by the microphone 12, the video and video of the participant captured by the camera 11, and the memo received via the operation unit 14. The communication control unit 26 controls the communication unit 18 to receive from the server device 30 the video of the shared screen material, the video and video of the other participants, and a list of memos.

[0032] The display control unit 27 controls the display of the image on the display unit 19. For example, the display control unit 27 controls the display unit 19 to display information resulting from the execution of a WEB conference application (hereinafter referred to as a "WEB conference screen").

[0033] The display control unit 27 may generate a WEB conference screen and cause the display control unit 27 to display it on the display unit 19. The display control unit 27 may receive a WEB conference screen generated by the server device 30 as video data and cause the display control unit 27 to display it on the display unit 19.

[0034] The application control unit 28 controls the operation of various applications that run on the information terminal device 10. The application control unit 28 controls the operation of an application used for a WEB conference, such as a WEB conference tool. The application control unit 28 may run an application such as a WEB conference tool in the server device 30. The application control unit 28 may cause the information terminal device 10 and the server device 30 to cooperate with each other to operate an application such as a WEB conference tool in a distributed manner. In other words, the application control unit 28 may be a functional block realized only in the information terminal device 10, a functional block realized only in the server device 30, or a functional block realized by the cooperation of the information terminal device 10 and the server device 30.

[0035] (Server device) 3 is a block diagram showing a configuration example of a server device according to the first embodiment. The server device 30 is, for example, a server device installed by a company that provides a web conference tool. The server device 30 includes a communication unit 31 and a server control device 40.

[0036] The communication unit 31 is a communication unit for performing wired communication or wireless communication. The communication unit 31 performs communication by a communication method using, for example, Wi-Fi or a mobile phone line, and is connected to an intranet or the Internet. The communication of the communication unit 31 is controlled by the communication control unit 41 of the server control device 40. The communication unit 31 performs data communication with the information terminal device 10. More specifically, the communication unit 31 receives the voice of the participant, the video and audio of the participant captured by the camera 11, and the accepted memo from the information terminal device 10. The communication unit 18 transmits the video of the screen-shared material, the video and audio of the other participants, and a list of memos to the information terminal device 10.

[0037] (Server control device) The server control device 40 controls each part of the server device 30. The server control device 40 has, for example, an utterance information display control device such as a CPU or an MPU, and a storage device such as a RAM or a ROM. The server control device 40 may be realized by an integrated circuit such as an ASIC or an FPGA. The server control device 40 may be realized by a combination of hardware and software. The server control device 40 executes a program that controls the operation of the server device 30 according to the present disclosure.

[0038] The server control device 40 includes a communication control unit 41 and a storage unit 42 as functional blocks realized by circuits or programs to be executed.

[0039] The communication control unit 41 controls the communication unit 31 to control data communication between the server device 30 and the information terminal device 10. More specifically, the communication control unit 41 controls, for example, the communication unit 31 to control communication between the server device 30 and the information terminal device 10. The communication control unit 41 controls, for example, the communication unit 31 to receive from the information terminal device 10 the voice of the participant, the video and audio of the participant captured by the camera 11, and the accepted memos. The communication control unit 41 controls, for example, the communication unit 31 to transmit to the information terminal device 10 the video of the shared screen material, the video and audio of the other participants, and a list of memos.

[0040] The storage unit 42 stores information such as the contents of calculations and programs of the communication control unit 41. The storage unit 42 is composed of, for example, a RAM, a main storage device such as a ROM, and a storage device such as an SSD or an HDD (Hard Disk Drive).

[0041] (Speech information display control device) FIG. 4 is a block diagram showing a configuration example of the utterance information display control device according to the first embodiment. The utterance information display control device 80 controls the display of utterance information of participants in a plurality of information terminal devices 10 used by the plurality of participants respectively. The utterance information display control device 80 is a conceptual device realized by one or two devices, the terminal control device 20 and the server control device 40. Either the information terminal device 10 or the server device 30 may be called the utterance information display control device 80, and the utterance information display system 1 composed of the information terminal device 10 and the server device 30 may be called the utterance information display control device 80. Each unit of the utterance information display control device 80 is realized by one or two devices, the terminal control device 20 and the server control device 40.

[0042] The speech information display control device 80 includes a segment extraction unit 81 , a topic information generation unit 82 , a meeting structure display control unit 83 , a memo acceptance unit 84 , a selection unit 85 , and a storage unit 89 .

[0043] The segmentation extraction unit 81 extracts segmentation information indicating a segmentation between topics from the voice data of the multiple participants acquired from the multiple information terminal devices 10. The segmentation extraction unit 81 extracts real-time conversation contents based on the acquired voices of the multiple participants. For example, when performing a voice recognition process on the voice data and recognizing that the speech of all participants has been interrupted for a predetermined time or more, or when detecting a predetermined speech that changes the topic, the segmentation extraction unit 81 extracts information indicating an elapsed time from the start of the conference as segmentation information. A known method can be used to extract a segmentation between topics from the voice data, and is not limited to this method. The space between the segments extracted by the segmentation extraction unit 81 is one topic. The information indicating the segmentation extracted by the segmentation extraction unit 81 is included in the topic segment data. The information indicating the segmentation may include, for example, the time when the segmentation is extracted, or the elapsed time from the start of the conference when the segmentation is extracted.

[0044] The topic information generating unit 82 sets the section up to the break as one topic, and generates text information, which is topic information, from the voices of all participants acquired for one topic. The topic information generating unit 82 sets the break as one topic, and generates text information indicating the gist, minutes, or keywords of the topic from at least one of the voices acquired between the breaks that are one topic and the video of the screen-shared materials, using a known method. Each time a break is extracted by the break extraction unit 81, the topic information generating unit 82 generates text information, which is topic information, from at least one of the voices and the video acquired for one topic. The topic information generating unit 82 associates the generated text information, which is topic information, with the topic in the topic category data.

[0045] The topic information generating unit 82 may generate topic category data linking various data for making it easier to review topics, such as text information indicating summaries of topics and parts of conference voice data.

[0046] The conference structure display control unit 83 generates a conference structure in chronological order in which topics and division information correspond in chronological order as a display showing the conference structure based on the topic division data, and displays the generated conference structure on the WEB conference screen of the display unit 19 of the multiple information terminal devices 10.

[0047] In this embodiment, the function of extracting segmentation information from real-time conversation content using a segmentation extraction unit 81, a topic information generation unit 82, and a meeting structure display control unit 83, generating text information for one topic, and displaying the chronological meeting structure on the WEB conference screen is called the topic classification system.

[0048] Furthermore, the conference structure display control unit 83 displays the memos stored in the storage unit 89 at the position of the corresponding topic in a time series display in which the topics correspond to the division information, and displays them on the WEB conference screen of the display unit 19 of the multiple information terminal devices 10. The conference structure display control unit 83 displays so that it is clear when during the conference the divided topics were dealt with and when during the conference the received memos correspond to the topics that were dealt with.

[0049] When an operation is received in which a memo is selected from the chronological display of the meeting structure displayed on the WEB conference screen, the meeting structure display control unit 83 displays text information indicating the inserted content of the memo and the topic linked to the memo on the WEB conference screen.

[0050] When a discussion is started with a memo selected, the meeting structure display control unit 83 displays the memo in chronological order in order to show that the memo is a continuation of the topic corresponding to the memo.

[0051] The meeting structure display control unit 83 erases the display of the mark image of the memo discussed during the meeting in the chronological display of the meeting structure or changes the icon image, thereby making it clear that the memo was discussed during the meeting.

[0052] The memo receiving unit 84 indicates whether or not a memo is required on the WEB conference screen of the display unit 19 of the information terminal device 10 used by the participant, and receives input of the memo from the information terminal device 10. More specifically, the memo receiving unit 84 displays, for example, a button image or the like that allows the participant to select whether or not to insert a memo on the WEB conference screen displayed on the information terminal device 10 used by the participant. When the button image corresponding to "Required" is selected using the operation unit 14 of the information terminal device 10, the memo receiving unit 84 displays a memo input screen on the WEB conference screen. The memo receiving unit 84 receives input of the memo from the information terminal device 10.

[0053] For example, when a participant comes up with an idea related to the topic of the meeting and wants to discuss the idea later, the participant selects the button image corresponding to "insert required."

[0054] The memo receiving unit 84 may constantly display on the display screen whether or not a memo is required to be inserted. The memo receiving unit 84 may start displaying whether or not a memo is required to be inserted only when a predetermined condition is satisfied.

[0055] The memo receiving unit 84 receives memos input using the operation unit 14 of each information terminal device 10.

[0056] The selection unit 85 selects a topic into which the memo accepted by the memo acceptance unit 84 is to be inserted. The selection unit 85 may select a topic that is highly related based on the text information of the memo, which is the content of the memo, and the topic. The selection of the topic may be manually performed by the participant who input the memo.

[0057] In this embodiment, the selection unit 85 may extract topics generated by the topic information generation unit 82 up until the time the memo was received by the memo receiving unit 84, and select a topic into which the memo is to be inserted.

[0058] In this embodiment, the selection unit 85 may present candidate topics to which the memo is to be inserted, along with a memo input screen, on the display screen of the information terminal device 10 used by the participant accepted by the memo accepting unit 84, allowing the participant to select a topic.

[0059] In this embodiment, the selection unit 85 determines highly relevant topics from the text information of the input note and the text information of topics generated up to the time the note is accepted by the note accepting unit 84, and selects a topic to associate the note with.

[0060] The selection unit 85 includes an insertion content extraction unit 851 and an insertion destination determination unit 852 .

[0061] The insertion content extraction unit 851 extracts text information indicating the insertion content from the memo accepted by the memo acceptance unit 84 .

[0062] The insertion destination determination unit 852 refers to the text information, which is the topic information added to each topic by the topic information generation unit 82, and determines a topic highly related to the text information of the memo as the insertion destination.

[0063] The storage unit 89 stores information such as the calculation contents and programs of the utterance information display control device 80. The storage unit 89 stores the text information of the memo in association with the topic selected by the selection unit 85.

[0064] FIG. 5 is a diagram showing an example of the progress of a WEB conference. In the WEB conference shown in FIG. 5, three topic changes have been made from the start of the conference to the current time before the scheduled end of the conference, and three topics, topic (1), topic (2), and topic (3), have come up. When topic (2) is being discussed, one of the participants inputs a note (1) about what he or she forgot to say about topic (1). When topic (3) is being discussed, another participant inputs an idea he or she came up with about topic (2) as note (2). The selection unit 85 of the speech information display control device 80 selects topic (1) as the topic to which note (1) is to be inserted. The selection unit 85 of the speech information display control device 80 selects topic (2) as the topic to which note (2) is to be inserted.

[0065] FIG. 6 is a diagram showing an example of a chronological display of a meeting structure. FIG. 6 is an example of a chronological display in a list format. The list shows a flag mark image corresponding to a note, the title of a divided topic, a brief summary of the topic, and the time period during which the topic was dealt with. When a note is stored in the storage unit 89, a part close to the text information of the note is cut out from the content dealt with in the topic and associated with the note. As a result, for example, when the cursor is placed on the mark image, the text information of the note is displayed in a speech bubble. In this way, a topic highly related to the note can be more appropriately accessed. When a discussion about the note begins, the time period during which the discussion took place is added to the time of the topic corresponding to the note. Such a chronological display of the meeting structure is displayed on a WEB conference screen.

[0066] If a discussion begins with a memo selected, it will be displayed in the chronological display of the meeting structure so that it is clear that it is a continuation of the topic corresponding to the memo. For example, if memo (2) is selected from the chronological display of the meeting structure shown in Figure 6, the time spent discussing memo (2) will be added to the "time" of topic (2), and after the discussion ends, the content of the topic will be updated to reflect the content of the discussion in memo (2).

[0067] Fig. 7 is a diagram showing another example of a chronological display of a meeting structure. Fig. 7 is an example of a chronological display in a timeline format. In this case, when the cursor is placed on the title of a topic or a mark image, the corresponding content may be displayed. Such a chronological display of a meeting structure may be displayed on a web conference screen.

[0068] (Method of displaying speech information) Fig. 8 is a flowchart showing the speech information display method according to the first embodiment. The process shown in Fig. 8 is started when a WEB conference is started. When the WEB conference is started, display of a WEB conference screen of the WEB conference is started on each information terminal device 10.

[0069] The speech information display control device 80 starts the topic classification system (step S101). More specifically, the speech information display control device 80 extracts, by the boundary extraction unit 81, boundary information indicating boundary between topics from the voice data of the multiple participants acquired from the multiple information terminal devices 10. The speech information display control device 80 sets up to the boundary as one topic by the topic information generation unit 82, and generates text information as topic information from the voice acquired for one topic. The speech information display control device 80 generates, by the conference structure display control unit 83, a conference structure of a time series display in which the topics and boundary information correspond in time series as a display indicating the conference structure based on the topic classification data, and displays the generated structure on the WEB conference screen. Topic classification data including information indicating the boundary and text information as topic information of one topic between the boundaries is generated in real time. The speech information display control device 80 proceeds to step S102.

[0070] The utterance information display control device 80 judges whether or not an operation has been performed on the memo receiving unit 84 (step S102). More specifically, the utterance information display control device 80 presents, by the memo receiving unit 84, a button image or the like that allows the participant to select whether or not to insert a memo on the WEB conference screen of the display unit 19 of the information terminal device 10 used by the participant, and accepts an input of a selection operation of whether or not to insert a memo from the information terminal device 10. When the utterance information display control device 80 judges that a selection operation of the button image corresponding to "insertion required" has been performed on the memo receiving unit 84 (Yes in step S102), the utterance information display control device 80 proceeds to step S103. When the utterance information display control device 80 does not judge that a selection operation of the button image corresponding to "insertion required" has been performed on the memo receiving unit 84 (No in step S102), the utterance information display control device 80 executes the process of step S102 again.

[0071] When it is determined that an operation has been performed on the memo accepting unit 84 (Yes in step S102), the utterance information display control device 80 displays a memo input screen and accepts the input of a memo (step S103). More specifically, the utterance information display control device 80 causes the memo accepting unit 84 to display a memo input screen on the WEB conference screen of the display unit 19 of the information terminal device 10 used by the participant. The utterance information display control device 80 accepts the input of a memo from the information terminal device 10 through the memo accepting unit 84. The utterance information display control device 80 proceeds to step S104.

[0072] The utterance information display control device 80 presents the extracted topic as a candidate topic of insertion (step S104). More specifically, the utterance information display control device 80 causes the insertion content extraction unit 851 to extract text information indicating the insertion content from the memo accepted by the memo acceptance unit 84. The utterance information display control device 80 causes the selection unit 85 to present candidate topics of insertion destinations of the memo together with a memo input screen on the display screen of the information terminal device 10 used by the participant accepted by the memo acceptance unit 84, and allows the participant to select a topic. The utterance information display control device 80 proceeds to step S105.

[0073] The utterance information display control device 80 links the topic selected by the destination determining unit 852 to the memo (step S105). More specifically, the topic highly related to the memo is determined as the destination by referring to the text information, which is the topic information added to each topic by the topic information generating unit 82. The utterance information display control device 80 proceeds to step S106.

[0074] The utterance information display control device 80 associates the memo with the topic and stores them in the storage unit 89 (step S106). More specifically, the utterance information display control device 80 associates the memo with the topic selected by the selection unit 85 and stores them in the storage unit 89. The utterance information display control device 80 proceeds to step S107.

[0075] The utterance information display control device 80 judges whether the conference has ended (step S107). More specifically, the utterance information display control device 80 judges that the conference has ended, for example, when the conference end time has passed or when the WEB conference has ended due to an operation by a participant. If it is judged that the conference has ended (Yes in step S107), the utterance information display control device 80 ends the processing of this flowchart. If it is not judged that the conference has ended (No in step S107), the utterance information display control device 80 executes the processing of step S102 again.

[0076] (effect) As described above, in this embodiment, a chronologically displayed conference structure is generated and displayed on a web conference screen. In this embodiment, accepted notes can be displayed at the position of the corresponding topic in the chronological display. According to this embodiment, notes on ideas that participants have come up with can be displayed in association with the topic of the conference. According to this embodiment, ideas that have not been spoken in a web conference or web seminar can be appropriately incorporated into the conference.

[0077] In this embodiment, a topic with a high relevance is determined from the text information of the inputted memo and the text information of the topic generated up to the time of receiving the memo, and a topic to which the memo is to be associated can be selected without depending on the operation of the participant. According to this embodiment, it is possible to easily associate the memo with the topic.

[0078] In this embodiment, a memo input screen and candidate topics for the memo insertion destination are presented on the display screen of the information terminal device 10 used by the participant to whom the memo has been accepted, allowing the participant to select a topic. According to this embodiment, the participant can appropriately associate the memo with the topic.

[0079] [Second embodiment] An utterance information display control device 80A according to the second embodiment will be described with reference to Figs. 9 and 10. Fig. 9 is a block diagram showing a configuration example of an utterance information display control device according to the second embodiment. Fig. 10 is a flowchart showing an utterance information display method according to the second embodiment. In the following description, components similar to those of the utterance information display control device 80 are given the same or corresponding symbols, and detailed description thereof will be omitted. This is also true for the following embodiments. The utterance information display control device 80A differs from the first embodiment in that it includes a behavior detection unit 86A and in the processing of a memo acceptance unit 84A and a selection unit 85A.

[0080] The behavior detection unit 86A detects a predetermined behavior of the participant. The behavior detection unit 86A performs image processing on the video captured by the camera 11 to detect the behavior of the participant.

[0081] The predetermined behavior is a preset behavior of a participant that triggers the start of accepting notes. The predetermined behavior is, for example, a behavior of stopping speech midway, such as when a short speech is detected after the microphone 12 of the information terminal device 10 used by the participant is turned on. The predetermined behavior may be, for example, an operation of switching the microphone 12 of the information terminal device 10 used by the participant between ON and OFF. The predetermined behavior may be, for example, a predetermined operation performed on the operation unit 14 of the information terminal device 10 used by the participant. The predetermined behavior may be, for example, a biometric authentication function of the information terminal device 10 used by the participant. The predetermined behavior may be, for example, a behavior of the participant, such as raising a hand or tapping a desk a predetermined number of times.

[0082] The memo receiving unit 84A receives the memo by indicating whether or not a memo is required to be inserted on the Web conference screen displayed on the Web conference screen of the information terminal device 10 used by the participant whose behavior has been detected by the behavior detecting unit 86A.

[0083] In this embodiment, the selection unit 85A may select, as the destination, the topic at the timing when the behavior is detected by the behavior detection unit 86A.

[0084] Next, the speech information display method will be described with reference to Fig. 10. The processes in steps S111, S114 to S117 in the flowchart shown in Fig. 10 are similar to those in steps S101, S104 to S107 in the flowchart shown in Fig. 8.

[0085] The utterance information display control device 80A determines whether or not the participant exhibited a specific behavior (step S112). More specifically, the utterance information display control device 80A detects the participant's behavior by performing image processing on the video captured by the camera 11 using the behavior detection unit 86A. If the utterance information display control device 80A determines that the participant exhibited a specific behavior (Yes in step S112), it proceeds to step S113. If the utterance information display control device 80A does not determine that the participant exhibited a specific behavior (No in step S112), it executes the process of step S112 again.

[0086] When it is determined that the participant has exhibited a specific behavior (Yes in step S112), the utterance information display control device 80A displays a memo input screen and accepts the input of the memo (step S113). More specifically, the utterance information display control device 80A displays a memo input screen on the WEB conference screen by the memo accepting unit 84. The utterance information display control device 80A accepts the input of the memo from the information terminal device 10 by the memo accepting unit 84. The utterance information display control device 80A proceeds to step S114.

[0087] (effect) As described above, in this embodiment, when a predetermined behavior is detected, the display screen of the information terminal device 10 used by the participant is displayed indicating whether or not a memo is required, and the participant can accept the memo.

[0088] In this embodiment, the topic at the timing when the behavior is detected can be selected as the destination. According to this embodiment, the memo and the topic can be easily associated with each other.

[0089] [Third embodiment] An utterance information display control device 80 according to the third embodiment will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an utterance information display method according to the third embodiment. A conference structure display control unit 83 is different from that in the first embodiment.

[0090] When the break information is extracted by the break extraction unit 81 and a new memo is stored in the storage unit 89, the conference structure display control unit 83 adds a display indicating the presence of a memo to the conference structure display. In more detail, when the break extraction unit 81 extracts a break between topics, the conference structure display control unit 83 first checks in the storage unit 89 whether or not there is a memo saved during the previous conference. Then, when there is a newly saved memo in the storage unit 89, the conference structure display control unit 83 displays the memo at the position of the corresponding topic in the chronological display of the conference structure, as shown in FIG. 6 or FIG. 7, and causes the display unit 19 of the multiple information terminal devices 10 to display the memo on the WEB conference screen.

[0091] Next, a method for displaying speech information will be described with reference to Fig. 11. The processes in steps S131 and S138 in the flowchart shown in Fig. 11 are similar to those in steps S101 and S107 in the flowchart shown in Fig. 8.

[0092] The utterance information display control device 80 judges whether the topic has changed (step S132). More specifically, the utterance information display control device 80 judges that the topic has changed when the boundary extraction unit 81 extracts boundary information. When the utterance information display control device 80 judges that the topic has changed (Yes in step S132), it proceeds to step S133. When the utterance information display control device 80 does not judge that the topic has changed (No in step S132), it executes the process of step S132 again.

[0093] The utterance information display control device 80 determines whether or not there is a new note (step S133). More specifically, the utterance information display control device 80 determines whether or not a new note is stored in the storage unit 89. When the utterance information display control device 80 determines that there is a new note (Yes in step S133), it proceeds to step S134. When the utterance information display control device 80 does not determine that there is a new note (No in step S133), it executes the process of step S132 again.

[0094] When it is determined that there is a new memo (Yes in step S133), the utterance information display control device 80 presents a topic list (step S134). More specifically, the utterance information display control device 80 causes the conference structure display control unit 83 to display the memo at the position of the corresponding topic in the chronological display of the conference structure as a topic list, and causes the display unit 19 of the multiple information terminal devices 10 to display the memo on the WEB conference screen. The utterance information display control device 80 proceeds to step S135.

[0095] The utterance information display control device 80 determines whether any of the notes has been selected (step S135). More specifically, the utterance information display control device 80 determines whether any of the notes has been selected from the chronological display of the meeting structure. If the utterance information display control device 80 determines that any of the notes has been selected (Yes in step S135), it proceeds to step S136. If the utterance information display control device 80 does not determine that any of the notes has been selected (No in step S135), it executes the process of step S132 again.

[0096] When it is determined that any of the notes has been selected (Yes in step S135), the utterance information display control device 80 presents information linked to the selected note (step S136). More specifically, the utterance information display control device 80 causes the meeting structure display control unit 83 to display on the screen text information indicating the inserted contents of the note and the topic linked to the note. The utterance information display control device 80 proceeds to step S137.

[0097] The utterance information display control device 80 reflects the fact that the memo is mentioned in the display system (step S138). More specifically, the utterance information display control device 80 causes the meeting structure display control unit 83 to display, for example, a chronological display of the meeting structure so that it is clear that the topic is a continuation of the topic corresponding to the memo. The utterance information display control device 80 causes the meeting structure display control unit 83 to, for example, erase the display of the mark image in the chronological display of the meeting structure or to change and display the icon image. The utterance information display control device 80 proceeds to step S138.

[0098] (effect) As described above, in this embodiment, when topic division information is extracted, if new notes are stored in the storage unit 89, a display indicating the presence of notes can be added to the meeting structure display. According to this embodiment, when a topic division is reached, it is possible to confirm whether or not to discuss the new notes.

[0099] Each component of the illustrated speech information display system is a functional concept, and does not necessarily have to be physically configured as illustrated. In other words, the specific form of each device is not limited to that shown in the figure, and all or part of the device may be functionally or physically distributed or integrated in any unit depending on the processing load and usage status of each device.

[0100] The configuration of each device of the speech information display system is realized, for example, as software, by a program loaded into a memory. In the above embodiment, the functional blocks are described as being realized by the cooperation of these hardware and software. In other words, these functional blocks can be realized in various forms by only hardware, only software, or a combination of both.

[0101] The components described above include those that a person skilled in the art can easily imagine and those that are substantially the same. Furthermore, the configurations described above can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the configurations are possible without departing from the scope of the present invention.

[0102] The above-mentioned speech information display control device may further include a topic comparison unit. The topic comparison unit compares the conversation content being handled in real time with text information, which is the content of the memo stored in the storage unit 89, and acquires a memo with a content similar to the real-time topic. The topic comparison unit collates the real-time conversation content extracted by the segment extraction unit 81 with the text information of the memo stored in the storage unit 89. For example, if the content similar to the content of the memo (2) is handled in the topic (3) of FIG. 6, the content of the memo (2) is popped up by the meeting structure display control unit 83 even during the discussion, so that the content of the memo (2) can be handled seamlessly in the discussion. If the text information of the memo is not picked up in the middle of the topic (3), a display is made that shows that the content of the memo (2) was similar to the topic (3), or a display is made with priority over other notes. As a result, the text information of the memo can be handled in a manner that follows the flow of the conversation during the meeting. [Explanation of symbols]

[0103] 1. Speech information display system 10 Information terminal equipment 11 Camera 12 Microphones 13 Audio output section 14 Control section 18 Communications Department 19 Display section 20 Terminal control device 21 Shooting control section 22 Voice Acquisition Unit 23 Audio output control section 24 Operation control section 25 Memory section 26 Communication control section 27 Display control section 28 Application control section 30 Server device 31 Communications Department 41 Communication control section 42 Storage section 80 Speech information display control device 81 Break extraction section 82 Topic information generation section 83 Meeting structure display control section 84 Memo Reception Section 85 Selection section 851 Insertion Content Extraction Unit 852 Insertion destination determination unit 89 Memory section

Claims

1. A speech information display control device that controls display of speech information of a plurality of participants in a plurality of information terminal devices respectively used by the plurality of participants, a segmentation extraction unit that extracts segmentation information indicating a segmentation between topics from the voice data of the participants acquired from the plurality of information terminal devices; a meeting structure display control unit that generates and displays a meeting structure in which the topics and the division information are displayed in chronological order; a memo receiving unit that indicates whether or not a memo needs to be inserted on a display screen of the information terminal device and receives input of a memo from a memo input screen displayed on the display screen of the information terminal device; a selection unit for selecting a topic into which the memo accepted by the memo acceptance unit is to be inserted; a storage unit for storing a memo in association with the selected topic; Equipped with the meeting structure display control unit causes the memo stored in the storage unit to be displayed at a position of the corresponding topic in the chronological display; Speech information display control device.

2. a topic information generating unit that sets a portion up to the boundary as one topic and generates text information from the speech acquired on one topic; Further equipped with the selection unit determines a topic having a high relevance from text information of the input memo and text information of topics generated up to the time of reception of the memo by the memo reception unit, and selects a topic to which the memo is to be associated. The speech information display control device according to claim 1 .

3. a behavior detection unit that detects a predetermined behavior of a participant; Further equipped with the memo receiving unit receives a request for inserting the memo by presenting the request on a display screen of the information terminal device used by the participant whose behavior has been detected; The speech information display control device according to claim 1 .

4. A speech information display method for controlling display of speech information of a plurality of participants in a plurality of information terminal devices respectively used by the plurality of participants, comprising: a segmentation extraction step of extracting segmentation information indicating a segmentation between topics from the voice data of the participants acquired from the plurality of information terminal devices; a meeting structure display control step of generating and displaying a meeting structure in which the topics and the division information are displayed in chronological order; a memo receiving step of indicating whether or not a memo needs to be inserted on a display screen of the information terminal device and receiving input of a memo from a memo input screen displayed on the display screen of the information terminal device; a selection step of selecting a topic into which the memo accepted by the memo acceptance step is to be inserted; a storage step of storing a memo in a storage unit in association with the selected topic; Including, The meeting structure display control step displays the memo stored in the storage step at a position of the corresponding topic in the chronological display. A method for displaying speech information.

5. A program for controlling display of speech information of a plurality of participants in a plurality of information terminal devices used by the plurality of participants, the program comprising: a segmentation extraction step of extracting segmentation information indicating a segmentation between topics from the voice data of the participants acquired from the plurality of information terminal devices; a meeting structure display control step of generating and displaying a meeting structure in which the topics and the division information are displayed in chronological order; a memo receiving step of indicating whether or not a memo needs to be inserted on a display screen of the information terminal device and receiving input of a memo from a memo input screen displayed on the display screen of the information terminal device; a selection step of selecting a topic into which the memo accepted by the memo acceptance step is to be inserted; a storage step of storing a memo in a storage unit in association with the selected topic; Including, The meeting structure display control step displays the memo stored in the storage step at a position of the corresponding topic in the chronological display. A program executed by a speech information display control device.

Images