Information display control device and information display method

Abstract

To display reference information appropriately from keywords based on utterance content.SOLUTION: There is provided an information display control device 20 that controls an information display device 10 to display information on a display part 12, which is provided with: an utterance recognition unit 21 for recognizing an utterance of a speaker; an operation control unit 32 for detecting an operation for a display screen displayed on the display part 12; an extraction unit 26 for extracting keywords requiring reference information from the utterance recognized by the utterance recognition unit 21; an information acquisition unit 27 for acquiring reference information for the keywords extracted by the extraction unit 26; and a display control unit 28 for displaying the reference information acquired by the information acquisition unit 27 on the display part 12. The display control unit 28, when it is detected that a display range related to the utterance displayed on the display part 12 is operated by the operation control unit 32, displays on the display part 12 the reference information for the keywords requiring the reference information extracted immediately before the time point of the detection by the extraction unit 26.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information display control device and an information display method. [Background technology]

[0002] 2. Description of the Related Art There is known a technique for displaying additional information about a phrase spoken by a person in a meeting or conversation (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-007003 Summary of the Invention [Problem to be solved by the invention]

[0004] The additional information displayed is information that explains abbreviations and difficult keywords. However, depending on the level of understanding of the party receiving the additional information, an explanation may be required for keywords for which no additional information is displayed. In this case, there is a risk that the recipient's understanding may not progress. Or, there may be cases where an explanation is not required for keywords for which additional information is displayed. In this case, the additional information is displayed even for keywords for which additional information is not required by the recipient, which may disrupt the flow of communication.

[0005] The present disclosure has been made in consideration of the above, and aims to appropriately display reference information from keywords based on the content of an utterance. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the objective, the information display control device of the present disclosure is an information display control device that controls an information display device to display information on a display unit, and includes an utterance recognition unit that recognizes a speaker's utterance, an operation control unit that detects operations on a display screen displayed on the display unit, an extraction unit that extracts keywords that require reference information from the utterance recognized by the utterance recognition unit, an information acquisition unit that acquires the reference information for the keywords extracted by the extraction unit, and a display control unit that displays the reference information acquired by the information acquisition unit on the display unit, and when the operation control unit detects that the display range related to the utterance displayed on the display unit has been operated, the display control unit displays on the display unit reference information for the keywords that require reference information extracted by the extraction unit immediately before the time of detection.

[0007] The information display method of the present disclosure is an information display method executed by an information display device that displays information on a display unit, and includes a speech recognition step of recognizing a speaker's utterance, an operation control step of detecting an operation on a display screen displayed on the display unit, an extraction step of extracting keywords that require reference information from the utterance recognized in the speech recognition step, an information acquisition step of acquiring reference information for the keywords extracted in the extraction step, and a display control step of displaying the reference information acquired in the information acquisition step on the display unit, wherein when it is detected by the operation control step that the display range related to the utterance displayed on the display unit has been operated, the display control step executes displaying on the display unit reference information for the keywords that require reference information extracted in the extraction step immediately before the time of detection. Effect of the Invention

[0008] According to the present disclosure, it is possible to provide an effect of appropriately displaying reference information based on keywords based on the content of an utterance. [Brief description of the drawings]

[0009] [Figure 1]FIG. 1 is a block diagram showing an example of the configuration of an information display device having an information display control device according to the first embodiment. [Diagram 2] FIG. 2 is a schematic diagram showing an example of a usage example of the information display device. [Diagram 3] FIG. 3 is a schematic diagram showing another example of use of the information display device. [Figure 4] FIG. 4 is a flowchart showing an example of a processing flow in the information display control device according to the first embodiment. [Diagram 5] FIG. 5 is a diagram showing an example in which a speaker display range is selected. [Figure 6] FIG. 6 is a flowchart showing an example of a process flow in the information display control device according to the second embodiment. [Figure 7] FIG. 7 is a diagram showing an example in which the speaker display range is divided into a plurality of sections. [Figure 8] FIG. 8 is a diagram showing an example in which the reference information display range is divided into a plurality of sections. [Figure 9] FIG. 9 is a schematic diagram for explaining an example of keywords extracted immediately before the operation time. [Figure 10] FIG. 10 is a flowchart showing an example of a process flow in the information display control device according to the third embodiment. [Figure 11] FIG. 11 is a flowchart showing an example of a process flow in the information display control device according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of an information display control device and an information display method according to the present disclosure will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the following embodiment.

[0011] [First embodiment] <Information display device> 1 is a block diagram showing a configuration example of an information display device 10 having an information display control device (hereinafter referred to as "control device") 20 according to a first embodiment. The information display device 10 displays reference information of keywords included in utterances of participants in at least speech-based communication such as a meeting or conversation.

[0012] The information display device 10 is used in a conference or conversation that is held online with the participation of multiple participants (hereinafter referred to as a "WEB conference"). The information display device 10 used in this case may be implemented as one of the functions of an information terminal device such as a notebook personal computer or smartphone that is used when participating in an online WEB conference or conversation via a WEB conference system (not shown).

[0013] In the following explanation, the case where the device is used for a web conference will be described as an example.

[0014] In a conference, the information display device 10 displays reference information for keywords based on the contents of speeches of participants. In this embodiment, when the information display device 10 detects that a display range related to speech displayed on the display unit 12 has been operated in a conference, the information display device 10 displays reference information for the keyword extracted immediately before the detected operation.

[0015] "Immediately before the time of operation" refers to a period going back a predetermined period from the time of operation, such as within 5 seconds from the time of operation.

[0016] 1, the information display device 10 includes a microphone 11, a display unit 12, an operation unit 17, a keyword information storage unit 18, a communication unit 19, and a control device 20. The information display device 10 used for a web conference further includes a camera (not shown).

[0017] The microphone 11 is a microphone that picks up the voices of participants in a conference using the information display device 10. The microphone 11 outputs the picked-up voice data to the speech recognition unit 21 of the control device 20. The microphone 11 picks up the voices of users who use the information display device 10 as participants.

[0018] Fig. 2 is a schematic diagram showing an example of a usage example of the information display device. Fig. 3 is a schematic diagram showing another usage example of the information display device. The display unit 12 is a display including, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display. The display unit 12 displays an image based on a video signal output from a display control unit 28 of the control device 20.

[0019] The display unit 12 displays reference information corresponding to keywords included in the speech of the conference participants. The range in which the reference information is displayed on the display unit 12 is referred to as a reference information display range 110. The reference information display range 110 may be displayed in a part of the display unit 12, for example, the lower part, upper part, left part, or right part. The reference information display range 110 may be displayed at all times even when the reference information is not displayed. The reference information display range 110 may be displayed as a pop-up on the display unit 12. The reference information display range 110 is, for example, a range in which about three lines or less of characters are displayed.

[0020] The display unit 12 displays the participants' videos and the shared materials on the screen.

[0021] The operation unit 17 is an input device capable of inputting various operations to the information display device 10. The operation unit 17 is, for example, a keyboard, a mouse, a touch panel arranged over the display unit 12, or the like. The operation unit 17 can accept, for example, a key input operation, a click operation, or a touch operation. The operation unit 17 can accept an operation within a display range related to an utterance displayed on the display unit 12. The operation unit 17 outputs operation information indicating the accepted operation to the operation control unit 32.

[0022] The display range relating to an utterance is, for example, a speaker display range 120 that is a display range that displays a speaker, or a reference information display range 110 that displays reference information.

[0023] 2, the operation unit 17 can accept operations at any position in the speaker display range 130-120 displayed on the display unit 12. In this embodiment, the operation unit 17 is not limited to the range in the speaker display range 120 where the speaker's video is displayed, but can accept operations in the range where any participant's video is displayed. The speaker display range 120 excludes the positions of operation icons for other commands.

[0024] As shown in FIG. 3, the operation unit 17 can accept an operation at any position in a reference information display area 110 displayed on the display unit 12.

[0025] The keyword information storage unit 18 stores reference information for each keyword as keyword information. The keyword information storage unit 18 is, for example, a recording medium such as a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory. The keyword information storage unit 18 may be an external recording unit wirelessly connected via the communication unit 19.

[0026] A keyword is a word that requires additional explanation, such as an unknown phrase, technical term, abbreviation, difficult word, or buzzword.

[0027] The reference information may be, for example, the meaning of the keyword or a specific explanation. If the keyword is an abbreviation, the reference information may be the full name without abbreviation. The explanation may be, for example, a photograph or a diagram other than text. The reference information may be, for example, additional information that is additional information to the keyword.

[0028] The reference information may be, for example, about 100 characters or less, which can be read by a person in a few seconds.

[0029] When the communication unit 19 searches for reference information on the Web, for example, information from a specific Web media such as Wikipedia (registered trademark) for the keyword, the latest news on the keyword, etc. are searched. Also, when the keyword is a product name or service name, the Web site of the provider of the product or service is searched. Furthermore, the contents displayed at the top of the results excluding advertisements in the results searched by a Web search engine may be used as reference information.

[0030] The communication unit 19 is a communication unit for performing wireless communication and the like. The communication unit 19 performs communication by a communication method using, for example, Wi-Fi (registered trademark) or a mobile phone line, and is connected to an intranet or the Internet. The communication of the communication unit 19 is controlled by a communication control unit 29 of the control device 20. The communication unit 19 exchanges voices and images of the participants with the information display devices 10 used by the other participants, and searches for reference information. More specifically, the communication unit 19 transmits the voices of the participants collected by the microphone 11 and the images of the participants captured by the camera to the information display devices 10 used by the other participants. The communication unit 19 receives the voices and images of the other participants from the information display devices 10 used by the other participants.

[0031] In the following description, the input voice refers to the voice picked up by the microphone 11 and the voice of other participants obtained via the communication unit 19 from the information display device 10 used by the other participants.

[0032] <Information display control device> The control device 20 controls each part of the information display device 10. The control device 20 controls the information display device 10 to display information on the display unit 12. The control device 20 is an arithmetic processing device (control device) composed of, for example, a CPU (Central Processing Unit) and an MPU (Micro Processing Unit), and has a storage device such as a RAM or a ROM (Read Only Memory). The control device 20 loads a stored program into the memory and executes instructions included in the program. The control device 20 includes an internal memory such as the above-mentioned RAM, and the internal memory is used for temporary storage of data in the control device 20. The control device 20 includes, as components such as functional blocks realized by executing the program, an utterance recognition unit 21, an operation control unit 32, an extraction unit 26, an information acquisition unit 27, a display control unit 28, and a communication control unit 29.

[0033] The speech recognition unit 21 recognizes at least the speech of participants who are participating in the speech communication. The speech recognition unit 21 analyzes the input voice and recognizes the speech content contained in the voice. The speech recognition unit 21 performs an acoustic model analysis for each phoneme or each word on the input voice and recognizes the speech content by comparing it with a phoneme model or a language model.

[0034] The operation control unit 32 acquires operation information of an operation on the operation unit 17. More specifically, the operation control unit 32 acquires operation information of an operation on the operation unit 17 in a display range related to an utterance displayed on the display unit 12. The operation control unit 32 accepts, for example, a key input operation, a click operation, or a touch operation on the speaker display range 120 or the reference information display range 110.

[0035] The extraction unit 26 extracts keywords requiring reference information from the utterance recognized by the utterance recognition unit 21.

[0036] The information acquiring unit 27 acquires reference information for the keywords extracted by the extracting unit 26, that is, for the keywords that the extracting unit 26 has determined require reference information. In this embodiment, the information acquiring unit 27 acquires reference information for the keywords based on information on the web.

[0037] The display control unit 28 controls the display of the image on the display unit 12. The display control unit 28 outputs to the display unit 12 a video signal of the image.

[0038] The display control unit 28 controls the display of the reference information acquired by the information acquisition unit 27 on the display unit 12. The display control unit 28 outputs a video signal for displaying the reference information corresponding to the keyword extracted by the extraction unit 26 in the reference information display range 110 of the display unit 12. The display control unit 28 displays the reference information for a predetermined period of time, for example, about 5 to 10 seconds. When displaying the next reference information while displaying the reference information, the display control unit 28 may change the display to the next reference information even before the predetermined period has elapsed.

[0039] The display control unit 28 controls the display of the video of the WEB conference. The display control unit 28 outputs a video signal of the video of the WEB conference including the video captured by the camera.

[0040] When the display control unit 28 detects that the display range related to the utterance displayed on the display unit 12 has been operated by the operation control unit 32, it causes the display unit 12 to display reference information for a keyword requiring reference information extracted by the extraction unit 26 immediately before the time of detection.

[0041] 2, when the display control unit 28 receives an operation on the speaker display range 130 from the operation control unit 32, the display control unit 28 displays reference information for the keyword extracted immediately before the operation is detected in a pop-up display. The display control unit 28 may change the display position of the pop-up display according to the position where the operation is detected on the display screen. For example, the display control unit 28 may set the position where the operation is detected to be the center, bottom end, top end, left end, or right end of the pop-up display.

[0042] As shown in FIG. 3, when the display control unit 28 receives an operation on the reference information display range 110 from the operation control unit 32, the display control unit 28 causes reference information for the keyword extracted immediately before the operation was detected to be displayed in the reference information display range 110.

[0043] The communication control unit 29 controls communication by the communication unit 19. The communication control unit 29 controls the communication between the information display device 10 used by the other participants and the information display device 10 used by the other participants to exchange voices and images of the participants, and to search for reference information, via the communication unit 19. More specifically, the communication control unit 29 controls the communication to transmit the voices of the participants picked up by the microphone 11 and the images of the participants captured by the camera to the information display device 10 used by the other participants, via the communication unit 19. The communication control unit 29 controls the communication to receive the voices and images of the other participants from the information display device 10 used by the other participants, via the communication unit 19.

[0044] <Information processing in information display control device> Next, the flow of processing in the control device 20 will be described with reference to FIG. 4. FIG. 4 is a flowchart showing an example of the flow of processing in the information display control device according to the first embodiment. The processing shown in FIG. 4 is started by starting an application for conducting a WEB conference or starting a WEB conference. In this case, the application for conducting a WEB conference may include a function for displaying reference information, and the function for displaying reference information may be selectable to be turned on or off by a user operation. The processing of the flowchart shown in FIG. 4 is started by starting the information display device 10.

[0045] The control device 20 starts speech recognition (step S101). More specifically, the control device 20 analyzes the input voice and recognizes the contents of the voice by the speech recognition unit 21. The control device 20 proceeds to step S102.

[0046] The control device 20 determines whether the display range related to the utterance has been operated (step S102). The control device 20 determines whether the operation control unit 32 has accepted an operation on the operation unit 17 in the display range related to the utterance displayed on the display unit 12. The control device 20 determines whether the operation control unit 32 has accepted, for example, a key input operation, a click operation, or a touch operation in the speaker display range 130 or the reference information display range 110. If the control device 20 determines that the display range related to the utterance has been operated (Yes in step S102), the control device 20 proceeds to step S103. If the control device 20 does not determine that the display range related to the utterance has been operated (No in step S102), the control device 20 proceeds to step S105.

[0047] When it is determined that the display range related to the utterance has been operated (Yes in step S102), the control device 20 determines whether or not a keyword requiring reference information has been extracted immediately before (step S103). More specifically, the control device 20 determines whether or not a keyword requiring reference information has been extracted by the extraction unit 26 from the utterance recognized by the utterance recognition unit 21 at the time of the operation of the display range related to the utterance. When the control device 20 determines that a keyword requiring reference information has been extracted immediately before (Yes in step S103), the control device 20 proceeds to step S104. When the control device 20 does not determine that a keyword requiring reference information has been extracted immediately before (No in step S103), the control device 20 proceeds to step S105.

[0048] When it is determined that a keyword has been extracted immediately before (Yes in step S103), the control device 20 acquires reference information for the keyword and displays it for a predetermined period (step S104). More specifically, the control device 20 acquires reference information for the immediately before extracted keyword by the information acquisition unit 27. The control device 20 outputs, for a predetermined period, a video signal of the reference information corresponding to the keyword extracted by the extraction unit 26 by the display control unit 28. The control device 20 proceeds to step S105.

[0049] The control device 20 judges whether or not to end the process (step S105). For example, it judges to end the process when the power of the information display device 10 is turned off, when an operation unit (not shown) is operated, or when an operation is performed to end an application that triggered the start of the flowchart shown in FIG. 4. If it is judged to end the process (Yes in step S105), the control device 20 ends this process. If it is not judged to end the process (No in step S105), the control device 20 executes the process of step S102 again.

[0050] For example, a case will be described in which a user of the information display device 10, who is a participant of a WEB conference, checks reference information for an unknown phrase (keyword) or the like for a statement made by another participant during the WEB conference. When the user of the information display device 10 wants to check reference information for an unknown keyword for a statement made by another participant, the user operates the speaker display range 130 or the reference information display range 110 displayed on the display unit 12. In step S102, the control device 20 detects an operation for the speaker display range 130 or the reference information display range 110 displayed on the display unit 12. When the control device 20 determines in step S103 that a keyword requiring reference information has been extracted from an utterance recognized at the time of the operation, the control device 20 causes the display unit 12 to display reference information for the immediately preceding keyword uttered within a predetermined retroactive period t1, for example, 5 seconds, from the time of the operation in step S104. The user of the information display device 10 checks the reference information displayed on the display unit 12.

[0051] <Effects> As described above, in this embodiment, when it is detected that the display range related to the utterance has been operated, the display unit 12 displays reference information for the keyword that requires reference information extracted by the extraction unit 26 immediately before the detection time. According to this embodiment, the user's operation on the speaker display range 130 or the reference information display range 110 can be used as a trigger to appropriately display the reference information for the keyword. This embodiment can appropriately support the user's understanding. This embodiment can prevent reference information for unnecessary keywords from being displayed to the user. This embodiment can support smooth flow of communication. In this way, this embodiment can appropriately display reference information from keywords based on the utterance content.

[0052] [Second embodiment] The information display device 10 according to this embodiment will be described with reference to FIG. 5 and FIG. 6. FIG. 5 is a diagram showing an example in which a speaker display range is selected. FIG. 6 is a flowchart showing an example of a process flow in an information display control device according to a second embodiment. The information display device 10 has a basic configuration similar to that of the information display device 10 of the first embodiment. In the following description, the same or corresponding reference symbols are used for components similar to those of the information display device 10, and detailed descriptions thereof will be omitted. In this embodiment, when it is detected that the display range related to an utterance displayed on the display unit 12 has been operated in a conference, the information display device 10 displays reference information for a keyword extracted immediately before the operation, corresponding to the operated range. In this embodiment, the processes in the operation unit 17, the utterance recognition unit 21, the operation control unit 32, and the display control unit 28 are different from those in the first embodiment.

[0053] As shown in Fig. 5, the operation unit 17 can accept operations for the speaker display ranges 130 in which the videos of the speakers are displayed on the display unit 12. In the example shown in Fig. 5, the operation unit 17 can accept operations for six speaker display ranges 130 in which the videos of the participants A to F are displayed, respectively.

[0054] For each speaker display range 130, a participant who is a speaker is associated with the speaker and stored in advance in a storage unit (not shown).

[0055] The speech recognition unit 21 identifies the speaker and recognizes the speech. For example, the speech recognition unit 21 may identify the speaker from participant information acquired together with the voices of the other participants from the information display device 10 used by the other participants via the communication unit 19.

[0056] The participant information is, for example, information for identifying a participant or information for identifying the information display device 10.

[0057] The operation control unit 32 acquires operation information of an operation on the operation unit 17 in the speaker display area 130 displayed on the display unit 12. The operation control unit 32 is capable of detecting and accepting the operated speaker.

[0058] When the operation control unit 32 detects that the display range related to the utterance displayed on the display unit 12 has been operated, the display control unit 28 causes the display unit 12 to display reference information for a keyword requiring reference information extracted by the extraction unit 26 immediately before the operation, corresponding to the operated range. More specifically, when the operation control unit 32 detects that the speaker display range 130 displayed on the display unit 12 has been operated, the display control unit 28 causes the display unit 12 to display reference information for a keyword requiring reference information extracted by the extraction unit 26 immediately before the operation, which is an utterance of the speaker corresponding to the operated speaker display range 130.

[0059] For example, the display control unit 28 may change the display position of the pop-up display according to the operated speaker display range 130. For example, the display control unit 28 may set the operated position in the operated speaker display range 130 to the center, bottom end, top end, left end, or right end of the pop-up display.

[0060] Next, the flow of processing in the control device 20 will be described with reference to Fig. 6. Processing in steps S111, S114, and S115 shown in Fig. 6 is similar to processing in steps S101, S104, and S105 in the flowchart shown in Fig. 4.

[0061] The control device 20 determines whether the speaker display area 130 has been operated (step S112). The control device 20 determines whether the operation control unit 32 has accepted an operation on the operation unit 17 in the speaker display area 130 displayed on the display unit 12. The control device 20 determines whether the operation control unit 32 has accepted a key input operation, a click operation, or a touch operation on any of the speaker display areas 130 set corresponding to multiple speakers. If the control device 20 determines that the speaker display area 130 has been operated (Yes in step S112), the control device 20 proceeds to step S113. If the control device 20 does not determine that the speaker display area 130 has been operated (No in step S112), the control device 20 proceeds to step S115.

[0062] When it is determined that the speaker display range 130 has been operated (Yes in step S112), the control device 20 determines whether or not the operated speaker has uttered a keyword requiring reference information immediately before (step S113). More specifically, the control device 20 determines whether or not the utterance recognition unit 21 recognizes the utterance of the operated speaker at the time of the operation of the speaker display range 130 by the extraction unit 26, and whether or not a keyword requiring reference information has been extracted from the recognized utterance. When the control device 20 determines that the operated speaker has uttered a keyword requiring reference information immediately before (Yes in step S113), the control device 20 proceeds to step S114. When the control device 20 does not determine that the operated speaker has uttered a keyword requiring reference information immediately before (No in step S113), the control device 20 proceeds to step S115.

[0063] For example, if the operated range is, for example, the speaker display range 130 that displays the video of participant D, participant D was speaking immediately before the time of the operation, in other words, participant D is speaking, and if a keyword is extracted, reference information regarding the keyword is displayed in a pop-up, for example starting from the speaker display range 130.

[0064] A case will be described in which a user of the information display device 10, who is a participant of a WEB conference, checks reference information for an unknown phrase (keyword) or the like in response to a statement made by another participant during the WEB conference. The user of the information display device 10 operates the speaker display range 130 of the speaker who spoke the keyword. In step S112, the control device 20 detects an operation on the speaker display range 130 displayed on the display unit 12. When the control device 20 determines in step S113 that the speech of the speaker operated at the time of the operation is recognized and a keyword requiring reference information is extracted from the speech, in step S114, the control device 20 causes the display unit 12 to display reference information of the immediately preceding keyword that was spoken within a predetermined retroactive period t1, for example, within 5 seconds, from the time of the operation. The user of the information display device 10 checks the reference information displayed on the display unit 12.

[0065] In this case, even if there is an utterance from a participant other than participant D corresponding to the operated speaker display range 130, for example, participant B, within the specified retroactive period t1 from the time of the operation, reference information for the keyword uttered by participant B will not be displayed.

[0066] <Effects> As described above, in this embodiment, when it is detected that the display range related to an utterance has been operated, reference information for a keyword requiring reference information extracted by the extraction unit 26 immediately before the operation, corresponding to the operated range, is displayed on the display unit 12. According to this embodiment, for example, it is possible to display on the display unit 12 reference information for a keyword requiring reference information extracted by the extraction unit 26 immediately before the operation, which is an utterance of a speaker corresponding to the operated speaker display range 130.

[0067] [Third embodiment] The information display device 10 according to the present embodiment will be described with reference to Figs. 7 to 10. Fig. 7 is a diagram showing an example in which the speaker display range is divided into a plurality of sections. Fig. 7 is a diagram showing an example in which the reference information display range is divided into a plurality of sections. Fig. 8 is a diagram showing an example in which the reference information display range is divided into a plurality of sections. Fig. 9 is a schematic diagram for explaining an example of a keyword extracted immediately before the operation time. Fig. 10 is a flowchart showing an example of a process flow in the information display control device according to the third embodiment. The basic configuration of the information display device 10 is the same as that of the information display device 10 of the second embodiment. In this embodiment, when it is detected that the display range related to the utterance displayed on the display unit 12 has been operated in a conference, the information display device 10 displays reference information for the keyword extracted immediately before the operation time in accordance with the operated range. In this embodiment, the process in the operation unit 17, the operation control unit 32, and the display control unit 28 is different from that in the second embodiment.

[0068] The operation unit 17 is capable of accepting operations for sections obtained by dividing a display range related to utterances displayed on the display unit 12.

[0069] As shown in Fig. 7, the operation unit 17 can accept operations on sections obtained by dividing the speaker display range 130 displayed on the display unit 12. In the example shown in Fig. 7, the speaker display range 130 is divided into a central range 130a located in the center and an outer range 130b located outside the central range 130a. The operation unit 17 can accept operations on the central range 130a and the outer range 130b of the speaker display range 130.

[0070] As shown in Fig. 8, the operation unit 17 can accept operations on sections obtained by dividing the reference information display range 110 displayed on the display unit 12. In the example shown in Fig. 8, the reference information display range 110 is divided into a left area 110a disposed on the left side and a right area 110b disposed to the right of the left area 110a. The operation unit 17 can accept operations on the left area 110a and the right area 110b of the reference information display range 110.

[0071] The operation control unit 32 acquires operation information on operations performed on sections obtained by dividing the display range related to the speech displayed on the display unit 12.

[0072] When the operation control unit 32 detects that the display range related to the utterance displayed on the display unit 12 has been operated, the display control unit 28 weights the reference information for the keyword requiring reference information extracted by the extraction unit 26 immediately before the time of the operation, corresponding to the operation position in the operated range, and displays it on the display unit 12.

[0073] For example, when operating speaker display range 130, display control unit 28 causes display unit 12 to display reference information based on utterances going back a longer period from the time of operation as the operation position approaches the center of speaker display range 130. When the operation position is in central range 130a of speaker display range 130, display control unit 28 causes display unit 12 to display reference information based on utterances going back a longer period from the time of operation, as compared to when the operation position is in outer range 130b, as shown in FIG.

[0074] For example, when operating the horizontally long reference information display range 110, if the operation position is on the right side, reference information based on the most recent comment is displayed on the display unit 12, and if the operation position is on the left side, reference information of multiple extracted keywords from the most recent to the second to third most recent keywords is displayed on the display unit 12. As shown in Fig. 8, the display control unit 28 causes the display unit 12 to display reference information based on the most recent comment if the operation position is on the right side range 110b of the reference information display range 110, and causes the display unit 12 to display reference information of multiple extracted keywords from the most recent to the second to third most recent keywords if the operation position is on the left side range 110a.

[0075] For example, when it is detected that the central range of the display range related to the utterance has been operated, the display control unit 28 causes the display unit 12 to display reference information of the immediately previous keyword uttered within a predetermined retroactive period t2 (t2>t1) from the time of the operation. When multiple immediately previous keywords uttered within a predetermined retroactive period t2 (t2>t1) from the time of the operation are extracted, multiple keywords may be displayed, or the following may be performed.

[0076] For example, when it is detected that the central range of the display range related to the utterance has been operated, the display control unit 28 may select and display the immediately previous keyword uttered within a predetermined retroactive period t2 from the time of the operation. For example, when it is detected that the central range 130a of the speaker display range 130 has been operated, the display control unit 28 selects and displays the immediately previous keyword uttered within a predetermined retroactive period t2 from the time of the operation. For example, when it is detected that the left range 110a of the reference information display range 110 has been operated, the display control unit 28 selects and displays the immediately previous keyword uttered within a predetermined retroactive period t2 from the time of the operation.

[0077] The keyword selection display means, for example, displaying a selection screen for selecting a plurality of keywords extracted during a predetermined period.

[0078] For example, when it is detected that the outer range of the display range related to the utterance has been operated, the display control unit 28 acquires and displays reference information for the immediately previous keyword uttered within a predetermined retroactive period t1 from the time of the operation. For example, when it is detected that the outer range 130b of the speaker display range 130 has been operated, the display control unit 28 acquires and displays reference information for the immediately previous keyword uttered within a predetermined retroactive period t1 from the time of the operation. For example, when it is detected that the right range 110b of the reference information display range 110 has been operated, the display control unit 28 acquires and displays reference information for the immediately previous keyword uttered within a predetermined retroactive period t2 from the time of the operation. For example, when there are multiple immediately previous keywords uttered within a predetermined retroactive period t2 from the time of the operation, the display control unit 28 may display reference information for multiple keywords, or multiple keywords may be selected and displayed, and reference information for the selected keyword may be displayed.

[0079] Fig. 9 is a schematic diagram for explaining an example of keywords extracted immediately before the operation time. In the example shown in Fig. 9, the keyword "word n" is extracted within a predetermined retroactive period t1 from the operation time. The keyword "word n" and the keyword "word n-1" are extracted within a predetermined retroactive period t2 from the operation time.

[0080] Next, the flow of processing in the control device 20 will be described with reference to Fig. 10. The processing from step S121 to step S123 and step S128 shown in Fig. 10 is similar to the processing from step S111 to step S113 and step S115 in the flowchart shown in Fig. 6.

[0081] When it is determined that the operated speaker has just uttered a keyword requiring reference information (Yes in step S123), the control device 20 determines whether or not the operation position is in the central range 130a of the speaker display range 130 (step S124). The control device 20 determines whether or not the operation position is in the central range 130a of the speaker display range 130 based on the operation information by the operation control unit 32. When the control device 20 determines that the operation position is in the central range 130a of the speaker display range 130 (Yes in step S124), the control device 20 proceeds to step S125. When the control device 20 does not determine that the operation position is in the central range 130a of the speaker display range 130 (No in step S124), the control device 20 proceeds to step S127.

[0082] When it is determined that the operation position is in the central range 130a of the speaker display range 130 (Yes in step S124), the control device 20 selectively displays the immediately previous keyword uttered within a predetermined retroactive period t2 from the time of the operation (step S125). More specifically, the control device 20 causes the display control unit 28 to display on the display unit 12 reference information of the immediately previous keyword uttered within a predetermined retroactive period t2 (t2>t1) from the time of the operation. For example, when there are multiple immediately previous keywords uttered within the predetermined retroactive period t2 from the time of the operation, the control device 20 may display reference information of the multiple keywords by the display control unit 28, or may selectively display multiple keywords and display reference information of the selected keyword. The control device 20 proceeds to step S126.

[0083] The control device 20 acquires the reference information of the keyword and displays it for a predetermined period (step S126). More specifically, the control device 20 acquires the reference information for the selected keyword by the information acquisition unit 27. The control device 20 outputs, for a predetermined period, a video signal of the reference information corresponding to the keyword extracted by the extraction unit 26 by the display control unit 28. The control device 20 proceeds to step S128.

[0084] If the control device 20 does not determine that the operation position is in the central range 130a of the speaker display range 130 (No in step S124), the control device 20 causes the display control unit 28 to display on the display unit 12 reference information for the keyword requiring reference information extracted by the extraction unit 26 immediately before the operation (step S127). More specifically, the control device 20 causes the information acquisition unit 27 to acquire reference information for the keyword extracted immediately before. The control device 20 causes the display control unit 28 to output a video signal of the reference information corresponding to the keyword extracted by the extraction unit 26 for a predetermined period. The control device 20 proceeds to step S128.

[0085] 10, an example of an operation in the speaker display range 130 has been described. For an operation in the reference information display range 110, "Is the operation position in the central range of the speaker display range?" in step S124 is replaced with "Is the operation position in the left range of the reference information display range?".

[0086] For example, when the user operates on the central range 130a of the speaker display range 130 as shown by the pointer image Pa in Fig. 7, or operates on the left range 110a of the reference information display range 110 as shown by the pointer image Pa in Fig. 8, the reference information of the immediately previous keyword uttered within a period t1, for example, 5 seconds from the time of the operation, is displayed. In the example shown in Fig. 9, the reference information of the keyword "word n" is displayed.

[0087] For example, when the user operates on the outer range 130b of the speaker display range 130 as shown by the pointer image Pb in FIG. 7, or operates on the right range 110b of the reference information display range 110 as shown by the pointer image Pb in FIG. 8, the reference information of the immediately preceding keyword uttered within a period t2, for example, 15 seconds, from the time of the operation is displayed. In these cases, when the extraction unit 26 extracts multiple keywords within a period t2 from the time of the operation, the reference information of the multiple keywords may be displayed, or the multiple keywords may be selectively displayed and the reference information of the selected keyword may be displayed. In the example shown in FIG. 9, the reference information of both the keyword "word n" and the keyword "word n-1" may be displayed. Alternatively, the keyword "word n" and the keyword "word n-1" may be selectively displayed. When the user selects the keyword "word n-1", the reference information of the keyword "word n-1" is displayed. When the keyword "word n" and the keyword "word n-1" are the same phrase, the selective display may not be performed.

[0088] <Effects> As described above, in this embodiment, when it is detected that the display range related to the utterance has been operated, the reference information for the keyword requiring the reference information extracted by the extraction unit 26 immediately before the operation is weighted in accordance with the operation position in the operated range, and is displayed on the display unit 12. According to this embodiment, it is possible to display or selectively display a plurality of keywords going back from the operation position according to the operation position. According to this embodiment, it is possible to appropriately display the reference information for the keyword required by the user.

[0089] [Fourth embodiment] The information display device 10 according to this embodiment will be described with reference to FIG. 11. FIG. 11 is a flowchart showing an example of a processing flow in an information display control device according to a fourth embodiment. The information display device 10 has a basic configuration similar to that of the information display device 10 according to the second embodiment. In this embodiment, when it is detected that a display range related to an utterance displayed on the display unit 12 has been operated during a conference, the information display device 10 displays reference information for a keyword extracted immediately before the operation, corresponding to the operated range. In this embodiment, the processing in the operation unit 17, the operation control unit 32, and the display control unit 28 is different from that in the second embodiment.

[0090] When the display control unit 28 detects that the display range related to the utterance displayed on the display unit 12 has been operated by the operation control unit 32, the display control unit 28 weights the reference information for the keyword requiring the reference information extracted by the extraction unit 26 at the time of detection in accordance with the period of operation, and displays the weighted reference information on the display unit 12.

[0091] For example, when it is detected that the display range related to an utterance has been operated, the display control unit 28 causes the display unit 12 to display reference information based on utterances that date back a longer period from the time of the operation as the period of operation is longer. For example, when it is detected that the display range related to an utterance has been operated, the display control unit 28 causes the display unit 12 to display reference information based on utterances that date back a shorter period from the time of the operation as the period of operation is shorter.

[0092] Corresponding to the period of operation will be described. For example, when the touch period and the operation period performed by the pointer such as a mouse on the speaker display range 130 and the reference information display range 110 are less than 1.0 second, the display control unit 28 displays reference information based on the most recent utterance, for example, reference information of the immediately preceding keyword uttered within a period t1 from the time of operation as shown in FIG. 9. For example, when the touch period and the operation period performed by the pointer such as a mouse on the speaker display range 130 and the reference information display range 110 are 1.0 second or more, the display control unit 28 displays reference information from the most recent to the previous 2 to 3 words, for example, up to the keyword uttered within a period t2 from the time of operation as shown in FIG. 9. When there are multiple extracted keywords, they may be selectively displayed. In this case, the operation time is preferably the operation start time.

[0093] Next, the flow of processing in the control device 20 will be described with reference to Fig. 11. The processing in steps S131 to S133 and steps S135 to S138 shown in Fig. 11 is similar to the processing in steps S121 to S123 and steps S125 to S128 in the flowchart shown in Fig. 10.

[0094] When it is determined that a keyword has been extracted (Yes in step S133), the control device 20 determines whether the operation has been performed for the period ta or more (step S134). More specifically, the control device 20 determines whether the operation has been performed for the period ta or more based on the operation information using the operation control unit 32. When the control device 20 determines that the operation has been performed for the period ta or more (Yes in step S134), it proceeds to step S135. The control device 20 determines whether the operation has been performed for the period ta or more based on the operation information using the operation control unit 32. When the control device 20 determines that the operation has been performed for the period ta or more (No in step S134), it proceeds to step S137.

[0095] <Effects> As described above, when it is detected that the display range related to the utterance has been operated, the reference information for the keyword requiring the reference information extracted by the extraction unit 26 at the time of detection is weighted in accordance with the period of the operation, and is displayed on the display unit 12. According to this embodiment, it is possible to display or selectively display a plurality of keywords going back from the time of the operation according to the period of the operation. According to this embodiment, it is possible to appropriately display the reference information for the keyword required by the user.

[0096] The information display device according to the present disclosure may be embodied in various different forms other than the embodiment described above.

[0097] Each component of the illustrated information display device is a functional concept, and does not necessarily have to be physically configured as illustrated. In other words, the specific form of each device is not limited to that shown in the figure, and all or part of the device may be functionally or physically distributed or integrated in any unit depending on the processing load and usage status of each device.

[0098] The configuration of the information display device is realized, for example, as software, by a program loaded into a memory. In the above embodiment, the functional blocks are described as being realized by the cooperation of these hardware and software. In other words, these functional blocks can be realized in various forms by only hardware, only software, or a combination of both.

[0099] The above-mentioned components include those that a person skilled in the art can easily imagine and those that are substantially the same. Furthermore, the above-mentioned configurations can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the configurations are possible without departing from the scope of the present invention. [Explanation of symbols]

[0100] 10 Information display device 11 Microphone 12 Display section 17 Control section 18 Keyword information storage section 19 Communications Department 20 Control device (information display control device) 21 Speech Recognition Unit 26 Extraction part 27 Information Acquisition Department 28 Display control section 29 Communication control section 32 Operation control section

Claims

1. An information display control device that controls an information display device to display information on a display unit, a speech recognition unit that recognizes the speech of a speaker displayed on the display unit; an operation control unit that detects an operation on a display screen displayed on the display unit; an extraction unit that extracts keywords requiring reference information from the utterance recognized by the utterance recognition unit; an information acquisition unit that acquires the reference information for the keyword extracted by the extraction unit; a display control unit that displays the reference information acquired by the information acquisition unit on the display unit; Equipped with when the operation control unit detects that a display range related to an utterance displayed on the display unit has been operated, the display control unit causes the display unit to display reference information for a keyword that requires reference information extracted by the extraction unit immediately before the detection. Information display control device.

2. when it is detected by the operation control unit that a display range related to the utterances corresponding to the plurality of speakers displayed on the display unit has been operated, the display control unit displays, on the display unit, reference information for a keyword requiring reference information extracted by the extraction unit, based on the utterance of the speaker corresponding to the operated display range related to the utterance. The information display control device according to claim 1 .

3. when the operation control unit detects that a display range related to an utterance displayed on the display unit has been operated, the display control unit weights reference information for a keyword that requires reference information extracted by the extraction unit immediately before the detection, in accordance with an operation position in the operated range, and displays the weighted reference information on the display unit. The information display control device according to claim 1 .

4. when it is detected by the operation control unit that a display range related to an utterance displayed on the display unit has been operated, the display control unit weights reference information for a keyword that requires reference information extracted by the extraction unit immediately before the detection time point, in accordance with a period of the operation. The information display control device according to claim 1 .

5. An information display method executed by an information display device that displays information on a display unit, comprising: a speech recognition step of recognizing the speech of the speaker displayed on the display unit; an operation control step of detecting an operation on a display screen displayed on the display unit; an extraction step of extracting keywords requiring reference information from the utterance recognized in the utterance recognition step; an information acquisition step of acquiring reference information for the keyword extracted in the extraction step; a display control step of displaying the reference information acquired in the information acquisition step on the display unit; Including, When the operation control step detects that a display range related to the utterance displayed on the display unit has been operated, the display control step displays, on the display unit, reference information for a keyword requiring reference information extracted in the extraction step immediately before the detection point in time. A method of displaying information that:

Images