Information display control device and information display method

Abstract

To appropriately display reference information, from a keyword based on utterance contents.SOLUTION: An information display control device 20 that controls an information display device 10 to display information on a display unit 12, comprises: an utterance recognition unit 21 that recognizes utterance of an utterer; a visual line detection unit 24 that detects a direction of a visual line of a user who uses the information display device 10; an extraction unit 26 that extracts a keyword requiring reference information, from the utterance recognized by the utterance recognition unit 21; an information acquisition unit 27 that acquires reference information to the keyword extracted from the extraction unit 26; and a display control unit 28 that displays the reference information acquired by the information acquisition unit 27 on the display unit 12. The display control unit 28, when it is detected that the direction of the user's visual line detected by the visual line detection unit 24 is directed to the display unit 12 or to a predetermined range 110 in the display unit where the reference information is to be displayed, displays the reference information to the keyword requiring the reference information that is extracted by the extraction unit immediately before the time point when it is detected, on the display unit 12.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information display control device and an information display method. [Background technology]

[0002] BACKGROUND ART There is known a technique for displaying additional information about a phrase spoken by a person in a meeting or conversation (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-007003 Summary of the Invention [Problem to be solved by the invention]

[0004] The additional information displayed is information that explains abbreviations and difficult keywords. However, depending on the level of understanding of the recipient of the additional information, an explanation may be necessary for keywords for which no additional information is displayed. In this case, it may be difficult for the recipient to understand. Alternatively, there may be cases where no explanation is necessary for keywords for which additional information is displayed. In this case, additional information is displayed even for keywords for which additional information is not required by the recipient, which may disrupt the flow of communication.

[0005] The present disclosure has been made in view of the above, and aims to appropriately display reference information based on keywords based on the content of an utterance. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the objective, the information display control device of the present disclosure is an information display control device that controls an information display device to display information on a display unit, and is equipped with an utterance recognition unit that recognizes the speaker's speech, a gaze detection unit that detects the direction of the user's gaze of the information display device, an extraction unit that extracts keywords that require reference information from the utterance recognized by the utterance recognition unit, an information acquisition unit that acquires the reference information for the keywords extracted by the extraction unit, and a display control unit that displays the reference information acquired by the information acquisition unit on the display unit, and when the display control unit detects that the direction of the user's gaze detected by the gaze detection unit is directed toward the display unit or a specified range on the display unit for displaying the reference information, it causes the display unit to display reference information for the keywords that require reference information extracted by the extraction unit immediately before the time of detection.

[0007] Another information display control device related to the present disclosure is an information display control device that controls an information display device to display information on a display unit, and includes: a speech recognition unit that recognizes the speaker's speech; a gaze detection unit that detects the direction of the user's gaze of the information display device; an extraction unit that extracts keywords that require reference information from the speech recognized by the speech recognition unit; an information acquisition unit that acquires the reference information for the keywords extracted by the extraction unit; and a display control unit that displays the reference information acquired by the information acquisition unit on the display unit, wherein the display control unit displays the reference information extracted by the extraction unit on the display unit, and when it detects that the direction of the user's gaze detected by the gaze detection unit is directed towards the display unit or a predetermined range on the display unit for displaying the reference information, it changes the display mode of the reference information being displayed.

[0008] The information display method of the present disclosure is an information display method executed by an information display device that displays information on a display unit, and includes a speech recognition step that recognizes the speaker's speech, a gaze detection step that detects the direction of the user's gaze of the information display device, an extraction step that extracts keywords that require reference information from the speech recognized in the speech recognition step, an information acquisition step that acquires reference information for the keywords extracted in the extraction step, and a display step that displays the reference information acquired in the information acquisition step on a display unit, wherein in the display step, when it is detected that the direction of the user's gaze detected in the gaze detection step is directed toward the display unit or a predetermined range on the display unit in which the reference information is to be displayed, reference information for the keywords that require reference information extracted in the extraction step just before the time of detection is displayed on the display unit.

[0009] Another information display method according to the present disclosure is an information display method executed by an information display device that displays information on a display unit, and includes a speech recognition step that recognizes the speaker's speech, a gaze detection step that detects the direction of the user's gaze of the information display device, an extraction step that extracts keywords that require reference information from the speech recognized in the speech recognition step, an information acquisition step that acquires reference information for the keywords extracted in the extraction step, and a display step that displays the reference information acquired in the information acquisition step on a display unit, wherein in the display step, the reference information extracted by the extraction unit is displayed on the display unit, and if the gaze detection step detects that the direction of the user's gaze is directed toward the display unit or a predetermined range on the display unit for displaying the reference information, the display mode of the displayed reference information is changed. [Effects of the Invention]

[0010] According to the present disclosure, it is possible to provide an effect of appropriately displaying reference information based on keywords based on the content of an utterance. [Brief explanation of the drawings]

[0011] [Figure 1]FIG. 1 is a schematic diagram showing an example of use of an information display device. [Figure 2] FIG. 2 is a schematic diagram showing another example of use of the information display device. [Figure 3] FIG. 3 is a block diagram showing an example of the configuration of an information display device having an information display control device according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of keyword information. [Figure 5] FIG. 5 is a flowchart showing an example of the flow of processing in the information display control device according to the first embodiment. [Figure 6] FIG. 6 is a flowchart showing an example of the flow of processing in the information display control device according to the second embodiment. [Figure 7] FIG. 7 is a flowchart showing an example of the flow of processing in the information display control device according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An information display control device and an information display method according to embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. However, the present invention is not limited to the following embodiments.

[0013] [First embodiment] <Information display device> FIG. 1 is a schematic diagram showing an example of a usage example of an information display device. FIG. 2 is a schematic diagram showing another example of a usage example of an information display device. FIG. 3 is a block diagram showing an example of the configuration of an information display device 10 having an information display control device (hereinafter referred to as "control device") 20 according to a first embodiment. The information display device 10 displays reference information for keywords included in the utterances of participants in at least speech-based communication such as a meeting or conversation.

[0014] As shown in FIG. 1, the information display device 10 is used in a conference or conversation where multiple participants gather in one place (hereinafter referred to as a "real conference"). The information display device 10 used in this case may be implemented as one of the functions of an information terminal device such as a smartphone used by the participants. The real conference shown in FIG. 1 includes a presenter M11, and other participants M12, M13, M14, and M15. The other participant M15 is using the information display device 10.

[0015] 2, information display device 10 is used in online communication such as a conference (hereinafter referred to as a "WEB conference") or conversation held online with the participation of multiple participants. Information display device 10 used in this case may be implemented as one of the functions of an information terminal device such as a laptop personal computer or smartphone used when participating in an online WEB conference or conversation via a WEB conference system (not shown).

[0016] In the following explanation, the term "conference" will be used in either a real conference or a web conference. When there is no need to distinguish between a real conference and a web conference, the term "conference" will be used.

[0017] When the information display device 10 detects that the direction of the user's gaze is directed toward the display unit 12 during a meeting, it causes the display unit 12 to display reference information for a keyword that requires reference information extracted immediately before the detection point (hereinafter referred to as the "most recent keyword"). The information display device 10 includes a microphone 11, a display unit 12, a camera 14, a keyword information storage unit 18, a communication unit 19, and a control device 20.

[0018] Microphone 11 is a microphone that picks up the voices of participants in a conference using information display device 10. Microphone 11 outputs the picked-up voice data to speech recognition unit 21 of control device 20. As shown in FIG. 1, when information display device 10 is used for a real conference, microphone 11 picks up the voice of the speaker. As shown in FIG. 2, when information display device 10 is used for a web conference, microphone 11 picks up the voice of the user, who is a participant using information display device 10.

[0019] The display unit 12 is, for example, a display including a liquid crystal display, an organic EL (Electro-Luminescence) display, etc. The display unit 12 displays an image based on a video signal output from the display control unit 28 of the control device 20.

[0020] Reference information corresponding to keywords included in the speech of conference participants is displayed on the display unit 12. The range in which the reference information is displayed on the display unit 12 is referred to as the reference information display range 110. As shown in FIG. 1, when the information display device 10 is used for a real conference, the reference information display range 110 may be displayed on the entire surface of the display unit 12. As shown in FIG. 2, when the information display device 10 is used for a web conference, the reference information display range 110 may be displayed on a part of the display unit 12, for example, the bottom, top, left side, or right side. When the information display device 10 is used for a web conference, the reference information display range 110 may be displayed as a pop-up on the display unit 12. The reference information display range 110 is, for example, a range in which approximately three lines of characters or less are displayed.

[0021] As shown in FIG. 2, when the information display device 10 is used for a web conference, the display unit 12 displays the video of the participants, the shared materials, and the like on the screen.

[0022] Camera 14 is disposed in a position where it can capture an image of the face of the user of information display device 10. More specifically, camera 14 captures an image including the face and eyes of the user in order to detect the line of sight of the user of information display device 10. Camera 14 is disposed in the periphery of display unit 12. Camera 14 outputs the captured image to line of sight detection unit 24.

[0023] The keyword information storage unit 18 stores reference information for each keyword as keyword information. The keyword information storage unit 18 is, for example, a recording medium such as a semiconductor memory element, such as a RAM (Random Access Memory) or a flash memory. The keyword information storage unit 18 may be an external recording unit wirelessly connected via the communication unit 19.

[0024] Keywords are words that require additional explanation, such as unknown phrases, technical terms, abbreviations, difficult words, and buzzwords.

[0025] The reference information may be, for example, the meaning of the keyword or a specific explanation. If the keyword is an abbreviation, the reference information may be the full name without abbreviation. The explanation may be, for example, a photograph or a diagram in addition to text. The reference information may be, for example, additional information that is additional information to the keyword.

[0026] The reference information may be, for example, 100 characters or less, which can be read by a person in a few seconds.

[0027] Fig. 4 is a diagram showing an example of keyword information. The keyword information shown in Fig. 4 stores reference information for each keyword. For example, for the keyword "XXX", reference information "○···○" is stored. For example, for the keyword "△△△", reference information "△···△" is stored.

[0028] When the communication unit 19 searches for reference information on the web, for example, information from a specific web media such as Wikipedia (registered trademark) for the keyword, the latest news related to the keyword, etc. are searched. Also, when the keyword is a product name or service name, the web site of the provider of the product or service is searched. Furthermore, the content displayed at the top of the results excluding advertisements in a search by a web search engine may be used as reference information.

[0029] The communication unit 19 is a communication unit for performing wireless communication and the like. The communication unit 19 communicates using a communication method such as Wi-Fi (registered trademark) or a mobile phone line, and is connected to an intranet or the Internet. The communication of the communication unit 19 is controlled by a communication control unit 29 of the control device 20. As shown in FIG. 1, when the information display device 10 is used for a real conference, the communication unit 19 acquires keyword information from a keyword information storage unit 18, which is an external storage unit, and searches for reference information. As shown in FIG. 2, when the information display device 10 is used for a web conference, the communication unit 19 exchanges audio and video of the participants with the information display devices 10 used by other participants and searches for reference information. More specifically, the communication unit 19 transmits the audio of the participants picked up by the microphone 11 and the video of the participants captured by the camera to the information display devices 10 used by the other participants. The communication unit 19 receives the audio and video of the other participants from the information display devices 10 used by the other participants.

[0030] In the following description, in the case of the information display device 10 used in a real conference, the input voice refers to the voice picked up by the microphone 11. In the case of the information display device 10 used in a web conference, the input voice refers to the voice picked up by the microphone 11 and the voice of other participants acquired via the communication unit 19 from the information display device 10 used by the other participants.

[0031] <Information display control device> The control device 20 controls each unit of the information display device 10. The control device 20 controls the information display device 10 to display information on the display unit 12. The control device 20 is an arithmetic processing device (control device) configured, for example, with a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), and has a storage device such as a RAM or a ROM (Read Only Memory). The control device 20 loads a stored program into the memory and executes instructions included in the program. The control device 20 includes an internal memory such as the RAM described above, which is used for temporary storage of data in the control device 20, etc. The control device 20 has, as components such as functional blocks realized by executing the program, an utterance recognition unit 21, a gaze detection unit 24, an extraction unit 26, an information acquisition unit 27, a display control unit 28, and a communication control unit 29.

[0032] The speech recognition unit 21 recognizes the speech of at least the participants taking part in the speech communication. The speech recognition unit 21 analyzes the input speech and recognizes the speech content contained in the speech. The speech recognition unit 21 performs acoustic model analysis on the input speech for each phoneme or each word, and recognizes the speech content by comparing it with a phoneme model or a language model.

[0033] The gaze detection unit 24 detects the direction of the gaze of the user of the information display device 10 based on the image data captured by the camera 14. There are no limitations on the method for detecting the gaze, but in this embodiment, the gaze is detected by the corneal reflex.

[0034] The gaze detection unit 24 detects, for example, that the direction of the user's gaze is directed toward the display unit 12. This is because, for example, during a real conference, when the user wants to refer to a keyword, it is predicted that the user will direct their gaze toward the display unit 12. Therefore, the gaze detection unit 24 detects, for example, that the direction of the user's gaze is directed toward the display unit 12 during a real conference.

[0035] For example, the gaze detection unit 24 detects that the direction of the user's gaze is directed toward a predetermined range on the display unit 12. This is because, for example, when a user wants to refer to a keyword during a web conference, the user is expected to direct their gaze toward a predetermined range on the screen of the display unit 12.

[0036] The predetermined range of the display unit 12 is, for example, the reference information display range 110 of the display unit 12, or a predetermined range of the screen that is set in advance.

[0037] The gaze detection unit 24 detects, for example, that the direction of the user's gaze has changed from facing a speaker to facing the display unit 12. This is because, for example, during a real-life conference or a web conference, when the user wants to refer to a keyword, it is predicted that the user will move their gaze from the speaker to the display unit 12.

[0038] The state in which the gaze direction is directed toward the speaker will be described. In the case of information display device 10 used in a real conference, for example, the direction of the speech sound is detected from the sound picked up by microphone 11. Alternatively, in the case of information display device 10 used in a web conference, for example, the direction in which the speaker or the material by the speaker is displayed on the screen is detected from the video captured by camera 14. The state in which the gaze direction of the user matches the direction of the speaker is the state in which the gaze direction is directed toward the speaker.

[0039] The extraction unit 26 extracts keywords that require reference information from the utterance recognized by the utterance recognition unit 21. In this embodiment, the extraction unit 26 starts extracting keywords at the same time that the utterance recognition unit 21 starts utterance recognition.

[0040] The information acquiring unit 27 acquires reference information for the keywords extracted by the extracting unit 26. In this embodiment, the information acquiring unit 27 acquires reference information for the keywords from the keyword information stored in the keyword information storage unit 18.

[0041] In this embodiment, when the gaze detection unit 24 detects that the direction of the user's gaze is directed toward the display unit 12, the information acquisition unit 27 acquires reference information for the most recent keyword.

[0042] The display control unit 28 controls the display of the video on the display unit 12. The display control unit 28 outputs to the display unit 12 a video signal of the video.

[0043] The display control unit 28 controls the display of the reference information acquired by the information acquisition unit 27 on the display unit 12. The display control unit 28 outputs a video signal for displaying the reference information corresponding to the keyword extracted by the extraction unit 26 in the reference information display area 110 of the display unit 12. The display control unit 28 displays the reference information for a predetermined period of time, for example, about 5 to 10 seconds. When displaying the next reference information while the reference information is being displayed, the display control unit 28 may change the display to the next reference information even before the predetermined period has elapsed.

[0044] When the display control unit 28 detects that the direction of the user's gaze detected by the gaze detection unit 24 is directed toward the display unit 12, the display control unit 28 causes the display unit 12 to display reference information for the most recent keyword. For example, during a real conference, when the user's gaze is directed toward the display unit 12 of a smartphone that is the information display device 10, the display unit 12 displays reference information for the most recent keyword.

[0045] In the present embodiment, the display control unit 28 may display the reference information acquired by the information acquisition unit 27 in a predetermined range on the display unit 12, and when it detects that the direction of the user's gaze detected by the gaze detection unit 24 is directed toward the predetermined range on the display unit 12, may display the reference information for the most recent keyword in the predetermined range on the display unit 12. For example, in a notebook personal computer that is the information display device 10 used for a web conference, if the reference information display range 110 is located at the bottom of the screen of the display unit 12, the display control unit 28 may display the reference information for the most recent keyword when the user looks at the bottom of the screen.

[0046] In this embodiment, when the display control unit 28 detects that the direction of the user's gaze, detected by the gaze detection unit 24, has changed from a state in which the gaze was directed toward the speaker to a state in which the gaze is directed toward the display unit 12, the display control unit 28 may cause the display unit 12 to display reference information for the most recent keyword. For example, in the information display device 10 used for a web conference, when the gaze direction is moved from the range in which the video of the participant or the document on the display unit 12 is displayed, the display control unit 28 displays reference information for the most recent keyword.

[0047] In this embodiment, when the display control unit 28 detects that the direction of the user's gaze detected by the gaze detection unit 24 is directed toward the display unit 12, the display control unit 28 may improve the visibility of the display of reference information for the most recent keyword.

[0048] Improving the visibility of the reference information display means displaying the reference information in a more noticeable manner, such as by increasing the number of characters, enlarging the characters, changing the character color, or changing the character thickness. For example, the display control unit 28 displays the reference information of the most recent keyword in a small number of characters, and when the user's gaze is directed thereto, displays the number of characters in a large number of characters. For example, the display control unit 28 displays the reference information of the most recent keyword in a light or transparent color, and when the user's gaze is directed thereto, displays the reference information in a dark color or as opaque characters.

[0049] When the information display device 10 is used for a web conference, the display control unit 28 controls the display of the video of the web conference. The display control unit 28 outputs a video signal of the video of the web conference including the video captured by the camera.

[0050] The communication control unit 29 controls communication by the communication unit 19. As shown in FIG. 1 , in the case of the information display device 10 used in a real conference, the communication control unit 29 controls the communication unit 19 to receive keyword information from the keyword information storage unit 18, which is an external storage unit, and to search for reference information. As shown in FIG. 2 , in the case of the information display device 10 used in a web conference, the communication control unit 29 controls the communication unit 19 to mutually exchange participant audio and video with the information display device 10 used by the other participant, and to search for reference information. More specifically, the communication control unit 29 controls the communication unit 19 to transmit participant audio picked up by the microphone 11 and participant video captured by the camera to the information display device 10 used by the other participant. The communication control unit 29 controls the communication unit 19 to receive participant audio and video from the information display device 10 used by the other participant.

[0051] <Information processing in information display control devices> Next, the flow of processing in the control device 20 will be described with reference to FIG. 5. As shown in FIG. 1, the processing shown in FIG. 5 is initiated by starting an application for displaying reference information when the information display device 10 is used for a real conference. As shown in FIG. 2, the processing is initiated by starting an application for conducting a web conference or by starting the web conference when the information display device 10 is used for a web conference. In this case, the application for conducting a web conference may include a function for displaying reference information, and the function for displaying reference information may be selectable on / off by a user operation. When the information display device 10 is started, the processing of the flowchart shown in FIG. 5 is initiated.

[0052] The control device 20 starts speech recognition (step S101). More specifically, the control device 20 analyzes the input speech using the speech recognition unit 21 and recognizes the content of the speech contained in the speech. Then, the control device 20 extracts keywords requiring reference information from the speech recognized by the speech recognition unit 21 using the extraction unit 26. The control device 20 proceeds to step S102.

[0053] The control device 20 determines whether the user's gaze is directed toward the display unit 12 (step S102). The control device 20 determines whether the direction of the user's gaze is directed toward the display unit 12 using the gaze detection unit 24. If the control device 20 determines that the user's gaze is directed toward the display unit 12 (Yes in step S102), the process proceeds to step S103. If the control device 20 does not determine that the user's gaze is directed toward the display unit 12 (No in step S102), the process proceeds to step S105.

[0054] The control device 20 determines whether a keyword requiring reference information has been extracted immediately before (step S103). More specifically, if the control device 20 determines that the extraction unit 26 has extracted the most recent keyword within, for example, two seconds (Yes in step S103), the control device 20 proceeds to step S104. If the control device 20 does not determine that the extraction unit 26 has extracted the most recent keyword within, for example, two seconds (No in step S103), the control device 20 proceeds to step S105.

[0055] When it is determined that the most recent keyword has been extracted (Yes in step S103), the control device 20 acquires reference information for the most recent keyword and displays it for a predetermined period (step S104). More specifically, the control device 20 acquires, for example, reference information for the most recent keyword using the information acquisition unit 27. The control device 20 outputs, for a predetermined period, a video signal of the reference information corresponding to the most recent keyword extracted by the extraction unit 26 using the display control unit 28. The control device 20 proceeds to step S105.

[0056] The control device 20 determines whether or not to end the process (step S105). For example, the control device 20 determines to end the process when the power of the information display device 10 is turned off, an operation on an operation unit (not shown) is performed, or an operation to end an application that triggered the start of the flowchart shown in FIG. 5 is performed. If the control device 20 determines to end the process (Yes in step S105), it ends this process. If the control device 20 does not determine to end the process (No in step S105), it executes the process of step S102 again.

[0057] <Effects> As described above, in this embodiment, when it is detected that the user's gaze is directed toward the display unit 12, reference information for the most recent keyword is displayed on the display unit 12. According to this embodiment, keyword reference information can be displayed while the user is looking at the display unit 12. This embodiment can appropriately support the user's understanding when the user needs it. This embodiment can prevent reference information for unnecessary keywords from being displayed to the user. This embodiment can help smooth the flow of communication. In this way, according to this embodiment, reference information can be appropriately displayed based on keywords based on the content of the utterance.

[0058] In this embodiment, reference information is displayed in a predetermined range on the display unit 12, and when it is detected that the user's line of sight is directed toward the predetermined range on the display unit 12, reference information for the most recent keyword is displayed in the predetermined range on the display unit 12. According to this embodiment, when the user is looking at the predetermined range on the display unit 12, reference information for the keyword can be displayed.

[0059] In this embodiment, when it is detected that the direction of the user's gaze has changed from facing the speaker to facing the display unit 12, reference information for the most recent keyword is displayed on the display unit 12. According to this embodiment, when the user moves their gaze from the speaker to the display unit 12, reference information for the keyword can be displayed.

[0060] In this embodiment, when it is detected that the user's gaze is directed toward the display unit 12, the visibility of the display of reference information for the most recent keyword is improved. This embodiment can appropriately support the user's understanding when the user needs it.

[0061] [Second embodiment] An information display device 10 according to this embodiment will be described with reference to FIG. 6. FIG. 6 is a flowchart showing an example of a processing flow in an information display control device according to a second embodiment. The basic configuration of the information display device 10 is similar to that of the information display device 10 according to the first embodiment. In the following description, components similar to those of the information display device 10 are denoted by the same or corresponding reference symbols, and detailed description thereof will be omitted. In this embodiment, when the information display device 10 detects that the user's gaze continues to be directed toward the display unit 12 for a predetermined period of time or longer during a meeting, the information display device 10 changes the display mode of reference information for keywords requiring reference information and displays it on the display unit 12. In this embodiment, the processing in the gaze detection unit 24 and the display control unit 28 differs from that in the first embodiment.

[0062] In this embodiment, the display control unit 28 causes the display unit 12 to display the reference information in the first display mode or the second display mode. In this embodiment, the first display mode and the second display mode differ in the amount of reference information displayed.

[0063] When the display control unit 28 displays the reference information in the first display mode and the second display mode, the keyword information storage unit 18 may store reference information with an information amount lv1 corresponding to the first display mode and reference information with an information amount lv2 corresponding to the second display mode. The relationship between the information amounts of the reference information with information amount lv1 and the reference information with information amount lv2 is information amount lv1<information amount lv2. For example, the reference information with information amount lv1 is reference information that is displayed in about 1 to 2 lines, and the reference information with information amount lv2 is reference information that is displayed in about 3 to 5 lines.

[0064] Furthermore, when the information acquisition unit 27 searches for reference information on the web, it acquires reference information for the first display mode and reference information for the second display mode that satisfy information amount lv1 < information amount lv2 based on reference information acquired from a web search or a specific web media. Furthermore, the information acquisition unit may set different information sources for the web media from which to acquire reference information for display in the first display mode and the web media from which to acquire reference information for display in the second display mode. For example, the reference information for display in the first display mode is acquired from web media that has a relatively small amount of information explaining keywords, and the reference information for display in the second display mode is acquired from web media that has a relatively large amount of information explaining keywords, in other words, that provides detailed explanations.

[0065] For example, the gaze detection unit 24 detects that the direction of the user's gaze continues to be directed for a predetermined period or longer within a predetermined range of the display unit 12. This is because, for example, when a user feels that they do not fully understand the reference information for a displayed keyword during a meeting, it is predicted that the state in which the gaze is directed toward the display unit 12 will continue for a long period of time.

[0066] When the display control unit 28 detects that a predetermined time or more has elapsed while the direction of the user's gaze detected by the gaze detection unit 24 is directed toward the display unit 12, the display control unit 28 changes the display mode and displays the reference information for the most recent keyword on the display unit 12. When the direction of the user's gaze is directed toward the display unit 12 for, for example, five seconds or more, the display control unit 28 changes the display mode and displays the reference information for the most recent keyword from the first display mode to the second display mode.

[0067] Next, the flow of processing in the control device 20 will be described with reference to Fig. 6. The processing from step S111 to step S113 and step S118 shown in Fig. 6 is the same as the processing from step S101 to step S103 and step S105 in the flowchart shown in Fig. 5.

[0068] In step S114, the control device 20 causes the display control unit 28 to display the reference information with the information amount lv1 corresponding to the most recent keyword extracted by the extraction unit 26 for a predetermined period of time.

[0069] The control device 20 determines whether or not a predetermined period of time has elapsed (step S115). If the control device 20 determines that the predetermined period of time has elapsed (Yes in step S115), the process proceeds to step S116. If the control device 20 does not determine that the predetermined period of time has elapsed (No in step S115), the control device 20 executes the process of step S115 again.

[0070] The control device 20 determines whether the user's gaze remains directed toward the display unit 12 (step S116). The control device 20 determines whether the direction of the user's gaze remains directed toward the display unit 12, using the gaze detection unit 24. If the control device 20 determines that the user's gaze remains directed toward the display unit 12 (Yes in step S116), the process proceeds to step S117. If the control device 20 does not determine that the user's gaze remains directed toward the display unit 12 (No in step S116), the process proceeds to step S118.

[0071] When it is determined that the user's gaze remains directed toward the display unit 12 (Yes in step S116), the control device 20 displays reference information corresponding to the most recent keyword with an information amount lv2 for a predetermined period (step S117). More specifically, the control device 20 causes the display control unit 28 to increase the information amount of the reference information corresponding to the most recent keyword extracted by the extraction unit 26 and display it. The control device 20 proceeds to step S118.

[0072] For example, when the reference information is displayed in step S114, the display control unit 28 starts displaying the reference information with a setting that displays the reference information for a predetermined period of time, such as five seconds. In step S115, even after the predetermined period of time, such as five seconds, it is detected that the user's gaze remains directed toward the display unit 12. In step S117, the display unit 12 displays the reference information that was displayed in the first display mode in the second display mode for another predetermined period of time, such as five seconds.

[0073] <Effects> As described above, when it is detected that a predetermined time or more has elapsed while the user's gaze is directed toward the display unit 12, the display mode of the reference information for the most recent keyword is changed and displayed on the display unit 12. According to this embodiment, it is possible to appropriately support the user's understanding when, for example, the user does not fully understand the keyword from the displayed reference information.

[0074] [Third embodiment] An information display device 10 according to this embodiment will be described with reference to FIG. 7. FIG. 7 is a flowchart showing an example of a processing flow in an information display control device according to a third embodiment. The basic configuration of the information display device 10 is similar to that of the information display device 10 according to the first embodiment. In the following description, components similar to those of the information display device 10 are denoted by the same or corresponding reference symbols, and detailed description thereof will be omitted. In this embodiment, when a keyword requiring reference information is extracted during a meeting, the information display device 10 displays the reference information on the display unit 12, and when it is detected that the user's gaze is directed toward the display unit 12, the information display device 10 changes the display mode of the reference information and displays it on the display unit 12. In this embodiment, the processing in the display control unit 28 differs from that in the first embodiment.

[0075] In this embodiment, the display control unit 28 causes the display unit 12 to display the reference information in the first display mode or the second display mode. In this embodiment, the first display mode and the second display mode differ in the amount of displayed reference information or the display visibility of the reference information.

[0076] When the first and second display forms in which the display control unit 28 displays the reference information are the amount of display information of the reference information, the first and second display forms are the same as the first and second display forms in the second embodiment. When the first and second display forms in which the display control unit 28 displays the reference information are the display visibility of the reference information, the display control unit 28 changes the display visibility of the reference information. Changing the display visibility of the reference information includes, for example, changing the character size, character color, character transparency, etc.

[0077] An example in which the display control unit 28 changes the display visibility of the reference information will be described. When the change in the display visibility of the reference information is a change in the character size, the display control unit 28 performs display with good visibility by changing the display mode to a second display mode in which the character size is larger than the character size displayed in the first display mode. Therefore, the display control unit 28 changes the display mode from the first display mode, which has relatively low visibility, to the second display mode, which has relatively high visibility. When displaying the reference information in the second display mode, the area of ​​the reference information display range 110, etc. may be increased in accordance with the change in character size.

[0078] When the change in display visibility of the reference information is a change in character color or character transparency, the display control unit 28 changes the display mode to a second display mode having a character color and character transparency that are more noticeable than the characters displayed in the first display mode, thereby providing a display with good visibility. The character color in the first display mode is, for example, black, and the character color in the second display mode is, for example, red. The character transparency, or transparency, in the first display mode is, for example, around 50%, which allows the background information to be seen through, and the character transparency in the second display mode is less than 10%, which makes the background information almost invisible. In addition to the above, the first and second display modes may also change the thickness of the characters, etc.

[0079] Next, the flow of processing in the control device 20 will be described with reference to Fig. 7. The processing in steps S121 and S126 shown in Fig. 7 is the same as that in steps S101 and S105 of the flowchart shown in Fig. 5.

[0080] In step S122, after the speech recognition is started, the control device 20 determines whether or not a keyword requiring reference information has been extracted (step S122). More specifically, the control device 20 determines whether or not a keyword requiring reference information has been extracted from the speech content by the extraction unit 26 based on the speech recognition result by the utterance recognition unit 21. If it is determined in step S122 that a keyword requiring reference information has been extracted (Yes in step S122), the control device 20 proceeds to step S123. If it is determined that a keyword requiring reference information has not been extracted (No in step S122), the control device 20 proceeds to step S126.

[0081] In step S123, the control device 20 acquires the reference information of the keyword requiring reference information extracted in step S122 and displays it in the first display format for a predetermined period of time. More specifically, the control device 20 acquires the reference information of the keyword requiring reference information using the information acquisition unit 27, and causes the display control unit 28 to display the extracted reference information on the display unit 12 in the first display format, i.e., a display format with relatively low visibility, for a predetermined period of time.

[0082] After displaying the reference information in the first display form in step S123, the control device 20 determines whether the user's gaze is directed toward the display unit 12 (step S124). The control device 20 determines whether the direction of the user's gaze is directed toward the display unit 12 using the gaze detection unit 24. If the control device 20 determines that the user's gaze is directed toward the display unit 12 (Yes in step S124), the process proceeds to step S125. If the control device 20 does not determine that the user's gaze is directed toward the display unit 12 (No in step S124), the process proceeds to step S126.

[0083] In step S125, the reference information displayed in the first display mode in step S123 is displayed in the second display mode for a predetermined period of time. In other words, in step S125, the display mode of the reference information displayed in the first display mode in step S123 is changed to the second display mode. That is, the display mode of the reference information displayed in a relatively low-visibility mode in step S123 is changed to a relatively high-visibility display mode. <Effects> As described above, in this embodiment, when a keyword requiring reference information is extracted, the reference information is displayed in a first display mode with relatively low visibility. When it is detected that the user's gaze is directed toward the display unit 12, the display mode of the reference information is changed to a second display mode with relatively high visibility. According to this embodiment, the user can clearly see the reference information of the keyword by looking at the display unit 12. This embodiment can appropriately support the user's understanding when needed. This embodiment displays reference information with high visibility for keywords for which the user does not need reference information, thereby preventing the user from becoming conscious of the reference information and disrupting the flow of communication in a meeting, etc. As such, according to this embodiment, reference information can be appropriately displayed based on keywords based on the content of the utterance.

[0084] The information display device according to the present disclosure may be implemented in various different forms other than the above-described embodiment.

[0085] The components of the illustrated information display device are conceptual and functional, and do not necessarily have to be physically configured as shown. In other words, the specific form of each device is not limited to that shown, and all or part of each device may be functionally or physically distributed or integrated in any unit depending on the processing load and usage status of each device.

[0086] The configuration of the information display device is realized, for example, as software, by a program loaded into memory. In the above embodiment, the functional blocks are described as being realized by the cooperation of these hardware and software. In other words, these functional blocks can be realized in various forms using only hardware, only software, or a combination of both.

[0087] The above-described components include those that can be easily imagined by a person skilled in the art and those that are substantially the same. Furthermore, the above-described configurations can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the configurations are possible within the scope of the gist of the present invention.

[0088] The gaze detection unit 24 in the above-described embodiment can achieve the same effect by detecting that the user has turned toward the display unit 12. Therefore, it can be replaced with a face direction detection unit that detects the direction of the user's face based on the image captured by the camera 14. The face direction detection unit detects that the user's face has turned toward the display unit 12 from the image of the user's face captured by the camera 14, based on the relative positional relationship between the position of the camera 14 and the display unit 12, using a known method. In this case, when the display control unit 28 detects that the user's face, detected by the face direction detection unit, has turned toward the display unit 12, it causes the display unit 12 to display reference information for keywords that require reference information extracted by the extraction unit 26 immediately before the detection. Alternatively, when the display control unit 28 detects that the user's face, detected by the face direction detection unit, has turned toward the display unit 12, it causes the display control unit 28 to change the display mode of the reference information displayed on the display unit 12. [Explanation of symbols]

[0089] 10 Information display device 11 Microphone 12 Display section 14 Camera 18 Keyword information storage unit 19 Communications Department 20 Control device (information display control device) 21 Speech Recognition Unit 24 Gaze detection unit 26 Extraction part 27 Information Acquisition Department 28 Display control unit 29 Communication control section

Claims

1. An information display control device that controls an information display device to display information on a display unit, a speech recognition unit that recognizes the speech of a speaker; a gaze detection unit that detects the direction of a gaze of a user of the information display device; an extraction unit that extracts keywords requiring reference information from the utterance recognized by the utterance recognition unit; an information acquisition unit that acquires the reference information for the keyword extracted by the extraction unit; a display control unit that displays the reference information acquired by the information acquisition unit on the display unit; Equipped with When the display control unit detects that the direction of the user's gaze detected by the gaze detection unit is directed toward the display unit or a predetermined range on the display unit in which the reference information is displayed, the display control unit causes the display unit to display reference information for a keyword that requires reference information extracted by the extraction unit immediately before the detection point in time. Information display control device.

2. When the display control unit detects that the gaze direction of the user detected by the gaze detection unit has changed from a state of being directed toward a speaker to the display unit or the predetermined range, the display control unit causes reference information for a keyword requiring reference information extracted by the extraction unit immediately before the detection time to be displayed on the display unit. The information display control device according to claim 1 .

3. An information display control device that controls an information display device to display information on a display unit, a speech recognition unit that recognizes the speech of a speaker; a gaze detection unit that detects the direction of a gaze of a user of the information display device; an extraction unit that extracts keywords requiring reference information from the utterance recognized by the utterance recognition unit; an information acquisition unit that acquires the reference information for the keyword extracted by the extraction unit; a display control unit that displays the reference information acquired by the information acquisition unit on the display unit; Equipped with the display control unit causes the display unit to display the reference information extracted by the extraction unit, and when it detects that the direction of the user's gaze detected by the gaze detection unit is directed toward the display unit or a predetermined range on the display unit in which the reference information is displayed, changes a display mode of the reference information being displayed. Information display control device.

4. An information display method executed by an information display device that displays information on a display unit, comprising: a speech recognition step of recognizing an utterance of a speaker; a gaze detection step of detecting a gaze direction of a user of the information display device; an extraction step of extracting keywords requiring reference information from the utterance recognized in the utterance recognition step; an information acquisition step of acquiring reference information for the keyword extracted in the extraction step; a display step of displaying the reference information acquired in the information acquisition step on a display unit; Including, In the display step, when it is detected that the direction of the user's gaze detected in the gaze detection step is directed toward the display unit or a predetermined range on the display unit in which the reference information is to be displayed, reference information for a keyword requiring reference information extracted in the extraction step immediately before the point in time of detection is displayed on the display unit. Information display method.

5. An information display method executed by an information display device that displays information on a display unit, comprising: a speech recognition step of recognizing an utterance of a speaker; a gaze detection step of detecting a gaze direction of a user of the information display device; an extraction step of extracting keywords requiring reference information from the utterance recognized in the utterance recognition step; an information acquisition step of acquiring reference information for the keyword extracted in the extraction step; a display step of displaying the reference information acquired in the information acquisition step on a display unit; Including, In the display step, the reference information extracted in the extraction step is displayed on the display unit, and when the gaze detection step detects that the direction of the user's gaze is directed toward the display unit or a predetermined range on the display unit where the reference information is displayed, a display mode of the displayed reference information is changed. Information display method.

Images