Graphic display control device, graphic display control method, and program
The graphic display control device facilitates easy identification and display of relevant graphics in graphic recordings by analyzing speaker utterances and movements, addressing the challenge of navigating graphical content.
Patent Information
- Application Number
- JP2022579221
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-02-03
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2041-02-03
AI Technical Summary
Existing graphic recording technologies lack support for easily identifying and highlighting graphics of interest or relevance to the speaker, making it difficult to read and navigate through graphical content.
A graphic display control device that acquires speaker utterances and movements, extracts relevant search information, selects corresponding graphics from a database, and outputs them based on a threshold and filter strength, allowing easy viewing of interest-related graphics.
Enables speakers to effortlessly identify and view graphics pertinent to their dialogue, enhancing readability and understanding of graphical content.
Smart Images

Figure 0007582342000001 
Figure 0007582342000002 
Figure 0007582342000003
Abstract
Description
[Technical field]
[0001] The present invention relates to techniques for controlling the presentation of graphics on a screen or display. [Background technology]
[0002] Currently, graphic recording technology, which graphically records the contents of lectures and multi-person conversations, is being introduced in many areas.
[0003] The graphics written in graphic recording are composed not only of words, but also of pictures, diagrams, and lines connecting them, and are characterized by their excellence in visualizing relationships that are difficult to convey with words alone. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2017-068742 A [Patent Document 2] JP 2017-004270 A Summary of the Invention [Problem to be solved by the invention]
[0005] However, graphic recording does not provide support for identifying parts that interest the speaker or that are relevant to the speaker, making it difficult to read.
[0006] On the other hand, in the case of documents such as minutes written in text, there are methods for searching for similar text, as disclosed in Patent Document 1. However, the technology disclosed in Patent Document 1 targets text information and cannot be applied to graphics.
[0007] The present invention has been made in consideration of the above points, and aims to provide a technique that enables a speaker to easily view graphics that interest him or her or that are related to him or her. [Means for solving the problem]
[0008] According to the disclosed technology, a data acquisition unit that acquires the content of a speaker's speech as text data; a search information extraction unit that extracts search information to be used for a search from the text data acquired by the data acquisition unit; a graphic selection unit that selects a graphic corresponding to the search information extracted by the search information extraction unit from a database; an output function unit that performs processing to present the graphic selected by the graphic selection unit on a graphic recording result; A graphic display control device, comprising: The graphic selection unit selects a graphic similar to the search information from the database using a threshold based on the information on the speaker's movements acquired by the data acquisition unit. A graphics display controller is provided. Effect of the Invention
[0009] The disclosed technology provides a technology that enables a speaker to easily view graphics that interest or are related to the speaker. [Brief description of the drawings]
[0010] [Figure 1] FIG. 1 is a system configuration diagram according to a first embodiment. [Diagram 2] 1 is a configuration diagram of a graphic display control device according to a first embodiment. [Diagram 3] 1 is a flowchart showing a process flow in the first embodiment. [Figure 4] FIG. 11 is a system configuration diagram according to a second embodiment. [Diagram 5] FIG. 11 is a configuration diagram of a graphic display control device according to a second embodiment. [Figure 6] 13 is a flowchart showing a process flow in the second embodiment. [Figure 7] FIG. 13 is a diagram showing an example of arranging a graphic in an image. [Figure 8] FIG. 13 is a diagram showing an example of arranging a graphic in an image. [Figure 9] FIG. 13 is a diagram showing an example of arranging a graphic in an image. [Figure 10] FIG. 2 illustrates an example of a hardware configuration of the apparatus. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.
[0012] (Outline of the embodiment) First, an outline of the present embodiment will be described. In this embodiment, a graphic display control device acquires information on the speech and actions of a person who uses graphics in graphic recording, and extracts information that is estimated to be of interest to that person.
[0013] Furthermore, the graphic display control device grasps the situation of the person when acquiring the speech and action information.The graphic display control device then selects a graphic to be shown to the person from this information and presents the selected graphic.
[0014] Since the graphic display control device performs the above-mentioned processing, the speaker can easily view parts of the graphics depicted in the graphic recording that are expected to be of interest to the speaker.
[0015] The graphic display control device can also adjust the way the graphics are output depending on the dialogue situation, making it possible to view the graphics in an easy-to-see manner according to the person's situation.
[0016] Hereinafter, examples of the system configuration and operation in this embodiment will be described in detail using examples 1 and 2.
[0017] Example 1 <System configuration> First, a first embodiment will be described. Fig. 1 is a configuration diagram of a graphic display control system in the first embodiment. In the first embodiment, it is assumed that one or more graphic recording results are displayed on a whiteboard, a screen, or the like, and speakers 5 and 6 are speaking in front of them. In the following description, unless otherwise specified, the speaker 5 will be the focus of attention.
[0018] As shown in Fig. 1, the graphic display control system of the first embodiment includes an imaging device 1, an utterance content acquisition device 2, a projector 3, and a graphic display control device 100. As shown in Fig. 1, the imaging device 1, the utterance content acquisition device 2, and the projector 3 are each connected to the graphic display control device 100.
[0019] The imaging device 1 is a device that captures the speaker 5. The imaging device 1 may be any device that can capture the shape of a person. For example, the imaging device 1 may be a color video camera, an infrared camera, a LiDAR that performs three-dimensional measurement, or the like.
[0020] The speech content acquirer 2 is a device for acquiring the speech content of the speaker 5. The speech content acquirer 2 is a device that, for example, inputs the speech of the speaker 5 from a microphone, transcribes the speech, and outputs text data.
[0021] In addition, when the speaker 5 leaves the contents of the utterance on paper, the utterance content acquisition device 2 may be a device that reads the contents by OCR or the like and outputs text data. In addition, when the speaker 5 inputs the contents of the utterance on a keyboard, the utterance content acquisition device 2 may be a keyboard.
[0022] Also, a part of the functions of the speech content acquisition device 2 described above may be implemented in the graphic display control device 100.
[0023] The projector 3 is a device that superimposes light on the graphic recording result. The projector 3 may be a projector or a movable light.
[0024] Also, regarding the projector 3, the graphic recording result may be projected as an image onto a screen or the like. Also, when the graphic recording result is displayed on a display, the projector 3 may not be provided. In this case, the light superimposition performed by the projector 3 can be reproduced on the display.
[0025] The graphic display control device 100 is a device implemented by a computer (such as a PC) and a program, for example. Also, the computer may be a virtual machine on the cloud. Also, the graphic display control device 100 may be implemented by one computer or a plurality of computers.
[0026] <Configuration of Graphic Display Control Device 100> FIG. 2 shows a functional configuration example of the graphic display control device 100. As shown in FIG. 2, the graphic display control device 100 includes a data acquisition unit 110, a search word extraction unit 120, an interaction situation grasping unit 130, a similar graphic selection unit 140, an output function unit 150, and a DB (database) 160. Each functional unit may be implemented by a different computer, or some of the functional units may be implemented on the cloud. Note that the search word extraction unit 120, the interaction situation grasping unit 130, and the similar graphic selection unit 140 may be called a search information extraction unit, a filter strength setting unit, and a graphic selection unit, respectively.
[0027] <Regarding the Information Held in DB160> In DB160, image information obtained by clustering graphics (image information) in advance into small groups and text data related to the information are stored. This small group can be set arbitrarily, and the same graphic may be stored as a plurality of pieces of image information in different groups.
[0028] Note that the above is an example, and the form may be any as long as the graphic (image information) and the text data are in a set.
[0029] In the first embodiment, it is assumed that the DB160 stores graphics based on the graphic recording results. For each graphic stored in the DB160, the DB160 has position information indicating where in the entire graphic recording result the graphic is drawn. The position information of a certain graphic may hold, for example, the centroid coordinates as the coordinates of a point in the graphic with a certain point in the entire graphic recording result as the origin. Alternatively, the entire graphic recording result may be regarded as a rectangular image, and pixel information including the graphic may be held.
[0030] <Operation example of the graphic display control system> Hereinafter, an operation example of the graphic display control system in the first embodiment will be described according to the procedure of the flowchart in FIG. 3.
[0031] Suppose that the speaker 5 is having a conversation with the speaker 6 while looking at the graphic recording result. In this situation, each process described below is performed in the graphic display control system.
[0032] <S101: Data acquisition process> The imaging device 1 acquires the video of the speaker 5, and recognizes the appearance when the speaker 5 enters the field of view and the disappearance when the speaker 5 exits the field of view. Regarding the appearance and disappearance of the speaker 5, for example, it can be recognized using openpose, which is open source. In addition to, or instead of, the appearance and disappearance, the imaging device 1 may recognize gestures such as the pointing motion of the speaker 5. Note that the actions recognized by the imaging device 1 are not limited to those described above, and actions other than those described above may also be recognized.
[0033] The imaging device 1 transmits the time code (time information) indicating the occurrence time of the acquired action and the ID for distinguishing the speaker when there are multiple speakers to the data acquisition unit 110 in the graphic display control device 100 as information attached to the action of the corresponding speaker.
[0034] In addition, the speech content acquisition device 2 acquires the speech content of the speaker 5, attaches the time information when the original speech was made, and transmits the text data representing the speech content to the data acquisition unit 110 in the graphic display control device 100.
[0035] The data acquisition unit 110 receives the text data with time information from the speech content acquisition device 2 and transmits the received text data to the search word extraction unit 120. In addition, the data acquisition unit 110 receives the information on the action of the speaker 5 with time information from the imaging device 1 and transmits the information on the action to the dialogue situation grasping unit 130.
[0036] <S102: Search Word Extraction Process> The search word extraction unit 120 extracts the information used for searching for graphics from the text data received from the data acquisition unit 110. The method of extracting information from the text data is not limited to a specific method, and various methods can be used.
[0037] For example, the search word extraction unit 120 may use existing technology to summarize the text data received from the data acquisition unit 110, divide the summarized document into sentences, and use the divided sentences as information for searching.
[0038] The search keyword extraction unit 120 transmits each of the sentences obtained as described above, together with the time information of the original text data, to the similar graphic selection unit 140.
[0039] In addition to the method as described above, the search keyword extraction unit 120 may perform morphological analysis on the text data received from the data acquisition unit 110, and utilize, as information for searching, words that appear n or more times within a certain period (here, within t1 seconds). It is assumed that t1 and n can be set to arbitrary values. Also, it is assumed that one or more arbitrary parts of speech can be set for the parts of speech of the words to be counted for n.
[0040] When one or more words as information for searching for graphics are extracted from the text data, for the one or more words, information numbering the frequency in descending order of the number of occurrences is assigned, and it is transmitted to the similar graphic selection unit 140 together with the time information of the original text data.
[0041] <S103: Dialogue situation grasping process> The dialogue situation grasping unit 130 receives, from the data acquisition unit 110, information on operations with time information attached, and sets the value of the filter strength variable corresponding to the operation. The value of the filter strength variable is a value representing the filter strength, and the value of the filter strength variable may also be referred to as the filter strength. The filter strength indicates the degree of selecting a specific graphic (that is, the degree of filtering other graphics) in the similar graphic selection process described later.
[0042] The value of the filter strength variable may be set for each operation, or different values may be set for each time-series change after the operation is performed. Regarding which filter strength to set for which operation / time-series change, for example, it may be determined in advance in a table or the like, and the determination may be made by referring to that table.
[0043] When setting for each action, for example, for two actions, "pointing action" and "appearing", set F1 for "pointing action" and set F2 for "appearing". F1 means that the filter strength is higher (stronger) than F2.
[0044] In the above example, when a pointing motion is performed, it is considered that the attention is high, and the filter strength is set to select a graphic with a high similarity, and when only the appearance is performed, it is considered that the attention is low, and the filter strength is set to select a graphic with a low similarity. Specifically, for example, F1 and F2 are set so that the threshold value α(F1) described later is 0.9 and the threshold value α(F2) is 0.7.
[0045] When setting the filter strength for each time series change, for example, the time since the appearance of each speaker is measured, and if the time is less than T1, the filter strength is set to F1, and if the time is T1 or more, the filter strength is set to F2.
[0046] Furthermore, the dialogue situation grasping unit 130 may set an arbitrary filter strength value based on a combination of an action and time information of the action.
[0047] In addition, the dialogue situation grasping unit 130 may receive the time since the appearance of the speaker 5 and / or the value of a filter strength variable corresponding to the speaker's action, and may use the output value by solving a predetermined function based on this information as the value of the filter strength variable.
[0048] Also, when there are multiple speakers, a new filter strength variable value may be set for each speaker from a combination of filter strength variable values set for each speaker based on the speaker's actions and time-series changes. Also, when there are multiple speakers, the filter strength variable value corresponding to the time since the appearance of each speaker and / or the action of each speaker may be received, and a value output by solving a predetermined function based on this information may be set as the filter strength variable value for each speaker.
[0049] The dialogue situation grasping unit 130 transmits the value of the filter strength variable corresponding to the operation of the speaker 5 to each of the similar graphic selection unit 140 and the output function unit 150.
[0050] <S104: Similar Graphic Selection Process> The similar graphic selection unit 140 receives information for searching for a graphic from the search word extraction unit 120, and saves the received information as graphic search information together with the time information attached to the original text data. The time information here may be the time when the graphic search information is received.
[0051] Furthermore, the similar graphic selection unit 140 receives the value of the filter strength variable from the dialogue situation grasping unit 130 and saves it together with the time information of the original operation. The time information here may be the time when the value of the filter strength variable is received.
[0052] When the similar graphic selection unit 140 receives either the graphic search information or the value of the filter strength variable, it executes the following process.
[0053] The similar graphic selection unit 140 checks the time information of the latest graphic search information currently held and the time information of the latest value of the filter strength variable currently held. When the time difference is T or less, the following process is performed using the latest graphic search information and the latest value of the filter strength variable. Note that T is a preset time value.
[0054] The similar graphic selection unit 140 uses the graphic search information to select a graphic corresponding to the graphic search information from the DB160.
[0055] As described above, the graphic search information is composed of text data such as sentences and words, and the graphics stored in the DB160 are also stored together with the text data related to the graphics.
[0056] The similar graphic selection unit 140 selects a graphic having text data similar to the graphic search information. The method for selecting a graphic having text data similar to the graphic search information is not limited to a specific method. For example, the method described in Patent Document 1 may be used.
[0057] That is, the similar graphic selection unit 140 obtains a similarity score between the text data that is the graphic search information and each text data associated with the graphic stored in the DB 160, and for example, selects one or more graphics corresponding to one or more text data whose similarity score is higher than a threshold value.
[0058] Regarding the threshold value for comparison with the similarity score obtained by the method described in Patent Document 1, a threshold value α(n) may be set for each value n of the filter strength variable. In that case, the similar graphic selection unit 140 uses, for example, the threshold value α(F) corresponding to the value F of the filter strength variable obtained from the operation corresponding to the graphic search information to be used, and selects one or more graphics corresponding to text data whose similarity score with the graphic search information is greater than the threshold value α(F).
[0059] The similar graphic selection unit 140 transmits the selected one or more graphics to the output function unit 150. When a plurality of graphics are obtained, all the graphics are transmitted to the output function unit 150. When transmitting this graphic, the similar graphic selection unit 140 may also transmit the information used for the search (for example, the text data transmitted from the data acquisition unit 110 or the graphic search information) together.
[0060] <S105: Output process> The output function unit 150 receives the value of the filter strength variable from the dialogue situation grasping unit 130. Furthermore, the output function unit 150 receives one or more pieces of graphic information from the similar graphic selecting unit 140. Note that it is assumed here that the output function unit 150 receives the value of the filter strength variable corresponding to the action of the speaker 5, and also receives one or more pieces of graphic information obtained from text search information related to the action (for example, within a time difference of T).
[0061] Based on the received graphic information, the output function unit 150 processes the graphic to make it stand out in the graphic recording result, and outputs the processed graphic to the projector 3 or the like.
[0062] Specifically, for example, the output function unit 150 may project light toward the location of the position information in the graphic recording result presented on a whiteboard, etc., based on the position information of the graphic to be highlighted. Also, the output function unit 150 may use image information of the graphic to be highlighted to project an image from the projector 3 onto the graphic recording result so as to highlight the outline of the graphic.
[0063] When the result of graphic recording is an image projected from the spotlight 3 and displayed on a screen, or when the result of graphic recording is an image displayed on a display, the output function unit 150 may convert the graphic to be highlighted into text data of the information used in the search, and display the text data as the words spoken by the speaker 5 on the screen or display, and then superimpose the graphic to be highlighted on the text data to display it prominently.
[0064] The above-mentioned method of displaying the graphic and text data is merely an example, and is not limited to the above. The method of displaying the graphic and text data may be different for each speaker, or may change over time.
[0065] <Effects of Example 1> According to the first embodiment, it becomes possible for a speaker to easily find a part that is of interest or relevant to him / her among the graphics depicted in the graphic recording.
[0066] Example 2 Next, a description will be given of Example 2. Example 2 may be implemented alone or in combination with Example 1.
[0067] In addition, Example 2 may assume that parts of the graphics drawn in graphic recording that are of interest to the speaker are selected and arranged in chronological order for easy viewing, or that graphics related to the speaker are arranged in chronological order for easy viewing, without assuming that graphic recording will be performed.
[0068] <System configuration> 4 is a configuration diagram of a graphic display control system in the second embodiment. In the second embodiment, the result of graphic recording may or may not be displayed on the display 17. Here, it is assumed that speakers 15 and 16 are speaking in front of the display 17. In the following description, unless otherwise specified, the speaker 15 is the focus of attention.
[0069] As shown in Fig. 4, the graphic display control system of the second embodiment includes an imaging device 11, an utterance content acquisition device 12, a projector 13, a display 17, and a graphic display control device 200. As shown in Fig. 4, the imaging device 11, the utterance content acquisition device 12, the projector 13, and the display 17 are each connected to the graphic display control device 200. Note that, regarding the projector 13 and the display 17, only one of them may be provided, or both may be provided.
[0070] The imaging device 11 is a device that captures the speaker 15. The imaging device 11 may be any device that can capture the shape of a person. For example, the imaging device 11 may be a color video camera, an infrared camera, a LiDAR that performs three-dimensional measurement, or the like.
[0071] The speech content acquirer 12 is a device for acquiring the speech content of the speaker 15. The speech content acquirer 12 is a device that inputs the speech of the speaker 15 from a microphone, transcribes the speech, and outputs text data.
[0072] In addition, when the speaker 15 leaves the contents of the utterance on paper, the utterance content acquisition device 12 may be a device that reads the contents by OCR or the like and outputs text data. In addition, when the speaker 15 inputs the contents of the utterance on a keyboard, the utterance content acquisition device 12 may be a keyboard.
[0073] Further, a part of the functions of the above-mentioned speech content acquirer 12 may be implemented in the graphic display control device 200 .
[0074] Both the light projector 13 and the display 16 display the reconstructed image of the graphic recording. The light projector 13 may be a projector or a movable light.
[0075] The graphic display control device 200 is a device implemented by a program, for example, a computer (such as a PC). Further, the computer may be a virtual machine on the cloud. Also, the graphic display control device 200 may be implemented by one computer or by a plurality of computers.
[0076] <Configuration of the Graphic Display Control Device 200> FIG. 5 shows a functional configuration example of the graphic display control device 200. As shown in FIG. 5, the graphic display control device 200 includes a data acquisition unit 210, a search word extraction unit 220, an interaction situation grasping unit 230, a similar graphic selection unit 240, an output function unit 250, and a DB (database) 260. The output function unit 250 includes a graphic reconstruction unit 255. Each functional unit may be implemented by a different computer, or some of the functional units may be implemented on the cloud. Note that the search word extraction unit 220, the interaction situation grasping unit 230, and the similar graphic selection unit 240 may be referred to as a search information extraction unit, a filter strength setting unit, and a graphic selection unit, respectively.
[0077] <Regarding the Information Stored in the DB 260> In the DB 260, image information obtained by clustering graphics (image information) in advance into small groups and text data related to the information are stored. This small group can be set arbitrarily, and the same graphic may be stored as a plurality of pieces of image information in different groups.
[0078] Note that the above is an example, and the form may be any as long as the graphic (image information) and the text data are in a set.
[0079] In Example 2, it is not necessary to store information about the graphic recording result in DB260. When storing information about the graphic recording result, for example, similar to Example 1, for each graphic stored in DB260, DB260 has position information indicating where in the entire drawn graphic recording result the graphic is located. The position information of a certain graphic may, for example, use a certain point in the entire graphic recording result as the origin and hold, as the coordinates of the points in that graphic, for example, the centroid coordinates. Also, the entire graphic recording result may be regarded as a rectangular image, and the pixel information including the graphic may be held.
[0080] <Operation Example of Graphic Display Control System> In Example 2, it is assumed that the speaker 15 is having a conversation with the speaker 16. The graphic display control system in Example 2 executes operations in accordance with the procedure of the flowchart in FIG. 6.
[0081] The imaging device 11, the utterance content acquisition device 12, the data acquisition unit 210, the search word extraction unit 220, the dialogue situation grasping unit 230, the similar graphic selection unit 240, and DB260 in Example 2 perform the same operations as the imaging device 1, the utterance content acquisition device 2, the data acquisition unit 110, the search word extraction unit 120, the dialogue situation grasping unit 130, the similar graphic selection unit 140, and DB160 in Example 1.
[0082] That is, the operations of S201 to S204 in FIG. 6 are the same as the operations of S101 to S104 shown in FIG. 3 in Example 1. Hereinafter, the operation of S205 in Example 2 will be described.
[0083] <S205: Reconfiguration / Output Processing> The output function unit 250 receives the value of the filter strength variable from the dialogue situation grasping unit 230. Furthermore, the output function unit 250 receives one or more pieces of graphic information from the similar graphic selecting unit 240. Note that, here, it is assumed that the output function unit 250 receives the value of the filter strength variable corresponding to the action of the speaker 15, and receives one or more pieces of graphic information obtained from text search information related to the action (for example, within a time difference of T). The same is true when multiple speakers are the target, and the output function unit 250 receives, for each speaker, the value of the filter strength variable corresponding to the action, and receives one or more pieces of graphic information obtained from text search information related to the action (for example, within a time difference of T).
[0084] The output function unit 250 passes the received filter strength variable value and graphic information to the graphic reconstructor 255. The graphic reconstructor 255 reconstructs the graphic based on the filter strength variable value and the graphic information, and outputs the reconstructed graphic via the output function unit 250.
[0085] More specifically, the graphic reconstruction unit 255 arranges the graphics sent to the output function unit 255, for example, in chronological order, on a rectangular image that can be projected on a screen by the projector 13, or on a rectangular image that can be displayed on the display 16. Here, the chronological order may be determined, for example, from the time information of the graphic search information that is the basis for selecting the graphic, or may be determined from the time when the output function unit 250 received the graphic information.
[0086] The rectangles are arranged on the image in a preset manner, and the setting method is arbitrary.
[0087] As an example, the graphic reconstruction unit 255 arranges the images from left to right and top to bottom in chronological order of the time when the output function unit 250 received the graphic information. An example of an arrangement in which the images are arranged from left to right and top to bottom is shown in Fig. 7. In Fig. 7, t1, t2, ... represent the proceeding time. Although t1, t2, ... are not written on the image, they are written in Fig. 7 for the purpose of explanation.
[0088] The graphic reconstructing unit 255 may also arrange the graphics from the left edge downward in the chronological order of the time when the output function unit 250 received the graphic information, and when it reaches the bottom of the image, arrange the graphics from the left edge of the remaining space downward again. An example of this case is shown in FIG.
[0089] The image on which the graphic is arranged is projected from the output function unit 250 via the projector 13, or displayed on the display 17. This image may also be newly generated every time the output function unit 250 receives graphic information.
[0090] Furthermore, with regard to the arrangement of graphics on an image, the graphics reconstructor 255 may group the graphics according to the values of the filter strength variables corresponding to the graphics, and arrange one or more of the grouped graphics.
[0091] The grouping method is not limited to a specific method, but for example, a group of one or more graphics corresponding to the same filter strength variable value may be arranged in chronological order of the time when the graphic information was received. As an example, as shown in Fig. 9, a graphic with a high filter strength may be arranged in chronological order in the center of the image, and graphics with a low filter strength may be arranged around it.
[0092] Furthermore, the graphic reconstructor 255 may color-code each graphic having the same filter strength variable value. In this case, the color may be preset for each filter strength variable value, or may be selected randomly.
[0093] Furthermore, the graphic reconstruction unit 255 may erase a graphic having a certain filter strength variable value from the image when a predetermined time T2 has elapsed since the graphic was projected or displayed by the output function unit 250. The same value for all filter strength variable values may be used for this T2, or a different value may be set for each filter strength variable value. As an example, in the case of setting T2 such that the higher the filter strength, the longer the graphic is displayed, the graphics displayed as shown in FIG. 7 will be erased as time passes, starting with the graphics with the lowest filter strength.
[0094] <Effects of Example 2> According to the second embodiment, the graphics can be displayed in an easy-to-view manner according to the speaker's situation.
[0095] (Hardware configuration example) As described above, the graphic display control devices 100 and 200 described in the first and second embodiments can be realized by, for example, executing a program on one or more computers. The computers may be physical computers or virtual machines.
[0096] That is, the graphic display control devices 100 and 200 can be realized by using hardware resources such as a CPU and memory built into a computer to execute a program corresponding to the processing performed by the graphic display control devices 100 and 200. The above program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The above program can also be provided via a network such as the Internet or email.
[0097] Fig. 10 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 10 has a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, and an input device 1007, which are all connected to each other via a bus B.
[0098] The program for realizing the processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.
[0099] The memory device 1003 reads out and stores the program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the graphic display control devices 100, 200 in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, and functions as an input means and an output means via the network. The display device 1006 displays a GUI (Graphical User Interface) or the like according to a program. The input device 1007 is composed of a keyboard and mouse, buttons, a touch panel, or the like, and is used to input various operation instructions. The output device 1008 outputs the results of calculations.
[0100] (Summary of the embodiment) This specification discloses at least the graphic display control device, the graphic display control method, and the program described in the following sections. (Section 1) A data acquisition unit that acquires the content of a speaker's speech as text data; a search information extraction unit that extracts search information to be used for a search from the text data acquired by the data acquisition unit; a graphic selection unit that selects a graphic corresponding to the search information extracted by the search information extraction unit from a database; an output function unit that performs processing for presenting the graphic selected by the graphic selection unit on a graphic recording result; A graphic display control device comprising: (Section 2) A data acquisition unit that acquires the content of a speaker's speech as text data; a search information extraction unit that extracts search information to be used for a search from the text data acquired by the data acquisition unit; a graphic selection unit that selects a graphic corresponding to the search information extracted by the search information extraction unit from a database; an output function unit that performs processing for displaying the graphics selected by the graphic selection unit in chronological order; A graphic display control device comprising: (Section 3) a filter strength setting unit that sets a filter strength based on the information on the movement of the speaker acquired by the data acquisition unit, The graphic selection unit selects a graphic similar to the search information from the database using a threshold value corresponding to the filter strength set by the filter strength setting unit. 3. A graphic display control device according to claim 1 or 2. (Section 4) The output function unit groups the graphics based on the filter strength. A graphic display control device according to claim 3 which is dependent on claim 2. (Section 5) The output function unit determines the time from when a graphic is displayed until when it is erased based on the filter strength. A graphic display control device according to claim 3 which is dependent on claim 2. (Section 6) A graphic display control method executed by a graphic display control device, comprising: A data acquisition step of acquiring the content of a speaker's speech as text data; a search information extraction step of extracting search information to be used for a search from the text data acquired by the data acquisition step; a graphic selection step of selecting, from a database, a graphic corresponding to the search information extracted by the search information extraction step; an output step of performing processing for presenting the graphic selected by the graphic selection step on a graphic recording result; A graphic display control method comprising: (Section 7) A graphic display control method executed by a graphic display control device, comprising: A data acquisition step of acquiring the content of a speaker's speech as text data; a search information extraction step of extracting search information to be used for a search from the text data acquired by the data acquisition step; a graphic selection step of selecting, from a database, a graphic corresponding to the search information extracted by the search information extraction step; an output step of performing processing for displaying the graphics selected by the graphic selection step in chronological order; A graphic display control method comprising: (Section 8) A program for causing a computer to function as each unit in the graphic display control device described in any one of claims 1 to 5.
[0101] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]
[0102] 1 Camera 2 Speech content acquisition device 3. Floodlight 5, 6 Speaker 100 Graphic display control device 110 Data Acquisition Section 120 Search word extraction section 130 Dialogue situation grasping section 140 Similar Graphics Selection Section 150 Output function section 160 DB 11 Camera 12 Speech content acquisition device 13 Floodlight 15, 16 Speaker 17 Display 200 Graphic display control device 210 Data Acquisition Section 220 Search word extraction section 230 Dialogue Situation Grasping Department 240 Similar Graphics Selection Section 250 Output function section 255 Graphics Reconstruction Unit 260 DB 1000 Drive Device 1001 Recording media 1002 Auxiliary storage 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device 1008 Output device
Claims
1. A data acquisition unit that acquires the content of a speaker's speech as text data; a search information extraction unit that extracts search information to be used for a search from the text data acquired by the data acquisition unit; a graphic selection unit that selects, from a database, a graphic corresponding to the search information extracted by the search information extraction unit; an output function unit that performs processing to present the graphic selected by the graphic selection unit on a graphic recording result, The graphic selection unit selects a graphic similar to the search information from the database using a threshold based on the information on the speaker's movements acquired by the data acquisition unit. Graphic display control device.
2. A data acquisition unit that acquires the content of a speaker's speech as text data; a search information extraction unit that extracts search information to be used for a search from the text data acquired by the data acquisition unit; a graphic selection unit that selects a graphic corresponding to the search information extracted by the search information extraction unit from a database; an output function unit that performs processing to display the graphics selected by the graphic selection unit in chronological order, The graphic selection unit selects a graphic similar to the search information from the database using a threshold based on the information on the speaker's movements acquired by the data acquisition unit. Graphic display control device.
3. a filter strength setting unit that sets a filter strength based on the information on the movement of the speaker acquired by the data acquisition unit, The graphic selection unit selects a graphic similar to the search information from the database using the threshold value corresponding to the filter strength set by the filter strength setting unit.
3. The graphic display control device according to claim 1 or 2.
4. The output function unit groups the graphics based on the filter strength. A graphic display control device according to claim 3 dependent on claim 2.
5. The output function unit determines the time from when a graphic is displayed until when it is erased based on the filter strength. A graphic display control device according to claim 3 dependent on claim 2.
6. A graphic display control method executed by a graphic display control device, comprising: A data acquisition step of acquiring the content of a speaker's speech as text data; a search information extraction step of extracting search information to be used for a search from the text data acquired by the data acquisition step; a graphic selection step of selecting, from a database, a graphic corresponding to the search information extracted by the search information extraction step; an output step of performing processing for presenting the graphic selected in the graphic selection step on a graphic recording result, In the graphic selection step, a graphic similar to the search information is selected from the database using a threshold based on the information on the speaker's movements acquired in the data acquisition step. Graphic display control method.
7. A graphic display control method executed by a graphic display control device, comprising: A data acquisition step of acquiring the content of a speaker's speech as text data; a search information extraction step of extracting search information to be used for a search from the text data acquired by the data acquisition step; a graphic selection step of selecting, from a database, a graphic corresponding to the search information extracted by the search information extraction step; an output step of performing processing for displaying the graphics selected in the graphic selection step in a chronological order, In the graphic selection step, a graphic similar to the search information is selected from the database using a threshold based on the information on the speaker's movements acquired in the data acquisition step. Graphic display control method.
8. A program for causing a computer to function as each unit in the graphic display control device according to any one of claims 1 to 5.
Citation Information
Patent Citations
Information processor, information processing method and program
JP2007281618A
Conference support system and conference support method
JP2017004270A
Relevant document retrieval device, model creation device, method and program therefor
JP2017068742A
Information processing apparatus, conference support system, conference support method, and image forming apparatus
JP2019095902A
Information processing device and information processing program
JP2019185676A
