Information processing device, terminal, information processing system, information processing method, and program

Abstract

To streamline the acquisition of information related to the information presented. [Solution] The information processing device includes: a similarity determination unit that determines whether there is similarity between first text data obtained by applying speech recognition to voice input from a terminal and second text data obtained by applying character recognition to drawing data handwritten into the terminal; a search unit that, if similarity is determined to exist, searches a storage unit for information related to the first text data and the second text data; and an output unit that outputs the searched information to the terminal.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a terminal, an information processing system, an information processing method, and a program.

Background Art

[0002] During a meeting using an electronic whiteboard, there are cases where a user stands in front of the electronic whiteboard to explain materials displayed on the electronic whiteboard, or manually writes information related to the materials on the electronic whiteboard. There are also cases where, in a corporate training room or a school classroom, an instructor or a teacher stands in front of an electronic whiteboard to give an explanation while displaying teaching materials on the electronic whiteboard, or manually writes down and explains information related to the teaching materials.

[0003] Note that Patent Document 1 discloses a matter in which an AI analyzes utterance content using natural language processing to automatically present related information.

[0004] Further, Patent Document 2 discloses a matter of extracting a keyword from audio data and displaying a keyword related to the keyword.

Summary of Invention

Problem to be Solved by the Invention

[0005] In the scenes as described above, a user standing in front of the electronic whiteboard wants to display and explain related information of the materials or teaching materials presented on the electronic whiteboard. However, if the user does not remember the storage destination of the related documents, there is a problem that it takes time to find and display the related documents.

[0006] Further, when a user standing in front of the electronic whiteboard wants to verbally explain the related information of the materials or teaching materials presented on the electronic whiteboard, and does not accurately remember the content of the related information, the user will search and check the related information on the Web or the like. In this case, there is a problem that it takes time to find and display the related information.

[0007] This invention has been made in view of the above points, and aims to improve the efficiency of acquiring information related to the presented information. [Means for solving the problem]

[0008] To solve the above problems, the information processing device includes: a similarity determination unit that determines whether there is similarity between first text data obtained by applying speech recognition to voice input from a terminal and second text data obtained by applying character recognition to drawing data handwritten into the terminal; a search unit that, if similarity is determined to exist, searches for information related to the first text data and the second text data from a storage unit; and an output unit that outputs the searched information to the terminal. [Effects of the Invention]

[0009] This allows for more efficient acquisition of information related to the presented information. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of the system configuration in the first embodiment. [Figure 2] This figure shows an example of the hardware configuration of the support server 1 in the first embodiment. [Figure 3] This figure shows an example of the overall hardware configuration of the electronic whiteboard 2 in the first embodiment. [Figure 4] This figure shows an example of the hardware configuration of the touch panel 31. [Figure 5] This figure shows an example of the hardware configuration of controller 32. [Figure 6] This figure shows an example of the functional configuration of the support server 1 in the first embodiment. [Figure 7] This figure shows an example of the functional configuration of the electronic whiteboard 2 in the first embodiment. [Figure 8] This figure shows the content drawn by the user by hand in the first embodiment. [Figure 9] This is a sequence diagram for describing an example of a processing procedure executed in the first embodiment. [Figure 10] This is a sequence diagram for describing an example of a processing procedure executed in the first embodiment. [Figure 11] This is a flowchart for describing an example of a processing procedure executed by the support server 1 in the first embodiment. [Figure 12] This is a diagram showing a display example of related information in the first embodiment. [Figure 13] This is a flowchart for describing an example of a processing procedure executed by the support server 1 in the second embodiment. [Figure 14] This is a diagram showing content hand-drawn by a user in the second embodiment. [Figure 15] This is a diagram showing a display example of related information in the second embodiment. [Figure 16] This is a diagram showing a system configuration example in the third embodiment. [Figure 17] This is a diagram showing a functional configuration example of the support server 1 in the third embodiment. [Figure 18] This is a diagram showing a functional configuration example of the electronic whiteboard 2 in the third embodiment. [Figure 19] This is a sequence diagram for describing an example of a processing procedure executed in the third embodiment. [Figure 20] This is a flowchart for describing an example of a processing procedure executed by the support server 1 in the third embodiment. [Figure 21] This is a diagram showing a display example of related information in the third embodiment. [Figure 22] This is a diagram showing a functional configuration example of the support server 1 in the fourth embodiment. [Figure 23] This is a flowchart for describing an example of a processing procedure executed by the support server 1 in the fourth embodiment. [Figure 24]This is a diagram showing a display example of related information in the fourth embodiment. [Figure 25] This is a diagram showing a functional configuration example of the electronic blackboard 2 in the fifth embodiment. [Figure 26] This is a flowchart for explaining an example of the processing procedure executed by the electronic blackboard 2 in the fifth embodiment. [Figure 27] This is a diagram showing a functional configuration example of the support server 1 in the sixth embodiment. [Figure 28] This is a diagram showing a functional configuration example of the electronic blackboard 2 in the sixth embodiment. [Figure 29] This is a flowchart for explaining an example of the processing procedure executed by the electronic blackboard 2 in the sixth embodiment. [Modes for Carrying Out the Invention]

[0011] Hereinafter, embodiments of the present invention will be described based on the drawings. Figure 1 is a diagram showing a system configuration example in the first embodiment. In Figure 1, the electronic blackboard 2 is connected to the support server 1 via a network such as a LAN (Local Area Network) or the Internet.

[0012] The support server 1 is a general-purpose computer (information processing apparatus).

[0013] The electronic blackboard 2 is an apparatus that includes a touch panel, supports handwritten input and screen operations using a pen or a finger, and is further equipped with a microphone to display meeting-related information by utilizing input audio. In the present embodiment, the electronic blackboard 2 is an example of a terminal [SA1] and an information processing apparatus.

[0014] Figure 2 is a diagram showing a hardware configuration example of the support server 1 in the first embodiment. The support server 1 in Figure 2 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, and an interface device 105, which are interconnected via a bus B, etc.

[0015] The program that enables processing on support server 1 is provided on a recording medium 101 such as a CD-ROM. When the recording medium 101 containing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101; it may also be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files and data.

[0016] The memory device 103 reads and stores a program from the auxiliary storage device 102 when a program startup command is received. The processor 104 is either a CPU or a GPU (Graphics Processing Unit), or both a CPU and a GPU, and executes functions related to the support server 1 according to the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

[0017] Figure 3 shows an example of the overall hardware configuration of the electronic whiteboard 2 in the first embodiment. The electronic whiteboard 2 shown in Figure 3 includes a display 30, a touch panel 31, and a controller 32.

[0018] The display 30 is a large-screen display, and for example, an LCD is used. The touch panel 31 has an infrared-emitting LED and a phototransistor facing each other, and determines that the touched position is the area where the phototransistor cannot detect light, that is, the area where the light is blocked.

[0019] Figure 4 shows an example of the hardware configuration of the touch panel 31. As shown in Figure 4, the touch panel 31 has 20 LEDs labeled LV1 to LV20 in the width direction (horizontal direction) and 15 LEDs labeled LH1 to LH15 in the height direction (vertical direction). Opposite LV1 to LV20 are 20 phototransistors labeled PV1 to PV20, and opposite LH1 to LH15 are 15 transistors labeled PH1 to PH15. The LEDs and phototransistors in a row are arranged at equal intervals. Note that if the touch panel 31 is larger than 40 inches, more LEDs and phototransistors will actually be required.

[0020] Figure 5 shows an example of the hardware configuration of the controller 32. As shown in Figure 5, the controller 32 includes a CPU 40, main memory 41, ROM 42, HD controller 43, hard disk 44, touch panel I / F 45, display controller 46, audio codec 47, microphone 48, LAN controller 49, LAN I / F 50, and bus 51.

[0021] The CPU 40 executes and processes control processing programs stored in the ROM 42, the OS (Operating System) and various application programs read from the hard disk 44 into the main memory 41. The main memory 41 is composed of DRAM (Dynamic Random Access Memory) and is used as the CPU 40's work area, etc. The ROM 42 has programs pre-written for system startup when the power is turned on and for controlling various devices. The hard disk 44 stores system software, various application programs, user-saved data, etc. The HD controller 43 has an interface with the hard disk 44, such as an IDE (Integrated Device Electronics) interface, and performs high-speed data transfer with the hard disk 44. The touch panel I / F 45 has a port for the touch panel and is controlled by a touch panel driver (control program). The display controller 46 performs D / A (Digital / Analog) conversion of character data, graphic data, etc., and controls the display of this data on the display 30. The audio codec 47 converts the audio signal input from the microphone 48 into audio data using an A / D conversion circuit (not shown in the diagram), and encodes this audio data using the AAC (Advanced Audio Coding) method. The LAN controller 49 executes a communication protocol compliant with, for example, the IEEE (Institute of Electrical and Electronics Engineers) 802.3 standard to control communication with other devices connected to Ethernet via the LAN I / F 61. Data is exchanged between each of the above components via the bus 51.

[0022] In this embodiment, unless otherwise specified, the CPU 40 uses the main memory 41 as a work area and executes processing according to the program.

[0023] Figure 6 shows an example of the functional configuration of the support server 1 in the first embodiment. As shown in Figure 6, the support server 1 includes a communication control unit 10, a speech recognition unit 11, an utterance segmentation determination unit 12, an utterance text generation unit 13, a drawing data discrimination unit 14, a keyword extraction unit 15, an image object recognition unit 16, a similarity determination unit 17, a text information retrieval unit 18, a similar image retrieval unit 19, a related information output unit 20, and an information storage unit 21. Each of these units is realized by processing that one or more programs installed on the support server 1 cause the processor 104 to execute. The information storage unit 21 can be realized using, for example, an auxiliary storage device 102, or a storage device that can be connected to the support server 1 via a network.

[0024] The communication control unit 10 performs and controls communication with the electronic whiteboard 2 via the network.

[0025] The speech recognition unit 11 decodes the encoded speech data received from the electronic whiteboard 2 and outputs text data corresponding to the speech data through speech recognition.

[0026] The speech segmentation determination unit 12 determines the end of speech uttered by the user of the electronic whiteboard 2. Specifically, the speech segmentation determination unit 12 determines that speech has been interrupted if, after receiving audio data from the electronic whiteboard 2, no further audio data is received for a predetermined period of time (silence). The audio of the speech is input by the electronic whiteboard 2. Therefore, silence corresponds to a state in which the display terminal (electronic whiteboard 2) does not input audio for a predetermined period of time.

[0027] The speech text generation unit 13 generates text based on the text data output from the speech recognition unit 11.

[0028] The drawing data discrimination unit 14 determines whether the drawing data received from the electronic whiteboard 2 is a character or shape (circle, square, etc.) or any other type of figure.

[0029] The keyword extraction unit 15 extracts keywords from the text generated by the speech text generation unit 13. The keyword extraction unit 15 also extracts keywords from text data converted from handwritten characters by character recognition.

[0030] The image object recognition unit 16 determines the character area in the screen data received from the electronic whiteboard 2, identifies the area outside the character area as an image area, and identifies objects within that image area. The image object recognition unit 16 extracts the feature data of the object and converts the feature data into vector data.

[0031] The similarity determination unit 17 determines the similarity between each type of data: the user's speech content on the electronic whiteboard 2 (first type), the handwritten drawing content on the electronic whiteboard 2 (second type), and the content of the materials displayed on the electronic whiteboard 2 (third type). In the first embodiment, the similarity determination unit 17 determines whether there is similarity between the first text data (user's speech content) obtained by applying speech recognition to the speech input by the electronic whiteboard 2 and the second text data (handwritten drawing content) obtained by applying character recognition to the handwritten drawing data on the electronic whiteboard 2. The presence or absence of similarity means whether or not they are similar. "Similar" means that the value of the similarity evaluation index of the text data satisfies a predetermined condition (for example, above a threshold). Any known index may be used as the evaluation index.

[0032] The text information retrieval unit 18 is an example of a retrieval unit that retrieves information related to the first text data and the second text data from the storage unit when the similarity determination unit 17 determines that there is similarity. Specifically, the text information retrieval unit 18 retrieves information related to the extracted keyword from information stored in the hard disk (auxiliary storage device 102) of the support server 1 or on an external server. Information related to a certain keyword is, for example, information that is retrieved when that keyword is entered in a known retrieval technique.

[0033] The similar image search unit 19 compares the vector data generated based on the feature data extracted by the image object recognition unit 16 with the vector data (feature data) of image objects stored on the hard disk of the support server 1 or on an external server, and extracts image objects with high similarity from the image objects stored on the hard disk of the support server 1 or on an external server. The similar image search unit 19 also retrieves the image captions (image descriptions) associated with the image objects with high similarity.

[0034] The related information output unit 20 outputs the information collected through the search to the communication control unit 10. The communication control unit 10 transmits the information to the electronic whiteboard 2. If the related information output unit 20 searches an external server, it stores the collected information in the information storage unit 21.

[0035] The information storage unit 21 stores information received from the electronic whiteboard 2 and information collected through searches of external servers (including web servers). The information storage unit 21 also stores a word vector dictionary in which words are vectorized.

[0036] Figure 7 shows an example of the functional configuration of the electronic whiteboard 2 in the first embodiment. As shown in Figure 7, the electronic whiteboard 2 has a contact position detection unit 60, a drawing data generation unit 61, a display control unit 62, a communication control unit 63, a speech encoding unit 64, a character recognition unit 65, a web browser control unit 66, and a data storage unit 67. Each of these units is realized by processing that one or more programs installed on the electronic whiteboard 2 cause the CPU 40 to execute. The data storage unit 67 can be realized using, for example, a hard disk 44 or a storage device that can be connected to the electronic whiteboard 2 via a network.

[0037] The contact position detection unit 60 obtains coordinate data of the part of the touch panel that is in contact with a finger, pen, etc. (the part where the phototransistor cannot detect the light from the LED).

[0038] The drawing data generation unit 61 generates drawing lines and shapes such as circles and rectangles from a sequence of coordinates input from the contact position detection unit 60 when the user writes.

[0039] The display control unit 62 performs control for displaying the display data on the display 30.

[0040] The communication control unit 63 connects to a network such as Ethernet and performs data transmission and reception with other devices via the network.

[0041] The audio encoding unit 64 encodes the audio data obtained by A / D conversion of the audio signal input from the microphone 48 using the AAC (Advanced Audio Coding) method.

[0042] The character recognition unit 65 refers to a dictionary to identify the character or string of characters that the user has entered by handwriting, and outputs its character code (text data).

[0043] The web browser control unit 66 uses the keywords entered by the user in the web browser displayed on the display 30 to perform a search on web servers connected to the internet, and displays the search results in the web browser.

[0044] The data storage unit 67 stores drawing data entered by the user, PC screen data displayed on the display 30, the address (URL) of the support server 1, a character recognition dictionary, and data received from other devices (including servers).

[0045] In the first embodiment, a user (chairperson) standing in front of the electronic whiteboard 2 draws a table (characters and lines) shown in Figure 8 on the touch panel 31 by hand (handwriting input), and discusses a service that records audio from a meeting, camera footage, text data transcribed from speech recognition, and data such as the PC display screen. At this point, the user (chairperson) wants to introduce a service that can identify speakers, but the user does not accurately remember the information about this service.

[0046] In this case, conventionally, one might display a web browser on the display 30 of the electronic whiteboard 2, search for this service, and display it. Alternatively, the user (chairperson) could search for information about this service on their own PC and display it on the display 30 of the electronic whiteboard 2.

[0047] This embodiment describes an example in which the following operations, performed in cooperation between the electronic whiteboard 2 and the support server 1, eliminate the need for the user to use a web browser to search or to find and display files on their PC.

[0048] Figures 9 and 10 are sequence diagrams illustrating an example of a processing procedure performed in the first embodiment. Figure 11 is a flowchart illustrating an example of a processing procedure performed by the support server 1 in the first embodiment. In Figure 11, steps that are the same as or correspond to the steps in Figure 9 or Figure 10 are given the same step number.

[0049] When the meeting begins and the user (chairperson) speaks (S101), the audio encoding unit 64 of the electronic whiteboard 2 performs A / D conversion on the audio signal input from the microphone 48 and encodes this audio data in AAC format (S102-S103). Subsequently, the audio encoding unit 64 transmits the encoded audio data to the support server 1 (S104). The speech recognition unit 11 of the support server 1 decodes the audio data and outputs text data corresponding to the audio data through speech recognition (S105, S106). Subsequently, the speech text generation unit 13 generates text based on the text data (S107).

[0050] On the other hand, when the user (chairperson) writes characters on the electronic whiteboard 2 by hand (S111, S112), the character recognition unit 65 applies character recognition to the handwritten characters to convert them into text data (S113), and sends the text data to the support server 1 (S114). The keyword extraction unit 15 of the support server 1 extracts keywords (words) from the received text data and stores them in the memory device 103 (S115). If the user (chairperson) writes the table in Figure 8, the keywords extracted are "meeting record service," "audio," "camera image," "transcription," "PC screen," "AAA," "BBB," and "CCC," and these are stored in the memory device 103.

[0051] Figure 9 shows that steps S101 to S107 are executed each time a speech is made, and steps S111 to S115 are executed each time handwritten characters are written.

[0052] Here, if the user utters, for example, "There is a service that can identify the speaker," the support server 1 generates the sentence "There is a service that can identify the speaker" through speech recognition (S101-S107). Subsequently, if the user becomes silent while trying to remember this service, the utterance break determination unit 12 of the support server 1 determines that the utterance has been interrupted (S121), and the keyword extraction unit 15 extracts "speaker identification" and "service" as keywords from the utterance (S122: S122-1 in Figure 11).

[0053] Next, the similarity determination unit 17 determines whether there is similarity between the keywords extracted from the handwritten characters and the spoken keywords if the handwritten keywords are stored in the memory device 103 (YES in S122-2 of Figure 11) (S123). The similarity between keywords can be determined using vectorized data of each keyword. The support server 1 vectorizes a large number of words using the Word2Vec (Word to Vector) model and stores this word vector dictionary in advance in the information storage unit 21. The similarity determination unit 17 obtains vector data corresponding to the keywords extracted from the handwritten characters and the spoken keywords using the Word2Vec model and the word vector dictionary. The similarity determination unit 17 calculates the cosine (cos) similarity from the vector data of both and determines that there is similarity if this value is greater than or equal to, for example, a threshold (for example, 0.7). The similarity determination unit 17 determines, through the above processing, that the "meeting record service" and "transcription" extracted from handwritten characters and the "speaker identification" extracted from the utterances are similar.

[0054] Next, the text information retrieval unit 18 of the support server 1 uses these keywords to first search the hard disk (auxiliary storage device 102) of the support server 1 (S124: S124-1 in Figure 11). If relevant information is found in the hard disk (auxiliary storage device 102) of the support server 1 (YES in S124-2 in Figure 11), the text information retrieval unit 18 outputs this information to the related information output unit 20. If there is no relevant information in the hard disk (auxiliary storage device 102) of the support server 1 (NO in S124-2 in Figure 11), the text information retrieval unit 18 performs a search on external web servers (web pages) connected to the internet (S125~S127: S125, S127-1 in Figure 11). If the text information retrieval unit 18 obtains relevant information (YES in S127-2 in Figure 11), it outputs the obtained information to the related information output unit 20. The related information output unit 20 adds data (predetermined data) indicating that it is related to the spoken content to the speaker-identifiable service information, and transmits it to the electronic whiteboard 2 via the communication control unit 10 (S128). If the text information retrieval unit 18 searches the external web server and does not obtain any related information (NO in S127-2 of Figure 11), it returns to receiving the voice data (S104). If the information collected from the external web server is not in the hard disk (auxiliary storage device 102) of the support server 1, the information storage unit 21 stores this information (S129).

[0055] When the electronic whiteboard 2 receives information to which data indicating that it is related information (predetermined data) has been added, it determines that the information is related to the spoken content and the handwritten input content, and the display control unit 62 displays this information in a pop-up window in a predetermined manner (for example, half brightness) (S130).

[0056] Figure 12 shows an example of the display of related information in the first embodiment. In Figure 12, an example is shown in which a pop-up window w1 containing related information is displayed on the electronic whiteboard 2. When the user checks this related information and touches the close button in the upper right corner of the pop-up window w1, the display control unit 62 closes the pop-up window w1. Note that this pop-up window w1 may be configured to close automatically after a predetermined amount of time has elapsed.

[0057] As described above, according to the first embodiment, when there is similarity between the content spoken by a user standing in front of the electronic whiteboard 2 and the content drawn by hand on the electronic whiteboard 2, information related to these contents is collected and displayed on the electronic whiteboard 2. Therefore, it is possible to efficiently acquire information related to the information presented (on the electronic whiteboard 2). As a result, the user is saved the trouble of searching for this related information, and the meeting can be made more efficient (time-saving).

[0058] Furthermore, by only searching for and outputting related information when the user standing in front of electronic whiteboard 2 tries to explain related information from the materials but cannot immediately recall it and stops speaking, it is possible to avoid frequently outputting related information.

[0059] Furthermore, if information related to the discussion is displayed on the electronic whiteboard 2 and shared with all meeting participants who are referring to it, the discussion may shift to that related information and potentially go in a direction unintended by the chairperson.

[0060] According to this embodiment, the relevant information is displayed in an inconspicuous manner, such as at half brightness, making it difficult for meeting participants who are far away from the electronic whiteboard 2 to see. As a result, it is possible to avoid the discussion shifting to the relevant information.

[0061] Furthermore, using the method described in Patent Document 1 presents a problem in that, every time someone speaks during a meeting, unintended related information is automatically displayed on the electronic whiteboard.

[0062] Furthermore, Patent Document 2 has the problem that it requires keywords to be registered in advance and associated with other information, and that information other than the registered keywords cannot be displayed. Additionally, it has the problem that related information cannot be collected through web searches or other internet-based methods.

[0063] [Second Embodiment] Next, a second embodiment will be described. The differences between the second embodiment and the first embodiment will be described. Therefore, points not specifically mentioned may be the same as in the first embodiment.

[0064] In the second embodiment, we will describe an example in which the process from determining the similarity of various keywords onward is executed when a state of no handwriting input continues for a predetermined period of time, rather than a state of silence.

[0065] Figure 13 is a flowchart illustrating an example of the processing procedure performed by the support server 1 in the second embodiment. In Figure 13, steps identical to those in Figure 11 are given the same step numbers.

[0066] When the meeting begins, the audio encoding unit 64 of the electronic whiteboard 2 converts the audio signal input from the microphone 48 using A / D conversion, encodes this audio data in AAC format, and sends it to the support server 1. The speech recognition unit 11 of the support server 1 decodes this audio data and outputs text corresponding to the audio data through speech recognition, and the speech text generation unit 13 generates a document based on this text data. The support server 1 stores this document in the memory device 103 (S104, S105~S107a).

[0067] In this meeting, the discussion revolves around services that record audio, camera footage, text data transcribed from speech recognition, and data from the PC display screen. The user (chairperson) standing in front of the electronic whiteboard 2 handwrites on the whiteboard 2 whether or not audio, camera footage, text data transcribed from speech, and PC display screen data can be recorded for meeting recording services AAA and BBB, as shown in the table in Figure 14(A). The handwritten entries "Meeting Recording Service," "Audio," "Camera Footage," "Transcription," "PC Screen," "AAA," "BBB," "○," and "×" by the user (chairperson) are converted into text data by the character recognition unit 65 of the electronic whiteboard 2. This text data is then sent to the support server 1 with the recognized character information added. When the support server 1 receives each of these text data entries (YES in S114), it adds the time information of receipt to each and stores each text data entry in the memory device 103 (No in S141, S142).

[0068] Next, the user (chairperson) writes "CCC" in the meeting record service column, as shown in Figure 14(B), in order to also record the meeting record service CCC in the table. The character recognition unit 65 of the electronic whiteboard 2 converts this handwritten character "CCC" into text data, adds the recognized character information to this text data, and sends it to the support server 1. When the support server 1 receives this text data (YES in S114), it adds the current time information and stores the text data in the memory device 103 (No in S141, S142).

[0069] Here, assume that the user (chairperson) and other meeting participants have no knowledge of the meeting record service CCC, and that the user (chairperson) does not write anything about the meeting record service CCC to the table. After the support server 1 receives text data with character recognition information attached, if a predetermined time has elapsed and it does not receive any more text data with character recognition information attached (i.e., if no handwritten input is made on the electronic whiteboard 2 for a predetermined time) (YES in S141), the keyword extraction unit 15 extracts a keyword from the last received text data with character recognition information attached (S143). That is, "CCC" is extracted as the keyword.

[0070] The keyword extraction unit 15 also extracts keywords from the speech generated from the audio data received after the time of the previous handwritten boundary, i.e., the time when "Meeting Record Service" was first handwritten in the table in Figure 14(A), if the period of time during which no text data with recognized character information is not received (non-reception time) is longer than a predetermined time (S144). If "Meeting Record Service" is written within a predetermined time after the start of the meeting and there is no handwritten boundary (non-reception time), the time from the start of the meeting until the first handwritten entry is considered the handwritten boundary. Therefore, if the user (chairperson) makes a speech containing a term present in the table in Figure 14(B) while filling it out (i.e., if text data corresponding to that term is stored in the memory device 103), the keyword extraction unit 15 extracts "Meeting Record Service", "Audio", "Camera Video", "Transcription", "PC Screen", "AAA", "BBB", and "CCC" as keywords of the speech in step S144.

[0071] Next, the similarity determination unit 17 determines whether there is similarity between the keyword extracted from the handwritten characters and the keyword of the utterance (S123). Specifically, the similarity determination unit 17 attempts to obtain vector data corresponding to the keyword "CCC" extracted from the handwritten characters and the keyword of the utterance using the Word2Vec model and the word vector dictionary, but assumes that "CCC" is not in the word vector dictionary. In this case, the similarity determination unit 17 compares the text data (character codes) of the keywords without using vector data. Then, the similarity determination unit 17 outputs the keyword "CCC," which was found to be a match through this comparison, to the text information retrieval unit 18.

[0072] The text information retrieval unit 18 uses this keyword to search the hard disk of the support server 1 and external web servers (web pages) connected to the internet (S125, S127-1). If the text information retrieval unit 18 obtains relevant information (YES in S127-2 in Figure 11), it outputs the obtained information to the relevant information output unit 20. The relevant information output unit 20 adds data to this information indicating that it is relevant to the spoken content and transmits it to the electronic whiteboard 2 via the communication control unit 10 (S128).

[0073] When the electronic whiteboard 2 receives information with the data attached, it determines that the information is related to the content of the speech, and the display control unit 62 displays this information in a pop-up window in a predetermined manner (for example, at half brightness).

[0074] Figure 15 shows an example of the display of related information in the second embodiment. In Figure 15, an example is shown in which a pop-up window w2 containing related information is displayed on the electronic whiteboard 2. When the user checks this related information and touches the close button in the upper right corner of the pop-up window w2, the display control unit 62 closes the pop-up window w2.

[0075] As described above, the second embodiment can obtain the same effects as the first embodiment.

[0076] Furthermore, by only searching for and outputting related information when a user explaining something while drawing on the electronic whiteboard 2 stops drawing because they cannot immediately recall the content of their explanation, it is possible to avoid frequently outputting related information.

[0077] [Third Embodiment] Next, a third embodiment will be described. The differences between the third embodiment and the first embodiment will be described. Therefore, points not specifically mentioned may be the same as in the first embodiment.

[0078] In the third embodiment, information related to an object (description data of the object) contained in the image data displayed on the electronic whiteboard 2 is retrieved from the storage unit, and the similarity determination unit 17 determines whether there is similarity between the first text data obtained by applying speech recognition to the speech input by the electronic whiteboard 2 and the related information. If similarity is determined, an example is described in which the related information is output to the electronic whiteboard 2.

[0079] In the third embodiment, we assume a situation where an electronic whiteboard 2 is installed in a school classroom and students participate in lessons using laptop computers.

[0080] Figure 16 shows an example of a system configuration in the third embodiment. In Figure 16, parts that are the same as or corresponding to those in Figure 1 are denoted by the same reference numerals, and their descriptions are omitted as appropriate.

[0081] In Figure 16, the electronic whiteboard 2 and the wireless access point 72 are connected to the school's internal network, which is behind the firewall 71. The firewall 71 monitors communication between devices connected to the school's internal network and external devices via the internet, allowing only authorized communications to pass through and blocking unauthorized communications. The wireless access point 72 is a wireless communication access point, and the notebook PC 74 can communicate with the outside world by connecting to the wireless access point 72.

[0082] Figure 17 shows an example of the functional configuration of the support server 1 in the third embodiment. In Figure 17, parts that are the same as or corresponding to those in Figure 6 are denoted by the same reference numerals, and their descriptions are omitted as appropriate.

[0083] In Figure 17, the support server 1 further includes a lesson control unit 80, a screen data distribution unit 81, a search execution determination unit 82, and a user authentication unit 83. Each of these units is realized by processing that one or more programs installed on the support server 1 cause the processor 104 to execute. The support server 1 also utilizes a user information storage unit 84. The user information storage unit 84 can be realized using, for example, an auxiliary storage device 102, or a storage device that can be connected to the support server 1 via a network.

[0084] The lesson control unit 80 stores user information of teachers and students, as well as equipment information of the electronic whiteboard 2, in the information storage unit 21 in association with lesson identification information (hereinafter referred to as "lesson ID"). For each lesson ID, it also permits connections between the teacher's and student's laptops and the electronic whiteboard 2, and between the teacher's and student's laptops. A lesson ID is generated when a lesson is scheduled, or, if there is no lesson scheduled, when the teacher's and student's laptops connect to the electronic whiteboard 2.

[0085] The screen data distribution unit 81 transmits screen data received from the interactive whiteboard 2 to the teacher's and students' laptops, and transmits screen data received from the teacher's and students' laptops to the interactive whiteboard 2.

[0086] The search execution determination unit 82 instructs the system to execute the search when it determines that it is ready to perform a search.

[0087] The user authentication unit 83 authenticates the user of the support server 1 by comparing the user ID and password received via the communication control unit 10 with the user ID and password stored in the user information storage unit 84.

[0088] The user information storage unit 84 stores the user ID and password of users (students, teachers) who use the support server 1.

[0089] Figure 18 shows an example of the functional configuration of the electronic whiteboard 2 in the third embodiment. In Figure 18, the same reference numerals are used for parts that are the same as or corresponding to those in Figure 7, and their descriptions are omitted as appropriate.

[0090] In Figure 18, the electronic whiteboard 2 further includes a screen data transmission and reception unit 90.

[0091] The screen data transmission / reception unit 90 transmits the display data to the support server 1 via the communication control unit 63 when there is a change in the display data on the screen, and also receives screen data from the teacher's or other students' laptops from the support server 1 via the communication control unit 63.

[0092] The hard disk (auxiliary storage device 102) of support server 1 stores vector data (feature data) and explanatory data (text data) of many image objects (objects within images (in other words, images showing objects)) related to the lesson, as a set. The explanatory data of an image object is text data that includes sentences describing the object shown by the image object.

[0093] On the teachers' and students' laptops, a class support application is running. The class support application sends a user ID and password to support server 1. Once the user is authenticated by support server 1, a data communication session is established, and screen data is sent and received through this session.

[0094] Figure 19 is a sequence diagram illustrating an example of a processing procedure performed in the third embodiment. Figure 20 is a flowchart illustrating an example of a processing procedure performed by the support server 1 in the third embodiment. In Figure 20, steps that are the same as or correspond to the steps in Figure 19 are given the same step number.

[0095] Let's assume that a certain student (hereinafter referred to as "Student A") found an unusual flower while visiting a park, took a picture of the flower with their smartphone, and copied that picture to notebook PC 74.

[0096] During a lesson about flowers, student A displays a photograph of a flower on laptop PC 74 in order to ask a question about the flower. The student then uses the screen sharing function of the lesson support application to send the screen data (image) displayed on laptop PC 74 to electronic whiteboard 2 via support server 1, where it is displayed (S201~S206).

[0097] In this process, the screen data distribution unit 81 of the support server 1 does not immediately erase the screen data received from the notebook PC 74 after sending it to the electronic whiteboard 2, the teacher's notebook PC, and each student's notebook PC 74, but instead stores it in the memory device 103.

[0098] Next, when student A says, "What kind of flower is this?" (S211), the electronic whiteboard 2 encodes the audio data obtained by A / D conversion of the audio signal input from microphone 48 (S212, S213), and sends the encoded audio data to support server 1 (S214). The speech recognition unit 11 of support server 1 decodes this audio data and outputs text corresponding to the audio data through speech recognition (S215, S216). Subsequently, the speech text generation unit 13 generates the sentence "What kind of flower is this?" based on the text data (S217). The keyword extraction unit 15 of support server 1 extracts the keywords "flower" and "what is" from the sentence (S218). The search execution decision unit 82 checks the keywords extracted by the keyword extraction unit 15, and if it detects "What is it?", it determines that it is a question and outputs a request to the similarity decision unit 17 to check whether there are any similar parts between the screen data stored in the memory device 103 and the student's utterance (S219). The similarity decision unit 17 outputs a request to the image object recognition unit 16 to investigate what kind of image objects are in the screen data.

[0099] The image object recognition unit 16 identifies image objects in the screen data and extracts their feature data (S220). More specifically, the image object recognition unit 16 first extracts edge pixels from the screen data and determines the continuity between pixels within a predetermined block of a predetermined size centered on these edge pixels and the edge pixels. If this continuity satisfies predetermined conditions, the unit determines that the block is a character area; otherwise, it determines that it is something other than characters (e.g., a photograph) (see, for example, Japanese Patent Application Publication No. 2007-288690). For areas other than characters (e.g., photographs), the image object recognition unit 16 extracts edge pixels and determines the area (object) enclosed by these edge pixels. If multiple objects are in a containment relationship, such as a camera body and lens, these objects are grouped together. The image object recognition unit 16 extracts the feature data of this object. Feature data includes the object's shape, brightness (luminance data), color (RGB color data), etc.

[0100] Next, the image object recognition unit 16 converts this feature data into vector data and outputs this vector data to the similar image search unit 19 (S221). The similar image search unit 19 compares this vector data with the vector data (feature data) of image objects stored on the hard disk (auxiliary storage device 102) of the support server 1 or on an external server, and extracts image objects with high similarity (S222). It is assumed that the external server also stores vector data (feature data) and their descriptive data (text data) as a set for many image objects. When the similar image search unit 19 extracts image objects with high similarity, it outputs the descriptive data (text data) stored in association with them to the image object recognition unit 16. The image object recognition unit 16 outputs the descriptive data (text data) stored in association with the image to the similarity determination unit 17.

[0101] The similarity determination unit 17 determines whether there is similarity between this descriptive data and "flower," which was extracted as a keyword for the utterance (S223). In other words, it determines whether there is similarity between the information related to the object (descriptive data) contained in the image data displayed on the electronic whiteboard 2 and the text data of the utterance obtained by applying speech recognition to the audio of the utterance input by the electronic whiteboard 2.

[0102] If the description data is "Peony blooms around May. It bears one large flower at the top of the current year's branch," the similarity determination unit 17 determines that the image displayed on the screen and the spoken content are similar because the keyword "flower" is present in the description data (YES in S224-1 of Figure 20). If the keyword of the spoken word is not present in the image's description data, the similarity determination unit 17 uses vectorized data of the keyword in the image's description data and the keyword of the spoken word to determine similarity. The similarity determination using vectorized data can be the same as in the first embodiment. If there are multiple image objects on the screen, the similarity determination unit 17 determines whether the description data (keywords) of each image object are similar to the keyword of the spoken word. The similarity determination unit 17 outputs the description data that it has determined to be similar to the keyword of the spoken word to the related information output unit 20. Furthermore, if the vector data (feature data) of highly similar image objects is not found on the hard disk of the support server 1 and is extracted from an external server, the similar image search unit 19 stores this vector data (feature data) and explanatory data on the hard disk (auxiliary storage device 102) of the support server 1.

[0103] The related information output unit 20 adds data to the explanatory data indicating that it is related information to the spoken content, and transmits it to the electronic whiteboard 2 via the communication control unit 10 (S224: S224-2 in Figure 20).

[0104] When the electronic whiteboard 2 receives the relevant information to which the data has been added, the display control unit 62 displays the relevant information in a pop-up window at half brightness (S225).

[0105] Figure 21 shows an example of the display of related information in the third embodiment. Figure 21 shows an example in which a pop-up window w3 containing related information is displayed on the electronic whiteboard 2.

[0106] As described above, according to the third embodiment, when a user in front of the electronic whiteboard 2 asks a question, if there is similarity between the user's utterance (first type) and the content of the displayed materials (third type), information related to these contents is collected and displayed on the electronic whiteboard 2. Therefore, the user is saved the trouble of searching for this related information, and the meeting can be made more efficient (time-saving).

[0107] [Fourth Embodiment] Next, a fourth embodiment will be described. The differences between the fourth embodiment and the first embodiment will be described. Therefore, points not specifically mentioned may be the same as in the first embodiment.

[0108] In the fourth embodiment, the support server 1 includes a module (service) that functions as a generative AI, and the use of this module (service) will be described. A generative AI is a machine learning model (e.g., a neural network) that has been trained to generate text corresponding to text input. For example, a generative AI using a Large Language Mode (LLM) may be used. An LLM is a machine learning model that has been trained on natural language processing using a large amount of text data.

[0109] Figure 22 shows an example of the functional configuration of the support server 1 in the fourth embodiment. In Figure 22, the same reference numerals are used for parts that are the same as those in Figure 6, and their descriptions are omitted.

[0110] The support server 1 shown in Figure 22 further includes a question generation unit 110 and a generation AI execution unit 111. Each of these units is realized by processing that one or more programs installed on the support server 1 cause the processor 104 to execute. On the other hand, the support server 1 shown in Figure 22 does not have a text information retrieval unit 18.

[0111] The question generation unit 110 generates question sentences (prompts) from spoken text and handwritten keywords.

[0112] The AI ​​generation execution unit 111 searches the hard disk of the support server 1 or an external server for information corresponding to the question text generated by the question generation unit 110, and generates an answer text for the question text from the retrieved information.

[0113] In the fourth embodiment, similar to the first embodiment, we will describe a scenario in which a speaker-identifiable service is displayed.

[0114] Figure 23 is a flowchart illustrating an example of the processing procedure performed by the support server 1 in the fourth embodiment. In Figure 23, steps identical to those in Figure 11 are given the same step numbers.

[0115] When the meeting begins and the user (chairperson) speaks, the audio encoding unit 64 of the electronic whiteboard 2 converts the audio signal input from the microphone 48 using A / D conversion, encodes this audio data in AAC format, and sends it to the support server 1. When the speech recognition unit 11 of the support server 1 receives the encoded audio data (YES in S104), it decodes the audio data and outputs text data corresponding to the audio data through speech recognition, and the speech text generation unit 13 generates text based on this text data (S105~S107).

[0116] On the other hand, when the user (chairperson) writes characters on the electronic whiteboard 2 by hand, the character recognition unit 65 applies character recognition to the handwritten characters to convert them into text data and sends the text data to the support server 1. The keyword extraction unit 15 of the support server 1 extracts keywords (words) from the received text data and stores them in the memory device 103 (YES in S114:S115 in Figure 23). If the user (chairperson) fills out the table in Figure 8, the keywords extracted are "meeting record service," "audio," "camera video," "transcription," "PC screen," "AAA," "BBB," and "CCC," and these are stored in the memory device 103.

[0117] For example, if the user says, "There is a service that can identify the speaker," the support server 1 generates the sentence "There is a service that can identify the speaker" through speech recognition. Then, if the user becomes silent while trying to remember this service, the speech segmentation determination unit 12 of the support server 1 determines that the utterance has been interrupted (YES in S121), and the keyword extraction unit 15 extracts "speaker identification" and "service" as keywords from the utterance (S122: S122-1 in Figure 23).

[0118] Next, the similarity determination unit 17 determines whether there is similarity between the keywords extracted from the handwritten characters and the keywords from the utterance (S123). If the similarity determination unit 17 determines that there is similarity (YES in S123), the question generation unit 110 generates a question sentence from the utterance sentence, "Is there a service that can identify the speaker?" and the handwritten keywords. For example, it generates the question sentence, "Is there a conference service that transcribes and records the audio for each speaker by identifying the speaker?" (S301). When the question generation unit 110 inputs the question sentence to the generation AI execution unit 111, the generation AI execution unit 111 searches for information related to the question sentence from the hard disk (auxiliary storage device 102) of the support server 1 or from an external server (S302~S305). If the relevant information is obtained (YES in S303 or YES in S305), the generation AI execution unit 111 generates a response sentence based on the obtained information and the learned parameters, and outputs the response sentence to the related information output unit 20 (S306). The related information output unit 20 adds data to the response sentence indicating that it is related information to the utterance content, and transmits it to the electronic whiteboard 2 via the communication control unit 10 (S307).

[0119] When the electronic whiteboard 2 receives information with the data attached, it determines that the information is related to the spoken content, and the display control unit 62 displays the information in a pop-up window at half brightness.

[0120] Figure 24 shows an example of the display of related information in the fourth embodiment. Figure 24 shows an example in which a pop-up window w4 containing related information is displayed on the electronic whiteboard 2.

[0121] As described above, according to the fourth embodiment, the same effects as in the first embodiment can be obtained using the generation AI.

[0122] In the embodiments described above, the process of recognizing characters from handwritten drawing data was shown to be performed on the electronic whiteboard 2, but this process may also be performed by the support server 1.

[0123] Next, a fifth embodiment will be described. The differences between the fifth embodiment and the first embodiment will be described. Therefore, points not specifically mentioned may be the same as in the first embodiment.

[0124] The fifth embodiment describes an example in which processing is performed using only the electronic whiteboard 2 without using the support server 1.

[0125] Figure 25 shows an example of the functional configuration of the electronic whiteboard 2 in the fifth embodiment. In Figure 25, the same reference numerals are used for parts that are the same as those in Figure 7, and their descriptions are omitted.

[0126] The electronic whiteboard 2 shown in Figure 25 further includes a speech recognition unit 120, an utterance segmentation determination unit 121, an utterance text generation unit 122, a drawing data discrimination unit 123, a keyword extraction unit 124, an image object recognition unit 125, a similarity determination unit 126, a text information retrieval unit 127, a similar image retrieval unit 128, and a search execution determination unit 129.

[0127] The speech recognition unit 120 applies speech recognition to the audio data obtained by A / D conversion of the audio signal input from the microphone 48, and outputs text data corresponding to the audio data.

[0128] The speech segmentation determination unit 121 determines that speech has been interrupted if there is no audio data input from the microphone 48 for a predetermined period of time (silence).

[0129] The speech text generation unit 122 generates text based on the text data output from the speech recognition unit 120.

[0130] The drawing data discrimination unit 123 determines whether the drawing data generated by the drawing data generation unit 61 is a character or a shape (such as a circle or square), or any other type of drawing.

[0131] The keyword extraction unit 124 extracts keywords from the text generated by the speech text generation unit 122. The keyword extraction unit 124 also extracts keywords from text data converted from handwritten characters by character recognition.

[0132] The image object recognition unit 125 identifies character areas in the screen data displayed on the display 30 (stored in the display memory) and considers everything outside of these character areas as an image area. The image object recognition unit 125 identifies objects within the image area. Furthermore, the image object recognition unit extracts feature data from these objects and converts this feature data into vector data.

[0133] The similarity determination unit 126 determines whether or not there is similarity between each type of data: the user's spoken content (first type), the handwritten drawing content (second type), and the content of the displayed material (third type).

[0134] The text information retrieval unit 127 uses the extracted keywords to perform a search within the device's hard disk or on an external server.

[0135] The similar image search unit 128 compares the vector data generated based on the feature data extracted by the image object recognition unit 125 with the vector data (feature data) of image objects stored on the device's hard disk or on an external server, and extracts image objects with high similarity. The similar image search unit 128 also retrieves image captions (image descriptions) associated with the image objects with high similarity.

[0136] The search execution determination unit 129 instructs the system to execute a search when it determines that it is ready to perform a search.

[0137] The data storage unit 67 stores drawing data entered by the user by hand, PC screen data displayed on the display 30, addresses (URLs) of on-premise servers (internal servers) to be searched, a speech recognition dictionary, a character recognition dictionary, data received from other devices (including servers), as well as a word vector dictionary in which words are vectorized.

[0138] Figure 26 is a flowchart illustrating an example of the processing procedure performed by the electronic whiteboard 2 in the fifth embodiment.

[0139] Let's assume a user (chairperson) standing in front of electronic whiteboard 2 is drawing a table (with text and lines) as shown in Figure 8 by hand, and discussing a service that records meeting audio, camera footage, text data transcribed from speech recognition, and data such as the PC display screen. At this point, the user (chairperson) wants to introduce a service that can identify speakers, but doesn't have an accurate memory of the information about this service.

[0140] When the meeting begins, the audio encoding unit 64 of the electronic whiteboard 2 performs A / D conversion on the audio signal input from the microphone 48 and inputs it to the speech recognition unit 120 (YES in S401). In this case, the speech recognition unit 120 outputs text data corresponding to the audio data through speech recognition, and the speech text generation unit 122 generates text based on this text data (S402).

[0141] On the other hand, when the user (chairperson) writes characters on the electronic whiteboard 2 by hand (YES in S403), the character recognition unit 65 converts the handwritten data into text data (S404). Subsequently, the keyword extraction unit 124 extracts keywords (words) from the text data and stores them in the main memory 41 (S405). For example, if the user (chairperson) writes the table in Figure 8, the keywords extracted are "meeting record service," "audio," "camera image," "transcription," "PC screen," "AAA," "BBB," and "CCC," and these are stored in the main memory 41.

[0142] For example, if the user says, "There is a service that can identify the speaker," the speech text generation unit 122 generates the sentence "There is a service that can identify the speaker" (YES in S401, S402).

[0143] Subsequently, if the user is silent while trying to recall the service, the speech segmentation determination unit 121 determines that the speech has been interrupted at this point (YES in S411). In this case, if the handwritten keywords are stored in the main memory 41 (YES in S412), the keyword extraction unit 124 extracts "speaker identification" and "service" as keywords for the speech from the text generated by the speech text generation unit 122 (S413).

[0144] Next, the similarity determination unit 126 determines whether there is similarity between the keywords extracted from the handwritten characters and the keywords from the utterance. The degree of similarity between these keywords is determined using vectorized data of each keyword (S414). If the similarity determination unit 126 determines that there is similarity (YES in S414), the search execution determination unit 129 instructs the text information retrieval unit 127 to execute the search.

[0145] The text information retrieval unit 127 first searches the hard disk 44 of the electronic whiteboard 2 using these keywords (S415). If relevant information is found in the hard disk 44 of the electronic whiteboard 2 (YES in S416), the display control unit 62 displays this information on the display 30 in a pop-up window at half brightness (S419).

[0146] If no relevant information is found in the hard disk 44 of the electronic whiteboard 2 (NO in S416), the text information retrieval unit 127 searches for the keyword on an external web server (web page) connected to the internet (S417). If relevant information is found (YES in S418), the text information retrieval unit 127 stores the information in the data storage unit 67. Subsequently, the display control unit 62 displays the information in a pop-up window at half brightness (S419). An example of the display of relevant information in step S419 is the same as in Figure 12.

[0147] As described above, according to the fifth embodiment, the same effects as the first embodiment can be obtained even without the support server 1.

[0148] Next, a sixth embodiment will be described. The differences between the sixth embodiment and the third embodiment will be described. Therefore, points not specifically mentioned may be the same as those in the third embodiment.

[0149] [Example of collecting and displaying relevant information from displayed images and audio] In the sixth embodiment, similar to the third embodiment, we assume a situation where an electronic whiteboard 2 is installed in a school classroom and students participate in lessons using laptop computers. Therefore, the system configuration is the same as in Figure 16.

[0150] Figure 27 shows an example of the functional configuration of the support server 1 in the sixth embodiment. In Figure 27, the same reference numerals are used for parts that are the same as those in Figure 17, and their descriptions are omitted.

[0151] The support server 1 shown in Figure 27 differs from the support server 1 shown in Figure 17 in that it does not have a speech recognition unit 11, an utterance segmentation determination unit 12, an utterance text generation unit 13, a drawing data discrimination unit 14, a keyword extraction unit 15, an image object recognition unit 16, a similarity determination unit 17, a text information retrieval unit 18, a similar image retrieval unit 19, a related information output unit 20, and a search execution determination unit 82.

[0152] Figure 28 shows an example of the functional configuration of the electronic whiteboard 2 in the sixth embodiment. In Figure 28, the same reference numerals are used for parts that are the same as those in Figure 25, and their descriptions are omitted.

[0153] The electronic whiteboard 2 shown in Figure 28 further includes a screen data transmission / reception unit 140.

[0154] The screen data transmission / reception unit 140 transmits the display data to the support server 1 via the communication control unit 63 when there is a change in the display data on the screen, and also receives screen data from the teacher's or other students' laptops from the support server 1 via the communication control unit 63.

[0155] The hard disk (auxiliary storage device 102) of support server 1 stores vector data (feature data) and explanatory data (text data) of many image objects related to the lesson, as a set.

[0156] On the teachers' and students' laptops, a class support application is running. The class support application sends a user ID and password to support server 1. Once the user is authenticated by support server 1, a data communication session is established, and screen data is sent and received through this session.

[0157] Figure 29 is a flowchart illustrating an example of the processing procedure performed by the electronic whiteboard 2 in the sixth embodiment.

[0158] Let's assume that a certain student (hereinafter referred to as "Student A") found an unusual flower while visiting a park, took a picture of the flower with their smartphone, and copied that picture to notebook PC 74.

[0159] During a lesson about flowers, student A displays a picture of a flower on laptop PC 74 in order to ask a question about it. The student then uses the screen sharing function of the lesson support application to send the screen data displayed on laptop PC 74 to electronic whiteboard 2 via support server 1, and displays it on electronic whiteboard 2.

[0160] In this process, the screen data distribution unit 81 of the support server 1 transmits the screen data received from the notebook PC 74 to the electronic whiteboard 2, the teacher's notebook PC, and each student's notebook PC 74. Therefore, the electronic whiteboard 2 receives the screen data (YES in S501).

[0161] Next, when student A says, "What kind of flower is this?", and an audio signal is input from microphone 48 on electronic whiteboard 2 (YES in S502), the audio encoding unit 64 performs A / D conversion on the audio signal and inputs it to the speech recognition unit 120. The speech recognition unit 120 outputs text corresponding to the audio data through speech recognition, and the speech text generation unit 122 generates the sentence "What kind of flower is this?" based on this text data (S503).

[0162] Next, the keyword extraction unit 124 of the electronic whiteboard 2 extracts "flower" and "what is it?" as keywords (S504). Subsequently, the search execution decision unit 129 checks the keywords extracted by the keyword extraction unit 124 and, upon detecting "what is it?", determines that it is a question (YES in S505), and outputs a request to the similarity decision unit 126 to check whether there are any similar parts between the screen data in the display memory and the student's utterance. The similarity decision unit 126 outputs a request to the image object recognition unit 125 to investigate what kind of image objects are in the screen data.

[0163] The image object recognition unit 125 identifies image objects in the screen data and extracts their feature data (S506). More specifically, the image object recognition unit 125 first extracts edge pixels from the screen data and determines the continuity between pixels within a predetermined block of a predetermined size centered on these edge pixels and the edge pixels. If this continuity satisfies predetermined conditions, the unit determines that the block is a character area; otherwise, it determines that it is something other than characters (e.g., a photograph) (see, for example, Japanese Patent Application Publication No. 2007-288690). For areas other than characters (e.g., photographs), the image object recognition unit 125 extracts edge pixels and determines the area (object) enclosed by these edge pixels. If multiple objects are in a containment relationship, such as a camera body and lens, these objects are grouped together. The image object recognition unit 125 extracts the feature data of this object. Feature data includes the object's shape, brightness (luminance data), color (RGB color data), etc.

[0164] Next, the image object recognition unit 125 converts this feature data into vector data and outputs this vector data to the similar image search unit 128 (S507). The similar image search unit 128 compares this vector data with the vector data (feature data) of image objects stored on the hard disk of the support server 1 or on an external server, and extracts image objects with high similarity (S508). It is assumed that the external server also stores vector data (feature data) and their descriptive data (text data) as a set for many image objects. When the similar image search unit 128 extracts image objects with high similarity, it outputs the corresponding descriptive data (text data) to the image object recognition unit 125. The image object recognition unit 125 outputs the descriptive data (text data) that has been stored in association with the image to the similarity determination unit 126.

[0165] The similarity determination unit 126 checks the similarity between this descriptive data and "flower," which has been extracted as a keyword for the utterance (S509). If the descriptive data is "Peony: Flowering season is around May. It bears one large flower at the top of this year's branch," then the similarity determination unit 126 determines that the image displayed on the screen and the utterance are similar because the keyword for the utterance, "flower," is present in the descriptive data (YES in S510). If the keyword for the utterance is not present in the image's descriptive data, the similarity determination unit 126 uses vectorized data of the keyword in the image's descriptive data and the keyword for the utterance to determine similarity. The similarity determination using vectorized data can be the same as in the first embodiment. If there are multiple image objects on the screen, the similarity determination unit 126 determines whether the descriptive data (keyword) of each image object is similar to the keyword for the utterance.

[0166] When the similarity determination unit 126 outputs the explanatory data to the display control unit 62, the display control unit 62 displays this data in a pop-up window at half brightness (S511). This display example is the same as in Figure 21.

[0167] As described above, the sixth embodiment provides the same effects as the third embodiment.

[0168] In each of the above embodiments, the audio subject to speech recognition includes not only speech from meeting participants but also audio played from PCs, smartphones, etc.

[0169] In each of the above embodiments, the text information search unit 18 or the similar image search unit 19 is an example of a search unit. The related information output unit 20 and the display control unit 62 are examples of output units.

[0170] Furthermore, each function of each embodiment described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" in this specification includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each function described above.

[0171] Although embodiments of the present invention have been described in detail above, the present invention is not limited to these specific embodiments, and various modifications and changes are possible within the scope of the gist of the present invention as described in the claims.

[0172] Examples of the present invention are as follows: <1> A similarity determination unit that determines whether or not there is similarity between first text data obtained by applying speech recognition to speech input from the terminal and second text data obtained by applying character recognition to drawing data handwritten into the terminal, If it is determined that there is a similarity, a search unit retrieves information related to the first text data and the second text data from the storage unit, An output unit that outputs the searched information to the terminal, An information processing device characterized by having the following features. <2> The similarity determination unit determines whether or not there is similarity when the terminal does not input voice for a predetermined period of time, or when the terminal does not input handwriting for a predetermined period of time. Characterized by <1> The information processing device described above. <3> A search unit that retrieves information related to the object contained in the image data displayed on the terminal from the storage unit, A similarity determination unit determines whether or not there is similarity between the first text data obtained by applying speech recognition to the voice input by the terminal and the related information. An output unit that outputs the relevant information to the terminal when it is determined that there is a similarity, An information processing device characterized by having the following features. <4> The output unit adds predetermined data to the related information and outputs it to the terminal. The terminal displays information to which the predetermined data has been added. Characterized by <1> ~ <3> An information processing device as described in any of the above. <5> It is a terminal, A similarity determination unit that determines whether or not there is similarity between first text data obtained by applying speech recognition to the voice input by the terminal and second text data obtained by applying character recognition to drawing data handwritten into the terminal, If it is determined that there is a similarity, a search unit retrieves information related to the first text data and the second text data from the storage unit, A display control unit that displays the aforementioned related information, A terminal characterized by having the following features. <6> It is a terminal, A search unit that retrieves information related to the object contained in the image data displayed on the terminal from the storage unit, A similarity determination unit determines whether or not there is similarity between first text data obtained by applying speech recognition to the voice input by the terminal and the aforementioned information. A display control unit that, when it is determined that there is a similarity, displays the relevant information, A terminal characterized by having the following features. <7> The display control unit displays the related information in a predetermined manner. Characterized by <5> or <6> The terminals listed. <8> An information processing system including a terminal and an information processing device, A similarity determination unit that determines whether or not there is similarity between first text data obtained by applying speech recognition to the voice input by the terminal and second text data obtained by applying character recognition to drawing data handwritten into the terminal, If it is determined that there is a similarity, a search unit retrieves information related to the first text data and the second text data from the storage unit, An output unit that outputs the searched information to the terminal, An information processing system characterized by having the following features. <9> A similarity determination procedure for determining whether there is similarity between first text data obtained by applying speech recognition to speech input from a terminal and second text data obtained by applying character recognition to drawing data handwritten on the terminal, If it is determined that there is a similarity, a search procedure is performed to retrieve information related to the first text data and the second text data from the storage unit. An output procedure for outputting the searched information to the terminal, An information processing method characterized by a computer executing the following. <10> A search procedure for retrieving information from the storage unit related to the object contained in the image data displayed on the terminal, A similarity determination procedure for determining whether there is similarity between first text data obtained by applying speech recognition to the voice input by the terminal and the related information, An output procedure for outputting the relevant information to the terminal when it is determined that there is a similarity, An information processing method characterized by a computer executing the following. <11> <9> or <10> A program that causes a computer to execute the information processing methods described above. [Explanation of symbols]

[0173] 1. Support Server 2. Electronic whiteboard 10 Communication Control Unit 11. Voice Recognition Unit 12. Utterance segment determination unit 13. Speech and Text Generation Unit 14 Drawing data discrimination unit 15 Keyword Extraction Section 16 Image Object Recognition Unit 17 Similarity judgment section 18 Text Information Retrieval Section 19 Similar Image Search Section 20 Related Information Output Unit 21 Information storage section 60 Contact position detection unit 61 Drawing Data Generation Unit 62 Display Control Unit 63 Communication Control Unit 64. Speech Coding Section 65 Character recognition section 66 Web Browser Control Unit 67 Data Storage Unit 71 Firewall 72 Wireless Access Points 74 Notebook PCs 80. Classroom Management Team 81 Screen Data Distribution Department 82 Search Execution Decision Unit 83 User Authentication Section 84 User Information Storage Unit 90 Screen data transmission / reception unit 120 Voice Recognition Unit 121 Utterance segmentation determination unit 122 Speech and Text Generation Unit 123 Drawing data discrimination unit 124 Keyword Extraction Unit 125 Image Object Recognition Unit 126 Similarity judgment part 127 Text Information Retrieval Unit 128 Similar Image Search Section 129 Search Execution Decision Unit 140 Screen data transmission / reception unit [Prior art documents] [Patent Documents]

[0174] [Patent Document 1] Japanese Patent Publication No. 2020-34897 [Patent Document 2] Japanese Patent Publication No. 2018-45676

Claims

1. A similarity determination unit determines whether or not there is similarity between first text data obtained by applying speech recognition to speech input from the terminal and second text data obtained by applying character recognition to drawing data handwritten on the terminal. If it is determined that there is a similarity, a search unit retrieves information related to the first text data and the second text data from the storage unit, An output unit that outputs the searched information to the terminal, An information processing device characterized by having the following features.

2. The similarity determination unit determines whether or not there is similarity when the terminal does not input voice for a predetermined period of time, or when the terminal does not input handwriting for a predetermined period of time. The information processing apparatus according to feature 1.

3. A search unit that retrieves information related to the object contained in the image data displayed on the terminal from the storage unit, A similarity determination unit determines whether or not there is similarity between the first text data obtained by applying speech recognition to the voice input by the terminal and the related information. An output unit that outputs the relevant information to the terminal when it is determined that there is a similarity, An information processing device characterized by having the following features.

4. The output unit adds predetermined data to the related information and outputs it to the terminal. The terminal displays information to which the predetermined data has been added. The information processing apparatus according to claim 1 or 3.

5. It is a terminal, A similarity determination unit that determines whether or not there is similarity between first text data obtained by applying speech recognition to the voice input by the terminal and second text data obtained by applying character recognition to drawing data handwritten into the terminal, If it is determined that there is a similarity, a search unit retrieves information related to the first text data and the second text data from the storage unit, A display control unit that displays the aforementioned related information, A terminal characterized by having the following features.

6. It is a terminal, A search unit that retrieves information related to the object contained in the image data displayed on the terminal from the storage unit, A similarity determination unit determines whether or not there is similarity between the first text data obtained by applying speech recognition to the voice input by the terminal and the aforementioned information. A display control unit that, when it is determined that there is a similarity, displays the relevant information, A terminal characterized by having the following features.

7. The display control unit displays the related information in a predetermined manner. The terminal according to claim 5 or 6.

8. An information processing system including a terminal and an information processing device, A similarity determination unit that determines whether or not there is similarity between first text data obtained by applying speech recognition to the voice input by the terminal and second text data obtained by applying character recognition to drawing data handwritten into the terminal, If it is determined that there is a similarity, a search unit retrieves information related to the first text data and the second text data from the storage unit, An output unit that outputs the searched information to the terminal, An information processing system characterized by having the following features.

9. A similarity determination procedure for determining whether there is similarity between first text data obtained by applying speech recognition to voice input from a terminal and second text data obtained by applying character recognition to drawing data handwritten on the terminal, If it is determined that there is a similarity, a search procedure is performed to retrieve information related to the first text data and the second text data from the storage unit. An output procedure for outputting the searched information to the terminal, An information processing method characterized by a computer executing the following.

10. A search procedure for retrieving information from the storage unit related to the object contained in the image data displayed on the terminal, A similarity determination procedure for determining whether there is similarity between first text data obtained by applying speech recognition to the voice input by the terminal and the related information, An output procedure for outputting the relevant information to the terminal when it is determined that there is a similarity, An information processing method characterized by a computer executing the following.

11. A program for causing a computer to execute the information processing method described in claim 9 or 10.

Images