Information processing system, information processing method, and information processing program

Abstract

To provide an information processing system, an information processing method, and an information processing program that allow users to easily confirm the content of a meeting.SOLUTION: The information processing system includes: an image processing unit for extracting a specific image, which is at least a part of an image being displayed in a display unit, at a specific timing; a voice processing unit for extracting a specific voice, which is a voice for a period according to the specific timing of a spoken voice; and a memory processing unit for storing the specific image extracted by the image processing unit and the specific voice extracted by the voice processing unit in relation to each other.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing method, and an information processing program. [Background technology]

[0002] 2. Description of the Related Art Conventionally, systems for generating minutes of a meeting have been known. For example, Patent Document 1 discloses a minutes-creating system capable of easily adding item information (headings) to the minutes. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-43046 A Summary of the Invention [Problem to be solved by the invention]

[0004] Here, for example, when reviewing the contents of a conference after the conference, it is necessary to play back all the recorded data of the conference while looking at the materials and minutes of the conference, which results in a problem that the review process takes a long time.

[0005] An object of the present invention is to provide an information processing system, an information processing method, and an information processing program that enable easy confirmation of the contents of a conference. [Means for solving the problem]

[0006] An information processing system according to one aspect of the present invention is a system comprising an image processing unit that extracts a specific image, which is at least a part of an image being displayed on a display unit, at a predetermined timing, an audio processing unit that extracts a specific audio, which is audio from spoken audio during a period corresponding to the predetermined timing, and a memory processing unit that stores the specific image extracted by the image processing unit and the specific audio extracted by the audio processing unit in association with each other.

[0007] An information processing method according to another aspect of the present invention is a method in which one or more processors execute an image processing step of extracting a specific image, which is at least a part of an image being displayed on a display unit, at a predetermined timing, an audio processing step of extracting a specific audio from spoken audio, which is audio for a period corresponding to the predetermined timing, and a storage processing step of storing the specific image extracted by the image processing step and the specific audio extracted by the audio processing step in association with each other.

[0008] An information processing program according to another aspect of the present invention is a program for causing one or more processors to execute the following steps: an image processing step for extracting a specific image, which is at least a part of an image being displayed on a display unit, at a predetermined timing; an audio processing step for extracting a specific audio from spoken audio, which is audio from a period corresponding to the predetermined timing; and a storage processing step for storing the specific image extracted by the image processing step and the specific audio extracted by the audio processing step in association with each other. Effect of the Invention

[0009] According to the present invention, there are provided an information processing system, an information processing method, and an information processing program that enable easy confirmation of the contents of a conference. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a functional block diagram showing a configuration of a conference support system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing an example of specific word information used in the conference support system according to the embodiment of the present invention. [Diagram 3] FIG. 3 is a diagram showing an example of minutes information used in the conference support system according to the embodiment of the present invention. [Figure 4] FIG. 4 is a diagram showing an example of a display screen displayed on a viewer terminal in the conference support system according to the embodiment of the present invention. [Diagram 5] FIG. 5 is a diagram showing an example of a display screen displayed on a viewer terminal in the conference support system according to the embodiment of the present invention. [Figure 6] FIG. 6 is a diagram showing an example of a display screen displayed on a viewer terminal in the conference support system according to the embodiment of the present invention. [Figure 7] FIG. 7 is a flowchart illustrating an example of a procedure of a conference supporting process in the conference supporting system according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. Note that the following embodiment is an example of the present invention, and does not limit the technical scope of the present invention.

[0012] [Meeting Support System 100] As shown in FIG. 1, a conference support system 100 according to an embodiment of the present invention includes a presenter terminal 1 and a viewer terminal 2. For example, the conference support system 100 includes one presenter terminal 1 and multiple viewer terminals 2. The presenter terminal 1 and the viewer terminals 2 can communicate with each other via a communication network N1 such as the Internet, a LAN, a WAN, or a public telephone line. The conference support system 100 is introduced, for example, in an online conference (Web conference) or seminar (Web seminar) using the communication network N1. The presenter terminal 1 is a terminal used by a presenter who is in charge of a presentation (explanation) at a conference, and displays conference materials according to the presenter's operation, acquires the presenter's voice, and transmits the voice data to the viewer terminal 2. The viewer terminal 2 is a terminal used by a viewer who participates in the conference, and displays the conference materials and outputs the presenter's voice based on the voice data. The viewer terminal 2 also creates and stores minutes of the conference based on the images and voice of the conference materials.

[0013] The conference support system 100 is an example of the information processing system of the present invention. The information processing system of the present invention may be configured with the viewer terminal 2 alone.

[0014] [Presenter Terminal 1] 2, the presenter terminal 1 includes a control unit 11, a storage unit 12, an operation display unit 13, a microphone speaker 14, and a communication unit 15. The presenter terminal 1 is, for example, an information processing device such as a tablet terminal, a personal computer, or a smartphone.

[0015] The communication unit 15 is a communication unit that connects the presenter terminal 1 to the communication network N1 by wire or wirelessly, and executes data communication with external devices such as the viewer terminal 2 via the communication network N1 in accordance with a predetermined communication protocol.

[0016] The operation display unit 13 is a user interface that includes a display unit, such as a liquid crystal display or an organic EL display, that displays various information, and an operation unit, such as a mouse, a keyboard, or a touch panel, that accepts operations.

[0017] The microphone speaker 14 is an audio device having a microphone function for collecting the voice of a speaker and a speaker function for outputting (emitting) the voice. The microphone speaker 14 may be built into the presenter terminal 1, or may be installed outside the presenter terminal 1 and connected to the presenter terminal 1 by wire or wirelessly. For example, the microphone speaker 14 collects the speech of the presenter operating the presenter terminal 1, and outputs the speech of the viewer operating the viewer terminal 2.

[0018] The storage unit 12 is a non-volatile storage unit such as a hard disk drive (HDD) or a solid state drive (SSD) that stores various information. Specifically, the storage unit 12 stores data such as materials used in a conference, conference information, and the like.

[0019] Various control programs are also stored in the storage unit 12. For example, the control programs are non-temporarily recorded on a computer-readable recording medium such as a CD or a DVD, read by a reading device (not shown) such as a CD drive or a DVD drive provided in the presenter terminal 1, and stored in the storage unit 22.

[0020] The control unit 11 has control devices such as a CPU, a ROM, and a RAM. The CPU is a processor that executes various arithmetic processes. The ROM is a non-volatile storage unit in which control programs such as a BIOS and an OS for causing the CPU to execute various arithmetic processes are stored in advance. The RAM is a volatile or non-volatile storage unit that stores various information, and is used as a temporary storage memory (work area) for various processes executed by the CPU. The control unit 11 controls the presenter terminal 1 by executing, by the CPU, various control programs stored in advance in the ROM or the storage unit 12.

[0021] Specifically, as shown in Fig. 1, the control unit 11 includes various processing units such as a display processing unit 111 and an acquisition processing unit 112. The control unit 11 functions as the various processing units by executing various processes according to the control program with the CPU. Some or all of the processing units may be configured with electronic circuits. The control program may be a program for causing a plurality of processors to function as the processing units.

[0022] The display processing unit 111 displays various information on the operation display unit 13. Specifically, when the conference starts and the presenter selects conference materials stored in the storage unit 12, the display processing unit 111 displays an image of the conference materials (document image) on the operation display unit 13. The display processing unit 111 also transmits image data of the document image to the viewer terminals 2 of the viewers participating in the conference. As a result, the document image of the conference materials is displayed on each viewer terminal 2. That is, the screens of the presenter terminal 1 and each viewer terminal 2 are shared. The presenter terminal 1 and each viewer terminal 2 switch the display screen (document image) according to the presenter's operation (such as a page turning operation).

[0023] The acquisition processing unit 112 acquires the speech of the presenter collected by the microphone speaker 14. The acquisition processing unit 112 transmits audio data of the speech of the presenter to each viewer terminal 2. As a result, the speech of the presenter is output from each viewer terminal 2. The acquisition processing unit 112 also acquires audio data of the speech of the viewer collected by the microphone speaker 24 of the viewer terminal 2 from the viewer terminal 2. Upon acquiring the audio data from the viewer terminal 2, the acquisition processing unit 112 causes the microphone speaker 14 to output the speech of the viewer.

[0024] [Viewer terminal 2] 2, the viewer terminal 2 includes a control unit 21, a storage unit 22, an operation display unit 23, a microphone speaker 24, and a communication unit 25. The viewer terminal 2 is, for example, an information processing device such as a tablet terminal, a personal computer, or a smartphone.

[0025] The communication unit 25 is a communication unit that connects the viewer terminal 2 to the communication network N1 by wire or wirelessly, and executes data communication with external devices such as the viewer terminal 2 via the communication network N1 in accordance with a predetermined communication protocol.

[0026] The operation display unit 23 is a user interface that includes a display unit, such as a liquid crystal display or an organic EL display, that displays various information, and an operation unit, such as a mouse, a keyboard, or a touch panel, that accepts operations.

[0027] The microphone speaker 24 is an audio device having a microphone function for collecting the voice of a speaker and a speaker function for outputting (emitting) the voice. The microphone speaker 24 may be built into the viewer terminal 2, or may be installed outside the viewer terminal 2 and connected to the viewer terminal 2 by wire or wirelessly. For example, the microphone speaker 24 collects the speech of a viewer who operates the viewer terminal 2, and outputs the speech of a presenter who operates the presenter terminal 1.

[0028] The storage unit 22 is a non-volatile storage unit such as an HDD or SSD that stores various information. Specifically, the storage unit 22 stores data such as specific word information D1 including information on specific words (keywords) and minutes information D2 including information on minutes of a meeting.

[0029] FIG. 2 is a diagram showing an example of the specific word information D1. As shown in FIG. 2, the specific word information D1 includes information such as "ID" and "specific word". The ID is identification information of the specific word. The specific word is a word (keyword) that specifies the timing of extracting a part of an image (specific image) displayed on the operation display unit 23 and the timing of extracting a part of an uttered voice (specific voice). For example, the specific word includes important words that are generally uttered in a meeting, such as "new product", "technology", "features", "trends", and "attention". The specific word also includes a predetermined word included in a document used in a meeting, such as "title". The specific word also includes synonyms and similar words of the above-mentioned specific word. The specific word information D1 is a dictionary database that is registered in advance. The control unit 21 may update the specific word of the specific word information D1 according to the meeting material.

[0030] FIG. 3 is a diagram showing an example of the minutes information D2. As shown in FIG. 2, the minutes information D2 includes information such as "ID", "image information", "audio information", "text information", "extraction date and time", and "importance". The ID is identification information of the minutes information. The image information is information of an image (specific image) extracted by the control unit 21 at a predetermined timing from among document images of the conference materials displayed on the operation display unit 23. For example, the image information is image information of a specific page extracted by the control unit 21 at a predetermined timing when the conference materials are composed of images of multiple pages. The audio information is information of audio (specific audio) extracted by the control unit 21 at a predetermined timing from among the speech of the presenter. For example, the audio information is audio information of the speech audio for several seconds before and after the predetermined timing extracted by the control unit 21. The text information is information of text obtained by converting the audio information (audio) into characters through speech recognition. The extraction date and time is information on the date and time when the control unit 21 extracts the image information and the audio information, that is, the date and time of the predetermined timing.

[0031] The importance is information indicating the importance of the image information. The control unit 21 determines the importance of the specific image based on at least one of the display time (continuous display time) of the specific image, the number of characters (amount of data) included in the specific image, and the frequency of a specific word included in the specific image, and registers the determined importance in the minutes information D2. The importance may be in two stages, "high" and "low", or may be in three stages or more. The viewer can extract the specific image and the specific sound by his / her own operation. For example, when the viewer determines that a document image displayed on the operation display unit 23 is important, the viewer performs an operation to save the document image (capture operation). In this case, the control unit 21 extracts the specific image and the specific sound at the timing of receiving the capture operation, and registers them in the minutes information D2. In this case, the control unit 21 registers "manual" as the importance.

[0032] In this way, the control unit 21 extracts the image information and the audio information at each specified timing and registers them in the minutes of meeting information D2, and also obtains text information corresponding to the audio information, the extraction date and time corresponding to the specified timing, and the importance of the image information and registers them in the minutes of meeting information D2.

[0033] The storage unit 22 also stores control programs such as a conference support program for causing the control unit 21 to execute a conference support process (see FIG. 7) described below. For example, the conference support program may be non-temporarily recorded on a computer-readable recording medium such as a CD or DVD, read by a reading device (not shown) such as a CD drive or DVD drive provided in the viewer terminal 2, and stored in the storage unit 22.

[0034] The control unit 21 has control devices such as a CPU, a ROM, and a RAM. The CPU is a processor that executes various arithmetic processes. The ROM is a non-volatile storage unit in which control programs such as a BIOS and an OS for causing the CPU to execute various arithmetic processes are stored in advance. The RAM is a volatile or non-volatile storage unit that stores various information, and is used as a temporary storage memory (work area) for various processes executed by the CPU. The control unit 21 controls the viewer terminal 2 by executing, by the CPU, various control programs stored in advance in the ROM or the storage unit 22.

[0035] However, in the conventional technology, for example, when checking the contents of a conference after the conference, it is necessary to play back all the recorded data of the conference while looking at the materials and minutes of the conference. This causes a problem that the checking work takes a long time. This causes a problem that the checking work takes a long time. In contrast, the viewer terminal 2 according to the present embodiment makes it possible to easily check the contents of the conference.

[0036] Specifically, as shown in Fig. 1, the control unit 21 includes various processing units such as an acquisition processing unit 211, a display processing unit 212, an image processing unit 213, an audio processing unit 214, and a storage processing unit 215. The control unit 21 functions as the various processing units by executing various processes according to the control program with the CPU. Some or all of the processing units may be configured with electronic circuits. The control program may be a program for causing a plurality of processors to function as the processing units.

[0037] The acquisition processing unit 211 acquires image data of the document images of the conference materials, audio data of the speech of the presenter, etc. from the presenter terminal 1. For example, the acquisition processing unit 211 acquires image data transmitted from the presenter terminal 1 every time the document image displayed on the presenter terminal 1 is updated. The acquisition processing unit 211 also acquires audio data transmitted from the presenter terminal 1 at a predetermined cycle. The acquisition processing unit 211 stores the acquired image data and audio data in the storage unit 22.

[0038] The display processing unit 212 displays the document image on the operation display unit 23 based on the image data acquired by the acquisition processing unit 211. FIG. 4 shows an example of a display screen F1 displayed on the operation display unit 23. A document image F11 is displayed in a predetermined display area of ​​the display screen F1. Also, an area designation button B11 for designating an area to be extracted from the document image F11 is displayed on the display screen F1. For example, a viewer selects the area designation button B11 on the display screen F1 displayed on the viewer terminal 2 to designate a desired range. The reference character R11 shown in FIG. 4 represents the designated range designated by the viewer.

[0039] When the specified range R11 is specified, the display processing unit 212 displays, for example, a display screen F2 shown in Fig. 5 on the operation display unit 23. The display screen F2 displays an automatic extraction button B21 for automatically executing an extraction process for extracting an image (specific image) of an area included in the specified range R11 from the document image F21, a manual extraction button B22 for manually executing the extraction process, and the like.

[0040] The image processing unit 213 extracts a specific image of at least a part (designated range R11) of the document image F21 being displayed on the display screen F2 at a predetermined timing. Specifically, the image processing unit 213 extracts the specific image at a timing when a specific word is detected in the spoken voice. For example, when a presenter speaks a specific word registered in the specific word information D1 (see FIG. 2) during a conference, the control unit 21 detects the specific word from the voice data. Then, the image processing unit 213 extracts a specific image of the designated range R11 from the document image F21 being displayed on the display screen F2 at a timing when the specific word is detected. The specific word may be the spoken voice of the presenter or the spoken voice of the audience. Note that, in the extraction process, the image processing unit 213 performs a capture process of the designated range R11 of the display screen F2. The image processing unit 213 stores image data of the extracted specific image in the image information of the minutes information D2 (see FIG. 3). Moreover, the image processing unit 213 stores information on the date and time when the specific image is extracted as the extraction date and time of the minutes information D2. The image processing unit 213 is an example of the image processing unit of the present invention.

[0041] The audio processing unit 214 extracts a specific audio, which is an audio for a period corresponding to the predetermined timing, from the spoken audio corresponding to the audio data acquired by the acquisition processing unit 211. Specifically, the audio processing unit 214 extracts the specific audio for a predetermined period before and after the timing at which the specific word is detected. For example, when a presenter utters a specific word registered in the specific word information D1 (see FIG. 2) during a conference, the control unit 21 detects the specific word from the audio data. Then, the audio processing unit 214 extracts audio data for several seconds before and after the timing at which the specific word is detected from the audio data. The audio processing unit 214 may extract audio data for a period from when the document image F21 displayed on the display screen F2 at the timing at which the specific word is detected is displayed on the display screen F2 until it is switched to another document image. The audio processing unit 214 stores the audio data of the extracted specific audio in the audio information of the minutes information D2 (see FIG. 3).

[0042] Furthermore, the voice processing unit 214 performs voice recognition on the extracted specific voice and converts it into text (character conversion). The voice processing unit 214 stores the converted text in the text information of the minutes information D2 (see FIG. 3). The voice processing unit 214 is an example of the voice processing unit of the present invention.

[0043] The storage processing unit 215 stores the specific image extracted by the image processing unit 213 and the specific sound extracted by the sound processing unit 214 in association with each other. Specifically, the storage processing unit 215 stores the specific image, the specific sound, the text, and the extraction date and time in the minutes information D2 (see FIG. 3) in association with each other with the ID attached. Note that in the present invention, it is sufficient for the storage processing unit 215 to store at least the specific image and the specific sound in association with each other.

[0044] This allows, for example, after the end of a conference, a viewer to review the contents of the conference (for example, important parts) by playing back only a specific image extracted at a predetermined timing and the audio corresponding to the specific image.

[0045] Here, the display processing unit 212 may display the text converted by the audio processing unit 214 superimposed on the specific image. For example, in the display screen F3 shown in FIG. 6, the display processing unit 212 displays the specific image P2 adjacent to the document image F21. Furthermore, the display processing unit 212 displays the balloon image P3 including the text T2 converted by the audio processing unit 214 superimposed on the specific image P2. For example, the display processing unit 212 displays the text T2 (balloon image P3) at a position corresponding to the specific voice in the area of ​​the specific image P2. Specifically, when the specific voice is a specific word included in the document image, the display processing unit 212 displays the balloon image P3 in association with the display position of the specific word. Furthermore, the display processing unit 212 displays the text T2 adjacent to the specific image P2 (below in FIG. 6). This allows the viewer to easily grasp the extracted specific image and the content of the spoken voice corresponding to the specific image. The viewer may be able to change the position of the balloon image P3 and the content of the text T2 on the display screen F3.

[0046] Furthermore, the display position of the balloon image P3 is not limited. For example, the display processing unit 212 may display the balloon image P3 at a position indicated (touched) by the presenter on the document image displayed on the presenter terminal 1, or at a position indicated (touched) by the viewer on the document image displayed on the viewer terminal 2.

[0047] Here, the image processing unit 213 may determine the importance of the specific image. Specifically, the image processing unit 213 determines the importance of the specific image based on at least one of the display time (continuous display time) of the specific image, the number of characters (amount of data) included in the specific image, and the frequency of a specific word included in the specific image. The image processing unit 213 stores the determined importance in the minutes information D2 (see FIG. 3).

[0048] The display processing unit 212 may display the specific image on the operation display unit 23 in a display mode corresponding to the importance. Specifically, the display processing unit 212 displays an identification image M21 corresponding to the importance on the display screen F2. For example, the display processing unit 212 displays the identification image M21 on a slide bar SB of the display screen F2 shown in FIG. 5. In the example shown in FIG. 5, an identification image M21 with a white circle symbol (leftmost) representing low importance and an identification image M21 with a black circle symbol (second and third from the left) representing high importance are shown. The display processing unit 212 displays the identification image M21 in a display mode corresponding to the importance on the slide bar SB every time the specific image and the specific sound are automatically extracted. Also, the display processing unit 212 displays an identification image M22 on the slide bar SB every time the specific image and the specific sound are manually extracted. The identification images M21 and M22 of the slide bar SB shown in Fig. 5 correspond to the registered contents of the minutes information D2 shown in Fig. 3. The identification images M21 and M22 allow the viewer to understand the extraction status and importance of the specific image and the specific sound.

[0049] Furthermore, when the viewer selects, for example, an identification image M21 on the display screen F2 shown in Fig. 5, the display processing unit 212 displays a display screen F3 (see Fig. 6) including a specific image P2 corresponding to the selected identification image M21 and text T2 corresponding to the specific voice. Furthermore, the display processing unit 212 displays the text T2 superimposed on the specific image P2. The display screen F3 shown in Fig. 6 shows a display state of the specific image P2 and the text T2 (ID "P002" in Fig. 3) when the second identification image M21 from the left is selected. Note that the display processing unit 212 may extract only the specific image with high importance and display it on the display screen F3.

[0050] According to the above configuration, for example, information on the specific image is stored in the minutes information D2 every time the specific word is detected, so that a situation may occur in which multiple identical or similar specific images are stored in the minutes information D2. Therefore, when the same or similar specific image has already been stored, the image processing unit 213 may delete the specific image with the older extraction date and store only the latest specific image. Note that, when the extraction dates and times of the multiple specific images are apart by a predetermined time or more, the image processing unit 213 may store each specific image without deleting any of them. Also, the image processing unit 213 may delete the specific image with the fewer number of characters (amount of data) included in the specific image, and store the specific image with the greater number of characters (amount of data).

[0051] In another embodiment, the display processing unit 212 may display the text T2 at a position in the area of ​​the specific image P2 according to the importance of the specific image P2. For example, when the importance of the specific image P2 is high, the display processing unit 212 displays the text T2 in the upper left of the area of ​​the specific image P2, and when the importance of the specific image P2 is low, the display processing unit 212 displays the text T2 in the lower right of the area of ​​the specific image P2.

[0052] [Meeting support processing] 7, an example of the procedure of the conference support process executed by the control unit 21 of the viewer terminal 2 will be described below. For example, when a conference starts, the control unit 11 of the presenter terminal 1 starts execution of the conference support program by transmitting image data of document images of the conference materials to each viewer terminal 2, thereby starting execution of the conference support process. The conference support process is executed individually and in parallel in each of the multiple viewer terminals 2.

[0053] The present invention can be understood as an invention of a conference support method that executes one or more steps included in the conference support process. The conference support method is an example of an information processing method of the present invention. One or more steps included in the conference support process described here may be omitted as appropriate. The steps in the conference support process may be executed in a different order as long as the same action and effect is achieved. Furthermore, although an example is described here in which the steps in the conference support process are executed by the control unit 21, in other embodiments, one or more processors may execute the steps in the conference support process in a distributed manner.

[0054] In step S1, the control unit 21 of the viewer terminal 2 determines whether or not image data of a document image of the conference material has been acquired from the presenter terminal 1. If the control unit 21 has acquired image data of the document image from the presenter terminal 1 (S1: Yes), the process proceeds to step S2. The control unit 21 waits until it acquires image data of the document image from the presenter terminal 1 (S1: No).

[0055] In step S2, the control unit 21 causes the document image to be displayed on the operation display unit 23 of the viewer terminal 2 based on the acquired image data. Fig. 4 shows an example of a display screen F1 displayed on the operation display unit 23. The control unit 21 causes a document image F11 to be displayed in a predetermined display area of ​​the display screen F1. Furthermore, upon receiving an operation from the viewer to designate an extraction area of ​​the document image, the control unit 21 sets a designated range R11 (see Fig. 4) corresponding to the extraction area.

[0056] Next, in step S3, the control unit 21 determines whether or not it has acquired audio data of the presenter's speech from the presenter terminal 1. If the control unit 21 has acquired the audio data of the speech from the presenter terminal 1 (S3: Yes), the process proceeds to step S4. If the control unit 21 has not acquired the audio data of the speech from the presenter terminal 1 (S3: No), the process returns to step S1.

[0057] In step S4, the control unit 21 outputs the spoken voice based on the acquired voice data from the microphone speaker 24. Through the above-mentioned steps S1 to S4, the control unit 21 realizes, for example, an online conference.

[0058] Next, in step S5, the control unit 21 determines whether or not a specific word registered in the specific word information D1 (see FIG. 2) has been detected from the acquired voice data. If the control unit 21 detects the specific word (S5: Yes), the process proceeds to step S6. On the other hand, if the control unit 21 does not detect the specific word (S5: No), the process returns to step S1. Note that in step S5, the control unit 21 may determine whether or not the image being displayed has been switched, or whether or not a specific user operation (described later) has been accepted.

[0059] In step S6, the control unit 21 extracts a specific image of the designated range R11 of the document image being displayed on the display screen F2 at the timing when the specific word is detected. For example, when a presenter utters a specific word registered in the specific word information D1 (see FIG. 2) during a conference, the control unit 21 detects the specific word from the voice data, and extracts a specific image of the designated range R11 from the document image being displayed on the display screen F2 at the timing when the specific word is detected. The control unit 21 stores image data of the extracted specific image in the image information of the minutes information D2 (see FIG. 3). Step S6 is an example of an image processing step of the present invention.

[0060] Next, in step S7, the control unit 21 extracts the specific voice from the spoken voice corresponding to the voice data acquired in step S3 for a predetermined period before and after the timing at which the specific word was detected. For example, when a presenter utters a specific word registered in specific word information D1 (see FIG. 2) during a meeting, the control unit 21 detects the specific word from the voice data, and extracts voice data from the voice data for several seconds before and after the timing at which the specific word was detected. The control unit 21 stores the extracted voice data of the specific voice in the voice information of the minutes information D2 (see FIG. 3). Step S7 is an example of a voice processing step of the present invention.

[0061] Next, in step S8, the control unit 21 performs voice recognition on the extracted specific voice and converts it into text (character conversion).

[0062] Next, in step S9, the control unit 21 stores the converted text in association with the specific image and the voice information (see FIG. 3). Step S9 is an example of a storage processing step of the present invention.

[0063] Next, in step S10, the control unit 21 displays an identification image corresponding to the importance of the specific image on a slide bar SB of the display screen F2 (see FIG. 5). For example, as shown in FIG. 5, the control unit 21 displays an identification image M21 with a white circle symbol (leftmost) representing low importance, an identification image M21 with black circle symbols (second and third from the left) representing high importance, and the like on the slide bar SB. The control unit 21 also displays an identification image M22 on the slide bar SB, which indicates that the specific image and the specific sound have been manually extracted.

[0064] Next, in step S11, the control unit 21 determines whether or not an operation to select the identification images M21 and M22 is accepted from the viewer. If the control unit 21 accepts the operation from the viewer (S11: Yes), the process proceeds to step S12. If the control unit 21 does not accept the operation from the viewer (S11: No), the process returns to step S1.

[0065] In step S12, the control unit 21 causes the operation display unit 23 to display a specific image P2 and text T2 corresponding to the selected identification images M21, M22. For example, when the viewer selects the identification image M21 on the display screen F2 shown in FIG. 5, the control unit 21 causes a display screen F3 (see FIG. 6) including a specific image P2 corresponding to the selected identification image M21 and text T2 corresponding to the specific voice. The control unit 21 also causes the text T2 to be displayed superimposed on the specific image P2. After step S12, the process returns to step S1, and the control unit 21 repeatedly executes the above-mentioned process. In this manner, the conference support process is executed.

[0066] As described above, the conference support system 100 according to this embodiment extracts a specific image, which is at least a part of an image being displayed on the viewer terminal 2, at a predetermined timing, and extracts a specific sound, which is a sound from the speech sound during a period corresponding to the predetermined timing. The conference support system 100 then stores the extracted specific image and specific sound in association with each other. This allows, for example, a viewer who has participated in a conference to obtain, after the conference ends, the document image that was being displayed at the predetermined timing among the document images used in the conference, and the speech sound corresponding to that timing, and thus easily check the contents of the conference.

[0067] Furthermore, the conference support system 100 stores the text of the specific voice in association with the specific image, and displays the text superimposed on the specific image, allowing the viewer to easily grasp the specific image and the content (text) of the spoken voice corresponding to the specific image.

[0068] In the conference support system 100, the presenter terminal 1 and the viewer terminal 2 may have the same functions. That is, a conference support program for executing the conference support process (see FIG. 7) may be stored in the memory unit 12 of the presenter terminal 1, and the control unit 11 may extract a specific image that is at least a part of an image being displayed on the presenter terminal 1 at a predetermined timing, and extract a specific voice that is a voice of a viewer during a period corresponding to the predetermined timing from the voice of the viewer. The control unit 11 may also store the text of the specific voice in association with the specific image, and display the text on the operation display unit 13 by superimposing the text on the specific image.

[0069] The present invention is not limited to the above-mentioned embodiment. In the above-mentioned embodiment, the predetermined timing is the timing when a specific word is detected in the uttered voice, but the predetermined timing of the present invention is not limited to this. For example, the predetermined timing may be at least one of the timing when the image (document image) being displayed is switched, the timing when a specific word is detected in the uttered voice, and the timing when a specific user operation is received.

[0070] Specifically, the image processing unit 213 may extract the specific image at the timing when the document image is switched in response to the presenter's page turning operation. For example, when the presenter switches the document image to the next page during a conference, the image processing unit 213 extracts the specific image of the designated range R11 from the document image of the switched page. In addition, the audio processing unit 214 extracts the specific voice from the speech voice corresponding to the audio data acquired by the acquisition processing unit 211 for a predetermined period before and after the timing when the document image is switched. The control unit 21 stores the specific image, the specific voice, and the text in the minutes information D2 in association with each other.

[0071] Furthermore, the image processing unit 213 may extract the specific image at the timing of receiving a viewer's manual instruction for the extraction process. For example, when a viewer presses the manual extraction button B22 on the display screen F2 (see FIG. 5) during a conference, the image processing unit 213 extracts a specific image of the specified range R11 from the displayed document image. Furthermore, the audio processing unit 214 extracts the specific voice from the speech corresponding to the audio data acquired by the acquisition processing unit 211 for a predetermined period before and after the timing when the manual extraction button B22 is pressed. The control unit 21 stores the specific image, the specific voice, and the text in the minutes information D2 in association with each other. The operation of the manual extraction button B22 is an example of a user operation of the present invention.

[0072] Furthermore, the predetermined timing may be, for example, a timing when the content of the document image of the new page changes significantly from the content of the document image of the previous page. For example, the predetermined timing may be a timing when the difference in the data amount between the previous and next document images exceeds a threshold.

[0073] The predetermined timing may be a timing when a viewer inputs a handwritten note or the like in the viewer terminal 2. The predetermined timing may be a timing when a presenter points to a part of the document image with a pointing means such as a pointer in the presenter terminal 1.

[0074] In the above-described embodiment, the conference support program is stored in the presenter terminal 1 and the viewer terminal 2, and each control unit 11 executes the conference support process, but in the information processing system of the present invention, a server (e.g., a cloud server) that manages the conference may store the conference support program and execute the conference support process. In this case, the server stores specific word information D1 (see FIG. 2) and minutes information D2 (see FIG. 3). In addition, the presenter terminal 1 and the viewer terminal 2 execute browser processing to display web pages of various screens (conference screen, display screens F1, F2, F3) provided from the server via the communication network N1 on the operation display unit and input operations on the operation display unit to the server.

[0075] In the above-described embodiment, the voice spoken by the presenter is the subject, but the speaker of the present invention is not limited to the presenter of the conference. For example, the speaker may be a minutes creator who creates the minutes of the conference. Specifically, the minutes creator who operates the viewer terminal 2 may speak the voice to be recorded together with the image information (specific image). In this case, the minutes creator can speak the matters he or she wants to record as comments, thereby creating minutes that include image information and text information corresponding to the spoken comments. This allows the minutes creator to easily create minutes that make it easy to look back on the conference later. In other words, the information processing system of the present invention can also be applied to a minutes creation support system.

[0076] Furthermore, the information processing system of the present invention can be constructed by freely combining the embodiments described above, or by appropriately modifying or omitting parts of each embodiment, within the scope of the invention described in each claim. [Explanation of symbols]

[0077] 1: Presenter terminal 2: Viewer terminal 100: Meeting support system 111: Display processing unit 112: Acquisition processing unit 211: Acquisition processing unit 212: Display processing unit 213: Image processing unit 214: Audio processing unit 215: Memory Processing Unit

Claims

1. an image processing unit that extracts a specific image, which is at least a part of an image being displayed on the display unit, at a predetermined timing; a voice processing unit that extracts a specific voice from the uttered voice, the specific voice being a voice for a predetermined period before and after the predetermined timing; a storage processing unit that stores the specific image extracted by the image processing unit and the specific sound extracted by the sound processing unit in association with each other; a display processing unit that causes the specific image to be displayed on the display unit; Equipped with the image processing unit extracts the specific image from the second image at the timing when the first image being displayed on the display unit is switched to the second image, and determines the importance of the specific image based on a continuous display time of the specific image; the display processing unit causes the display unit to display an identification image according to the importance level in association with the specific image; The voice processing unit extracts the specific voice from a predetermined period before and after the timing, and converts the extracted specific voice into text; The storage processing unit stores the specific image, the specific voice, and the text in association with each other. Information processing system.

2. The display processing unit displays the text converted by the audio processing unit so as to be superimposed on the specific image. The information processing system according to claim 1 .

3. the display processing unit displays the text at a position corresponding to the specific voice in an area of ​​the specific image; The information processing system according to claim 2 .

4. the display processing unit displays the text at a position in the specific image area designated by a user; The information processing system according to claim 2 .

5. the display processing unit displays the text at a position in the area of ​​the specific image according to the importance of the specific image; The information processing system according to claim 2 .

6. the display processing unit causes the specific image to be displayed on the display unit when the identification image is selected. The information processing system according to any one of claims 1 to 5.

7. One or more processors an image processing step of extracting a specific image, which is at least a part of an image being displayed on the display unit, at a predetermined timing; a voice processing step of extracting a specific voice, which is a voice for a predetermined period before and after the predetermined timing, from the uttered voice; a storage processing step of storing the specific image extracted by the image processing step and the specific sound extracted by the sound processing step in association with each other; a display processing step of displaying the specific image on the display unit; Run In the image processing step, the specific image is extracted from the second image at the timing when the first image being displayed on the display unit is switched to the second image, and an importance of the specific image is determined based on a continuous display time of the specific image; In the display processing step, the specific image is displayed on the display unit in a display mode corresponding to the importance level, In the voice processing step, the specific voice is extracted from a predetermined period before and after the timing, and the extracted specific voice is converted into text; In the storage processing step, the specific image, the specific voice, and the text are stored in association with each other. Information processing methods.

8. an image processing step of extracting a specific image, which is at least a part of an image being displayed on the display unit, at a predetermined timing; a voice processing step of extracting a specific voice, which is a voice for a predetermined period before and after the predetermined timing, from the uttered voice; a storage processing step of storing the specific image extracted by the image processing step and the specific sound extracted by the sound processing step in association with each other; a display processing step of displaying the specific image on the display unit; An information processing program for causing one or more processors to execute the above, In the image processing step, the specific image is extracted from the second image at the timing when the first image being displayed on the display unit is switched to the second image, and an importance of the specific image is determined based on a continuous display time of the specific image; In the display processing step, the specific image is displayed on the display unit in a display mode corresponding to the importance level, In the voice processing step, the specific voice is extracted from a predetermined period before and after the timing, and the extracted specific voice is converted into text; In the storage processing step, the specific image, the specific voice, and the text are stored in association with each other. Information processing program.

Images