Viewing support system, viewing support method and program

Abstract

To simply and properly record contents to be displayed.SOLUTION: A viewing support system comprises a display unit, a captured image generation unit, a voice recognition unit, and a linking storage unit. The display unit has a display screen which can display contents having a plurality of pages. The captured image generation unit generates captured images of the contents displayed on the display screen. The voice recognition unit recognizes voice included in the contents. The linking storage unit stores a voice recognition result which is a recognition result of the voice included in the contents by the voice recognition unit linked with the captured image generated by the captured image generation unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to a viewing support system, a viewing support method, and a program. [Background technology]

[0002] Conventionally, document creation and viewing devices have been known that create documents based on video data or audio data recorded from a conference or lecture, and allow the document creator or participants to view the summarized document together with the video or audio. In conventional document creation and viewing devices, a conference is recorded to generate video data, and the contents of the conference are recorded to generate audio data. Document data is also generated that includes a draft of the minutes entered by the minutes creator. Furthermore, based on the audio or video data and the document data, a correspondence relationship between the audio or video and the document is derived, and correspondence relationship table data is generated. Based on the correspondence relationship table data, the audio or video and the document are displayed in correspondence with each other.

[0003] Further, methods have been developed for linking and utilizing video data recorded from conferences, lectures, etc., or audio data recorded from audio. However, in online meetings using online meeting apps, presenters would often turn the pages of their documents one after another, making it difficult to view the pages before the currently displayed page. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2005 / 027092 Summary of the Invention [Problem to be solved by the invention]

[0005] An object of the present invention is to provide a viewing assistance system, a viewing assistance method, and a program that can easily and appropriately record displayed content. [Means for solving the problem]

[0006] The viewing support system of the embodiment has a display unit, a capture image generation unit, a voice recognition unit, and a linked storage unit. The display unit has a display screen capable of displaying content having multiple pages. The capture image generation unit generates a capture image of the content to be displayed on the display screen. The voice recognition unit recognizes voice included in the content. The linked storage unit links a voice recognition result, which is a result of the voice recognition unit recognizing the voice included in the content, to the capture image generated by the capture image generation unit and stores the voice recognition result. [Brief description of the drawings]

[0007] [Figure 1] 1 is a diagram illustrating an example of a configuration of a viewing support system according to a first embodiment. [Diagram 2] 11 is a diagram for explaining an example in which a captured image and a voice recognition result linked and stored by a linked storage unit are displayed on a second display screen of the display unit. FIG. [Diagram 3] 11 is a diagram for explaining an example of a page turning determination algorithm of a page turning detection unit. FIG. [Figure 4] 10 is a diagram for explaining an example of a moving image determination algorithm of the moving image determination unit; [Diagram 5] 13 is a diagram for explaining an example of a process performed by a record page deletion unit of the association storage unit; FIG. [Figure 6] 4 is a flowchart illustrating an example of processing executed in the viewing support system of the first embodiment. [Figure 7] FIG. 4 is a diagram showing an example of a second display screen of the display unit. [Figure 8] FIG. 13 is a diagram illustrating an example of the configuration of a viewing support system according to a fourth embodiment. [Figure 9]FIG. 13 is a diagram illustrating an example of the configuration of a viewing support system according to a fifth embodiment. [Figure 10] 11 is a diagram for explaining an example of a moving image area determination algorithm of the moving image determination unit. FIG. [Figure 11] FIG. 23 is a diagram for explaining an application example of the viewing support system of the eighth embodiment, in which minutes are included in a page of content displayed on the display screen of the display unit. [Figure 12] FIG. 13 is a diagram illustrating an example of the configuration of a viewing support system according to a ninth embodiment. [Figure 13] FIG. 13 is a diagram for explaining an example of a scroll determination function of the viewing support system according to the ninth embodiment. [Figure 14] FIG. 23 is a diagram illustrating an example of the configuration of a viewing support system according to a tenth embodiment. [Figure 15] FIG. 23 is a diagram illustrating an example of the configuration of a viewing support system according to an eleventh embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, a viewing support system, a viewing support method, and a program according to an embodiment will be described with reference to the drawings.

[0009] First Embodiment FIG. 1 is a diagram showing an example of the configuration of a viewing support system 1 according to the first embodiment. 1, the viewing support system 1 is a terminal device having, for example, a function of an online conference app and a function of linking and saving, which will be described later. The viewing support system 1 includes a display unit 1A, a capture image generation unit 1B, a voice recognition unit 1C, a linking and saving unit 1D, a page turning detection unit 1E, a distribution unit 1F, a communication unit 11, a processing unit 12, an input unit 13, and an output unit 14. The display unit 1A has a first display screen and a second display screen. The display unit 1A displays content having multiple pages (such as presentation materials used in online conferences) on the first display screen. The capture image generating unit 1B generates a capture image of the content displayed on the first display screen of the display unit 1A. For example, the capture image generating unit 1B generates a plurality of capture images at a preset capture image generation interval (e.g., every 3 seconds) during the period in which the content is displayed on the first display screen of the display unit 1A. The voice recognition unit 1C recognizes the voice included in the content (for example, the speech of a presenter of an online conference) and outputs the voice recognition result. In detail, the voice recognition unit 1C recognizes the voice data included in the content and generates text data.

[0010] The linked storage unit 1D stores the voice recognition result by the voice recognition unit 1C in association with the capture image generated by the capture image generation unit 1B. In addition to the voice recognition result by the voice recognition unit 1C, the linked storage unit 1D may also store notes, which are memos input by the user of the viewing support system 1, in association with the capture image generated by the capture image generation unit 1B. The linked storage section 1D includes a recording page generation section 1D1 and a recording page deletion section 1D2. The record page generation unit 1D1 generates a plurality of record pages in which the voice recognition results are linked to a plurality of capture images generated by the capture image generation unit 1B at intervals of, for example, 3 seconds, as candidates for storage pages in which the voice recognition results are linked to the capture images stored by the linking storage unit 1D. The recording page deleting section 1D2 has a function of deleting some of the recording pages generated by the recording page generating section 1D1. When the recording page deletion unit 1D2 deletes a portion of a plurality of recording pages, the linking storage unit 1D links the voice recognition result linked to the capture image constituting the recording page to be deleted to the capture image constituting the recording page that is not deleted, and stores it as a saved page.

[0011] The above-mentioned display unit 1A not only has the function of displaying content having multiple pages (such as presentation materials used in online conferences, etc.) on the first display screen, but also has the function of displaying captured images and voice recognition results linked and stored by the linking storage unit 1D on the second display screen. In the example shown in FIG. 1, the display unit 1A has a first display screen and a second display screen, but in other examples, a video of an online conference and a captured image may be displayed on the same display screen.

[0012] FIG. 2 is a diagram for explaining an example in which a captured image and a voice recognition result that are linked and stored by the linked storage unit 1D are displayed on the second display screen of the display unit 1A. 2, the viewing support system 1 of the first embodiment is used in an online conference. A presenter of the online conference makes a presentation on an "online medical consultation system," an "operator system," a "shopping system," etc., by using a multi-page presentation material. When the presenter of the online conference starts the presentation on the "Shopping System," the user of the viewing support system 1 causes the second display screen of the display unit 1A to display the captured image and voice recognition results of the "Online Consultation System" that have been linked and stored by the linking storage unit 1D (i.e., the presentation has already ended) and the captured image and voice recognition results of the "Operator System." In the example shown in Fig. 1 and Fig. 2, the content displayed on the first display screen of the display unit 1A can be easily and appropriately recorded (saved). In detail, the pages ("Online Consultation System" and "Operator System" pages) displayed before the currently displayed page ("Shopping System" page) can be viewed on the second display screen, and presentation materials for an online conference can be displayed on the first display screen of the display unit 1A. In addition, a captured image of the "Online Consultation System" can be linked to a voice recognition result and displayed on the second display screen of the display unit 1A, and a captured image of the "Operator System" can be linked to a voice recognition result and displayed on the second display screen of the display unit 1A.

[0013] 1, page turning detection unit 1E detects a page turning of the content displayed on the first display screen of display unit 1A. For example, page turning detection unit 1E detects the switching to the next page of a presentation material used in an online conference as a "page turning." The linked storage unit 1D links the voice recognition result to the capture image generated by the capture image generation unit 1B based on the page turning detection result by the page turning detection unit 1E and stores the result. In other words, the linked storage unit 1D links the voice recognition result to the capture image generated by the capture image generation unit 1B and stores the result without the need for an instruction from the user of the viewing support system 1.

[0014] FIG. 3 is a diagram for explaining an example of a page turning determination algorithm of the page turning detection unit 1E. In the example shown in FIG. 3, the capture image generating section 1B generates a plurality of capture images at a preset capture image generating interval. Specifically, at time t1, the capture image generating unit 1B generates a capture image. The page turning detection unit 1E compares the captured image with images saved before time t1 and determines whether there has been a change. In the example shown in Fig. 3, since there is no image saved before time t1, the captured image generated at time t1 is saved. Next, at time t2 (e.g., 3 seconds after time t1), the capture image generating unit 1B generates a capture image. The page turning detection unit 1E compares the capture image generated at time t2 with the capture image generated and stored at time t1, and determines whether or not there has been a change. In the example shown in FIG. 3, the capture image generated at time t2 is the same as the capture image generated at time t1, and it is determined that a page has not been turned. As a result, the capture image generated at time t2 is discarded without being stored. The voice recognition result at time t2 is associated with the capture image generated and stored at time t1. Next, at time t3 (e.g., 3 seconds after time t2), the capture image generating unit 1B generates a capture image. The page turning detection unit 1E compares the capture image generated at time t3 with the capture image generated and stored at time t1, and determines whether or not there has been a change. In the example shown in FIG. 3, the capture image generated at time t3 and the capture image generated at time t1 are the same, and it is determined that a page has not been turned. As a result, the capture image generated at time t3 is discarded without being stored. The voice recognition result at time t3 is associated with the capture image generated and stored at time t1. Next, at time t4 (e.g., 3 seconds after time t3), the capture image generating unit 1B generates a capture image. The page turning detection unit 1E compares the capture image generated at time t4 with the capture image generated and stored at time t1, and determines whether or not there has been a change. In the example shown in FIG. 3, the capture image generated at time t4 differs from the capture image generated at time t1, and it is determined that a page has been turned. As a result, the capture image generated at time t4 is stored. The voice recognition result at time t4 is associated with the capture image generated and stored at time t4. In this way, the captured image is saved only when the page is turned.

[0015] In the example shown in FIG. 3, in order for the page turning detection unit 1E to determine whether or not a page has been turned at time t2, the page turning detection unit 1E performs, for example, the following processing on the capture image generated by the capture image generation unit 1B at time t1 and the capture image generated by the capture image generation unit 1B at time t2. First, the page turning detection unit 1E performs gray conversion on those captured images in order to reduce the calculation cost. Next, the page turning detection unit 1E performs resolution conversion (reducing the resolution) on those captured images to reduce calculation costs, and normalizes them by the size of the captured images so that they can be compared with a threshold value for judgment. Next, the page turning detection unit 1E executes processing using a Gaussian filter in order to improve the tolerance to slight misalignment by blurring the image. Next, the page turning detection unit 1E calculates the sum of the differences to calculate the total value of the difference. Next, the page turning detection unit 1E compares the sum of the differences with a first threshold value, and determines that a page has been turned if the sum of the differences exceeds the first threshold value.

[0016] 1, the page turning detection unit 1E includes a moving image determination unit 1E1. The moving image determination unit 1E1 determines whether or not a moving image is being displayed on the first display screen of the display unit 1A.

[0017] FIG. 4 is a diagram for explaining an example of a moving image determination algorithm of the moving image determination unit 1E1. In the example shown in FIG. 4, the moving image determination section 1E1 divides the first display screen of the display section 1A into a plurality of windows of a fixed size in order to determine whether or not a moving image is being displayed on the first display screen. In the example shown in Fig. 3, as described above, captured images generated at intervals of, for example, 3 seconds are used to determine whether a page has been turned or not. On the other hand, in the example shown in Fig. 4, frames of a moving image acquired at intervals of, for example, 1 / 30 seconds are used to determine whether a moving image is being displayed on the first display screen of the display unit 1A. Specifically, in the example shown in Fig. 4, the moving image determination unit 1E1 compares an image displayed in a predetermined window on the first display screen of the display unit 1A at a predetermined time with an image displayed in the same window on the first display screen of the display unit 1A 1 / 30 seconds after the predetermined time, and calculates the amount of change. The calculation of the amount of change in the example shown in Fig. 4 is performed by using, for example, an algorithm similar to the algorithm used in the example shown in Fig. 3. Moving image determination unit 1E1 determines that moving images are being displayed on the first display screen of display unit 1A when the ratio of the number of changed windows to the total number of windows is equal to or greater than a second threshold value. 1, when the moving image determination unit 1E1 determines that a moving image is displayed on the first display screen of the display unit 1A, the page turning detection unit 1E stops the function of detecting page turning. As a result, it is possible to reduce the risk that the page turning detection unit 1E erroneously detects that a page has been turned when in fact no page has been turned. When the page turning detection unit 1E stops the function of detecting page turning, the user of the viewing support system 1 manually determines whether or not a page has been turned.

[0018] In the example shown in FIG. 1, when there are multiple capture images that are targets (candidates) for linking a voice recognition result, the allocation unit 1F executes a process of allocating the linked voice recognition result to one of the multiple capture images. For example, when the capture image generation unit 1B generates a capture image (pre-page turning capture image) (in the example shown in FIG. 3, the capture image at time t1) of the content displayed on the first display screen of the display unit 1A before the timing of page turning (the timing between time t3 and time t4 in the example shown in FIG. 3) and a capture image (post-page turning capture image) (in the example shown in FIG. 3, the capture image at time t4) of the content displayed on the first display screen of the display unit 1A after the timing of page turning, and when the voice recognized by the voice recognition unit 1C straddles the timing of page turning, the allocation unit 1F allocates the result of recognition by the voice recognition unit 1C of the voice straddling the timing of page turning (straddle voice recognition result) to either the pre-page turning capture image (in the example shown in FIG. 3, the capture image at time t1) or the post-page turning capture image (in the example shown in FIG. 3, the capture image at time t4). That is, the allocating unit 1F has a function of determining whether the cross-page speech recognition result is allocated to the pre-page-turning capture image or the post-page-turning capture image.

[0019] In a first example of the viewing support system 1 of the first embodiment, the allocation unit 1F allocates a spanning voice recognition result (for example, the recognition result of the speech of the presenter of an online conference saying "Now, I will explain the next page.") to a capture image of the content displayed on the first display screen of the display unit 1A when the voice included in the content is interrupted (capture image at the time of voice interruption), which is one of the capture image before page turning (capture image at time t1 in the example shown in FIG. 3) and the capture image after page turning (capture image at time t4 in the example shown in FIG. 3). The linking storage unit 1D links the spanning voice recognition result to the capture image at the time of voice interruption and stores it. In a second example of the viewing support system 1 of the first embodiment, the allocation unit 1F allocates a spanning voice recognition result (for example, the recognition result of the speech of the presenter of an online conference saying "Now, I will explain the next page.") to a capture image (capture image at the time of outputting the voice recognition result) of the content displayed on the first display screen of the display unit 1A when the voice recognition unit 1C outputs the spanning voice recognition result, which is one of the capture image before page turning (capture image at time t1 in the example shown in FIG. 3) and the capture image after page turning (capture image at time t4 in the example shown in FIG. 3). The linking storage unit 1D links the spanning voice recognition result to the capture image at the time of outputting the voice recognition result and stores it.

[0020] In the example shown in FIG. 1, the communication unit 11 performs processing such as receiving the content to be displayed on the first display screen of the display unit 1A from outside the viewing support system 1. The processing unit 12 executes various processes other than those executed by the display unit 1A, the capture image generating unit 1B, the voice recognizing unit 1C, the association storing unit 1D, the page turning detecting unit 1E, the sorting unit 1F and the communication unit 11 described above. The input unit 13 accepts input operations by a user of the viewing support system 1 or the like. The output unit 14 executes a process for outputting data generated in the viewing support system 1, excluding display by the display unit 1A.

[0021] FIG. 5 is a diagram for explaining an example of a process performed by the recorded page deletion section 1D2 of the association storage section 1D. In the example shown in FIG. 5, when the record page deletion unit 1D2 deletes some (the lower three record pages of the four record pages shown in FIG. 5(A)) of the multiple record pages (the voice recognition results (and notes (not shown in FIG. 5)) linked to the saved captured images), the linking storage unit 1D links the voice recognition result “Today, I will explain about patent topics” linked to the captured image constituting the record page to be deleted (the second record page from the top in FIG. 5(A)), the voice recognition result “Feel free to ask me any questions” linked to the captured image constituting the record page to be deleted (the third record page from the top in FIG. 5(A)), and the voice recognition result “Let’s get started” linked to the captured image constituting the record page to be deleted (the fourth record page from the top in FIG. 5(A)), to the captured image constituting the record page not to be deleted (the first record page from the top in FIG. 5(A)), and stores them as the saved pages shown in FIG. 5(B).

[0022] Although not shown in the figure, when the record page deletion unit 1D2 deletes some of multiple record pages (storage capture images to which voice recognition results are linked) and there are multiple stored pages stored by the linking storage unit 1D, the voice recognition result linked to the capture image constituting the deleted record page is linked to the capture image constituting the non-deleted record page that was generated by the capture image generation unit 1B at a generation time that is earlier than the generation time of the capture image constituting the deleted record page and that is immediately prior to the generation time of the capture image constituting the deleted record page.

[0023] Although not shown, when the record page deletion unit 1D2 deletes a portion of a plurality of record pages, and there is a stored page stored by the linked storage unit 1D, and there is no record page that will not be deleted and was generated by the capture image generation unit 1B at a generation time earlier than the generation time of the capture image that constitutes the record page to be deleted, the voice recognition result that was linked to the capture image that constitutes the record page to be deleted is linked to the capture image that constitutes the record page that will not be deleted and was generated by the capture image generation unit 1B at a generation time later than the generation time of the capture image that constitutes the record page to be deleted.

[0024] FIG. 6 is a flowchart illustrating an example of processing executed in the viewing support system 1 of the first embodiment. In the example shown in FIG. 6, in step S1, the display unit 1A of the viewing support system 1, which has the function of linking and saving the function of an online conference app, displays content having multiple pages (e.g., presentation materials used in online conferences, etc.) on a first display screen. Next, in step S2, the capture image generating section 1B of the viewing support system 1 generates a capture image of the content displayed on the first display screen of the display section 1A in step S1. Furthermore, in step S3, the voice recognition unit 1C of the viewing support system 1 recognizes the voice included in the content displayed on the first display screen of the display unit 1A in step S1. Next, in step S4, the association storage unit 1D of the viewing support system 1 acquires the capture image generated in step S2. In addition, the association storage unit 1D of the viewing support system 1 associates the voice recognition result, which is the result of recognizing the voice included in the content in step S3, with the acquired capture image and stores it.

[0025] FIG. 7 is a diagram showing an example of the second display screen (specifically, the main screen) of the display unit 1A. In the example shown in FIG. 7, the viewing support system 1 of the first embodiment is used for viewing presentation materials of an online conference. The second display screen of the display unit 1A of the viewing support system 1, which has the function of linking and saving the function of the online conference app shown in Figure 7, is composed of, for example, a touch panel, and also functions as the input unit 13 of the viewing support system 1 that accepts input operations by the user of the viewing support system 1. 7 accepts an input operation by the user of the viewing support system 1 for saving (generating) a captured image of the presentation materials (contents). In other words, the "Capture" button is a button for manually saving (generating) a captured image in response to an input operation by the user of the viewing support system 1, rather than for the captured image generating unit 1B of the viewing support system 1 to automatically generate a captured image. The button shown in Figure 7 as "Page forward of recorded image" is a capture image stored by the linking storage unit 1D of the viewing support system 1, and is a button for page forward of the capture image displayed on the second display screen of the display unit 1A of the viewing support system 1. The button shown in Figure 7 as "Next to Latest Page" is a capture image stored by the linked storage unit 1D of the viewing support system 1, and is a button for updating the capture image displayed on the second display screen of the display unit 1A of the viewing support system 1 to the latest one (i.e., the captured image that was most recently saved). In FIG. 7, a “captured still image” indicates a captured image stored by the linked storage unit 1D of the viewing support system 1 and displayed on the second display screen of the display unit 1A of the viewing support system 1.

[0026] The button indicated in FIG. 7 as "Automatic capture / recording / voice recognition ON" is a button for switching between an ON mode in which the capture image generation unit 1B of the viewing support system 1 automatically generates a capture image of the content displayed on the first display screen of the display unit 1A of the viewing support system 1, the processing unit 12, for example, of the viewing support system 1 automatically records the audio contained in the content, and the audio recognition unit 1C of the viewing support system 1 automatically recognizes the audio contained in the content, and a stopped mode in which these operations are not performed automatically. The button indicated as "Mute" in FIG. 7 is a button for switching the mute function (more specifically, for switching whether or not the speech of the user of the viewing support system 1 is recorded). The button shown in Figure 7 as "Page-turning detection ON / OFF" is a button for switching between an ON mode in which the page-turning detection unit 1E of the viewing support system 1 automatically detects page turning of the content displayed on the first display screen of the display unit 1A of the viewing support system 1, and an OFF mode in which the function of the page-turning detection unit 1E of the viewing support system 1 is turned OFF and the determination of whether or not a page turning has occurred is made manually by the user of the viewing support system 1. An icon indicated as "delete image" in FIG. 7 accepts an input operation of the viewing support system 1 when a user of the viewing support system 1 deletes a captured image stored by the association storage unit 1D of the viewing support system 1. The portion indicated as "playback of recorded audio" in FIG. 7 accepts input operations of the viewing support system 1 when a user of the viewing support system 1 plays back audio recorded by, for example, the processing unit 12 of the viewing support system 1.

[0027] The button shown in Figure 7 as "Audio jump to page position" is a button for cueing playback of audio (audio recorded by, for example, processing unit 12 of viewing support system 1) contained in the content that was displayed on the first display screen of display unit 1A of viewing support system 1 when the capture image displayed on the second display screen of display unit 1A of viewing support system 1 was generated by capture image generation unit 1B of viewing support system 1. The portion shown in Figure 7 as "Voice recognition result display" shows the result of voice recognition performed by the voice recognition unit 1C of the viewing support system 1 when the captured image displayed on the second display screen of the display unit 1A of the viewing support system 1 was saved by the linking storage unit 1D of the viewing support system 1. In the example shown in FIG. 7, the voice recognition unit 1C performs voice recognition without identifying the speaker (i.e., information on whether the speaker is the same or different is not included in the voice recognition result), but in other examples, the voice recognition unit 1C may identify the speaker and perform voice recognition. The identification method may use a known speaker identification technique, or may simply separate the voice from the system side and the voice from the microphone side.

[0028] In an example where the speech recognition results show both the speech of the user (user of viewing support system 1) and the other party (e.g., the other party (presenter) in an online conference) being displayed on the second display screen of display unit 1A of viewing support system 1, for example, the following is displayed on the second display screen of display unit 1A of viewing support system 1. [Me] How much does it cost per month to use this system? [Other party] Fees start from 10,000 yen per month. [Me] Could you please send me detailed information about the table on this page? [Recipient] Understood. I will send it immediately. In the above example (an example in which the viewing support system 1 is used for an online conference), the interaction between "yourself" and "the other party" in the online conference is automatically recorded by the viewing support system 1 and displayed as a voice recognition result on the second display screen of the display unit 1A of the viewing support system 1. In other words, the viewing support system 1 of the first embodiment can be conveniently used not only for one-way webinar viewing but also for online conferences.

[0029] The portion shown as "Note" in Figure 7 indicates a note entered by the user of the viewing support system 1 when the captured image displayed on the second display screen of the display unit 1A of the viewing support system 1 was saved by the linking storage unit 1D of the viewing support system 1. In other words, in the example shown in Figure 7, not only is the voice recognition result by the voice recognition unit 1C of the viewing support system 1 linked to the captured image stored by the linking storage unit 1D of the viewing support system 1, but the "notes," which are notes entered by the user of the viewing support system 1, are also linked to the captured image stored by the linking storage unit 1D of the viewing support system 1.

[0030] As described above, in the viewing support system 1 of the first embodiment having the function of storing in association with the function of an online conference app, for example, screen capture (generation of a captured image of the content displayed on the first display screen of the display unit 1A) can be performed at an appropriate timing from materials displayed on the first display screen of the display unit 1A of the viewing support system 1 in an online conference, and can be recorded in association with the voice recognition result. In this case, the capture can be performed appropriately according to changes in the first display screen of the display unit 1A of the viewing support system 1, and the voice recognition result can be appropriately associated with the captured image based on the voice recognition result and the captured content. Furthermore, when deleting an image, appropriate processing can be performed so as not to delete the recognition result.

[0031] The viewing support system 1 of the first embodiment, which has a function of saving in association with the function of an online conference app, can be used as, for example, an online conference app. The display unit 1A of the viewing support system 1 of the first embodiment can be applied to, for example, video content, animation content, and the like having multiple pages (i.e., requiring page turning). When the viewing support system 1 of the first embodiment having a function of saving in association with the function of an online conference app is applied to an online conference, the user of the viewing support system 1 who is an attendee of the online conference can properly record the online conference. In addition, the user of the viewing support system 1 can view a page different from the page on which the presentation is being made during the online conference, thereby deepening his / her understanding of the online conference. In each of the above-described examples, the viewing support system 1 of the first embodiment is a single terminal device having the function of an online conference app and the function of linking and saving, but in other examples, the viewing support system 1 of the first embodiment may separately include a first terminal device having the function of an online conference app and a second terminal device having the function of linking and saving.

[0032] <Second embodiment> A second embodiment of the viewing support system, the viewing support method, and the program of the present invention will be described below. The viewing support system 1 of the second embodiment is configured similarly to the viewing support system 1 of the above-mentioned first embodiment, except for the points described below. Therefore, the viewing support system 1 of the second embodiment can achieve the same effects as the viewing support system 1 of the above-mentioned first embodiment, except for the points described below.

[0033] In the viewing support system 1 of the second embodiment, similarly to the viewing support system 1 of the first embodiment, when there are multiple capture images that are targets (candidates) for linking the voice recognition result, the allocation unit 1F executes a process of allocating the linked voice recognition result to one of the multiple capture images.

[0034] As described above, in the first example of the viewing support system 1 of the first embodiment, the allocation unit 1F allocates the spanning voice recognition result to a capture image of the content displayed on the first display screen of the display unit 1A of the viewing support system 1 when the voice included in the content is interrupted (capture image at the time of the voice interruption), which is one of the capture image before the page turning and the capture image after the page turning. The linking storage unit 1D links the spanning voice recognition result to the capture image at the time of the voice interruption and stores it. As described above, in the second example of the viewing support system 1 of the first embodiment, the allocation unit 1F allocates the spanning voice recognition result to a capture image (capture image at the time of outputting the voice recognition result) of the content displayed on the first display screen of the display unit 1A of the viewing support system 1 when the voice recognition unit 1C outputs the spanning voice recognition result, which is one of the capture image before page turning and the capture image after page turning. The linking storage unit 1D links the spanning voice recognition result to the capture image at the time of outputting the voice recognition result and stores it.

[0035] On the other hand, in the viewing support system 1 of the second embodiment, the allocation unit 1F allocates the spanning speech recognition result (for example, the recognition result of the speech of the presenter of the online conference saying "Now, I will explain the next page.") to a capture image (capture image at the end of the speech) of the content displayed on the first display screen of the display unit 1A of the viewing support system 1 at the end of the speech corresponding to the spanning speech recognition result (that is, the timing of the end of the spoken sentence. The timing when the speech recognition result of Japanese becomes a period.) which is one of the capture image before the page turning (capture image at time t1 in the example shown in FIG. 3) and the capture image after the page turning (capture image at time t4 in the example shown in FIG. 3). The linking storage unit 1D links the spanning speech recognition result to the capture image at the end of the speech and stores it.

[0036] <Third embodiment> A third embodiment of the viewing support system, the viewing support method, and the program of the present invention will be described below. The viewing support system 1 of the third embodiment is configured similarly to the viewing support system 1 of the above-mentioned first embodiment, except for the points described below. Therefore, the viewing support system 1 of the third embodiment can achieve the same effects as the viewing support system 1 of the above-mentioned first embodiment, except for the points described below.

[0037] In the viewing support system 1 of the third embodiment, the allocation unit 1F allocates the spanning voice recognition result (for example, the recognition result of the speech of the presenter of the online conference saying "Up to this point, I have explained the contents of the previous page.") to a capture image (capture image at the start of speech) of the content displayed on the first display screen of the display unit 1A of the viewing support system 1 at the start of the speech corresponding to the spanning voice recognition result (that is, for example, the timing when the presenter of the online conference starts speaking), which is one of the capture image before page turning (capture image at time t1 in the example shown in FIG. 3) and the capture image after page turning (capture image at time t4 in the example shown in FIG. 3). The linking storage unit 1D links the spanning voice recognition result to the capture image at the start of speech and stores it.

[0038] <Fourth embodiment> A fourth embodiment of the viewing support system, the viewing support method, and the program of the present invention will be described below. The viewing support system 1 of the fourth embodiment is configured similarly to the viewing support system 1 of the above-mentioned first embodiment, except for the points described below. Therefore, the viewing support system 1 of the fourth embodiment can achieve the same effects as the viewing support system 1 of the above-mentioned first embodiment, except for the points described below.

[0039] FIG. 8 is a diagram showing an example of the configuration of a viewing support system 1 according to the fourth embodiment. In the example shown in FIG. 8, the allocating unit 1F includes an utterance voice content recognizing unit 1F1, an image content recognizing unit 1F2, and a similarity calculating unit 1F3. The speech content recognition unit 1F1 recognizes the content of the speech corresponding to the straddling speech recognition result (for example, the recognition result of the speech of a presenter in an online conference saying, "So far, I have been explaining about XX. Next, I will explain about XX."). The image content recognition unit 1F2 recognizes the contents of the captured image before page turning (the captured image at time t1 in the example shown in FIG. 3) and the captured image after page turning (the captured image at time t4 in the example shown in FIG. 3). The similarity calculation unit 1F3 calculates a first similarity which is a similarity between the content of the speech voice recognized by the speech voice content recognition unit 1F1 and the content of the capture image before turning the page recognized by the image content recognition unit 1F2. The similarity calculation unit 1F3 also calculates a second similarity which is a similarity between the content of the speech voice recognized by the speech voice content recognition unit 1F1 and the content of the capture image after turning the page recognized by the image content recognition unit 1F2.

[0040] When the first similarity is higher than the second similarity, the allocating unit 1F allocates the spanning speech recognition result to the pre-page-turning capture image, and the associated storage unit 1D stores the spanning speech recognition result in association with the pre-page-turning capture image. On the other hand, if the first similarity is lower than the second similarity, the allocating unit 1F allocates the spanning speech recognition result to the captured image after page turning. The linked storage unit 1D links the spanning speech recognition result to the captured image after page turning and stores it.

[0041] For example, in an example where the cross-over speech recognition result is the recognition result of the speech of a presenter in an online conference saying, "Up to this point, I have been explaining about XX. Next, I will explain about XX," no matter when the page-turning detection unit 1E detects a page turn in the speech saying, "Up to this point, I have been explaining about XX. Next, I will explain about XX," this cross-over speech recognition result is assigned to the capture image before the page turn if the first similarity is higher than the second similarity, and assigned to the capture image after the page turn if the first similarity is lower than the second similarity.

[0042] <Fifth embodiment> A fifth embodiment of the viewing support system, the viewing support method, and the program of the present invention will be described below. The viewing support system 1 of the fifth embodiment is configured similarly to the viewing support system 1 of the first embodiment described above, except for the points described below. Therefore, the viewing support system 1 of the fifth embodiment can achieve the same effects as the viewing support system 1 of the first embodiment described above, except for the points described below.

[0043] FIG. 9 is a diagram showing an example of the configuration of a viewing support system 1 according to the fifth embodiment. 9, the sorting unit 1F includes a keyword determination unit 1F4. The keyword determination unit 1F4 determines whether or not a preset keyword (e.g., "Up to here," "It was," "Next," "Continued," etc.) is included in the speech corresponding to the speech recognition result. When the keyword determination unit 1F4 determines that a preset keyword is included in the speech corresponding to the spanning speech recognition result, the allocation unit 1F allocates the spanning speech recognition result to either the capture image before page turning (the capture image at time t1 in the example shown in FIG. 3) or the capture image after page turning (the capture image at time t4 in the example shown in FIG. 3) based on the keyword included in the speech corresponding to the spanning speech recognition result. The linking storage unit 1D links the spanning speech recognition result to the capture image to which it is allocated and stores it. When the keyword "up to here..." is included in the speech corresponding to the spanning speech recognition result, the allocating unit 1F allocates the spanning speech recognition result to the pre-page-turning capture image (the capture image at time t1 in the example shown in FIG. 3). When the keyword "was..." is included in the speech corresponding to the spanning speech recognition result, the allocating unit 1F allocates the spanning speech recognition result to the pre-page-turning capture image (the capture image at time t1 in the example shown in FIG. 3). When the keyword "next..." is included in the speech corresponding to the spanning speech recognition result, the allocating unit 1F allocates the spanning speech recognition result to the post-page-turning capture image (the capture image at time t4 in the example shown in FIG. 3). When the keyword "continues..." is included in the speech corresponding to the spanning speech recognition result, the allocating unit 1F allocates the spanning speech recognition result to the post-page-turning capture image (the image captured at time t4 in the example shown in FIG. 3).

[0044] Sixth Embodiment A sixth embodiment of the viewing support system, the viewing support method, and the program of the present invention will be described below. The viewing support system 1 of the sixth embodiment is configured similarly to the viewing support system 1 of the first embodiment, except for the points described below. Therefore, the viewing support system 1 of the sixth embodiment can achieve the same effects as the viewing support system 1 of the first embodiment, except for the points described below.

[0045] As described above, in the viewing support system 1 of the first embodiment, the moving image determination unit 1E1 determines whether or not a moving image is being displayed on the first display screen of the display unit 1A. On the other hand, in the viewing support system 1 of the sixth embodiment, the moving image determination unit 1E1 determines whether or not a moving image is included in each of the multiple pages of the content displayed on the first display screen of the display unit 1A.

[0046] In the viewing support system 1 of the sixth embodiment, for a page that is determined by the moving image determination unit 1E1 not to contain a moving image, the page turning detection unit 1E does not stop the function of detecting page turning. On the other hand, for a page that is determined by the video determination unit 1E1 to include a video, the page turning detection unit 1E stops the function of detecting page turning. As a result, the determination of whether a page has been turned is manually performed by the user of the viewing support system 1. Also, the capture image generation unit 1B generates a capture image of the content displayed on the first display screen of the display unit 1A in response to an input operation by the user of the viewing support system 1.

[0047] Seventh embodiment A seventh embodiment of the viewing support system, the viewing support method, and the program of the present invention will be described below. The viewing support system 1 of the seventh embodiment is configured similarly to the viewing support system 1 of the above-mentioned first embodiment, except for the points described below. Therefore, the viewing support system 1 of the seventh embodiment can achieve the same effects as the viewing support system 1 of the above-mentioned first embodiment, except for the points described below.

[0048] As described above, in the viewing support system 1 of the first embodiment, when the video determination unit 1E1 determines that a video is being displayed on the first display screen of the display unit 1A, the page turning detection unit 1E stops the function of detecting page turning. On the other hand, in the seventh embodiment of the viewing support system 1, when the video determination unit 1E1 determines that a video is being displayed on the first display screen of the display unit 1A, the page turning detection unit 1E detects a page turning by excluding the portion of the video included in the first display screen of the display unit 1A.

[0049] FIG. 10 is a diagram for explaining an example of a moving image area determination algorithm of the moving image determination section 1E1. In the example shown in FIG. 10, moving image determination section 1E1 divides the first display screen of display section 1A into a plurality of windows of a fixed size in order to determine a portion of moving images (moving image area) included in the first display screen. In the example shown in FIG. 10, in order to determine the moving image area included in the first display screen of the display unit 1A, frames of a moving image acquired at intervals of, for example, 1 / 30 seconds are used, similarly to the example shown in FIG. Specifically, in the example shown in Fig. 10, the moving image determination unit 1E1 compares an image displayed in a predetermined window on the first display screen of the display unit 1A at a predetermined time with an image displayed in the same window on the first display screen of the display unit 1A 1 / 30 seconds after the predetermined time, and calculates the amount of change. The calculation of the amount of change in the example shown in Fig. 10 is performed by using, for example, an algorithm similar to the algorithm used in the example shown in Fig. 3. Moving image determination unit 1E1 determines an area of ​​the first display screen of display unit 1A that includes a changed window as a moving image area. In the viewing support system 1 of the seventh embodiment, the risk of the page turning detection unit 1E erroneously detecting that a page has been turned due to a change in the display state of the video area, even though a page has not actually been turned, can be reduced.

[0050] In the viewing support system 1 of the seventh embodiment, the moving image determination unit 1E1 has a function of determining whether or not a person is displayed on the first display screen of the display unit 1A. As a technique for determining whether or not a person is displayed on the first display screen, for example, a commonly used technique is used. When the moving image determination unit 1E1 determines that a person is displayed on the first display screen of the display unit 1A, the page turning detection unit 1E detects the page turning by excluding the part of the person included in the first display screen of the display unit 1A.

[0051] Eighth embodiment An eighth embodiment of the viewing support system, the viewing support method, and the program of the present invention will be described below. The viewing support system 1 of the eighth embodiment is configured similarly to the viewing support system 1 of the above-mentioned first embodiment, except for the points described below. Therefore, the viewing support system 1 of the eighth embodiment can achieve the same effects as the viewing support system 1 of the above-mentioned first embodiment, except for the points described below.

[0052] As described above, in the viewing support system 1 of the first embodiment, the page turning detection unit 1E compares the amount of change (e.g., the sum of the above-mentioned differences) with a first threshold value, and determines that a page turning has occurred if the amount of change exceeds the first threshold value. On the other hand, when the content having multiple pages displayed on the first display screen of the display unit 1A is minutes (the speed of change is slow, so the amount of change per unit time is small) or animation (only a part of the page changes, so the amount of change of the entire page is small), the amount of change does not exceed the first threshold, and it may be difficult for the page turning detection unit 1E to determine that a page has been turned. As a result, there is a risk that the necessary captured image will not be saved by the association saving unit 1D. Therefore, in the viewing support system 1 of the eighth embodiment, measures to be described later are taken.

[0053] In the viewing support system 1 of the eighth embodiment, similar to the viewing support system 1 of the first embodiment described above, the capture image generation unit 1B generates multiple capture images at a preset capture image generation interval (e.g., every 3 seconds) during the period when the content is displayed on the first display screen of the display unit 1A. In other words, in the viewing support system 1 of the eighth embodiment, the capture image generation unit 1B generates a first capture image (in the example shown in FIG. 3, a capture image at time t3) as a capture image of the content displayed on the first display screen of the display unit 1A at a first time point (in the example shown in FIG. 3, time t3). In addition, the capture image generation unit 1B generates a second capture image (in the example shown in FIG. 3, a capture image at time t4) as a capture image of the content displayed on the first display screen of the display unit 1A at a second time point (in the example shown in FIG. 3, time t4), which is the time point when a preset capture image generation interval (e.g., a 3-second interval) has elapsed from the first time point.

[0054] In the viewing support system 1 of the eighth embodiment, when the amount of change between the second capture image (in the example shown in Figure 3, the capture image at time t4) and the first capture image (in the example shown in Figure 3, the capture image at time t3) exceeds a first threshold value, the page turning detection unit 1E detects a page turning, similar to the viewing support system 1 of the first embodiment. In the viewing support system 1 of the eighth embodiment, unlike the viewing support system 1 of the first embodiment, when the amount of change in the second captured image relative to the first captured image exceeds a first threshold, the linking storage unit 1D not only links the voice recognition result to the second captured image and stores it based on the page turning detection result by the page turning detection unit 1E (in the example shown in Figure 3, the page turning occurs between time t3 and time t4), but also links the voice recognition result to the first captured image and stores it.

[0055] Fig. 11 is an application example of the viewing support system 1 of the eighth embodiment, and is a diagram for explaining an example in which minutes are included in a page of content displayed on the first display screen of the display unit 1A. In detail, Fig. 11(A) shows the first display screen of the display unit 1A, and Fig. 11(B) shows the relationship between the amount of change in the first display screen of the display unit 1A (vertical axis of Fig. 11(B)) and time (horizontal axis of Fig. 11(B)). In the example shown in FIG. 11, at time t11, the page of the content displayed on the first display screen of the display unit 1A switches from a page not shown to a page shown in FIG. 11(A) (specifically, at time t11, the "first line" to the "sixth line" shown in FIG. 11(A) are not yet displayed on the first display screen of the display unit 1A). Therefore, the amount of change in the first display screen of the display unit 1A becomes greater than the first threshold value, and the page turning detection unit 1E detects a page turning. As a result, the association storage unit 1D associates the voice recognition result with the capture image generated by the capture image generation unit 1B (a capture image in which only the characters "minutes" shown in FIG. 11(A) are captured) and stores it. Next, at time t12, the "first line" of the minutes shown in Fig. 11(A) is displayed on the first display screen of the display unit 1A. Therefore, the amount of change in the captured image (the captured image in which the characters "minutes" and the "first line" are captured) increases with respect to the captured image in which only the characters "minutes" shown in Fig. 11(A) are captured. Next, at time t13, the "first line" and "second line" of the minutes shown in Fig. 11(A) are displayed on the first display screen of the display unit 1A. Therefore, the amount of change in the captured image (the captured image in which the characters "minutes", the "first line", and the "second line" are captured) further increases with respect to the captured image in which only the characters "minutes" shown in Fig. 11(A) are captured.

[0056] Next, at time t14, the "first line," "second line," and "third line" of the minutes shown in Fig. 11(A) are displayed on the first display screen of the display unit 1A. Therefore, the amount of change in the captured image (the captured image in which the characters "minutes," "first line," "second line," and "third line" are captured) compared to the captured image in which only the characters "minutes" shown in Fig. 11(A) are captured further increases. Next, at time t15, the first, second, third, and fourth lines of the minutes shown in Fig. 11A are displayed on the first display screen of the display unit 1A. Therefore, the amount of change in the captured image (the captured image in which the characters of "minutes", the first, second, third, and fourth lines) compared to the captured image in which only the characters of "minutes" shown in Fig. 11A are captured further increases and exceeds the first threshold, causing the page turning detection unit 1E to detect page turning. As a result, the linking storage unit 1D links the voice recognition result to the captured image (the captured image including the first, second, third, and fourth lines of the minutes shown in Fig. 11A) generated by the captured image generation unit 1B. Moreover, since the page turning detection section 1E detects a page turning, the amount of change in the first display screen of the display section 1A becomes zero (is reset).

[0057] Next, at time t16, the "1st line", "2nd line", "3rd line", "4th line", and "5th line" of the minutes shown in Fig. 11(A) are displayed on the first display screen of the display unit 1A. Therefore, the amount of change in the captured image (the captured image in which the characters "1st line", "2nd line", "3rd line", "4th line", and "5th line" of "minutes" are captured) increases with respect to the captured image in which the characters "1st line", "2nd line", "3rd line", "4th line", and "5th line" of "minutes" shown in Fig. 11(A) are captured. Next, at time t17, the "1st line", "2nd line", "3rd line", "4th line", "5th line", and "6th line" of the minutes shown in Fig. 11(A) are displayed on the first display screen of the display unit 1A. Therefore, the amount of change in the captured image (the captured image in which the characters "1st line", "2nd line", "3rd line", "4th line", "5th line", and "6th line" of "minutes" are captured) from the captured image in which the characters "1st line", "2nd line", "3rd line", "4th line", "5th line", and "6th line" of "minutes" shown in Fig. 11(A) are captured further increases, but is smaller than the first threshold value. Next, at time t18, the page of the content displayed on the first display screen of the display unit 1A is switched from the page shown in Fig. 11(A) to a page not shown. Therefore, the amount of change in the first display screen of the display unit 1A becomes larger than the first threshold, and the page turning detection unit 1E detects a page turning. As a result, the association storage unit 1D associates the voice recognition result with the capture image (not shown in Fig. 11) generated by the capture image generation unit 1B and stores it.

[0058] In detail, in the example shown in FIG. 11 in which the viewing support system 1 of the eighth embodiment is applied, the linking storage unit 1D not only links and stores the voice recognition result to the captured image including the "first line," "second line," "third line," and "fourth line" of the minutes shown in FIG. 11(A) based on the page turning detection result by the page turning detection unit 1E (in the example shown in FIG. 11, the timing of the page turning is time t15), but also links and stores the captured image including the "first line," "second line," and "third line" of the minutes shown in FIG. 11(A) with the voice recognition result. Furthermore, based on the page-turning detection result by the page-turning detection unit 1E (in the example shown in FIG. 11, the page turning occurs at time t18), the linked storage unit 1D not only links and stores the voice recognition result to the capture image (not shown in FIG. 11) generated by the capture image generation unit 1B at time t18, but also links and stores the voice recognition result to the capture image including the "first line," "second line," "third line," "fourth line," "fifth line," and "sixth line" of the minutes shown in FIG. 11(A).

[0059] <Ninth embodiment> A ninth embodiment of the viewing support system, the viewing support method, and the program of the present invention will be described below. The viewing support system 1 of the ninth embodiment is configured similarly to the viewing support system 1 of the above-mentioned first embodiment, except for the points described below. Therefore, the viewing support system 1 of the ninth embodiment can achieve the same effects as the viewing support system 1 of the above-mentioned first embodiment, except for the points described below.

[0060] FIG. 12 is a diagram showing an example of the configuration of a viewing support system 1 according to the ninth embodiment. In the example shown in FIG. 12, a page turning detection section 1E includes a moving image determination section 1E1 and a character string area determination section 1E2. The character string area determination unit 1E2 has a function of calculating the number of characters contained in each of the multiple capture images generated by the capture image generation unit 1 B. As a technique for calculating the number of characters contained in an image, for example, a commonly used technique is used. In the example shown in FIG. 11, character string area determination unit 1E2 calculates the number of characters contained in a captured image in which only the characters "minutes" shown in FIG. 11(A) is captured, the number of characters contained in a captured image in which the characters "minutes" and "first line" are captured, the number of characters contained in a captured image in which the characters "minutes", "first line" and "second line" are captured, the number of characters contained in a captured image in which the characters "minutes", "first line", "second line" and "third line" are captured, the number of characters contained in a captured image in which the characters "minutes", "first line", "second line", "third line" and "fourth line" are captured, the number of characters contained in a captured image in which the characters "minutes", "first line", "second line", "third line", "fourth line" and "fifth line" are captured, and the number of characters contained in a captured image in which the characters "minutes", "first line", "second line", "third line", "fourth line", "fifth line" and "sixth line" are captured.

[0061] In the example shown in FIG. 12, the character string area determining section 1E2 has a function of selecting the capture image containing the largest number of characters from among the multiple capture images generated by the capture image generating section 1B. In the example shown in FIG. 11, the character string area determination unit 1E2 selects the captured image containing the characters “minutes” in the “1st line”, “2nd line”, “3rd line”, “4th line”, “5th line”, and “6th line” as the captured image containing the largest number of characters.

[0062] In the example shown in FIG. 12, the association storage unit 1D associates the voice recognition result with the capture image selected by the character string area determination unit 1E2 and stores the result. In the example shown in FIG. 11, the linked storage unit 1D stores the voice recognition result in association with a captured image in which the characters “minutes” (line 1), “line 2”, “line 3”, “line 4”, “line 5”, and “line 6” are captured.

[0063] The viewing support system 1 of the ninth embodiment may have a function of determining that the capture image generated by the capture image generating unit 1B is a capture image of the minutes being created. In this example, the viewing support system 1 extracts a character area by performing optical character recognition (OCR) on the capture image generated by the capture image generating unit 1B. When the extracted character area gradually increases over time, the viewing support system 1 determines that the capture image generated by the capture image generating unit 1B is a capture image that captures the minutes being created.

[0064] The viewing support system 1 of the ninth embodiment may have a function of determining that the first display screen of the display unit 1A is scrolling, based on the capture image generated by the capture image generating unit 1B.

[0065] FIG. 13 is a diagram for explaining an example of the scroll determination function of the viewing support system 1 of the ninth embodiment. In the example shown in Figure 13, the viewing support system 1 extracts the "portion between the third and fourth lines" as a matching portion among the capture image generated by the capture image generating unit 1B at time t21 (the capture image shown in Figure 13(A)), the capture image generated by the capture image generating unit 1B at time t22 (the capture image shown in Figure 13(B)), and the capture image generated by the capture image generating unit 1B at time t23 (the capture image shown in Figure 13(C)). The viewing support system 1 determines that the matching portion, "the portion between the third and fourth lines," is moving up and down, and therefore determines that the scrolling is "vertical scrolling." For example, when the matching portion moves left and right, the viewing support system 1 determines that the event is "left and right scrolling."

[0066] In the example shown in FIG. 13, since the proportion of the matching portion "the portion from the second to fourth lines" between the capture image generated by the capture image generating unit 1B at time t21 (the capture image shown in FIG. 13(A)) and the capture image generated by the capture image generating unit 1B at time t22 (the capture image shown in FIG. 13(B)) to the entire capture image is equal to or greater than the third threshold, the page turning detection unit 1E does not detect that a page has been turned at time t22. Since the proportion of the matching portion "the portion between the third and fourth lines" between the capture image generated by capture image generation unit 1B at time t21 (the capture image shown in FIG. 13(A)) and the capture image generated by capture image generation unit 1B at time t23 (the capture image shown in FIG. 13(C)) in the entire capture image is less than the third threshold, at time t23, page turning detection unit 1E detects that a page has been turned, and linking storage unit 1D links the voice recognition result to the capture image generated by capture image generation unit 1B at time t23 and stores it. In the example shown in FIG. 13, the data displayed on the first display screen of the display unit 1A can be appropriately (without omission) saved as a captured image while preventing the linked storage unit 1D from saving more captured images than necessary.

[0067] Tenth embodiment A tenth embodiment of the viewing support system, the viewing support method, and the program of the present invention will be described below. The viewing support system 1 of the tenth embodiment is configured similarly to the viewing support system 1 of the first embodiment described above, except for the points described below. Therefore, the viewing support system 1 of the tenth embodiment can achieve the same effects as the viewing support system 1 of the first embodiment described above, except for the points described below.

[0068] FIG. 14 is a diagram showing an example of the configuration of a viewing support system 1 according to the tenth embodiment. As described above, in the example shown in FIG. 1, the linked storage unit 1D includes a recording page generation unit 1D1 and a recording page deletion unit 1D2. On the other hand, in the example shown in FIG. 14, the linked storage section 1D includes a recording page generation section 1D1 and a recording page change section 1D3.

[0069] In the example shown in FIG. 14, the record page generation unit 1D1 generates a plurality of record pages in which the voice recognition results are linked to a plurality of capture images generated by the capture image generation unit 1B (in the example shown in FIG. 3, the capture image generated at time t1 and the capture image generated at time t4) as candidates for storage pages in which the voice recognition results are linked to the capture images stored by the linking storage unit 1D.

[0070] In the example shown in FIG. 14, a recording page changing section 1D3 has a function of changing a part of a plurality of recording pages generated by a recording page generating section 1D1. When the recording page change unit 1D3 changes a part of a plurality of recording pages, the linking storage unit 1D changes the captured image constituting the changed recording page to a blank image. Furthermore, the linking storage unit 1D links the voice recognition result linked to the captured image constituting the changed recording page to the blank image and stores it as a saved page. 14, when it is not necessary to save the captured image generated by the captured image generating unit 1B, the captured image that does not need to be saved is converted into a blank image. Furthermore, the voice recognition result that was linked to the captured image that does not need to be saved is linked to the blank image and saved as a saved page. Therefore, in the example shown in Figure 14, even if there is no page before or after it, such as the first page of a presentation document, the speech recognition results when that page (e.g. the first page) was displayed can be saved.

[0071] <Eleventh embodiment> An eleventh embodiment of the viewing support system, the viewing support method, and the program of the present invention will now be described. The viewing support system 1 of the 11th embodiment is configured similarly to the viewing support system 1 of the above-mentioned first embodiment, except for the points described below. Therefore, the viewing support system 1 of the 11th embodiment can achieve the same effects as the viewing support system 1 of the above-mentioned first embodiment, except for the points described below.

[0072] FIG. 15 is a diagram showing an example of the configuration of a viewing support system 1 according to the eleventh embodiment. As described above, in the example shown in FIG. 1, the linked storage unit 1D includes a recording page generation unit 1D1 and a recording page deletion unit 1D2. On the other hand, in the example shown in FIG. 15, the linked storage unit 1D includes a recording page generation unit 1D1, a recording page deletion unit 1D2, and an optical character recognition unit 1D4.

[0073] 15, the record page generation unit 1D1 generates a plurality of record pages in which the voice recognition results are linked to a plurality of capture images generated by the capture image generation unit 1B (in the example shown in FIG. 3, the capture image generated at time t1 and the capture image generated at time t4) as candidates for stored pages in which the voice recognition results are linked to the capture images stored by the linked storage unit 1D. The record page deletion unit 1D2 has a function of deleting some of the record pages generated by the record page generation unit 1D1. 15, when the recording page deletion unit 1D2 deletes a part of a plurality of recording pages, the optical character recognition unit 1D4 performs optical character recognition on a previous capture image, which is a capture image constituting a recording page that will not be deleted and which was generated by the capture image generation unit 1B at a generation time earlier than the generation time of the capture image constituting the recording page to be deleted. Furthermore, the optical character recognition unit 1D4 performs optical character recognition on a subsequent capture image, which is a capture image constituting a recording page that will not be deleted and which was generated by the capture image generation unit 1B at a generation time later than the generation time of the capture image constituting the recording page to be deleted.

[0074] The association storage unit 1D calculates a similarity (first similarity) between the optical character recognition result of the previous captured image and the voice recognition result associated with the captured image constituting the recording page to be deleted. Moreover, the association storage unit 1D calculates a similarity (second similarity) between the optical character recognition result of the subsequent captured image and the voice recognition result associated with the captured image constituting the recording page to be deleted. Furthermore, the linking storage unit 1D links the voice recognition result linked to the capture image constituting the recording page to be deleted to either the front capture image or the back capture image based on the first similarity and the second similarity, and stores it as a saved page.

[0075] <Twelfth embodiment> A twelfth embodiment of the viewing support system, the viewing support method, and the program of the present invention will be described below. The viewing support system 1 of the 12th embodiment is configured similarly to the viewing support system 1 of the above-mentioned first embodiment, except for the points described below. Therefore, the viewing support system 1 of the 12th embodiment can achieve the same effects as the viewing support system 1 of the above-mentioned first embodiment, except for the points described below.

[0076] The viewing support system 1 of the twelfth embodiment is a terminal device that does not have the function of an online conference app, but has the function of linking and saving. As described above, the viewing support system 1 of the first embodiment includes the communication unit 11, but the viewing support system 1 of the twelfth embodiment does not include the communication unit 11. The viewing support system 1 of the twelfth embodiment has a function of recording the operation method of a terminal device (PC (personal computer)) and creating a manual for the terminal device. A user of the viewing support system 1 of the 12th embodiment performs a specified operation on the terminal device, and the capture image generation unit 1B generates a capture image of the window at that time (i.e., the capture image generation unit 1B generates a capture image of the first display screen of the display unit 1A at that time). Furthermore, the user of the viewing support system 1 of the twelfth embodiment verbally explains the operation, and the voice recognition unit 1C recognizes the voice (the voice uttered by the user of the viewing support system 1) and outputs the voice recognition result. The associated storage unit 1D stores the voice recognition result by the voice recognition unit 1C in association with the capture image generated by the capture image generation unit 1B.

[0077] <Thirteenth embodiment> Hereinafter, a thirteenth embodiment of the viewing support system, the viewing support method, and the program of the present invention will be described. The viewing support system 1 of the 13th embodiment is configured similarly to the viewing support system 1 of the above-mentioned first embodiment, except for the points described below. Therefore, the viewing support system 1 of the 12th embodiment can achieve the same effects as the viewing support system 1 of the above-mentioned first embodiment, except for the points described below.

[0078] The viewing support system 1 of the thirteenth embodiment is a terminal device that does not have the function of an online conference app, but has the function of linking and saving. As described above, the viewing support system 1 of the first embodiment includes the communication unit 11, but the viewing support system 1 of the thirteenth embodiment does not include the communication unit 11. The viewing support system 1 of the thirteenth embodiment has a function of creating a record of operations performed on a terminal device (PC) by a user of the viewing support system 1. A user of the viewing support system 1 of the 13th embodiment performs a specified operation on the terminal device, and the capture image generation unit 1B generates a capture image of the window at that time (i.e., the capture image generation unit 1B generates a capture image of the first display screen of the display unit 1A at that time). The user of the viewing support system 1 of the thirteenth embodiment orally explains the operation as necessary, and the voice recognition unit 1C recognizes the voice (the voice spoken by the user of the viewing support system 1) and outputs the voice recognition result. The linked storage unit 1D stores the voice recognition result by the voice recognition unit 1C in association with the capture image generated by the capture image generation unit 1B (if the user of the viewing support system 1 does not speak and there is no voice recognition result, only the capture image is stored).

[0079] According to at least one of the embodiments described above, the viewing support system 1 has a display unit 1A, a capture image generation unit 1B, a voice recognition unit 1C, and a linked storage unit 1D. The display unit 1A has a display screen (first display screen) capable of displaying content having multiple pages. The capture image generation unit 1B generates a capture image of the content displayed on the display screen (first display screen). The voice recognition unit 1C recognizes voice included in the content. The linked storage unit 1D links the voice recognition result, which is the result of the voice recognition unit 1C recognizing the voice included in the content, to the capture image generated by the capture image generation unit 1B and stores it. This makes it possible to easily and appropriately record the displayed content.

[0080] At least some of the functions of the viewing support system 1 in the above-mentioned embodiment (including the modified example) may be realized by a computer. In that case, a program for realizing the functions may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the functions. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memories, and storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may include a medium that dynamically holds a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. The above-mentioned program may be a program for realizing some of the above-mentioned functions, or may be a program that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0081] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents described in the claims, as well as in the scope and spirit of the invention. [Explanation of symbols]

[0082] 1...viewing support system, 1A...display unit, 1B...capture image generation unit, 1C...voice recognition unit, 1D...linking storage unit, 1D1...recorded page generation unit, 1D2...recorded page deletion unit, 1D3...recorded page change unit, 1D4...optical character recognition unit, 1E...page turning detection unit, 1E1...video determination unit, 1E2...character string area determination unit, 1F...sorting unit, 1F1...speech content recognition unit, 1F2...image content recognition unit, 1F3...similarity calculation unit, 1F4...keyword determination unit, 11...communication unit, 12...processing unit, 13...input unit, 14...output unit

Claims

1. A display unit having a display screen capable of displaying content having a plurality of pages; a capture image generating unit that generates a capture image of the content to be displayed on the display screen; a voice recognition unit that recognizes voice included in the content; a page turning detection unit that detects a page turning of the content displayed on the display screen; a linking and storing unit that links a voice recognition result, which is a result of the voice recognition unit recognizing the voice included in the content based on the page turning detection result by the page turning detection unit, to the capture image generated by the capture image generation unit and stores the voice recognition result; a capture image generating unit generating a capture image before turning a page, which is a capture image of the content displayed on the display screen before the timing of turning the page, and a capture image after turning a page, which is a capture image of the content displayed on the display screen after the timing of turning the page, When the voice recognized by the voice recognition unit spans before and after the timing of turning the page, an allocating unit that allocates a spanning voice recognition result, which is a result of the voice recognition unit recognizing a voice spanning before and after the timing of the page turning, to one of the captured image before the page turning and the captured image after the page turning; Equipped with the allocation unit has a function of determining whether the result of the speech recognition across pages is to be allocated to the captured image before turning a page or the captured image after turning a page, Viewing assistance system.

2. The allocating unit includes: a speech content recognition unit that recognizes the content of a speech corresponding to the result of the cross-word speech recognition; an image content recognition unit that recognizes the contents of the pre-page-turning capture image and the post-page-turning capture image; a similarity calculation unit that calculates a first similarity, which is a similarity between the content of the utterance voice recognized by the utterance voice content recognition unit and the content of the capture image before turning a page recognized by the image content recognition unit, and a second similarity, which is a similarity between the content of the utterance voice recognized by the utterance voice content recognition unit and the content of the capture image after turning a page recognized by the image content recognition unit, When the first similarity is higher than the second similarity, the allocating unit allocates the result of the inter-page speech recognition to the capture image before turning the page, and the linked storage unit stores the result of the inter-page speech recognition in association with the capture image before turning the page; When the first similarity is lower than the second similarity, the allocating unit allocates the result of the inter-page speech recognition to the captured image after the page turning, and the linked storage unit stores the result of the inter-page speech recognition in a linked manner to the captured image after the page turning. The viewing support system according to claim 1 .

3. The allocating unit includes: a keyword determination unit that determines whether a predetermined keyword is included in the speech corresponding to the cross-domain speech recognition result; When the keyword determination unit determines that the keyword is included in the speech voice, the allocating unit allocates the result of the speech recognition to one of the captured image before turning the page and the captured image after turning the page based on the keyword included in the speech; The linked storage unit links the cross-speech recognition result to a capture image to which the cross-speech recognition result is assigned and stores the result. The viewing support system according to claim 1 .

4. The page turning detection unit is a moving image determination unit that determines whether or not a moving image is being displayed on the display screen; The viewing support system according to claim 1 .

5. When the moving image determination unit determines that the moving image is being displayed on the display screen, The page turning detection unit stops the function of detecting the page turning. The viewing support system according to claim 4.

6. When the moving image determination unit determines that the moving image is being displayed on the display screen, the page turning detection unit detects the page turning while excluding a portion of the video included in the display screen; The viewing support system according to claim 4.

7. A display unit having a display screen capable of displaying content having multiple pages; a capture image generating unit that generates a capture image of the content to be displayed on the display screen; a voice recognition unit that recognizes voice included in the content; a linking storage unit that links a voice recognition result, which is a result of the voice recognition unit recognizing the voice included in the content, to the capture image generated by the capture image generation unit and stores the result; Equipped with the capture image generating unit generates a plurality of capture images at a preset capture image generating interval; The linking storage unit includes a recording page generating unit and a recording page deleting unit, the record page generation unit generates a plurality of record pages in which the voice recognition results are linked to the plurality of capture images generated by the capture image generation unit as candidates for a storage page in which the voice recognition results are linked to the capture images stored by the linking storage unit; the recording page deletion unit has a function of deleting a part of the plurality of recording pages generated by the recording page generation unit, When the recording page deletion unit deletes a part of the plurality of recording pages, the linking storage unit links the voice recognition result linked to the capture image constituting the recording page to be deleted to the capture image constituting the recording page not to be deleted, and stores the result as the saved page. Viewing assistance system.

8. A display unit having a display screen capable of displaying content having multiple pages; a capture image generating unit that generates a capture image of the content to be displayed on the display screen; a voice recognition unit that recognizes voice included in the content; a linking storage unit that links a voice recognition result, which is a result of the voice recognition unit recognizing the voice included in the content, to the capture image generated by the capture image generation unit and stores the result; Equipped with the capture image generating unit generates a plurality of capture images at a preset capture image generating interval; The linking storage unit includes a recording page generating unit and a recording page changing unit, the record page generation unit generates a plurality of record pages in which the voice recognition results are linked to the plurality of capture images generated by the capture image generation unit as candidates for a storage page in which the voice recognition results are linked to the capture images stored by the linking storage unit; the recording page changing unit has a function of changing a part of the plurality of recording pages generated by the recording page generating unit, When the recording page changing unit changes a part of the plurality of recording pages, The linking storage unit is changing the captured images constituting the recording page being modified to blank images; the voice recognition result linked to the capture image constituting the changed recording page is linked to the blank image, and the blank image is stored as the saved page; Viewing assistance system.

9. A display unit having a display screen capable of displaying content having multiple pages; a capture image generating unit that generates a capture image of the content to be displayed on the display screen; a voice recognition unit that recognizes voice included in the content; a linking storage unit that links a voice recognition result, which is a result of the voice recognition unit recognizing the voice included in the content, to the capture image generated by the capture image generation unit and stores the result; Equipped with the capture image generating unit generates a plurality of capture images at a preset capture image generating interval; the linking storage unit includes a recording page generating unit, a recording page deleting unit, and an optical character recognition unit; the record page generation unit generates a plurality of record pages in which the voice recognition results are linked to the plurality of capture images generated by the capture image generation unit as candidates for a storage page in which the voice recognition results are linked to the capture images stored by the linking storage unit; the recording page deletion unit has a function of deleting a part of the plurality of recording pages generated by the recording page generation unit, When the recording page deletion unit deletes a part of the plurality of recording pages, The optical character recognition unit includes: performing optical character recognition on a previous capture image, which is a capture image constituting the recording page that is not to be deleted and which was generated by the capture image generating unit at a generation time prior to the generation time of the capture image constituting the recording page to be deleted; performing optical character recognition on a subsequent capture image, which is a capture image constituting the recording page that is not to be deleted and which was generated by the capture image generating unit at a generation time later than the generation time of the capture image constituting the recording page to be deleted; The linking storage unit is a degree of similarity between the optical character recognition result of the previous captured image and the voice recognition result associated with the captured image constituting the recording page to be deleted; Based on a similarity between the optical character recognition result of the captured image and the voice recognition result associated with the captured image constituting the recording page to be deleted, the voice recognition result associated with the capture image constituting the recording page to be deleted is associated with either the front capture image or the rear capture image, and is stored as the saved page. Viewing assistance system.

10. a capture image acquisition step of acquiring a capture image of a content having a plurality of pages displayed on a display screen; a voice recognition step of recognizing a voice included in the content; a page turning detection step of detecting a page turning of the content displayed on the display screen; a linking and storing step of linking a voice recognition result, which is a result of recognizing a voice included in the content in the voice recognition step, to the capture image acquired in the capture image acquisition step, based on the page turning detection result in the page turning detection step, and storing the voice recognition result; a capture image acquiring step of acquiring a pre-page-turning capture image which is a capture image of the content displayed on the display screen before the timing of turning the page, and a capture image after the page-turning capture image which is a capture image of the content displayed on the display screen after the timing of turning the page, When the voice recognized by the voice recognition step spans before and after the timing of turning the page, a distribution step of distributing a result of the speech recognition process in which a speech that spans before and after the timing of the page turning is recognized by the speech recognition step, to one of the captured image before the page turning and the captured image after the page turning; Equipped with The allocating step determines whether the result of the speech recognition across pages is to be allocated to the captured image before the page turning or the captured image after the page turning. Viewing support methods.

11. To your computer a capture image acquisition step of acquiring a capture image of a content having a plurality of pages displayed on a display screen; a voice recognition step of recognizing a voice included in the content; a page turning detection step of detecting a page turning of the content displayed on the display screen; a linking and storing step of linking a voice recognition result, which is a result of recognizing a voice included in the content in the voice recognition step, to the capture image acquired in the capture image acquisition step, based on the page turning detection result in the page turning detection step, and storing the voice recognition result; a capture image acquiring step of acquiring a pre-page-turning capture image which is a capture image of the content displayed on the display screen before the timing of turning the page, and a capture image after the page-turning capture image which is a capture image of the content displayed on the display screen after the timing of turning the page, When the voice recognized by the voice recognition step spans before and after the timing of turning the page, a distribution step of distributing a result of the speech recognition process in which a speech that spans before and after the timing of the page turning is recognized by the speech recognition step, to one of the captured image before the page turning and the captured image after the page turning; Run the command, The allocating step determines whether the result of the speech recognition across pages is to be allocated to the captured image before the page turning or the captured image after the page turning. program.

Images