Information presentation system, device, method and program

Abstract

According to one embodiment of the present invention, when explanatory information in a format using a visual material in a launch or exhibition event is transmitted from an information presentation control device to a viewer terminal via a network and presented, the information presentation control device acquires information relating to a viewing operation on the presented explanatory information by a viewer from the viewer terminal and estimates the viewing state of the viewer on the basis of the acquired information relating to the viewing operation. Auxiliary explanatory information that includes spoken voice information corresponding to the explanatory information is generated on the basis of the estimated viewing state, and the generated auxiliary explanatory information is transmitted to the viewer terminal, causing the spoken voice information to be outputted.

Description

[Technical Field]

[0001] One aspect of the present invention relates to an information presentation system that presents presentation information at an online lecture or exhibition information at an online exhibition to viewers in a format using visual materials, and an apparatus, method, and program used in this system. [Background technology]

[0002] In recent years, for example, in online lectures and exhibitions, systems have been considered for making presentations and exhibits in the form of visual materials such as text and diagrams, such as posters and slides (see, for example, Non-Patent Document 1). In such systems, various mechanisms have been proposed, such as allowing viewers to access the website of the presentation or exhibition from their own terminal, select a desired poster by scrolling, and view its contents, and further allowing for question and answer sessions between viewers and presenters for each poster via chat or web conference. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Rui Fukui and Tetsuharu Watanabe, "An attempt to make a poster-style academic conference online using Facebook as a platform," Transactions of the Japan Society of Mechanical Engineers, Vol. 87, No. 895, 2021 Summary of the Invention [Problem to be solved by the invention]

[0004] However, because browsing by scrolling only presents the presentation or exhibition information written on the poster in a uniform visual manner, it is difficult for the presenter to fully convey the content of the presentation or exhibition to the viewer. Furthermore, although the viewer can view the information in real time, they simply read the content of the presentation or exhibition, which does not always allow them to fully understand it. In other words, there are cases where information presentation that is satisfactory for both the presenter and the viewer is not possible.

[0005] On the other hand, using a system for Q&A via chat or web conferencing makes it possible to present the content of a presentation or exhibition in more detail, but as the number of viewers increases, the burden on the presenter increases and the responsiveness of Q&A decreases, making it impossible to maintain real-time performance.

[0006] This invention was made with the above-mentioned circumstances in mind, and aims to provide a technology that enables information to be presented accurately in accordance with the viewer's viewing state while maintaining real-time performance when making a presentation or exhibition using online visual materials, thereby enabling information presentation that is highly satisfying for both the presenter and the viewer. [Means for solving the problem]

[0007] In order to solve the above problems, one aspect of the information presentation system and its device or method according to the present invention is that, when explanatory information in a format using visual materials at a presentation or exhibition event is transmitted and presented online via a network from an information presentation control device used by an information presenter to a viewer terminal used by a viewer, the viewer terminal receives the explanatory information transmitted from the information presentation control device via the network and displays it on a display device, detects information regarding the viewer's viewing operation with respect to the displayed explanatory information, and transmits the detected information regarding the viewing operation to the information presentation control device via the network.In response to this, the information presentation control device acquires information regarding the viewing operation via the network and, based on the acquired information regarding the viewing operation, Detects whether the scrolling direction is reverse, and determines whether the scrolling in the reverse direction has been completed to the end, and determines whether the purpose of the scrolling in the reverse direction is "reconfirmation" or "supplement" based on the result of the determination.Estimate Depending on whether the purpose of the backward scrolling is "reconfirmation" or "supplementation" The system generates auxiliary explanatory information including voice information to be read aloud corresponding to the explanatory information, and transmits the generated auxiliary explanatory information to the viewer terminal via the network. The viewer terminal receives the transmitted auxiliary explanatory information via the network, and outputs the voice information to be read aloud included in the received auxiliary explanatory information from an acoustic device.

[0008] According to one aspect of the present invention, the viewing state of the viewer is estimated based on the viewing operation of the viewer on the presented explanatory information, and based on the estimated viewing state, read-aloud audio information corresponding to the explanatory information is generated and output to the viewer. This allows the viewer to visually view the explanatory information on the poster while simultaneously audibly confirming the content through the read-aloud audio information.

[0009] Furthermore, the read-aloud audio information is output with the appropriate read content or audio source selected according to the viewer's viewing status, such as the level of detail and purpose of the viewing. This makes it possible to not only simply read out explanatory information, but also to present audio messages with content or audio sources that correspond to the viewer's viewing status, and furthermore, to present information in a more real-time manner than when using chat or web conferencing. [Effects of the Invention]

[0010] In other words, according to one aspect of the present invention, when making a presentation or exhibition using online visual materials, it is possible to accurately present information according to the viewer's viewing state while maintaining real-time performance, thereby providing a technology that enables the presentation of information in a manner that is highly satisfying to both the presenter and the viewer. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an information presentation system according to an embodiment of the present invention. [Figure 2]FIG. 2 is a block diagram showing an example of the hardware configuration of a viewer terminal used by a viewer in the information presentation system shown in FIG. [Figure 3] FIG. 3 is a block diagram showing an example of a software configuration of a viewer terminal used by a viewer in the information presentation system shown in FIG. [Figure 4] FIG. 4 is a block diagram showing an example of the hardware configuration of a server device used as an information presentation control device by an information presenting side in the information presentation system shown in FIG. [Figure 5] FIG. 5 is a block diagram showing an example of the software configuration of a server device used as an information presentation control device by the information presenting side in the information presentation system shown in FIG. [Figure 6A] FIG. 6A is a diagram showing an example of supplementary explanatory information stored in the server device shown in FIG. [Figure 6B] FIG. 6B is a diagram showing an example of supplementary explanatory information stored in the server device shown in FIG. [Figure 7] FIG. 7 is a diagram showing an example of visit history information stored in the server device shown in FIG. [Figure 8] FIG. 8 is a flowchart showing an example of a processing procedure and processing content of a viewing control process executed by the viewer terminal shown in FIG. [Figure 9] FIG. 9 is a flowchart showing an example of a processing procedure and processing content of the information presentation control processing executed by the server device shown in FIG. [Figure 10] FIG. 10 is a flowchart showing an example of the procedure and content of the viewing state estimation process in the information presentation control process shown in FIG. [Figure 11] FIG. 11 is a diagram showing an example of explanatory information in a format using visual materials distributed by a server device. [Figure 12] FIG. 12 is a diagram showing an example of the explanatory information shown in FIG. 11 downloaded to a viewer terminal and displayed. [Figure 13] FIG. 13 is a diagram showing a first example of a scroll operation on explanation information displayed on a viewer terminal. [Figure 14] FIG. 14 is a diagram showing a second example of a scroll operation on the explanation information displayed on the viewer terminal. [Figure 15] FIG. 15 is a diagram showing a first example of a reference point for a scroll operation on explanation information displayed on a viewer terminal. [Figure 16] FIG. 16 is a diagram showing a second example of the reference point for the scroll operation on the explanation information displayed on the viewer terminal. [Figure 17] FIG. 17 is a diagram showing a third example of the reference point for the scroll operation on the explanation information displayed on the viewer terminal. [Figure 18] FIG. 18 is a diagram showing an example of the positional relationship between a presenter, a viewer, their companions, and other participants in an exhibition. [Figure 19] FIG. 19 is a diagram showing an example of setting the sound image position of the presenter in the positional relationship shown in FIG. [Figure 20] FIG. 20 is a diagram showing an example of setting the sound image position of the accompanying person in the positional relationship shown in FIG. [Figure 21] FIG. 21 is a diagram showing an example of setting the sound image positions of other participants in the positional relationship shown in FIG. [Figure 22] FIG. 22 is a diagram for explaining a modified example of the output process of the auxiliary explanatory information. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0013] [One embodiment] In one embodiment of the present invention, an example will be described in which explanatory information about exhibits is presented in the form of visual materials such as letters and diagrams, such as posters and slides, at an online exhibition.

[0014] (Configuration example) (1) System FIG. 1 is a diagram showing an example of the configuration of an information presentation system according to an embodiment of the present invention.

[0015] An information presentation system according to one embodiment includes a server device SV operated as an information presentation control device by, for example, an exhibition organizer, who is the information presenting party. This server device SV is accessible via a network NW by multiple exhibitor terminals ET1-ETm used by exhibitors of exhibits and multiple viewer terminals UT1-UTn used by viewers of the exhibits. Hereinafter, the above-mentioned ET1-ETm and UT1-UTn will be simply referred to as ET and UT, respectively.

[0016] The exhibitor terminal ET is, for example, a personal computer, and has the function of uploading and registering explanatory information about exhibits created by exhibitors and their audio reading files to the server device SV via the network NW. The explanatory information explains the contents of the exhibit in a format using visual materials such as text and diagrams, such as posters and slides, and is created, for example, by combining text data with diagrams and photographs for each major category. The audio reading files represent the content of each major category included in the explanatory information by audio reading. The audio reading files include multiple audio files with different granularity of information content for each major category. The structure of the supplementary explanatory information will be explained in the operation example.

[0017] The network NW comprises, for example, a wide area network with the Internet at its core, and an access network for accessing this wide area network. The access network may be, for example, a public communication network using wired or wireless, a local area network (LAN) using wired or wireless, or a cable television (CATV) network.

[0018] (2) Equipment (2-1) Viewer terminals UT1 to UTn 2 and 3 are block diagrams showing an example of the hardware configuration and software configuration of the viewer terminal UT, respectively.

[0019] The viewer terminal UT is, for example, a personal computer equipped with input / output functions necessary for viewing information and a browser necessary for transmitting information data to and from the server device SV. Note that the viewer terminal UT may also be a tablet terminal, a smartphone, or the like, as long as it has similar functions.

[0020] The viewer terminal UT includes a control unit 1 that uses a hardware processor such as a central processing unit (CPU). Connected to the control unit 1 are a storage unit having a program storage unit 2 and a data storage unit 3, a communication interface (hereinafter, the interface will be referred to as I / F) 4, and an input / output I / F unit 5.

[0021] Under the control of the control unit 1, the communication I / F unit 4 performs data communication with the server device SV using a communication protocol defined by the network NW.

[0022] An input device 51, a display device 52, and a speaker 53 are connected to the input / output I / F unit 5. The input device 51 includes, for example, a keyboard and a mouse. The display device 52 is a device that uses liquid crystal or organic electroluminescence (EL). The display device 52 is primarily used to visually display explanatory information about the exhibits downloaded from the server device SV. The input device 51 is primarily used to scroll the display position of the explanatory information displayed on the display device 52. The speaker 53 is used to amplified and output the read-aloud audio file sent from the server device SV.

[0023] Additionally, a camera may be connected to the input / output I / F unit 5. By providing a camera, it becomes possible to detect the viewer's line of sight with respect to the displayed explanatory information from, for example, a facial image of the viewer, and to recognize the viewer's viewing position with respect to the explanatory information based on the detected line of sight.

[0024] The program storage unit 2 is configured by combining, for example, a non-volatile memory such as a solid-state drive (SSD) as a storage medium that can be written to and read from at any time, and a non-volatile memory such as a read-only memory (ROM), and stores middleware such as an operating system (OS), as well as application programs required to execute various control processes according to one embodiment. Hereinafter, the OS and each application program will be collectively referred to as the program.

[0025] The data storage unit 3 is, for example, a combination of a non-volatile memory such as an SSD that can be written to and read from at any time as a storage medium, and a volatile memory such as RAM (Random Access Memory), and is equipped with an explanatory information storage unit 31 and an auxiliary explanatory information storage unit 32 as the main storage units required to implement one embodiment.

[0026] The explanation information storage unit 31 is used to store explanation information downloaded from the server device SV.

[0027] The auxiliary explanatory information storage unit 32 is used to store auxiliary explanatory information including a read-aloud voice file downloaded from the server device SV.

[0028] The control unit 1 includes, as processing functions necessary for implementing one embodiment, an explanatory information reception processing unit 11, an explanatory information display processing unit 12, a scrolling operation detection processing unit 13, a browsing operation information transmission processing unit 14, an auxiliary explanatory information reception processing unit 15, and a reading voice output processing unit 16. These processing units 11 to 16 are all realized by causing a hardware processor of the control unit 1 to execute an application program stored in the program storage unit 2. Note that the application program may be stored in advance in the program storage unit 2, or may be downloaded from the server device SV when necessary and stored in the program storage unit 2.

[0029] The explanatory information receiving processing unit 11 accesses the server device SV in response to the viewer's viewing access operation, receives explanatory information about the exhibit from the server device SV via the communication I / F unit 4, and stores it in the explanatory information storage unit 31.

[0030] The explanatory information display processing unit 12 reads explanatory information from the explanatory information storage unit 31, and outputs and displays the read explanatory information to the display device 52 via the input / output I / F unit 5. At this time, the explanatory information display processing unit 12 also performs processing to control the display position of the explanatory information in response to a scroll operation of the input device 51.

[0031] The scroll operation detection processing unit 13 performs a process of detecting the amount and direction of the scroll operation on the displayed explanatory information at a predetermined detection period.

[0032] The browsing operation information transmission processing unit 14 generates browsing operation information including identification information representing the explanatory information and major item being viewed, and information representing the amount and direction of the detected scrolling operation, and performs processing to transmit the generated browsing operation information from the communication I / F unit 4 to the server device SV.

[0033] The auxiliary explanatory information receiving processing unit 15 receives auxiliary explanatory information corresponding to the explanatory information currently being viewed, transmitted from the server device SV, via the communication I / F unit 4, and stores it in the auxiliary explanatory information storage unit 32.

[0034] The reading audio output processing unit 16 reads out the auxiliary explanatory information from the auxiliary explanatory information storage unit 32, converts the reading audio file contained in this auxiliary explanatory information into an analog audio signal, and then outputs it to the speaker 53 via the input / output I / F unit 5, thereby amplified and outputting the reading audio.

[0035] (2-2) Server device SV 4 and 5 are block diagrams showing an example of the hardware configuration and software configuration of the server device SV, respectively.

[0036] The server device SV is, for example, a server computer located on a cloud, and includes a control unit 6 that uses a hardware processor such as a CPU. A storage unit having a program storage unit 7 and a data storage unit 8, and a communication I / F unit 9 are connected to the control unit 6 via a bus.

[0037] Under the control of the control unit 6, the communication I / F unit 9 transmits and receives information data to and from the exhibitor terminal ET and the viewer terminal UT using a communication protocol defined by the network NW.

[0038] The program storage unit 7 is configured, for example, by combining a non-volatile memory such as an HDD or SSD as a storage medium that can be written to and read from at any time, with a non-volatile memory such as a ROM, and stores middleware such as an OS as well as programs necessary to execute various control processes according to one embodiment of the present invention.

[0039] The data storage unit 8 is, for example, a combination of a non-volatile memory such as an HDD or SSD as a storage medium that can be written to and read at any time, and a volatile memory such as RAM, and is equipped with an explanation information storage unit 81, a viewing operation information storage unit 82, a visit history information storage unit 83, a viewing status storage unit 84, and an auxiliary explanation information storage unit 85 as the main storage units required to implement one embodiment of the present invention.

[0040] The explanatory information storage unit 81 is used to store explanatory information uploaded from the exhibitor terminal ET. As mentioned above, the explanatory information is created in a format that uses visual materials, for example, by combining text data with diagrams or photographs for each major category.

[0041] The browsing operation information storage unit 82 is used to store the browsing operation information sent from the viewer terminal UT. As mentioned above, the browsing operation information includes the explanatory information being viewed, identification information indicating its main item, and information indicating the amount and direction of the scrolling operation.

[0042] The visit history information storage unit 83 is used to store the viewer's viewing history (for example, the number of views (also called visit history)) of the explanation information being viewed by each major category based on the above-mentioned viewing operation information.

[0043] The viewing state storage unit 84 is used to store the estimation result, obtained by the control unit 6, of the viewing state of the viewer with respect to the explanation information.

[0044] The auxiliary explanatory information storage unit 85 stores, for each major category of explanatory information, auxiliary explanatory information prepared in advance corresponding to each of a plurality of viewing states of the viewer that are assumed while the viewer is viewing the explanatory information. The auxiliary explanatory information storage unit 85 also stores, as auxiliary explanatory information, audio files that represent the content of supplementary or advice that is actually provided or is assumed to be provided by the viewer's companion or other participants while the viewer is viewing the explanatory information.

[0045] The control unit 6 includes, as processing functions according to an embodiment of the present invention, an explanatory information transmission processing unit 61, a viewing operation information acquisition processing unit 62, a viewing state estimation processing unit 63, an auxiliary explanatory information generation processing unit 64, and an auxiliary explanatory information transmission processing unit 65. These processing units 61 to 65 are all realized by causing the hardware processor of the control unit 6 to execute application programs stored in the program storage unit 7.

[0046] The explanatory information transmission processing unit 61 receives explanatory information uploaded from the exhibitor terminal ET via the communication I / F unit 9 and stores it in the explanatory information memory unit 81, and also reads out corresponding explanatory information from the explanatory information memory unit 81 in response to a viewing request from the viewer terminal UT, and transmits the read-out explanatory information from the communication I / F unit 9 to the requesting viewer terminal UT.

[0047] The viewing operation information acquisition processing unit 62 receives viewing operation information periodically transmitted from the viewer terminal UT while viewing the explanatory information via the communication I / F unit 9, and performs processing to store the received viewing operation information in the viewing operation information.

[0048] The browsing operation information acquisition processing unit 62 also performs processing to update the browsing history (visit history) for each major category of explanation information stored in the visit history information storage unit 83 based on the received browsing operation information.

[0049] Every time new browsing operation information is received, the browsing state estimation processing unit 63 reads the newly received browsing operation information and the latest updated visit history information from the browsing operation information storage unit 82 and the visit history information storage unit 83, respectively. Then, the browsing state estimation processing unit 63 performs processing to estimate the browsing state of the viewer based on the read browsing operation information and visit history information. Note that this browsing state estimation processing will be described in an operation example.

[0050] Each time a new estimation result of the viewing state is obtained, the auxiliary explanatory information generation processing unit 64 searches the auxiliary explanatory information storage unit 85 using the estimation result and the viewing operation information used to obtain this estimation result as search conditions, and thereby performs a process of reading out the auxiliary explanatory information corresponding to the above search conditions.

[0051] The auxiliary explanatory information transmission processing unit 65 performs processing to transmit the read auxiliary explanatory information from the communication I / F unit 9 to the viewer terminal UT currently viewing the information.

[0052] (Example of operation) Next, an example of the operation of the information presentation system configured as above will be described. Figure 8 is a flowchart showing the processing procedure and processing content of the viewing control processing executed by the control unit 1 of the viewer terminal UT, and Figure 9 is a flowchart showing the processing procedure and processing content of the information presentation control processing executed by the control unit 6 of the server device SV.

[0053] (1) Structure of supplementary explanatory information First, the structure of the auxiliary explanatory information will be described. Figures 6A and 6B are diagrams showing an example of the structure of the auxiliary explanatory information. The auxiliary explanatory information corresponds, for each major item that constitutes explanatory information in a format using visual materials, to the identification information (major item ID) of that major item, multiple viewing states that are expected during viewing, sound sources corresponding to each viewing state, and a read-aloud audio file corresponding to the explanatory content of the major item.

[0054] In this example, the following five types of viewing states are defined. The first state is "skip" viewing, which corresponds to speed reading or skipping. The second state is "normal" viewing, which corresponds to viewing at a normal speed. The third state is "detailed" viewing, which corresponds to careful reading. The fourth state is "reconfirmation," in which after reaching the end of the page, the user returns to any major item on the page and views it again. The fifth state is "supplementary" explanation, in which the user returns to any major item in the middle of the page and views it again.

[0055] In this example, the sound sources are defined as "explainer," "companion" of the viewer, and "other participants." The "explainer" is associated with the above-mentioned "skip" viewing, "normal" viewing, and "detail" viewing. Furthermore, "reconfirmation" is associated with "other participants," and "companion" is associated with "supplementary" explanation.

[0056] In this example, multiple audio files are prepared as audio files for reading aloud, with different levels of granularity in the explanation depending on the viewing state. For example, for "skip" viewing, an audio file is prepared that reads out only the "summary" of the explanation of each major item, for "normal" viewing, an audio file is prepared that reads out the explanation written in the major item as is, and for "detailed" viewing, an audio file is prepared that reads out the explanation of the major item and any additional explanations. Each of the audio files is generated in advance by the exhibitor and uploaded from the exhibitor terminal ET.

[0057] In addition, for "reconfirmation," audio files are provided that provide supplementary explanations of the major items or FAQs (Frequently Asked Questions) as read-alouds, and for "supplementary" explanations, audio files are provided for supplementary explanations by an accompanying person.

[0058] Furthermore, each read-aloud audio file contains sound image position (also called sound image localization) designation information to represent the position of the sound source by the read-aloud audio. The sound image position designation information is composed of audio control data that controls the phase of the read-aloud audio and sound image position display data that displays the audio source position on the display screen of the viewer terminal UT.

[0059] (2) Present explanatory information The explanation will be given assuming that the explanation information uploaded from the exhibitor terminal ET is pre-stored in the explanation information memory unit 81 of the server device SV, and that the auxiliary explanation information configured as described above is pre-stored in the auxiliary explanation information memory unit 85.

[0060] Suppose a viewer accesses the URL (Uniform Resource Locator) of the online exhibition on his / her viewer terminal UT in order to view exhibits at the online exhibition. Then, under the control of the explanation information receiving processing unit 11, the control unit 1 of the viewer terminal UT detects the access operation in step S10 shown in Fig. 8, proceeds to step S11, and transmits an access request to the server device SV of the exhibition site.

[0061] In response to this, when the server device SV receives the access request in step S30 shown in Fig. 9 under the control of the explanation information transmission processing unit 61, in step S31 it first reads out the menu information of the exhibit from the explanation information storage unit 81 and downloads it to the viewer terminal UT, so that the menu information of the exhibit is displayed on the display device 52 of the viewer terminal UT.

[0062] In this state, if a viewer specifies an exhibit that he or she wishes to view, the specified information for the exhibit is transmitted from the viewer terminal UT. In response, the control unit 6 of the server device SV, under the control of the explanatory information transmission processing unit 61, reads explanatory information for the poster corresponding to the specified information from the explanatory information storage unit 81, and downloads the read explanatory information to the viewer terminal UT that made the request. Figure 11 shows an example of the explanatory information to be downloaded. As shown in the figure, the explanatory information has a plurality of major items I1 to Ik, and each of the major items I1 to Ik is composed of, for example, a combination of text data and diagrams or photographs.

[0063] The control unit 1 of the viewer terminal UT, under the control of the explanatory information reception processing unit 11, receives the explanatory information in step S12 and temporarily stores the received explanatory information in the explanatory information storage unit 31. Then, the control unit 1 reads out the received explanatory information from the explanatory information storage unit 31 and outputs the read-out explanatory information to the display device 52 via the input / output I / F unit 5. Thus, the explanatory information of the exhibit is displayed on the display device 52 of the viewer terminal UT from the top position of the page. Figure 12 shows an example of the display result of the explanatory information. Note that in this figure, ME indicates a margin area.

[0064] (3) Present supplementary explanatory information (3-1) Detecting browsing operations When the viewer performs a scroll operation using the input device 51 while the explanatory information is displayed, the explanatory information displayed on the display device 52 is scrolled as shown in SC in FIG. 12, for example.

[0065] When the control unit 1 of the viewer terminal UT detects the scrolling operation in step S13 under the control of the scrolling operation detection processing unit 13, the control unit 1 detects the amount and direction of the scrolling operation at a preset detection period (for example, any period less than one second) in step S14. Then, under the control of the browsing operation information transmission processing unit 14, in step S15, the control unit 1 generates browsing operation information including identification information indicating the explanatory information and main item being viewed, and the detected amount and direction of the scrolling operation, and transmits the generated browsing operation information from the communication I / F unit 4 to the server device SV.

[0066] In order for the server device SV to be able to identify the viewer's focus position on the explanation information after the scroll operation, a reference point for the focus position is set in the scroll operation detection processing unit 13. As a setting method, for example, the following method is used.

[0067] The first method assumes that the viewer is focusing on a position P0 that is a predetermined distance Lth below the top edge of the currently displayed page, as shown in Figure 15, and uses this position as the reference point for scrolling.

[0068] The second method assumes that the position pointed to by the mouse pointer or cursor CU is the position P0 that the viewer is focusing on, and uses the coordinates of that position as the reference point for scrolling, as shown in FIG. 16, for example.

[0069] In the third method, when a camera is provided, as shown in Fig. 17, for example, the viewer's line of sight L is detected from the viewer's face image captured by the camera, and the viewer's focus position P0 is calculated based on this line of sight L. Then, the position coordinates of the calculated focus position P0 are used as the reference point for scrolling.

[0070] The coordinate P0 of the reference point of the scroll set as described above is inserted into the browsing operation information together with the detection results of the amount and direction of the scroll, and is transmitted to the server device SV.

[0071] When the browsing operation information is transmitted from the viewer terminal UT, in step S32, the control unit 6 of the server device SV, under the control of the browsing operation information acquisition processing unit 62, receives the browsing operation information transmitted from the viewer terminal UT via the communication I / F unit 9 and temporarily stores the received browsing operation information in the browsing operation information storage unit 82. At the same time, in step S33, the browsing operation information acquisition processing unit 62 updates the number of views (number of visits) for the corresponding major category stored in the visit history information storage unit 83 based on the identification information of the major category being viewed, which is included in the browsing operation information. Fig. 7 shows an example of the browsing history information after the update.

[0072] (3-2) Estimation of browsing status When the viewing operation information is received, the control unit 6 of the server device SV, under the control of the viewing state estimation processing unit 63, executes a process of estimating the viewing state of the viewer with respect to the explanation information in step S34 as follows.

[0073] FIG. 10 is a flowchart showing the processing procedure and processing contents of the browsing state estimation processing. That is, in step S41, the viewing state estimation processing unit 63 reads the amount and direction of the scrolling operation included in the viewing operation information and the reference position coordinates of the scrolling from the viewing operation information storage unit 82, and further reads the number of views (number of visits) for the major item currently being viewed from the visit history information storage unit 83.

[0074] When the above-mentioned reading is completed, the viewing state estimation processing unit 63 first determines in step S43 whether the direction of the scrolling operation SC is the "forward" direction or the "reverse" direction. If the result of this determination is that the scrolling direction is the "forward" direction, for example, as shown in FIG. 13, the viewing state estimation processing unit 63 calculates the speed Vcur of the scrolling operation based on the amount of the scrolling operation. The speed Vcur of the scrolling operation is calculated, for example, as an average value over a predetermined period (for example, one second).

[0075] The browsing state estimation processing unit 63 then compares the calculated scroll speed Vcur with a first threshold value Vth1 in step S44. As a result of this comparison, if Vcur ≧ Vth1, since the scroll speed is high and it can be considered that the user is reading quickly or skipping, in step S46, it is estimated that the browsing state at this time is "skip" browsing.

[0076] On the other hand, if Vcur ≧ Vth1 does not hold, the browsing state estimation processing unit 63 compares the scroll speed Vcur with a second threshold value Vth2 (Vth2 < Vth1) in step S45. As a result of this comparison, if Vcur > Vth2, since it can be considered that the user is browsing at a general speed, in step S47, it is estimated that the browsing state at this time is "normal" browsing. On the contrary, if Vcur > Vth2 does not hold, since the scroll speed is slow and it can be considered that the user is reading carefully, the browsing state estimation processing unit 63 estimates that the browsing state at this time is "detailed" browsing in step S48.

[0077] On the other hand, assume that in step S43 above, the scroll direction is determined to be the "reverse" direction as shown in FIG. 14 for example. In this case, the browsing state estimation processing unit 63 determines whether the scroll has reached the end of the page based on the browsing history information in step S50. As a result of this determination, if the scroll has reached the end of the page, the browsing state estimation processing unit 63 estimates in step S51 that the current browsing state is "reconfirmation" of returning to an arbitrary major item within the page and browsing again after reaching the end of the page. On the contrary, if the scroll has not reached the end of the page, in step S52, it is estimated that the current browsing state is a "supplementary" explanation of returning to an arbitrary major item in the middle of the page and browsing again.

[0078] (3-3) Generation of Supplementary Explanation Information When the estimation process of the browsing state ends, the control unit 6 of the server device SV then executes the generation process of supplementary explanation information as follows under the control of the supplementary explanation information generation processing unit 64 in step S35.

[0079] That is, the auxiliary explanatory information generation processing unit 64 acquires the estimation result of the browsing information from the browsing state estimation processing unit 63, and reads the browsing operation information and visit history information used to obtain the estimation result from the browsing operation information storage unit 82 and the visit history information storage unit 83, respectively. Then, the auxiliary explanatory information generation processing unit 64 searches the auxiliary explanatory information storage unit 85 using the estimation result of the browsing state and the browsing operation information as search conditions, thereby reading out auxiliary explanatory information corresponding to the search conditions.

[0080] For example, suppose the current position of interest (viewing position) has a major item ID of 1 and the estimated viewing state is "skip" viewing. In this case, the auxiliary explanatory information generation processing unit 64 reads out the sound source "explainer" and the audio file "1-1.wav" corresponding to the search criteria from the auxiliary explanatory information (e.g., FIG. 6A) stored in the auxiliary explanatory information storage unit 85. At this time, the audio file "1-1.wav" contains a low-granularity readout audio file that reads out only the "summary" of the explanatory content corresponding to the "skip" viewing. The audio file "1-1.wav" also contains sound image position designation information for representing the position of the sound source "explainer." This sound image position designation information is composed of sound control data for controlling the phase of the readout audio and sound image position display data for displaying the audio generation position on the display screen.

[0081] Also, for example, suppose the viewing position is the major item ID=1 and the estimated viewing state is "normal" viewing. In this case, the auxiliary explanatory information generation processing unit 64 reads out the sound source "explainer" and the audio file "1-2.wav" corresponding to the above search conditions from the auxiliary explanatory information (e.g., FIG. 6A) stored in the auxiliary explanatory information storage unit 85. At this time, the audio file "1-2.wav" contains a medium-granularity read-out audio file that reads out the explanatory content written in the major item as is, corresponding to the above "normal" viewing. Also, in this case, since the sound source is "explainer," the audio file "1-2.wav" contains sound image position designation information for expressing the position of the "explainer."

[0082] Furthermore, suppose the viewing position is the major item ID=1 and the estimated viewing state is "detailed" viewing. In this case, the auxiliary explanatory information generation processing unit 64 reads out the sound source "explainer" and the audio file "1-3.wav" corresponding to the above search conditions from the auxiliary explanatory information (e.g., FIG. 6A) stored in the auxiliary explanatory information storage unit 85. At this time, the audio file "1-3.wav" contains a highly granular audio file that reads out the explanation content of the major item and the content of the additional explanation corresponding to the above "detailed" viewing. Also, in this case, since the sound source is "explainer," the audio file "1-3.wav" contains sound image position designation information for expressing the position of the "explainer."

[0083] On the other hand, for example, suppose the viewing position has a major item ID of 1 and the estimated viewing state is "supplementary" explanation. In this case, the auxiliary explanatory information generation processing unit 64 reads out the sound source "companion" and the audio file "1-sup1.wav" or "1-sup2.wav" corresponding to the above search criteria from the auxiliary explanatory information (e.g., FIG. 6B) stored in the auxiliary explanatory information storage unit 85. At this time, the audio file "1-sup1.wav" or "1-sup2.wav" contains an audio file reading out the supplementary explanation of the companion. Furthermore, since the sound source is "companion," the audio file "1-sup1.wav" or "1-sup2.wav" contains sound image position designation information for expressing the position of the "companion."

[0084] The reason why two types of audio files, "1-sup1.wav" and "1-sup2.wav", are provided is to accommodate cases where there are two accompanying persons. These audio files are registered in advance in association with the identification information of the accompanying persons who have been registered in advance. The accompanying person's audio file may be obtained from the accompanying person's terminal (not shown) and registered whenever necessary during viewing.

[0085] Also, for example, suppose the viewing position is the major item ID=1 and the estimated viewing state is "reconfirm." In this case, the auxiliary explanatory information generation processing unit 64 reads out the sound source "other participants" and the audio file "1-con1.wav" corresponding to the above search criteria from the auxiliary explanatory information (e.g., FIG. 6B) stored in the auxiliary explanatory information storage unit 85. At this time, the audio file "1-con1.wav" contains an audio file that provides supplementary explanation or FAQs for the explanation content of the major item corresponding to the above "reconfirm." Furthermore, since the sound source is "other participants," the audio file "1-sup1.wav" contains sound image position designation information for representing the position of the "other participants." Note that, in this case, assuming that there are multiple other participants, audio files corresponding to each of the multiple participants are prepared.

[0086] (3-4) Output of supplementary information In step S36, the control unit 6 of the server device SV, under the control of the auxiliary explanatory information transmission processing unit 65, transmits the auxiliary explanatory information generated by the auxiliary explanatory information generation processing unit 64 from the communication I / F unit 9 to the viewer terminal UT.

[0087] In response to this, when the control unit 1 of the viewer terminal UT detects the reception of auxiliary explanatory information in step S16, under the control of the auxiliary explanatory information reception processing unit 15, in step S17, the control unit 1 temporarily stores the auxiliary explanatory information received by the communication I / F unit 4 in the auxiliary explanatory information storage unit 32. Next, under the control of the reading audio output processing unit 16, the control unit 1 of the viewer terminal UT reads out a reading audio file from the stored auxiliary explanatory information. Then, the read reading audio file is converted into an analog audio signal, and then output to the speaker 53 via the input / output I / F unit 5, where it is amplified and output as audio.

[0088] As a result, for example, if the audio file included in the auxiliary explanatory information corresponds to "skip" viewing, a read-out voice is output indicating an "overview" of the explanatory content of the major item being viewed. Also, for example, if the audio file included in the auxiliary explanatory information corresponds to "normal" viewing, a read-out voice is output indicating the explanatory content itself written in the major item being viewed. Furthermore, for example, if the audio file included in the auxiliary explanatory information corresponds to "detailed" viewing, a read-out voice is output indicating the explanatory content of the major item being viewed and the content of additional explanations.

[0089] At this time, the auxiliary explanatory information receiving processor 15 controls the phase of the reading voice in accordance with the sound source position designation information included in the voice file, thereby expressing the position of the sound source as follows:

[0090] That is, let us assume that the positional relationship of the presenter ES, viewer US, their companion AS, and other participants PS with respect to the exhibits at an actual exhibition is, for example, as shown in Figure 18. In this case, if the sound source is the "presenter," the phase of the reading voice is controlled so that the voice is reading out as if it were coming from in front of the exhibits. Also, if the sound source is the "companion," the phase of the reading voice is controlled so that the voice is reading out as if it were coming from the side of the viewer. Furthermore, if the sound source is the "other participants," the phase of the reading voice is controlled so that the voice is reading out as if it were coming from behind the viewer.

[0091] Furthermore, the auxiliary explanatory information receiving processing unit 15 determines in step S19 whether or not sound source position display data is included in the sound source position designation information included in the auxiliary explanatory information. If sound source position display data is included, the auxiliary explanatory information receiving processing unit 15 superimposes the sound source position display data on the explanatory information and displays it on the display device 52 under the control of the explanatory information display processing unit 12 in step S20.

[0092] For example, if the sound source is the "presenter," sound source position SL1 is visually displayed on the upper edge of the margin area ME of the display screen as shown in Fig. 19. If the sound source is the "accompanying person," sound source position SL2 is visually displayed on the side of the margin area ME of the display screen as shown in Fig. 20. If the sound source is the "other participant," sound source position SL3 is visually displayed on the lower edge of the margin area ME of the display screen as shown in Fig. 21. Each of the sound source positions SL1, SL2, and SL3 is displayed in the form of, for example, an animation of spreading sound waves.

[0093] As described above, by controlling the phase of the read-out voice, the position of the sound source is audibly expressed, and the position of the sound source is also visually displayed on the display screen, so that the viewer can get a sense of the positions and distances of the presenter, companions, and other participants through the read-out voice and display, even while viewing online. In other words, it is possible to achieve a sense of realism in online viewing.

[0094] When the end of browsing is detected in steps S21 and S37, the viewer terminal UT and the server device SV respectively end the series of information presentation processes.

[0095] (Actions and Effects) As described above, in one embodiment, while a viewer is online viewing explanatory information in the form of visual materials such as characters and figures, such as posters and slides, the viewer terminal UT detects the viewer's scrolling operation and transmits the amount and direction of the scrolling to the server device SV. Meanwhile, the server device SV estimates the viewer's viewing state based on the amount and direction of the scrolling operation, generates read-aloud audio files corresponding to the content of the explanatory information and with different granularity according to the estimated viewing state, transmits these to the viewer terminal UT as auxiliary explanatory information, and outputs the read-aloud audio to the viewer in a loud voice.

[0096] Therefore, the viewer can confirm the content of the explanatory information not only by visually displayed explanatory information but also by audio reading. Moreover, audio reading with different information content and granularity is output depending on the viewer's viewing state, so the viewer can confirm the desired content and granularity of information by audio reading according to their own viewing state. Therefore, the viewer can confirm the desired granularity of information in real time without using chat or web conferencing.

[0097] Furthermore, sound source position specification information is added to the read-aloud audio file, and this sound source position specification information controls the phase of the read-aloud audio and also visually displays the sound source position. As a result, even while viewing online, viewers can sense the position and distance of the sound source, such as the presenter, accompanying persons, and other participants, making information viewing more realistic.

[0098] [Other embodiments] (1) In the above embodiment, the system detects which major item the viewer focuses on through a scroll operation, and outputs a voice readout corresponding to the explanatory content of the major item where the focus is located. However, this is not limited to this. For example, as shown in FIG. 22, a margin area ME may be set between major items Ik-1 and Ik, and when the viewer's viewing position through a scroll operation SC moves from P5 on major item Ik-1 to position P6 on the margin area ME, a voice readout corresponding to the explanatory information of the major item Ik-1 before the movement may be output. In this case, the width of the margin area ME may be fixed in advance, or may be set to, for example, a value obtained by multiplying the average length of the writing areas of major items Ik-1 and Ik by a specific magnification factor, or a value obtained by multiplying the average amount of scrolling by the user per scroll by a specific magnification factor.

[0099] (2) When switching the output of the reading voice in response to the movement of the focus position due to scrolling, the output of the next reading voice may be controlled so that the current reading voice finishes outputting to the end. This prevents the reading voice from being interrupted midway. When switching the output of the reading voice, effect processing such as fade-in / fade-out may also be performed.

[0100] (3) In the above embodiment, the viewing state is estimated based on the scroll speed. However, the viewing state may also be estimated based on factors other than the scroll speed, such as the length of time the mouse pointer or cursor is stopped.

[0101] (4) In the above embodiment, the auxiliary explanatory information is registered in advance in the auxiliary explanatory information storage unit 85, and the corresponding auxiliary explanatory information is read out depending on the result of the estimation of the viewing state, etc. However, without being limited to this, a learning model configured using deep learning, etc. may be prepared, and the feature amount of the viewing operation may be input to this learning model as an explanatory variable, and an audio file of the auxiliary explanatory information or its label may be output as a target variable.

[0102] (5) In the above embodiment, the case where the viewing state estimation process is performed on the server device SV has been described as an example. However, the present invention is not limited to this. The viewing state estimation process may be performed on the viewer terminal UT, and the estimation result may be transmitted to the server device SV.

[0103] (6) In the embodiment described above, the explanatory information and auxiliary explanatory information are stored in advance in the server device SV, and the explanatory information and auxiliary explanatory information are transmitted from the server device SV to the viewer terminal UT in response to the viewer's browsing operation. However, this is not limiting, and for example, the explanatory information and additional explanatory information may be stored in the exhibitor terminal ET, and the explanatory information and auxiliary explanatory information may be transmitted from the exhibitor terminal ET directly or via the server device SV to the viewer terminal UT in response to the viewer's browsing operation.

[0104] (7) In the above embodiment, an online exhibition was used as an example, but the present invention may also be applied to a system for conducting online lectures or presentations, and may also be applied to websites that present information according to a certain purpose and context, such as public relations pages for companies or educational institutions, product introduction pages in online stores, etc. For example, in the case of a product introduction page, the presenter would be a company product planner or developer, and the other participants would be users who have actually used the product.

[0105] (8) In addition, the method for detecting viewing operations, the method for estimating viewing states, the types of viewing states, the configuration of the audio file to be read, the configuration of the explanatory information, and the configuration and functions of the viewer terminal and the information presentation control device can be modified and implemented in various ways without departing from the spirit of this invention.

[0106] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0107] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0108] SV...Server device UT1~UTn...Viewer device ET1~ETm: Exhibitor terminals NW...Network 1...Control unit of viewer terminal 2...Program memory section of the viewer terminal 3...Data storage unit of the viewer's terminal 4...Communication I / F section of the viewer's terminal 5... Input / output I / F section of viewer terminal 6...Control unit of server device 7...Program storage unit of server device 8...Data storage unit of server device 9...Communication I / F section of server device 11...Explanatory information receiving processing unit 12...Explanatory information display processing section 13...Scroll operation detection processing unit 14... Browsing operation information transmission processing unit 15...Auxiliary explanatory information receiving and processing unit 16...Read-out audio output processing unit 31...Explanatory information storage unit 32...Auxiliary explanation information storage unit 61...Explanatory information transmission processing unit 62... Browsing operation information acquisition processing unit 63... Browsing state estimation processing unit 64...Auxiliary explanation information generation processing unit 65...Auxiliary explanatory information transmission processing unit 81...Explanation information storage unit 82... Browsing operation information storage unit 83...Visit history information storage unit 84... Viewing state memory section 85...Auxiliary explanation information storage unit

Claims

1. An information presentation system that transmits explanatory information in a form using visual materials at a presentation or exhibition event from an information presentation control device used by an information presenter to a viewer terminal used by a viewer online via a network and presents the information, The viewer terminal includes: a display processing unit that receives the explanatory information transmitted from the information presentation control device via the network and displays the received explanatory information on a display device; a viewing operation transmission processing unit that detects information regarding a viewing operation of the viewer with respect to the explanatory information displayed on the display device, and transmits the detected information regarding the viewing operation to the information presentation control device via the network; Equipped with The information presentation control device includes: an estimation processing unit that acquires information about the browsing operation via the network, detects whether the scrolling direction is reverse based on the acquired information about the browsing operation, and determines whether the scrolling in the reverse direction has been completed to the end, and estimates whether the purpose of the scrolling in the reverse direction is "reconfirmation" or "supplement" based on the determination result; a generation processing unit that generates supplementary explanatory information including voice reading information corresponding to the explanatory information depending on whether the purpose of the backward scrolling is "reconfirmation" or "supplementation"; an auxiliary explanatory information transmission processing unit that transmits the generated auxiliary explanatory information to the viewer terminal via the network; Equipped with The viewer terminal includes: An information presentation system comprising an audio output processing unit that receives the auxiliary explanatory information via the network and outputs the read-aloud audio information included in the received auxiliary explanatory information from an acoustic device.

2. An information presentation control device that transmits explanatory information in a form using visual materials at a presentation or exhibition event to a viewer terminal used by a viewer via a network and presents the information, an estimation processing unit that acquires information about the viewer's viewing operation for the presented explanation information from the viewer terminal via the network, detects whether the scrolling direction is reverse based on the acquired information about the viewing operation, and determines whether the scrolling in the reverse direction has been completed to the end, and estimates whether the purpose of the scrolling in the reverse direction is "reconfirmation" or "supplement" based on the determination result; a generation processing unit that generates supplementary explanatory information including voice reading information corresponding to the explanatory information depending on whether the purpose of the backward scrolling is "reconfirmation" or "supplementation"; a transmission processing unit that transmits the generated auxiliary explanatory information to the viewer terminal via the network and causes the viewer terminal to output the read-aloud voice information; An information presentation control device comprising:

3. the estimation processing unit estimates a browsing detail level of the viewer for the explanation information based on information related to the browsing operation; the generation processing unit generates the read-aloud voice information with different information granularity corresponding to the explanation information based on the estimated browsing detail level; The transmission processing unit transmits the generated auxiliary explanatory information including the read-aloud voice information to the viewer terminal, and causes the read-aloud voice information to be output from the viewer terminal. The information presentation control device according to claim 2 .

4. the estimation processing unit estimates a viewing purpose of the viewer for the explanation information based on information related to the viewing operation; the generation processing unit selects a corresponding sound source from a plurality of explanation sound sources prepared in advance based on the estimated viewing purpose, and generates the reading voice information reflecting a sound image position of the selected sound source; The transmission processing unit transmits the generated auxiliary explanatory information including the read-aloud voice information to the viewer terminal, and causes the read-aloud voice information to be output from the viewer terminal. The information presentation control device according to claim 2 .

5. the generation processing unit further generates sound image position display information representing the sound image position; The transmission processing unit transmits the generated auxiliary explanatory information including the sound image position display information to the viewer terminal, and displays the sound image position display information on the viewer terminal in a state where the sound image position display information is superimposed on the explanatory information. The information presentation control device according to claim 4 .

6. 1. An information presentation method implemented by an information presentation system that transmits explanatory information in a form using visual materials at a presentation or exhibition event from an information presentation control device used by an information presenter to a viewer terminal used by a viewer online via a network, comprising: receiving, by the viewer terminal, the explanatory information transmitted from the information presentation control device via the network, and displaying the received explanatory information on a display device; a step of detecting, by the viewer terminal, information regarding a viewing operation of the viewer with respect to the explanatory information displayed on the display device, and transmitting the detected information regarding the viewing operation to the information presentation control device via the network; a step of acquiring information about the browsing operation via the network by the information presentation control device, detecting whether the scrolling direction is reverse based on the acquired information about the browsing operation, and determining whether the scrolling in the reverse direction has been completed to the end, and estimating whether the purpose of the scrolling in the reverse direction is "reconfirmation" or "supplement" based on the determination result; generating, by the information presentation control device, auxiliary explanatory information including voice reading information corresponding to the explanatory information, depending on whether the purpose of the backward scrolling is "reconfirmation" or "supplement"; a step of transmitting the generated auxiliary explanatory information to the viewer terminal via the network by the information presentation control device; receiving the auxiliary explanatory information via the network by the viewer terminal, and outputting the read-out audio information included in the received auxiliary explanatory information from an acoustic device; An information presentation method comprising:

7. A program that causes a processor provided in an information presentation control device to execute processing by each of the processing units provided in the information presentation control device described in any one of claims 2 to 5.

Images