Content providing system, content providing method, and content providing program

The content providing system addresses inappropriate audio association by using line-of-sight detection and priority conditions to ensure timely and accurate audio playback, enhancing user experience.

JP2024160487AActive Publication Date: 2024-11-14友永 那雄

Patent Information

Application Number
JP2023075537
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-01
Publication Date
2024-11-14
Estimated Expiration
2043-05-01

AI Technical Summary

Technical Problem

Existing content providing systems struggle with inappropriate association of audio with user's line of sight in documents, leading to overlapping voices or missed dialogues due to unpredictable user reading patterns.

Method used

A content providing system that includes a display processing section, line of sight detection, decision processing, and audio provision, using priority conditions based on line-of-sight information to determine appropriate audio playback timing.

Benefits of technology

Enables more accurate and effective association of audio with user's reading patterns, preventing voice overlap and ensuring proper playback of dialogues.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024160487000001_ABST
    Figure 2024160487000001_ABST
Patent Text Reader

Abstract

To provide a content providing system, a content providing method, and a content providing program.SOLUTION: A content providing system 1 includes a display processing unit, a gaze detection unit, a determination processing unit, an audio providing unit, and a memory unit. The memory unit stores document information and audio information about audio corresponding to a document. The display processing unit performs display processing on a content including the document and transmits a display processing result to a terminal used by a user. The gaze detection unit detects gaze information including a position of the user's gaze toward the document and a staying time of the gaze at the position, and the determination processing unit determines priority conditions including a waiting time from when the gaze is detected until the audio is reproduced for each document based on the gaze information that changes over time. The audio providing unit transmits the audio corresponding to the document to the terminal based on the gaze information, the priority conditions, and the audio information.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a content providing system, a content providing method, and a content providing program. [Background technology]

[0002] Systems for providing contents such as electronic documents have been developed for some time. Recently, in addition to devising methods for providing the contents, various efforts have been made to enhance the presentation effects of the contents.

[0003] For example, Patent Document 1 discloses an electronic document terminal that uses a camera module to detect the user's gaze position, calculates the gaze position and the time the gaze stays in the detection area, and determines that the user is reading the detection area in a comic book or electronic document when the gaze position, etc. meets specified conditions.

[0004] Patent Document 2 discloses a manga playback system that includes a display means, an audio playback means, an audio storage means, and an audio data selection means. This manga playback system allows users to select audio for a manga work, thereby maintaining the user's interest. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2015-212981 A [Patent Document 2] JP 2023-11069 A Summary of the Invention [Problem to be solved by the invention]

[0006] Incidentally, it is conceivable that by combining the technology of Patent Document 1 and the technology of Patent Document 2, a content providing system that is highly convenient for users and has a high performance effect can be realized.

[0007] In other words, in addition to using the technology of Patent Document 1 to detect the user's gaze (detect the document being read by the user) in content such as manga, the technology of Patent Document 2 can be used to provide content with audio corresponding to the detected document (for example, the dialogue of a character in a manga).

[0008] However, while manga and other content often have a certain degree of expectation of the order in which lines are read, in reality, the user's gaze does not necessarily move in the expected order. Also, users may skip over or reread lines in manga and other content. In this case, if the gaze movement speed is fast, the next sound may be played while the previous sound is being played (the sounds may overlap), and the sound of the lines may not be played properly.

[0009] The present invention has been made in consideration of the problems with the conventional technology described above, and has an object to provide a content providing system that can provide content to which audio is associated more appropriately and effectively than conventional systems. [Means for solving the problem]

[0010] In order to solve the above problems, the present invention provides a content providing system for providing content, comprising: The content providing system includes a display processing unit, a gaze detection unit, a determination processing unit, a sound providing unit, and a storage unit, The storage unit stores document information relating to a document and voice information relating to a voice corresponding to the document; the display processing unit performs display processing on the content including the document, and transmits a display processing result to a terminal used by a user; the gaze detection unit detects gaze information including a gaze position from the user to the document and a duration of the gaze at that position; the determination processing unit determines, for each of the documents, a priority condition including a waiting time from when the gaze is detected until when the audio is reproduced, based on the gaze information that changes over time; The voice providing unit transmits a voice corresponding to the document to the terminal based on the line of sight information, the priority condition, and the voice information.

[0011] The present invention also provides a content providing method executed by a content providing system for providing content, comprising: The content providing system includes a display processing unit, a gaze detection unit, a determination processing unit, a sound providing unit, and a storage unit, The storage unit stores document information relating to a document and voice information relating to a voice corresponding to the document, the display processing unit performs display processing on the content including the document, and transmits a display processing result to a terminal used by a user; the gaze detection unit detects gaze information including a gaze position from the user to the document and a duration of the gaze at that position; the determination processing unit determines, for each of the documents, a priority condition including a waiting time from when the gaze is detected until when the audio is reproduced, based on the gaze information that changes over time; The method includes a step in which the voice providing unit transmits a voice corresponding to the document to the terminal based on the line of sight information, the priority condition, and the voice information.

[0012] The present invention also provides a content providing program for providing content, comprising: A computer is caused to function as a display processing unit, a gaze detection unit, a decision processing unit, a voice providing unit, and a storage unit; The storage unit stores document information relating to a document and voice information relating to a voice corresponding to the document; the display processing unit performs display processing on the content including the document, and transmits a display processing result to a terminal used by a user; the gaze detection unit detects gaze information including a gaze position from the user to the document and a duration of the gaze at that position; the determination processing unit determines, for each of the documents, a priority condition including a waiting time from when the gaze is detected until when the audio is reproduced, based on the gaze information that changes over time; The voice providing unit transmits a voice corresponding to the document to the terminal based on the line of sight information, the priority condition, and the voice information.

[0013] With this configuration, it is possible to provide content associated with audio more appropriately and effectively than ever before, by utilizing a predetermined priority condition based on line-of-sight information.

[0014] In a preferred embodiment of the present invention, the document information includes a progress number indicating an order in which the user is to read the document, The determination processing unit determines the priority condition for a candidate document whose progress number is the next for the progress number of the target document in which the gaze is located as a first priority condition, the priority condition for a read document whose progress number is the previous for the progress number of the target document as a second priority condition, and the priority condition for unread documents assigned other progress numbers as a third priority condition.

[0015] With this configuration, it is possible to provide content while predicting the document or the like that will be viewed next by the user.

[0016] In a preferred embodiment of the present invention, a waiting time is set for the first priority condition for the candidate document, which is shorter than a waiting time for the second priority condition for the already-read document, A longer waiting time is set for the third priority condition for the unread documents than for the second priority condition for the read documents.

[0017] With this configuration, it is possible to provide content with more appropriate audio association.

[0018] In a preferred embodiment of the present invention, the document includes a main document to which the progress number is associated and other sub-documents; the storage unit stores a sub-audio playback condition of the sub-document based on the progress number; The audio providing unit transmits audio of the sub-document to the terminal based on the line-of-sight information and the sub-audio reproduction conditions.

[0019] By adopting such a configuration, it is possible to realize a more effective presentation in providing content.

[0020] In a preferred embodiment of the present invention, the document information is associated with position information indicating a position of the document in a display range of the content and area information indicating a display range of the document; The gaze detection unit detects the gaze information by using the position information and / or the area information.

[0021] With this configuration, the user's line of sight can be detected with high accuracy.

[0022] In a preferred embodiment of the present invention, the content is an electronic comic book content, The document includes lines of characters in an electronic comic; The audio information includes lines spoken by characters in the electronic comic.

[0023] With this configuration, dialogue can be appropriately provided in electronic comic content.

[0024] In a preferred embodiment of the present invention, the content providing system further comprises a sound selection unit, The storage unit stores a plurality of the dialogue voices, The voice selection unit is capable of selecting one line voice from among a plurality of line voices for the same line based on an input from the user.

[0025] With this configuration, it is possible to provide voice of lines according to the user's request. Effect of the Invention

[0026] According to the present invention, a novel technique for a content providing system can be provided by utilizing a predetermined priority condition based on line-of-sight information. [Brief description of the drawings]

[0027] [Figure 1] 1 shows a block diagram of a system configuration according to an embodiment of the present invention. [Diagram 2] 1A and 1B are schematic diagrams illustrating an example of a hardware configuration of an information processing device and a terminal according to an embodiment of the present invention. [Diagram 3] 1 shows a schematic image diagram of a content providing system according to an embodiment of the present invention. [Figure 4] 2 shows a flowchart of a processing procedure of a content providing system according to an embodiment of the present invention. [Diagram 5] 4 shows an example of each piece of information used in the content providing system according to one embodiment of the present invention. [Figure 6] 13A and 13B show schematic image diagrams of display processing results according to an embodiment of the present invention. [Figure 7] 4 shows an example of each piece of information used in the content providing system according to one embodiment of the present invention. [Figure 8] FIG. 2 shows a schematic image diagram of priority conditions according to an embodiment of the present invention. [Figure 9] FIG. 2 shows a schematic image diagram of priority conditions according to an embodiment of the present invention. [Figure 10] FIG. 2 shows a schematic image diagram of priority conditions according to an embodiment of the present invention. [Figure 11] 4 shows a detailed process flow of the content providing system according to the present embodiment. [Figure 12] FIG. 1 shows a schematic image diagram for explaining a sub-document according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] The present invention will now be described more fully with reference to the accompanying drawings, in which preferred embodiments are shown, but which may be embodied in many different forms and are not limited to the embodiments set forth herein.

[0029] For example, the configuration, operation, etc. of the content providing system will be described in this embodiment, but the same effects can be achieved by an executed method (steps), device, computer program, etc. The program in this embodiment may be provided as a non-transient computer-readable recording medium, or may be provided so as to be downloadable from an external server.

[0030] In addition, in this embodiment, the term "unit" may include, for example, a combination of hardware resources implemented by circuits in a broad sense and software information processing that can be specifically realized by these hardware resources.

[0031] In this embodiment, "information" is represented by, for example, the physical value of a signal representing a voltage or current, the high or low value of a signal as a collection of binary bits consisting of 0 or 1, or a quantum superposition (so-called quantum bit), and communication and calculations can be performed on a circuit in a broad sense.

[0032] A circuit in the broad sense is a circuit realized by appropriately combining a circuit, circuitry, a processor, memory, etc. In other words, it includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), etc.

[0033] <System configuration> Fig. 1 is a block diagram showing a system configuration according to an embodiment of the present invention. As shown in Fig. 1, a content providing system 1 includes an information processing device 10 and a database DB. The content providing system 1 is configured to be able to communicate with a plurality of terminals 2 (in Fig. 1, reference numerals 2(a) to 2(c)) and a plurality of voice providing servers 3 (in Fig. 1, reference numerals 3(a) to 3(c)) via a network NW.

[0034] The information processing device 10 operates as a server, the terminal 2 is a terminal where the user can view content such as manga, and the voice providing server 3 is a server for providing voice corresponding to a document included in the content. Note that the voice providing server 3 is not necessarily required to realize the present invention, and the information processing device 10 may be configured to include the function of the voice providing server 3.

[0035] In this embodiment, the network NW is an IP (Internet Protocol) network, but there is no restriction on the type of communication protocol, and further, there is no restriction on the type and scale of the network.

[0036] It should be noted that a general-purpose server computer, a personal computer, or the like can be used as the information processing device 10. It is also possible to configure the content providing system 1 by implementing the functional components described below in a plurality of computers.

[0037] A smartphone, a tablet terminal, a personal computer, a wearable device, or the like can be used as the terminal 2. The terminal 2 stores a content providing application program for a user, and this application program is configured to have a function for the user to view the content provided by the content providing system 1, or to transmit information required for the user to view the content to the information processing device 10.

[0038] The terminal 2 may be configured not to have a content providing application program for the user. In this case, the terminal 2 can use a web browser or the like to browse various pieces of information.

[0039] The voice providing server 3 stores voices (voice data) corresponding to documents included in the content. As shown in Fig. 1, a plurality of voice providing servers 3 are communicatively connected to the information processing device 10 via a network NW. For example, in a content such as a manga, when the voice actors of each character belong to different companies, the information processing device 10 can receive the respective voice data from the plurality of voice providing servers 3.

[0040] <Hardware configuration> 2(a) is a diagram showing an example of a hardware configuration of the information processing device 10. The information processing device 10 includes a control unit 11, a storage unit 12, and a communication unit 13 as the hardware configuration.

[0041] The control unit 11 includes one or more processors such as a CPU, and controls the overall operation and processing of the information processing device 10 by executing the content providing program according to the present invention, the OS, and other applications.

[0042] The storage unit 12 is an HDD, SSD, ROM, RAM, etc., and stores the content providing program according to the present invention and data used when the control unit 11 executes processing based on the program. The control unit 11 executes processing based on the content providing program stored in the storage unit 12, thereby realizing a functional configuration described below.

[0043] The communication unit 13 executes communication control with the network NW, and performs input required to operate the information processing device 10 and output related to the operation results.

[0044] 2(b) is a diagram showing an example of a hardware configuration of a terminal 90 (terminal 2 in FIG. 1). The terminal 90 includes, as its hardware configuration, a control unit 91, a storage unit 92, a communication unit 93, an input unit 94, and an output unit 95.

[0045] The control unit 91 of the terminal 90 includes one or more processors such as a CPU, and controls the overall operation processing of the terminal 90. The storage unit 92 of the terminal 90 is an HDD, SSD, ROM, RAM, or the like, and stores the above-mentioned content providing application, as well as data and the like used when the control unit 91 executes processing based on a program.

[0046] A communication unit 93 of the terminal 90 controls communication with the network. An input unit 94 of the terminal 90 is a touch panel, a mouse, a keyboard, or the like, and inputs operation requests from a user to the control unit 91. An output unit 95 of the terminal 90 is a display, or the like, and displays the results of processing by the control unit 91, etc.

[0047] <Functional configuration> 2(a), the information processing device 10 includes, as functional components, a registration unit 101, a display processing unit 102, a gaze detection unit 103, a decision processing unit 104, a voice provision unit 105, and a voice selection unit 106. These are information processing performed by software (stored in the storage unit 12) that is specifically realized by hardware (the control unit 11, etc.).

[0048] The registration unit 101 registers various information necessary for providing content to a user. In this embodiment, the registration unit 101 registers document information and content information. For example, the registration unit 101 can obtain voice information from the voice providing server 3 and register (store) it in the storage unit 12.

[0049] The display processing unit 102 performs display processing of the content based on a request from the user. In this embodiment, the display processing unit 102 performs display processing of the content including a document, and transmits the display processing result to the terminal 2 used by the user.

[0050] For example, the display processing unit 102 can operate and display the content image in the target area based on gaze information (described later) (when the gaze is located in the target area). Specifically, when the user's gaze is located in the target area, the display processing unit 102 can display an animation of the facial expressions, eyes, mouths, etc. of the characters in an electronic comic book, etc., moving, and display them as a still image after the gaze leaves the target area.

[0051] Similarly, the display processing unit 102 can change the color of the content image in the target area based on the gaze information. For example, when the user's gaze is located in the target area, the display processing unit 102 can display the target frame in an electronic comic or the like in color, and display it in black and white after the gaze leaves the target area. In addition, it is also possible to display content in a virtual space using VR equipment. By performing these display processes, the content providing system can provide further dramatic effects along with the sound.

[0052] The gaze detection unit 103 detects the gaze of the user toward the terminal 2. The gaze detection unit 103 in this embodiment detects gaze information including the position of the gaze of the user toward a document included in the content and the duration of the gaze at that position.

[0053] Furthermore, the gaze detection unit 103 can detect gaze information by using position information indicating the position of a document in the display range of the content and / or area information indicating the display range of the document.

[0054] This gaze detection can be performed by acquiring the user's actions (facial movements and eye movements) using a camera (imaging unit) provided in the terminal 2, and detecting gaze information from the user's actions. In addition, for example, the information processing device 10 (display processing unit 102) can also perform display processing of a transparent layer that does not affect the viewing of documents included in the content by overlaying the content, and detect the gaze when the user's gaze is located on the transparent layer. This transparent layer functions as a button that responds to the user's gaze. In this way, general eye tracking technology (gaze measurement technology) can be used for gaze detection to realize the present invention.

[0055] The determination processing unit 104 determines priority conditions for playing back audio corresponding to a document. In this embodiment, the determination processing unit 104 determines priority conditions for each document, including a waiting time from when the gaze is detected until the audio is played back, based on gaze information that changes over time. The determination processing unit 104 can determine the priority conditions for each document based on the gaze information, and determine the priority conditions for each document based on the determination processing result.

[0056] Furthermore, the determination processing unit 104 can determine, among the documents included in the content, the priority condition for a candidate document (document expected to be read next) with the next (one greater) progress number relative to the progress number of the target document where the user's gaze is currently located (the progress number indicating the order in which the user reads the document) as the first priority condition, the priority condition for a read document (document read one time ago) with the previous (one smaller) progress number relative to the progress number of the target document as the second priority condition, and the priority condition for an unread document with a progress number other than those as the third priority condition.

[0057] The voice providing unit 105 transmits a voice corresponding to the document to the terminal 2, etc. The voice providing unit 105 in this embodiment transmits a voice corresponding to the document to the terminal 2 based on the gaze information, the priority condition, and the voice information.

[0058] Furthermore, the audio providing unit 105 can also transmit the audio of the sub-document to the terminal 2 based on the line-of-sight information and the sub-audio reproduction conditions in the sub-document to which no progress number is assigned.

[0059] The voice selection unit 106 selects a voice corresponding to a document. In this embodiment, the voice selection unit 106 can be configured to select one dialogue voice from among multiple dialogue voices for the same document (e.g., dialogue in a manga) based on an input from a user.

[0060] For example, if the memory unit 12 stores voice data by multiple voice actors for the same line, the voice selection unit 106 can select one voice data from among the voice data by multiple voice actors based on user input (request).

[0061] <Database> 1 stores document information on documents included in the content, audio information on audio corresponding to the documents, content information, priority condition information, other information necessary for browsing the content, etc. Some or all of these may be stored in the storage unit 12, etc., or some of these may be stored in another database, etc.

[0062] The content providing system 1 and the processing performed by each functional component will be described below with reference to FIGS.

[0063] <Content provision system overview> The content providing system 1 according to this embodiment is a system for providing content with audio in response to a request from a user. Specifically, it is possible to detect the gaze of the user directed toward the terminal 2 on which the content is displayed, and provide audio corresponding to the dialogue along with the content such as a cartoon based on the gaze.

[0064] A schematic image diagram of a content providing system according to an embodiment of the present invention is shown in Fig. 3. As shown in the figure, an information processing device 10 detects the position of a user's gaze directed toward a content (e.g., a text such as dialogue in the case of a comic book) displayed on a terminal 2 as gaze information (Fig. 3(a)).

[0065] Then, the information processing device 10 uses the gaze information to transmit voice data based on characteristic predetermined conditions to the terminal 2 (FIG. 3(b)). The terminal 2 plays (outputs) voice corresponding to the document of the content on which the user's gaze is directed (FIG. 3(c)).

[0066] By repeating the series of processes from (a) to (c) in FIG. 3, the user can enjoy ordinary electronic comic content with added dramatic effects, that is, electronic comic content with sound.

[0067] The content providing system 1 according to this embodiment performs each process in the flow roughly described above. Specific processing procedures of the content providing system 1 according to this embodiment will be described in detail below.

[0068] <Registering various information> 4 is a flowchart showing a processing procedure of the content providing system according to an embodiment. In S201, the registration unit 101 of the information processing device 10 registers content information. The content information is associated with document information and audio information.

[0069] As shown in Fig. 5(a), document information includes information such as the text of the document included in the content, position information indicating the position of the document within the display range of the content, area information indicating the display range of the document, document type such as main document or sub-document, audio ID of the audio corresponding to the document, progress number indicating the order in which the document is read, etc., and is managed by user ID. For example, in Fig. 5(a), the document will be read in the order of progress numbers (1), (2), (3), and (4).

[0070] As shown in Fig. 5(b), the audio information includes information such as audio data and the names of the voice actors who are responsible for each piece of audio data, and is managed by an audio ID. As shown in Fig. 5(c), the content information includes information such as image data, document ID, audio ID, and is managed by a content ID.

[0071] 1, and stores (registers) the voice data as voice information in the storage unit 12. The registration unit 101 can also obtain voice data from the user's terminal 2 as voice information, and stores (registers) the voice data in the storage unit 12.

[0072] For example, the registration unit 101 can obtain voice data by multiple voice actors from the voice providing servers 3(a)-(c) to which the voice actors belong, and register voice data by different voice actors for the same lines, etc. In this case, as will be described in detail later, voice data by multiple voice actors can be selectively associated with the same lines or characters.

[0073] 5(d), the registration unit 101 registers priority condition information indicating the priority and conditions for playing back audio. The priority condition information includes information such as document type such as a main document or sub-document, which will be described later, items such as a first priority condition and a second priority condition, information regarding the waiting time from when the user's gaze is detected until audio is played back, other audio playback conditions, and the like, and is managed by a priority condition ID.

[0074] In this embodiment, image data and documents are registered separately, but it is also possible to register image data including, for example, characters and lines from a manga etc. In this case, the document information does not include text information, but includes information such as the position and range of lines in the image data, and in addition, includes information about a button (the transparent layer mentioned above) for detecting line-of-sight information when the line of sight is directed at the document.

[0075] Also, for example, users who wish to use this content providing system may be required to register in advance, in which case only registered users may use this content providing system.

[0076] <Display processing and gaze detection> In S202, the display processing unit 102 of the information processing device 10 performs display processing of content such as a comic book based on a request from a user, and transmits the display processing result to the terminal 2. An example of the display processing result according to one embodiment of the present invention is shown in Fig. 6. As shown in the figure, in this embodiment, a content image W10 displayed on the terminal 2 is an electronic comic book or the like.

[0077] In content image W10 in Fig. 6, characters 1 and 2 are displayed as characters (appearances), and lines for each character are displayed. Here, the lines are the document (document information) in Fig. 5(a), and characters 1 and 2 (as well as background images, etc., not shown) are the image data in Fig. 5(c). As mentioned above, the document and characters, etc. can also be displayed together as image data. The characters are not limited to characters.

[0078] For example, in Fig. 6, lines are assigned progress numbers from (1) to (7). The progress numbers indicate the order in which the user reads the document. In Fig. 6, (1) to (7) are displayed in content image W10 to explain the progress numbers, but in actual display processing, these progress numbers do not need to be displayed.

[0079] In S203, the gaze detection unit 103 of the information processing device 10 detects the gaze of the user. Specifically, the gaze detection unit 103 detects gaze information including the position of the gaze of the user directed toward the document in FIG. 6 and the duration of the gaze at that position.

[0080] As described above, the gaze detection unit 103 can detect the gaze information from the user's actions (face movement and eye movement) obtained by a camera provided in the terminal 2. Alternatively, the display processing unit 102 can overlay a transparent layer that does not affect the viewing of a document included in the content on the content and perform display processing, and the gaze detection unit 103 can detect the gaze when the user's gaze is located on the transparent layer.

[0081] As shown in Figure 7(a), gaze information includes position information regarding the position of the user's gaze toward the document, the time the user stayed at the position of the gaze toward the document, and the detection time when the gaze was detected, and is managed by a gaze information ID.

[0082] For example, the position information in the gaze information may be coordinates indicating the position of the document relative to the entire content display. For the area information, a so-called bounding box (a rectangular boundary line surrounding a character string, etc.) technique may be used. Based on this gaze information, priority conditions for audio playback, which will be described later, are determined.

[0083] <Priority conditions> In S204, the determination processing unit 104 of the information processing device 10 determines priority conditions for each document. Based on gaze information that changes over time, the determination processing unit 104 determines priority conditions for each document, including a waiting time from when the gaze is detected until the sound is played back. In this embodiment, the sound corresponding to the document is played back based on the priority conditions.

[0084] The determination processing unit 104 determines the priority condition for a candidate document (document expected to be read next) with the next (one greater) progress number relative to the progress number of the target document where the user's gaze is currently located (the progress number indicating the order in which the user reads the document) as the first priority condition, the priority condition for a read document (document read one time ago) with the previous (one smaller) progress number relative to the progress number of the target document as the second priority condition, and the priority condition for unread documents with other progress numbers as the third priority condition.

[0085] The order of the progress numbers is not particularly limited, and for example, the progress numbers can be set in reverse order, i.e., so that the document with the smallest number next to the document with the largest number is read, rather than the document with the smallest number being read first.

[0086] The priority conditions are determined for each document based on the gaze information that changes over time, and are managed as audio playback information. For example, as shown in Fig. 7(b), the audio playback information is managed including items such as gaze ID, audio ID, content ID, progress number, priority condition ID, first priority, and information such as waiting time and other audio playback conditions. The audio playback information in Fig. 7(b) corresponds to the content image W10 in Fig. 8 described later (gaze ID 40002 is audio playback).

[0087] In addition, since the user's line of sight changes over time and the priority conditions for each document change in a short period of time, the voice playback information does not necessarily need to be managed in the storage unit 12. For example, the priority conditions can be repeatedly determined or changed each time according to changes in the user's line of sight, and the voice playback information can be configured not to be managed as data. By configuring in this way, it is expected that the data capacity in the storage unit 12 can be reduced.

[0088] Fig. 8 shows a schematic image diagram of priority conditions according to one embodiment of the present invention. In order to make the explanation easier to understand in Fig. 8, the progress numbers of each document expressed as a speech bubble are displayed as (1) to (7), and the speech bubble of each document is colored or patterned according to the priority conditions. Hereinafter, the document (dialogue) assigned the progress number (1) in Fig. 8 will be referred to as document (1), and the document assigned the progress number (2) as document (2) (the same applies to (3) to (7)).

[0089] For example, as shown in FIG. 8, when the user directs his / her gaze toward a line of dialogue in document (2), that is, when gaze detection unit 103 detects the user's gaze, audio is played from terminal 2 for the line of dialogue in document (2) (details will be described later).

[0090] At this time, the priority condition for document (2) is determined by the determination processing unit 104 as the first priority condition, which has the highest priority for audio playback. It is also possible not to set a priority condition for a document that is being played back as audio, such as document (2). For example, it is also possible to set the document that is being played back as being gradually changed from the first priority condition to the second priority condition.

[0091] 8, when document (2) is being played back audibly based on the gaze information, the determination processor 104 determines the priority condition of document (3), which is next to the document (2) being played back (the document to be read next in the story order), as the first priority condition, which has the highest priority condition. Since a user is usually likely to read document (3) after document (2), the determination processor 104 in this embodiment determines document (3) as the first priority condition.

[0092] Furthermore, the determination processing unit 104 determines the priority condition of the document (1) that precedes the document (2) being audibly reproduced (the document that has already been read and is located just before the document (2)) as the second priority condition with the second highest priority, taking into consideration the possibility that the user will reread the previous document (1).

[0093] Then, while the audio of document (2) is being played back, the determination processing unit 104 determines the priority conditions for documents (4), (5), (6), and (7) other than documents (1) and (3) as the third priority conditions, taking into consideration that documents (5) and the like are unlikely to be looked at by the user compared to documents (1) and (3).

[0094] In this way, the determination processing unit 104 determines the priority condition according to the possibility that the user will direct his / her gaze at each document based on the gaze information. In this embodiment, the waiting time from when the user gazes at the document until the audio is played back is set according to the priority condition. For example, in FIG. 7(b), the waiting time for the first priority condition is set to 1.0 seconds, the waiting time for the second priority condition to 3.0 seconds, and the waiting time for the third priority condition to 6.0 seconds. These waiting times can be arbitrarily changed based on a setting change input from the user (from the terminal 2).

[0095] For example, the waiting time for the first priority condition is preferably about 0.1 to 2.0 seconds, and more preferably about 0.5 to 1.0 seconds. This is because the document with the first priority condition is likely to be read next by the user, and it is necessary to play the audio instantly.

[0096] The waiting time under the second priority condition is preferably about 2.0 to 5.0 seconds, and more preferably about 3.0 to 4.0 seconds. For example, while the user may re-read a line, even in such a case the user may not desire audio playback and may instantaneously move their eyes to a document with the second priority condition, so the waiting time is set in this way to allow appropriate audio playback.

[0097] Furthermore, the waiting time under the third priority condition is preferably about 5.0 to 10.0 seconds, and more preferably about 6.0 to 8.0 seconds. A document to which the third priority condition is applied is basically unlikely to be read next by a user, but there is a possibility that the user may move his / her eyes quickly to check the entire content, for example, and this is to ensure that audio can be played appropriately even in such a case. In other words, this is to prevent audio from being played when the user moves his / her eyes quickly to check the entire content.

[0098] Then, as shown in FIG. 9, when document (3) is being played back as audio due to the user's gaze, the determination processing unit 104 determines (or changes) document (4), which has a progress number one value higher than document (3), as the first priority condition, determines document (2), which has a progress number one value lower than document (3), as the second priority condition, and determines the other documents (1), (5), (6), and (7) as the third priority condition.

[0099] Furthermore, as shown in FIG. 10, when the user has not yet directed his / her gaze at a document included in the content, i.e., when the page of the content is displayed for the first time, the determination processing unit 104 determines document (1) as the first priority condition, and determines the other documents (2) to (7) as the third priority condition.

[0100] In this embodiment, as the user directs his / her gaze at document (1) in FIG. 10 and continues reading the document, the determination processing unit 104 determines or changes the priority conditions to the states shown in FIG. 8 or FIG. 9 described above based on the user's gaze information that changes over time.

[0101] Then, in S205, the line of sight information and the voice based on the priority conditions are transmitted to the terminal 2 used by the user. The terminal 2 receives the voice (voice data) based on the predetermined priority conditions.

[0102] In S206, the terminal 2 that has received the voice data outputs a voice corresponding to the document from the terminal 2. Specifically, as described above, a priority condition is determined for each document shown in Fig. 8 to 10, so the voice is reproduced from the terminal 2 after a waiting time corresponding to each priority condition has elapsed based on the line-of-sight information (position information, area information) from the user.

[0103] By setting the wait time according to the priority conditions in this way, for example, when the user wants to check the entire dialogue of the content (when the user is quickly changing the position of his or her eyes to look at the whole picture), no audio will be played, and the audio of the text included in the content can be played at the appropriate time when the user wants to read the dialogue.

[0104] Fig. 11 shows a detailed processing flow of the content providing system according to this embodiment. As shown in the figure, when the user's line of sight enters the target area of ​​the terminal 2 (S301), the information processing device 10 refers to the progress log (S302) and decides (and judges) the priority (S303). Then, it is judged whether the previously executed audio is currently being played (S304), and if it is being played, the audio track being played is stopped (skip) (S305). If it is not being played, the audio playback is put on hold based on the waiting time of the priority condition (S306).

[0105] Thereafter, it is determined whether the gaze has left the target area (S307), and if the gaze has left the target area, the execution process is canceled (S308), and if the gaze has not left the target area, the audio associated with the document is played from terminal 2 (S309), the progress log is updated (S310), it is determined whether the gaze has left the target area (S311), and this series of processes is repeated for the next audio algorithm.

[0106] When performing these processes in this embodiment, the gaze detection unit 103 executes processes related to S301, S302, S307, S310, and S311, the decision processing unit 104 executes processes related to S303, S304, and S305, the audio providing unit 105 (or the terminal 2 that received the audio) executes processes related to S306, S308, and S309, and the memory unit 12 manages progress logs, etc.

[0107] For example, by using the progress log and gaze information, documents that the user has focused on and viewed multiple times can be analyzed as numerical values, and the results of this analysis can be used to provide irregular content that changes in content content or structure.

[0108] <Subdocument audio playback> In this embodiment, documents are divided into two types, and audio playback can be managed according to these multiple document types. Specifically, documents in this embodiment have a main document to which a progress number is associated, and other sub-documents (FIG. 5(a)). Note that document types are not limited to two types, and audio playback can also be managed by classifying documents into three or more types.

[0109] Fig. 12 shows a schematic image diagram for explaining a sub-document according to one embodiment of the present invention. Fig. 12 shows a content image W10 shown in Fig. 8 to which a sub-document and the like have been added. This sub-document is a document that is less relevant to the main story in the content, and in Fig. 8, it is a document that expresses the sound of rain as "drops."

[0110] The registration unit 101 of the information processing device 10 can register the sub-audio playback conditions of the sub-document according to the progress number in advance. For example, as shown in Fig. 5(d), the sub-audio playback conditions can be registered as being included in the priority condition information.

[0111] 12(b) shows a setting image of the sub-audio playback information. As shown in the figure, when the user's line of sight is in the range of progress numbers (2) to (4), that is, when the user is reading documents (2) to (4), the audio provider 105 can transmit audio data of "drops" expressing the sound of rain to the terminal 2.

[0112] In this manner, in this embodiment, audio can be reproduced based on a predetermined condition for sub-documents and the like that are less relevant to the story of the content but are intended to enhance the dramatic effect. Note that the sub-audio reproduction conditions can be appropriately changed in response to input from the user.

[0113] In addition, as shown in Fig. 12(b), if you want to use "BGM" as another production effect, you can set it so that the BGM sounds are played when the line of sight is on the progression numbers (1) to (7). In this case, the dialogue of the document (2) and the like is played over the BGM sound.

[0114] It is also possible to output (play) sound effects during the scene of document (5). In this case, the sound effects and the audio of document (5) are played over the background music. For example, it is possible to set sound effects (or audio of sub-documents) to be played after or before the dialogue of document (5) is played.

[0115] In addition, the number of times the sound effect is to be played and the duration of the playback can be set. For example, the sound effect can be set to be played repeatedly while the dialogue of document (5) is being played in Fig. 12. The information processing device 10 can receive a setting change signal from the terminal 2 based on an input from the user, change these settings based on the setting change signal, and store the changes in the storage unit 12.

[0116] In this manner, in this embodiment, priority conditions with different waiting times can be set for main documents related to the main story in the content, and further, for sub-documents less related to the story, flexible presentation effects can be achieved according to the progress number without compromising the sound of the story.

[0117] As described above, according to the content providing system 1 of the present invention, by utilizing gaze information and predetermined priority conditions based on the gaze information, it is possible to provide content with audio added more appropriately and effectively than before.

[0118] Furthermore, although the present embodiment has been described with respect to electronic comic book content, the same effects as the present invention can be obtained when the content providing system 1 is used for games such as television games, video games, and computer games, as well as other content, in addition to electronic books such as novels, newspapers, magazines, and learning materials. [Explanation of symbols]

[0119] 1 Content provision system 2. Terminal 3. Voice provision server 10. Information processing device 11 Control section 12 Storage section 13. Communications Department 90 Terminal (Terminal 2) 91 Control Unit 92 Memory section 93 Communications Department 94 Input section 95 Output section 101 Registration Department 102 Display processing unit 103 Detection processing section 104 Decision Processing Unit 105 Audio playback section 106 Audio selection section NW Network W10 Content Image

Claims

1. A content providing system for providing content, comprising: The content providing system includes a display processing unit, a gaze detection unit, a determination processing unit, a sound providing unit, and a storage unit, The storage unit stores document information relating to a document and voice information relating to a voice corresponding to the document; the display processing unit performs display processing on the content including the document, and transmits a display processing result to a terminal used by a user; the gaze detection unit detects gaze information including a gaze position from the user to the document and a duration of the gaze at that position; the determination processing unit determines, for each of the documents, a priority condition including a waiting time from when the gaze is detected until when the audio is reproduced, based on the gaze information that changes over time; the voice providing unit transmits to the terminal a voice corresponding to the document based on the line of sight information, the priority condition, and the voice information; Content delivery system.

2. the document information includes a progression number indicating an order in which the user is to read the document; the determination processing unit determines a priority condition for a candidate document whose progress number is the next to the progress number of the target document where the gaze is located as a first priority condition, a priority condition for a read document whose progress number is the previous to the progress number of the target document as a second priority condition, and a priority condition for an unread document assigned a progress number other than the progress number as a third priority condition. The content providing system according to claim 1 .

3. a waiting time is set for the first priority condition for the candidate document, which is shorter than a waiting time for the second priority condition for the already-read document; a third priority condition for the unread document is set to a waiting time longer than a second priority condition for the read document; The content providing system according to claim 2 .

4. The document includes a main document to which the progress number is associated and other sub-documents; the storage unit stores a sub-audio playback condition of the sub-document based on the progress number; the audio providing unit transmits audio of the sub-document to the terminal based on the line-of-sight information and the sub-audio playback conditions; The content providing system according to claim 2 .

5. The document information is associated with position information indicating a position of the document in a display range of the content and area information indicating a display range of the document; The gaze detection unit detects the gaze information by utilizing the position information and / or the area information. The content providing system according to claim 1 .

6. The content is an electronic comic book content, The document includes lines of characters in an electronic comic; The audio information includes dialogue of characters in the electronic comic.

6. A content providing system according to claim 1.

7. The content providing system further includes an audio selection unit, The storage unit stores a plurality of the dialogue voices, the voice selection unit is capable of selecting one line voice from among a plurality of line voices for the same line based on an input from the user; The content providing system according to claim 6 .

8. A content providing method executed by a content providing system for providing content, comprising: The content providing system includes a display processing unit, a gaze detection unit, a determination processing unit, a sound providing unit, and a storage unit, The storage unit stores document information relating to a document and voice information relating to a voice corresponding to the document, the display processing unit performs display processing on the content including the document, and transmits a display processing result to a terminal used by a user; the gaze detection unit detects gaze information including a gaze position from the user to the document and a duration of the gaze at that position; the determination processing unit determines, for each of the documents, a priority condition including a waiting time from when the gaze is detected until when the audio is reproduced, based on the gaze information that changes over time; The voice providing unit transmits a voice corresponding to the document to the terminal based on the line of sight information, the priority condition, and the voice information. How content is provided.

9. A content providing program for providing content, A computer is caused to function as a display processing unit, a gaze detection unit, a decision processing unit, a voice providing unit, and a storage unit; The storage unit stores document information relating to a document and voice information relating to a voice corresponding to the document; the display processing unit performs display processing on the content including the document, and transmits a display processing result to a terminal used by a user; the gaze detection unit detects gaze information including a gaze position from the user to the document and a duration of the gaze at that position; the determination processing unit determines, for each of the documents, a priority condition including a waiting time from when the gaze is detected until when the audio is reproduced, based on the gaze information that changes over time; the voice providing unit transmits to the terminal a voice corresponding to the document based on the line of sight information, the priority condition, and the voice information; Content provision program.

Citation Information

Patent Citations

  • Electronic book device and sound reproduction system

    JP2001265566A

  • Page recognition method for comics and comics information reproduction system

    JP2007114878A

  • Electronic comic viewer device, electronic comic browsing system, viewer program and recording medium recording viewer program

    JP2012133662A

  • Information processing device and program

    JP2020046842A

  • Interaction and management of devices using gaze detection

    US20130307771A1

Cited By

  • Game machine

    JP2024166283A

  • Game machine

    JP2024166284A

  • Game machine

    JP2024166285A

  • Game machine

    JP2024169530A

  • Game machine

    JP2024169531A