Stimulus information generation device and program

Abstract

To provide a rich media experience to a user by synchronizing with progress of contents and presenting an appropriate stimulus at appropriate timing when the user receives information from static contents.SOLUTION: A user operation setting unit 11-1 of a stimulus information generation device 1-1 sets a target area on the basis of gazing points of a predetermined number of users on target contents, and calculates an average value of residence time of the gazing points as trigger time on the basis of the fixation points of a target user to sample contents. A stimulus setting unit 12 sets a stimulus type, intensity, etc. for the target area, and generates a stimulus generation rule. A stimulus information generation unit 30-1 generates stimulus information including the stimulus type, intensity, etc. corresponding to the target area from the stimulus generation rule, when the gazing points of the target user on the target contents are within the target area and the trigger time has elapsed. Then, a stimulus according to the stimulus information is presented to the target user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a stimulus information generation device and program that generate stimulus information, such as tactile information, as static media content progresses. [Background technology]

[0002] Television and the internet offer a wide variety of video content. In recent years, with the development of AR / VR technology, attention has been focused on methods that present tactile and olfactory stimuli to enhance the sense of realism and impact when enjoying video content. However, most of these methods present tactile and olfactory stimuli in sync with the progression of the video content. Users passively receive visual stimuli from automatically updated dynamic images and auditory stimuli from sound, and then receive tactile and olfactory stimuli that are linked to these.

[0003] However, the media available in the world includes not only dynamic media like video content, where the information presented is automatically updated (changes) on the display device, but also a large amount of static content such as books like manga, novels, and picture books, as well as web content.

[0004] Users who receive information from such static content progress through the content by manually refreshing it at their own pace. Here, refreshing information from static content means moving your eyes, turning pages, etc., if the static content is a book, and scrolling with the mouse, etc., if it is web content.

[0005] An example of static content is a pop-up book (see, for example, Non-Patent Document 1). In this pop-up book, the user is presented with the olfactory stimulus of maple syrup scent by rubbing a picture of a pancake.

[0006] To realize such pop-up books, a scent printing technology is known that generates a fragrance when the book is rubbed (see, for example, Non-Patent Document 2). However, in this type of pop-up book, a conscious action of rubbing is required to receive olfactory stimulation, which is not expected during natural reading.

[0007] Furthermore, as an example of static content, there is a known technology that presents tactile stimuli by clicking on designated areas when enjoying content on electronic media (see, for example, Non-Patent Document 3). However, this content also requires a conscious action, such as clicking, which is not expected during natural reading, in order to receive tactile stimuli.

[0008] In contrast, as an example of static content, there are known technologies that present tactile stimuli without requiring conscious actions such as rubbing or clicking (see, for example, Patent Documents 1 and 2). The technologies in Patent Documents 1 and 2 measure the user's gaze, generate tactile effects corresponding to the point of focus the user is looking at, and present tactile stimuli to the user.

[0009] Specifically, the technology described in Patent Document 1 analyzes a medium for identifying an image, identifies the point of gaze of the user viewing the image, and, based on the identified image and the user's point of gaze, generates a tactile effect corresponding to the object at the point of gaze, and presents the tactile stimulus to the user.

[0010] Furthermore, the technology described in Patent Document 2 determines the user's gaze direction, generates a first tactile effect if the gaze direction includes an interactable object, generates a second tactile effect if the gaze direction does not include an interactable object, and presents the user with a tactile stimulus corresponding to the first or second tactile effect. [Prior art documents] [Patent Documents]

[0011] [Patent Document 1] Japanese Patent Publication No. 2016-126773 [Patent Literature 2] Japanese Patent No. 6703044 Publication [Non-Patent Literature]

[0012] [Non-Patent Literature 1] Kimura Yuichi, "Hot Hot Pancake with the scent of maple syrup!", [online], February 2016, Sekai Bunka Publishing, [retrieved July 28, Reiwa 4], Internet <URL:https: / / www.e-hon.ne.jp / bec / SA / Detail?refISBN=9784418168071> [Non-Patent Literature 2] "Scent Printing", [online], Print On Co., Ltd., [retrieved July 28, Reiwa 4], Internet <URL:https: / / www.print-on.jp / doujin / comic / tokusyu / kaori.htm> [Non-Patent Literature 3] Kazi Masudul Alam, et al., "HE-book: A prototype haptic interface for immersive e-book reading experience", [online], 2011, IEEE Xplore, [retrieved July 28, Reiwa 4], Internet <URL:https: / / ieeexplore.ieee.org / stamp / stamp.jsp?tp=&arnumber=<5945514> [Summary of the Invention] [Problem to be Solved by the Invention]

[0013] However, when a user receives information from static content, in order to achieve a richer experience by being presented with tactile stimuli and the like, it is necessary to present appropriate tactile stimuli and the like at appropriate timing in synchronization with the user-specific content progression.

[0014] When tactile stimuli are presented at the appropriate time, they greatly enhance the user's media experience. Conversely, when presented at an inappropriate time (including timing discrepancies or inappropriate types of stimuli), they can actually degrade the quality of the media experience.

[0015] Therefore, in the case of static content that is not synchronized with video, it is important to synchronize the user's progress through the content with tactile stimuli, etc., and that the stimuli presented are appropriate.

[0016] On the other hand, the technologies described in the aforementioned Patent Documents 1 and 2 measure the user's gaze, generate tactile effects corresponding to the user's gaze point, and present tactile stimuli to the user. However, these technologies do not provide any technical information on how to appropriately determine the user's gaze point for generating tactile effects, or at what timing it is appropriate to present the stimuli.

[0017] Therefore, even if tactile stimuli or the like are presented to the user when the user receives information from static content (for example, when the user is reading a manga) using the technologies described in the aforementioned Patent Documents 1 and 2, it is not guaranteed that the quality of the media experience will be sufficiently improved.

[0018] Therefore, the present invention has been made to solve the aforementioned problems, and its objective is to provide a stimulus information generating device and program that can provide users with a rich media experience by presenting appropriate stimuli at appropriate times in synchronization with the progression of the content when users receive information from static content. [Means for solving the problem]

[0019] To solve the above problem, the stimulus information generating device of claim 1 generates stimulus information for presenting a stimulus to a user as the content progresses according to the operation of a user who receives information from static content, and outputs the stimulus information to a stimulus presentation device, wherein the target area is a predetermined area that is the region where the user's gaze point remains over the content and depicts the content, the user to whom the stimulus is presented is the target user, and the content that progresses according to the operation of the target user is the target content, the gaze data of a predetermined number of users when they are looking at the target content is input from a gaze measurement device that measures the gaze of the user, a plurality of target areas are determined based on the trajectories of the gaze points indicated by the gaze data of the predetermined number of users and the predetermined area, the positions of the plurality of target areas are set as a plurality of target area position data, and the gaze data of the target user when they are looking at sample content other than the target content is input from the gaze measurement device, and the The system is characterized by comprising: a user operation setting unit that sets the dwell time of the point of fixation indicated by the gaze data as the trigger time and stores the multiple target area position data and the trigger time in a first database; a stimulus setting unit that sets the stimulus type and intensity corresponding to each of the multiple target areas and stores the stimulus type and intensity in a second database; and a stimulus information generation unit that inputs gaze data from the gaze measurement device when the target user is viewing the target content, reads the multiple target area position data and the trigger time from the first database, reads the stimulus type and intensity for each of the multiple target areas from the second database, determines the conditions that the point of fixation indicated by the gaze data is within the target area and that the time the point of fixation has been within the target area has elapsed since the trigger time, and when it is determined that the conditions are met, generates stimulus information including the stimulus type and intensity corresponding to the target area and outputs the stimulus information to the stimulus presentation device.

[0020] Furthermore, the stimulus information generating device of claim 2 is characterized in that, in the stimulus information generating device of claim 1, the user operation setting unit determines, for each of the predetermined number of users, the area where the gaze point remains based on the trajectory of the gaze point indicated by the gaze data of the user, and determines a region that includes all the areas where the gaze point remains for the predetermined number of users, and the predetermined region as the plurality of target regions, or averages all the areas where the gaze point remains determined for each of the predetermined number of users, determines the averaged area, and the predetermined region as the plurality of target regions, and stores the plurality of target region position data in the 1 DB.

[0021] Furthermore, the stimulus information generating device of claim 3 is characterized in that, in the stimulus information generating device of claim 1, the user operation setting unit calculates multiple dwell times in multiple regions where the gaze point remains, based on the gaze point indicated by the gaze data of the target user, calculates an average value from the multiple dwell times, sets the average value as the trigger time, and stores the trigger time in the first DB.

[0022] Furthermore, the stimulus information generating device of claim 4 generates stimulus information for presenting a stimulus to a user as the content progresses according to the user's operation of receiving information from static content, and outputs the stimulus information to a stimulus presentation device, wherein the target area is a predetermined area that depicts the content and is the region where the user's gaze point remains over the content, the user to whom the stimulus is presented is the target user, and the content that progresses according to the target user's operation is the target content, eye gaze data is input from an eye gaze measuring device that measures the gaze of the user, for a predetermined number of users when they are looking at the target content, a plurality of target areas are determined based on the trajectories of the gaze points indicated by the eye gaze data of the predetermined number of users and the predetermined area, the positions of the plurality of target areas are set as a plurality of target area position data, the number of characters per frame in the target content is set, and eye gaze data is input from the eye gaze measuring device when the target user is looking at sample content other than the target content, and the trajectory of the gaze point indicated by the eye gaze data A user operation setting unit that calculates the reading speed of the target user when reading characters based on the number of characters, calculates the movement speed of the target user's gaze point based on the trajectory of the gaze point indicated by the gaze data, calculates the arrival time for the target user's gaze point to reach each of the multiple target areas included in each of the multiple target areas included in each of the multiple pages constituting the target content, based on the multiple target area position data, the number of characters per frame, the reading speed, and the movement speed, and stores the arrival time in a first database; a stimulus setting unit that sets the stimulus type and intensity corresponding to each of the multiple target areas, and stores the stimulus type and intensity in a second database; reads the arrival time for each of the multiple target areas included in each of the multiple target areas included in each of the multiple target areas from the first database, reads the stimulus type and intensity for each of the multiple target areas from the second database, counts the elapsed time since the page was turned for each of the pages, determines the condition for the elapsed time to reach the arrival time, and when it is determined that the condition is met,a stimulation information generation unit configured to generate said stimulation information including said stimulation type and intensity corresponding to said target region for said arrival time, and output said stimulation information to said stimulation presentation device; and the apparatus is characterized by comprising the foregoing.

[0023] Also, the stimulation information generation apparatus according to claim 6 is the stimulation information generation apparatus according to claim 4, characterized in that, when the gaze point indicated by said gaze data of said target user moves from a first target region to a second target region, said user action setting unit obtains said movement speed by dividing the distance from said first target region to said second target region by the movement time from said first target region to said second target region.

[0024] Also, the stimulation information generation apparatus according to claim 6 is the stimulation information generation apparatus according to claim 4, wherein said user action setting unit sets said reading speed as V C , said movement speed as V L , the number of characters in the n-th frame as C n , the center position of said target region in the n-th frame as (P xn , P yn ), the time from when said target user starts reading the n-th frame until said gaze point moves to said target region of the next (n+1)-th frame as T n , and said arrival time as TT n , using said reading speed V C , said movement speed V L , said number of characters C in the n-th frame n , the center position (P xn , P yn ) of said target region in the n-th frame and the center position (P xn+1 , P yn+1 ) of said target region in the (n+1)-th frame, the apparatus is characterized in that it calculates said time T by means of TIFF0007923663000001.tif21170, n calculates said time T n and uses said time T to calculate said arrival time TT by means of TIFF0007923663000002.tif23170, n and is characterized by calculating said arrival time TT.

[0025] Furthermore, the stimulus information generating device of claim 7 is a stimulus information generating device according to claim 1 or 4, further comprising a sound DB storing a plurality of set data consisting of onomatopoeia and voice data, wherein when vibration is set as the stimulus type corresponding to the target area, the stimulus setting unit selects voice data corresponding to onomatopoeia present in the frame including the vibration target area from the sound DB, stores the vibration and intensity of the stimulus type and the voice data in the second DB, reads the vibration and intensity of the stimulus type and the voice data from the second DB, and when it determines that the vibration target area satisfies the conditions, generates stimulus information including the vibration and intensity of the stimulus type corresponding to the vibration target area and the voice data, and outputs the stimulus information to the stimulus presentation device in which the vibrator operates according to the intensity and the voice data.

[0026] Furthermore, the program of claim 8 generates stimulus information to present stimuli to a user as the content progresses according to the user's operation of receiving information from static content, and outputs the stimulus information to a stimulus presentation device. The computer constituting the stimulus information generation device has a target area that is a predetermined area where the user's gaze point remains on the content and which depicts the content's contents, the user to whom the stimulus is presented is the target user, and the content that progresses according to the target user's operation is the target content. The program inputs gaze data from an eye-tracking device that measures the user's gaze, for each of a predetermined number of users when they are looking at the target content, and determines a plurality of target areas based on the trajectories of the gaze points indicated by the gaze data of the predetermined number of users and the predetermined area, sets the positions of the plurality of target areas as a plurality of target area position data, inputs gaze data from the eye-tracking device when the target user is looking at sample content other than the target content, and the gaze data The system is characterized by functioning as a stimulus information generation unit that inputs gaze data from the gaze measurement device when the target user is viewing the target content, reads the multiple target area position data and the trigger time from the first DB, reads the stimulus type and intensity for each of the multiple target areas from the second DB, determines whether the gaze point indicated by the gaze data is within the target area and whether the time the gaze point has been within the target area has elapsed since the trigger time, generates stimulus information including the stimulus type and intensity corresponding to the target area when it is determined that the conditions are met, and outputs the stimulus information to the stimulus presentation device.

[0027] Furthermore, the program of claim 9 generates stimulus information to present stimuli to a user as the content progresses according to the user's operation of receiving information from static content, and outputs the stimulus information to a stimulus presentation device. The computer constituting the stimulus information generation device has a target area that is a region where the user's gaze point remains on the content and which depicts the content's contents, the user to whom the stimulus is presented is the target user, and the content that progresses according to the target user's operation is the target content. The program inputs gaze data from an eye-tracking device that measures the user's gaze, for each of a predetermined number of users when they are looking at the target content, determines a plurality of target areas based on the trajectories of the gaze points indicated by the gaze data of the predetermined number of users and the predetermined area, sets the positions of the plurality of target areas as a plurality of target area position data, sets the number of characters per frame in the target content, inputs gaze data from the eye-tracking device when the target user is looking at sample content other than the target content, and the gaze data A user operation setting unit calculates the reading speed when the target user reads characters based on the trajectory of the gaze point indicated by the gaze data and the number of characters, calculates the movement speed of the target user's gaze point based on the trajectory of the gaze point indicated by the gaze data, calculates the arrival time for the target user's gaze point to reach each of the multiple target areas included in each of the multiple pages constituting the target content, based on the multiple target area position data, the number of characters per frame, the reading speed, and the movement speed, and stores the arrival time in a first database; a stimulus setting unit sets the stimulus type and intensity corresponding to each of the multiple target areas and stores the stimulus type and intensity in a second database; and reads the arrival time for each of the multiple target areas included in each of the multiple target areas from the first database for each page of the target content, reads the stimulus type and intensity for each of the multiple target areas from the second database, counts the elapsed time since the page was turned for each page, and determines the condition that the elapsed time has reached the arrival time.The system is characterized by functioning as a stimulus information generation unit that, when it determines that the above conditions are met, generates stimulus information including the type and intensity of the stimulus corresponding to the target region of the arrival time, and outputs the stimulus information to the stimulus presentation device. [Effects of the Invention]

[0028] As described above, according to the present invention, when a user receives information from static content, it is possible to present appropriate stimuli at the appropriate time in synchronization with the progression of the content. In this way, a rich media experience can be provided to the user. [Brief explanation of the drawing]

[0029] [Figure 1] This is a schematic diagram showing an example of the overall system configuration including the stimulus information generation device of Examples 1 and 2. [Figure 2] This is a block diagram showing an example configuration of the user operation setting unit and reading tendency database in Example 1. [Figure 3] This flowchart shows an example of the processing in the target area setting section. [Figure 4] This flowchart shows an example of the process in the trigger time setting section. [Figure 5] This is a diagram illustrating an example of a target region. [Figure 6] This figure shows an example of target area data stored in the reading tendency database. [Figure 7] This is a block diagram showing an example of the configuration of the stimulus setting unit in Example 1. [Figure 8] This is a flowchart showing an example of the processing of the stimulus setting unit in Example 1. [Figure 9] This figure shows an example of the display screen of the display device 2 when setting the stimulus generation rules. [Figure 10] This flowchart shows an example of the processing performed by the audio data retrieval unit. [Figure 11] This figure shows an example of the data structure of a voice database. [Figure 12]This figure shows an example of stimulus generation rules stored in the stimulus generation rule database. [Figure 13] This is a block diagram showing an example of the configuration of the stimulus information generation unit in Example 1. [Figure 14] This flowchart shows an example of the processing of the stimulus information generation unit in Example 1. [Figure 15] This is a block diagram showing an example configuration of the user operation setting unit and reading tendency database in Example 2. [Figure 16] This figure shows an example of the number of characters Cn per frame. [Figure 17] This flowchart shows an example of the processing performed by the reading speed calculation unit. [Figure 18] This flowchart shows an example of the processing in the arrival time calculation unit. [Figure 19] This diagram illustrates an example of the processing in the arrival time calculation unit. [Figure 20] This figure shows an example of arrival time TTn. [Figure 21] This is a block diagram showing an example of the configuration of the stimulus information generation unit in Example 2. [Figure 22] This is a flowchart showing an example of the processing of the stimulus information generation unit in Example 2. [Figure 23] This flowchart shows the continuation of the processing example in Figure 22. [Modes for carrying out the invention]

[0030] The embodiments for carrying out the present invention will be described in detail below with reference to the drawings. [Points of focus of the invention] Before describing Examples 1 and 2 of the present invention, let's explain the focus of the present invention. In the case of static content that is not video-synchronized, it is important that the static content is accurately synchronized with the user's progress and that the stimuli presented are appropriate. Two methods for achieving this are envisioned below. (1) A method that constantly tracks the user's gaze and determines the point of fixation to present stimuli appropriate to the area the user is looking at (hereinafter referred to as "the first method"). (2) A method that automatically presents stimuli in accordance with the user's reading tendencies (reading speed, eye movement speed, etc.) (hereinafter referred to as "the second method").

[0031] The first method, when limited to the presentation of vibrational stimuli alone, overlaps with the technology described in Patent Document 2, but as mentioned above, the technology described in Patent Document 2 alone is insufficient in providing users with a rich media experience.

[0032] For example, in the case of static content such as manga, manga has panels, and onomatopoeia, which are words that represent sound effects as well as words that represent smells, the appearance of objects, and the actions of characters, are written in various sizes and in various places in distinctive handwritten characters. In addition, in manga, emotions may be expressed through illustrations of flames or vertical lines or the color of the panel background, and a variety of forms are used to express emotions. The technology of Patent Document 2 cannot handle such onomatopoeia and diverse forms of expression.

[0033] In order to present the right stimuli (vibration, temperature, wind, scent, etc.) at the right time in accordance with the user's progression through static content, in response to these diverse forms of expression, the following five points are key.

[0034] Firstly, it is important to set the area that responds when the user's gaze point is treated as a trigger input (hereinafter referred to as the "target area"). When the user's gaze point is treated as a trigger input, if the target area is too small, it will not respond, and if it is too large, it will respond in an unintended place. Therefore, it is important to set the appropriate position and size of the target area.

[0035] Secondly, setting the dwell time of the point of focus used as a trigger input (how long the gaze must remain within the target area before a trigger is activated) is also important. This depends on the user's specific reading habits, so it needs to be adjusted to suit each individual user.

[0036] Thirdly, the timing of stopping the stimulus presented by the trigger input is also important. For example, in a section depicting continuous onomatopoeia, the stimulus should be presented continuously while the viewer is fixated on it, and stopped when the viewer's gaze is broken. However, it is necessary to accurately determine when the viewer's gaze has been broken.

[0037] Fourthly, constantly performing eye-tracking limits the positional relationship between the eye-tracking device and the user. When enjoying long, static content, restricting the user's posture can be burdensome for the user.

[0038] Fifth, if the viewer mistakenly identifies a specific area they were looking at the moment they turned the page, it could lead to an error in the order in which stimuli are presented. Even in such cases, it is necessary to present the stimuli in the correct order.

[0039] On the other hand, in order to implement the second method, it is essential to first accurately understand the user's reading habits and to automatically present stimuli at a highly precise timing (synchronized with the user's progression through static content).

[0040] Secondly, it is important to prevent a time lag from accumulating between the user's progression through static content and the presentation of pre-set, automated stimuli. Thirdly, it is important to accommodate the user's irregular actions (such as rereading or pausing reading).

[0041] Here, we assume that in the first and second methods, as static content progresses, stimulus information is generated with the type of stimulus being vibration according to the user's gaze point, and this vibration stimulus information is output to a tactile stimulus presentation device.

[0042] When using a vibrator that inputs audio data as a tactile stimulus presentation device, in order to present appropriate vibrations to the user, the audio data needs to be appropriate data corresponding to, for example, onomatopoeia at the point of focus.

[0043] Here, it is necessary to pre-select appropriate audio data corresponding to the onomatopoeia, but selecting one from the vast amount of existing audio data is difficult. For example, trying out vibration presentations with each audio data and then selecting one would be extremely time-consuming.

[0044] Therefore, one possible approach is to prepare an audio database that allows searching for audio data using onomatopoeia as a search term, and then select audio data using this database (hereinafter referred to as the "third method").

[0045] Example 1, described below, corresponds to the first method described above, and Example 2, described below, corresponds to the second method described above. In Examples 1 and 2, the third method described above is implemented.

[0046] [Example of the overall system configuration including the stimulus information generation device] Next, the overall system including the stimulus information generation device of Examples 1 and 2 will be described. Figure 1 is a schematic diagram showing an example configuration of the overall system including the stimulus information generation device of Examples 1 and 2. This system is composed of a stimulus information generation device 1, a display device 2, an eye-tracking device 3, an acceleration sensor 4, tactile stimulus presentation devices 5A, 5B, 5C and an olfactory stimulus presentation device 6, etc. The stimulus information generation devices 1-1 and 1-2 and the tactile stimulus presentation devices 5A, 5B, 5C and the olfactory stimulus presentation device 6, etc. are connected by wireless or wired communication means.

[0047] The stimulus information generator 1 generates and outputs stimulus information for static content, presenting appropriate stimuli (tactile stimuli, olfactory stimuli, etc.) at appropriate timings in accordance with the user's unique content progression speed. This enables the provision of a rich media experience to the user.

[0048] The stimulus information generating device 1 outputs and displays pages (images) of static content (e.g., manga) to the display device 2 in accordance with the user's page-turning operation.

[0049] The stimulus information generator 1 receives gaze data from the gaze measurement device 3 indicating the area the user is fixated on as the content progresses, for example, by reading through panels of a manga. Then, in Example 1, the stimulus information generator 1 generates stimulus information when conditions are met, such as the point of fixation indicated by the gaze data being within a pre-set target area, or in Example 2, when the elapsed time since the page was turned reaches a predetermined time. The stimulus information generator 1 outputs the stimulus information to one of the tactile stimulus presentation devices 5A, 5B, 5C and the olfactory stimulus presentation device 6, etc., according to the type of stimulus indicated by the stimulus information.

[0050] Here, the content that the stimulus information generator 1 receives as input, generates stimulus information, and outputs is referred to as the target content. The content used for pre-processing other than the target content, among the content input to the stimulus information generator 1, is referred to as the sample content. Furthermore, the user to whom stimuli based on the stimulus information generated by the stimulus information generator 1 are presented is referred to as the target user.

[0051] The stimulus information generation device 1 comprises a pre-processing unit 10, a reading tendency DB (first DB) 20, a voice DB 21, a stimulus generation rule DB (second DB) 22, a stimulus information generation unit 30, and a device control unit 31. The pre-processing unit 10 comprises a user operation setting unit 11 and a stimulus setting unit 12.

[0052] Specifically, the stimulus information generation device 1-1 of Example 1 includes a pre-processing unit 10 equipped with a user operation setting unit 11-1 and a stimulus setting unit 12, as well as a reading tendency DB 20-1, a voice DB 21, a stimulus generation rule DB 22, a stimulus information generation unit 30-1, and a device control unit 31.

[0053] Furthermore, the stimulus information generation device 1-2 of Example 2 includes a pre-processing unit 10 equipped with a user operation setting unit 11-2 and a stimulus setting unit 12, as well as a reading tendency DB 20-2, a voice DB 21, a stimulus generation rule DB 22, a stimulus information generation unit 30-2, and a device control unit 31.

[0054] In Example 1, the user action setting unit 11-1 inputs target content, outputs and displays the page of the target content on the display device 2, and, for each of a predetermined number of users, sequentially inputs the gaze data of the user viewing the page on the display device 2 from the gaze measurement device 3. The user action setting unit 11-1 also inputs sensor data (hand) from the acceleration sensor 4, determines a page turn based on the sensor data (hand), and sets the page. Page turns may be performed by pressing a button on the UI (user interface) of the display device 2. The predetermined number of users may or may not include the target users described later.

[0055] The user action setting unit 11-1 sets target regions as areas where the gaze point lingers and where stimulus presentation is appropriate as a depiction of the target content, based on the gaze trajectories (gaze trajectory data including position information and time information) of a predetermined number of users' gaze data, and sets their positions as target region position data. The user action setting unit 11-1 stores the target region data of the target content in the reading tendency DB 20-1 in Example 1, using the data relating to multiple target regions, including the multiple target region position data, as target region data.

[0056] As a result, the target area is set based on the gaze data of a predetermined number of users, ensuring that the area is neither too small nor too large, and that it is set at a location where the user's gaze tends to linger, thus providing information of appropriate size and position. In other words, the first problem in the first method described above can be solved.

[0057] Furthermore, in Example 1, the user operation setting unit 11-1 receives sample content as input, outputs the sample content page to the display device 2 for display, and simultaneously receives eye-tracking data from the eye-tracking device 3 of the target user who is viewing the page displayed on the display device 2.

[0058] The user action setting unit 11-1 calculates the dwell time of the gaze point based on the trajectory of the gaze point indicated by the target user's eye gaze data, calculates the average value of the dwell time and sets this as the trigger time, and stores the trigger time in the reading tendency DB 20-1 in Example 1.

[0059] As a result, the trigger time is set based on the target user's eye-tracking data, allowing for individualized adjustments (adjustment of the timing of stimulus presentation) for each target user. In other words, this solves the second problem in the first method described above.

[0060] In Example 2, the user action setting unit 11-2 generates target area data based on the gaze point trajectories indicated by the eye-tracking data of a predetermined number of users, using the target content, similar to the user action setting unit 11-1 described above. The user action setting unit 11-2 then stores the target area data of the target content in the reading tendency DB 20-2 in Example 2.

[0061] Furthermore, the user action setting unit 11-2 in Embodiment 2 recognizes the characters in each frame of the target content using an existing method and calculates the number of characters C in each frame of the target content. n The reading tendency DB20-2 in Example 2 is stored. Alternatively, the user operation setting unit 11-2 sets the number of characters C per frame of the target content which is set in advance. n This is stored in the reading trend DB20-2 in Example 2. n represents the frame number for each page.

[0062] Furthermore, in Example 2, the user operation setting unit 11-2 receives sample content as input, outputs the sample content page to the display device 2 for display, and simultaneously receives eye-tracking data from the eye-tracking device 3 of the target user who is viewing the page displayed on the display device 2.

[0063] The user action setting unit 11-2 calculates the reading speed V based on the number of characters read and the time it takes for the eyes to move, using the trajectory of the gaze point indicated by the target user's eye-tracking data. CIn addition to calculating the distance and time of eye movement, the eye movement velocity V is calculated from the distance and time of eye movement. L The user operation setting unit 11-2 then calculates the reading speed V of the target user. C and movement speed V L This is stored in the reading tendency DB20-2 in Example 2.

[0064] Furthermore, the user operation setting unit 11-2 in Embodiment 2 includes multiple target area position data included in the target area data, and the number of characters C for each frame. n , reading speed V C and movement speed V L Based on this, for each page that makes up the target content, the time it takes for the target user's attention point to reach each of the multiple target areas present on that page is calculated as the arrival time TT. n The calculation is performed as follows. Then, the user action setting unit 11-2 determines the arrival time TT of the target user. n This data is stored in the reading tendency DB20-2 in Example 2.

[0065] This will increase the target user's arrival time (TT). n This represents the time that reflects the reading tendencies of the target user, and in the stimulus information generation unit 30-2 in Example 2 described later, the arrival time TT n Stimulus information is generated and output at the appropriate timing. Therefore, stimuli can be presented to the target user at a highly accurate and appropriate timing. In other words, the first problem in the second method described above can be solved.

[0066] The stimulus setting unit 12 reads the target area data of the target content from the reading tendency databases 20-1 and 20-2. Then, in accordance with the operator's instructions specifying various data necessary to operate the stimulus information generation device 1, the stimulus setting unit 12 generates a stimulus generation rule by setting the stimulus type, intensity, stimulus duration, etc., corresponding to each of the multiple target areas indicated by the target area data for each of the pages constituting the target content. The stimulus setting unit 12 then stores the stimulus generation rule in the stimulus generation rule database 22.

[0067] Examples of stimulus types include "vibration," "temperature," "wind," "scent," "lighting," "sound," and "taste." "Vibration," "temperature," and "wind" belong to tactile stimuli, "scent" belongs to olfactory stimuli, "lighting" belongs to visual stimuli, "sound" belongs to auditory stimuli, and "taste" belongs to gustatory stimuli.

[0068] This allows for the setting of stimulation duration, in addition to stimulation type and intensity, for each of the multiple target regions, enabling the stimulation to be stopped at the appropriate time. In other words, it solves the third problem in the first method described above.

[0069] Furthermore, if "vibration" is set as the stimulus type corresponding to the target region, the stimulus setting unit 12 reads audio data corresponding to the onomatopoeia present in the target region from the audio DB 21. The stimulus setting unit 12 then generates a stimulus generation rule for the target region, including the stimulus type "vibration," intensity, audio data, and stimulus duration, and stores it in the stimulus generation rule DB 22.

[0070] This allows for the creation of audio data corresponding to onomatopoeia using a simple method. In other words, it solves the problem described in the third method mentioned above.

[0071] The details of the data structure for Reading Tendency DB20, Audio DB21, and Stimulus Generation Rule DB22 will be described later.

[0072] In Example 1, the stimulus information generation unit 30-1 receives target content as input, outputs and displays the page of the target content on the display device 2, and simultaneously receives eye-tracking data from the eye-tracking device 3 of the target user viewing the page displayed on the display device 2. The stimulus information generation unit 30-1 also reads the target area data and trigger time of the target content from the reading tendency DB 20-1 in Example 1, and reads stimulus generation rules from the stimulus generation rule DB 22. The stimulus information generation unit 30-1 receives sensor data (hand) from the acceleration sensor 4, determines a page turn based on the sensor data (hand), and sets the page. Page turns may also be performed by pressing a button on the UI of the display device 2.

[0073] The stimulus information generation unit 30-1 determines whether the gaze point indicated by the gaze data is within the target region indicated by the target region position data, and whether the time the gaze point has continuously resided in the target region has exceeded the trigger time.

[0074] In this case, the stimulus information generation unit 30-1 determines, in the process of determining whether the point of gaze is within the target area and whether the trigger time has elapsed, whether the point of gaze is moving sequentially through pre-set target areas on the page.

[0075] This allows stimuli to be presented to the target user in the correct order. In other words, it solves the fifth problem in the first method described above.

[0076] If the aforementioned conditions are met, the stimulus information generation unit 30-1 reads the stimulus type and intensity corresponding to the target region from the stimulus generation rules, generates stimulus information based on this data, and outputs the stimulus information to the device control unit 31.

[0077] When the stimulus type is "vibration," the stimulus information generation unit 30-1 generates tactile stimulus information, for example, audio data and the intensity of the "vibration." When the stimulus type is "temperature," the stimulus information generation unit 30-1 generates tactile stimulus information, for example, hot / cold data (data indicating hot or cold) and the intensity of the hot or cold. When the stimulus type is "wind," the stimulus information generation unit 30-1 generates tactile stimulus information, for example, the strength and direction of the wind.

[0078] Furthermore, when the stimulus type is "fragrance," the stimulus information generation unit 30-1 generates olfactory stimulus information including, for example, the type of fragrance (fragrance based on essential oils of trees, herbs, citrus fruits, etc., or fragrance of flavor oils) and the intensity of the fragrance. Furthermore, when the stimulus type is "lighting," the stimulus information generation unit 30-1 generates visual stimulus information including, for example, the color and brightness of the lighting. Furthermore, when the stimulus type is "sound," the stimulus information generation unit 30-1 generates auditory stimulus information including, for example, sound data and the intensity of the sound. Furthermore, when the stimulus type is "taste," the stimulus information generation unit 30-1 generates gustatory stimulus information including, for example, the type of taste (sweetness, sourness, saltiness, bitterness, umami, etc.). The same applies to the stimulus information generation unit 30-2 in Example 2.

[0079] In Example 2, the stimulus information generation unit 30-2 receives the target content as input and outputs the page of the target content to the display device 2 for display. Furthermore, the stimulus information generation unit 30-2 also uses the reading tendency DB 20-2 in Example 2 to determine the arrival time TT. n The system reads the data and also reads the stimulus generation rules from the stimulus generation rule DB22. The stimulus information generation unit 30-2 receives sensor data (hand) from the acceleration sensor 4, determines a page turn based on the sensor data (hand), and sets the page. Page turns may also be performed by pressing a button on the UI of the display device 2.

[0080] The stimulus information generation unit 30-2 counts the elapsed time t since the page was turned for each page, and the elapsed time t corresponds to the arrival time TT. n The system determines whether or not the target has been reached. The stimulus information generation unit 30-2 then determines whether the elapsed time t is the target time TT.n If it is determined that the time TT has been reached, the stimulus generation rule will determine the time TT to be reached. n The stimulation type and intensity corresponding to the target area are read, and stimulation information is generated based on this data, and the stimulation information is output to the device control unit 31. In this case, the stimulation information generation unit 30-2 resets the elapsed time t each time a page is turned.

[0081] This prevents the accumulation of a temporal lag between the progress of static content and the presentation of stimuli with each page turn. In other words, it solves the second problem in the second method described above.

[0082] The device control unit 31 receives stimulus information from the stimulus information generation units 30-1 and 30-2 and controls devices such as the tactile stimulus presentation devices 5A, 5B, and 5C and the olfactory stimulus presentation device 6 so that the stimulus indicated by the stimulus information is presented to the target user. In other words, the device control unit 31 outputs the stimulus information (or its control signal) to one of the stimulus presentation devices such as the tactile stimulus presentation devices 5A, 5B, and 5C and the olfactory stimulus presentation device 6, according to the type of stimulus included in the stimulus information.

[0083] Display device 2 receives content pages from stimulus information generators 1-1 and 1-2 and displays the corresponding pages on the screen.

[0084] The eye-tracking device 3 outputs eye-tracking data indicating the direction the user is fixating on to the stimulus information generation devices 1-1 and 1-2. When the user is looking at a page of content displayed on the display device 2, the eye-tracking device 3 outputs eye-tracking data that includes the position information of the user's point of fixation within the page.

[0085] If the eye-tracking device 3 is installed at a distance from the user, the user's reading posture will be restricted. Therefore, if posture restriction is not desired, a wearable device (for example, a device attached to eyeglasses) should be used as the eye-tracking device 3.

[0086] This allows the user's reading posture to be unrestricted and eliminates the burden on their posture. In other words, it solves the fourth problem in the first method described above.

[0087] The acceleration sensor 4 is attached to the hand of the user viewing the content page displayed on the display device 2. Sensor data (hand) for detecting the user's hand movements is output to the stimulus information generation devices 1-1 and 1-2 when a hand movement corresponding to the user turning a page is detected. This sensor data (hand) is used by the stimulus information generation devices 1-1 and 1-2 to determine whether a page has been turned.

[0088] The tactile stimulus presentation device 5A is a device that presents tactile stimuli of vibration to the user. It receives stimulus information, including, for example, audio data and intensity, from stimulus information generation devices 1-1 and 1-2, and the vibrator vibrates in conjunction with the audio data and at the level indicated by the intensity, according to the stimulus information.

[0089] The tactile stimulus presentation device 5B is a device that presents temperature tactile stimuli to the user, and receives stimulus information, including, for example, hot / cold data and intensity, from the stimulus information generation devices 1-1 and 1-2. The tactile stimulus presentation device 5B then operates so that a current flows through the Peltier element according to the stimulus information, and if the hot / cold data indicates hot, the stimulus presentation surface of the Peltier element heats up to the level indicated by the intensity, and if the hot / cold data indicates cold, the stimulus presentation surface of the Peltier element cools down to the level indicated by the intensity. The Peltier element is an element that can respond to both hot and cold presentations by controlling the polarity of the current.

[0090] The tactile stimulus presentation device 5C is a device that presents the user with tactile stimuli of wind. It receives stimulus information, such as wind strength and direction, from stimulus information generation devices 1-1 and 1-2, and blows air by rotating a motor in the direction specified by the level indicated by the wind strength, according to the stimulus information.

[0091] The olfactory stimulus presentation device 6 is a device that presents olfactory stimuli of scent to the user. It receives stimulus information, such as the type and intensity of the scent, from the stimulus information generation devices 1-1 and 1-2, and generates the scent by rotating a motor at the level indicated by the intensity, according to the stimulus information.

[0092] Here, the tactile stimulus presentation devices 5A and 5B are devices that present touch-type tactile stimuli (vibration and temperature) to the user. The size of these devices is made small so that users can easily handle them and so that they can be held in a natural way when enjoying static media such as comics.

[0093] The grip-type miniature device does not require attachment or removal like wearable devices, and can be used efficiently in a variety of situations. Furthermore, if the user wants to immediately move away from the stimulus being presented, no control of the miniature device is required; they can simply release their hand from the device.

[0094] Furthermore, for the grip-type miniature device, vibration and temperature are presented over an area roughly the size of a palm, and the user touches it when gripping it. This improves versatility compared to glove-type devices and devices that fix the touch area, and also reduces the impact of individual differences.

[0095] For example, with respect to the tactile stimulus presentation devices 5A, 5B, and 5C that present tactile stimuli, it is desirable that vibrations, temperature, etc., can be presented without the user changing their posture, in order to adapt to various scenes of static content. For this reason, in order to allow the user to grasp the device together with the display device 2 that displays the content in a natural way, for example, if the display device 2 is a tablet PC, the tactile stimulus presentation devices 5A, 5B, and 5C should be able to be attached to its back.

[0096] For example, the olfactory stimulus presentation device 6, which presents olfactory stimuli, is installed in a position that does not interfere with the user's enjoyment of the content (for example, above the display device 2 on which the content is displayed). See below for details. [Non-patent literature] “Aroma Shooter (registered trademark)”, [online], Aromajoin Corporation, [Retrieved August 5, 2022], Internet<URL:https: / / aromajoin.com / products / aroma-shooter>

[0097] The acceleration sensor 4-2 is provided when the stimulus information generation device 1 functions as the stimulus information generation device 1-2 of Embodiment 2. The acceleration sensor 4-2 is attached to the head of a target user, for example, who is viewing a page of content displayed on the display device 2, and outputs sensor data (head) for detecting head movement to the stimulus information generation device 1-2 when the target user's head moves. This sensor data (head) is used by the stimulus information generation device 1-2 to determine irregular movements of the target user.

[0098] In this case, the stimulus information generation unit 30-2 in Example 2, upon determining that there is irregular movement by the target user based on sensor data (head), determines that the target user has reread something, temporarily interrupted reading, etc., and interrupts the processing of the stimulus information generation unit 30-2.

[0099] This allows us to determine whether the target user is exhibiting irregular behavior, enabling us to respond to situations such as the user rereading or temporarily interrupting their reading. In other words, it solves the third problem in the second method described above.

[0100] Furthermore, the system shown in Figure 1 may also include, for example, a visual stimulus presenter, an auditory stimulus presenter, and a gustatory stimulus presenter, in addition to the tactile stimulus presenters 5A, 5B, and 5C and the olfactory stimulus presenter 6 that present stimuli to the user.

[0101] The visual stimulus presentation device is a device that presents a visual stimulus of lighting to the user. It receives stimulus information, such as the color and brightness of the lighting, from the stimulus information generation devices 1-1 and 1-2, and the lighting device turns on with the color and brightness corresponding to the stimulus information. The auditory stimulus presentation device is a device that presents an auditory stimulus of sound to the user. It receives stimulus information, such as audio data and the intensity of the audio, from the stimulus information generation devices 1-1 and 1-2, and outputs the audio corresponding to the stimulus information at the level indicated by the intensity through a speaker or headphones. The gustatory stimulus presentation device is a device that presents a gustatory stimulus of taste to the user. It receives stimulus information, such as the type of taste, from the stimulus information generation devices 1-1 and 1-2, and presents the user with the taste corresponding to the stimulus information.

[0102] [Example 1] Next, we will describe in detail the stimulus information generation device 1-1 of Embodiment 1 shown in Figure 1. As mentioned above, the stimulus information generation device 1-1 of Embodiment 1 implements a first method that constantly tracks the target user's gaze and determines the point of fixation, thereby presenting stimuli appropriate to the area the target user is looking at.

[0103] Figure 2 is a block diagram showing an example configuration of the user operation setting unit 11-1 and the reading tendency DB 20-1 in Example 1.

[0104] This user operation setting unit 11-1 includes a target area setting unit 40 and a trigger time setting unit 41. The reading tendency DB 20-1 stores target content eye-tracking data, target area data, sample content eye-tracking data acquired from each of a predetermined number of users, and the trigger time of the target user.

[0105] (User operation setting unit 11-1 / Target area setting unit 40) First, the target area setting unit 40 provided in the user operation setting unit 11-1 will be described. Figure 3 is a flowchart showing an example of processing by the target area setting unit 40. The target area setting unit 40 takes the target content as input and outputs the page of the target content to the display device 2 for display (step S301).

[0106] In this case, the target area setting unit 40 determines whether a page has been turned based on sensor data (hand) input from, for example, the acceleration sensor 4, and then outputs and displays the next page to the display device 2.

[0107] Steps S302 and S303, described later, are performed for each of a predetermined number of users.

[0108] The target area setting unit 40 receives eye-tracking data for each user from the eye-tracking device 3, and stores the trajectory of the gaze point indicated by the eye-tracking data as eye-tracking trajectory data, and stores the target content eye-tracking trajectory data for each user in the reading tendency DB 20-1 (step S302).

[0109] The target area setting unit 40 reads target content eye-tracking data for each user from the reading tendency DB 20-1 (step S303). Then, for each user, the target area setting unit 40 draws the areas where the gaze point lingers based on the target content eye-tracking data of that user, and sets the area that encompasses all the areas drawn for a predetermined number of users (or a predetermined number of users less than or equal to a predetermined number) or an averaged area, and in which stimulus presentation is appropriate as a depiction of the target content, as the target area (step S304). The averaged area is, for example, a rectangular area obtained by the following process. That is, for each of the predetermined number of users, the target area setting unit 40 draws the area where the gaze point lingers as a rectangle and finds the coordinates of its four vertices. Then, for each area where the gaze point lingers, the target area setting unit 40 finds the average position coordinate of the four vertices by averaging the coordinates of the four vertices for all users, and sets the rectangular area indicated by the average position coordinate of the four vertices as the averaged area. As a result, for each page that makes up the target content, all areas that have been drawn for a predetermined number of users and for which stimulus presentation is appropriate as a depiction of the target content are set as multiple target areas.

[0110] An area where the point of focus lingers is an area where the movement of the point of focus is minimal from frame to frame of the target content, and when the point of focus remains within a predetermined area for a predetermined period of time, the area containing that point of focus is considered an area where focus lingers. Furthermore, an area where stimulus presentation is appropriate as a depiction of the target content is a predetermined area that depicts the content of the target content, such as the area containing a character's face, the box for dialogue, the area where onomatopoeia is depicted, or the area of ​​an illustration (flames, impact, etc.). In other words, the target area is an area where the user's point of focus lingers over the target content, and is a predetermined area that depicts the content of the target content.

[0111] The target area setting unit 40 sets the position of the target area (top-left coordinate, width, and height of the target area) as target area position data.

[0112] Figure 5 illustrates an example of a target area. As shown in Figure 5, the target area setting unit 40 sets, for example, target area TR1 in the first frame (n=1), target area TR2 in the second frame (n=2), ..., target area TR5 in the fifth frame (n=5) on a page containing the target content.

[0113] Note that while Figure 5 shows an example where one target area is set for each frame, there are also cases where multiple target areas are set for a single frame.

[0114] Returning to Figures 2 and 3, after step S304, the target area setting unit 40 assigns a target area number to each of the multiple target areas on each page, indicating the order in which stimuli are presented (step S305). Then, the target area setting unit 40 stores the target area data, including the target area number and target area position data for each page, in the reading tendency DB 20-1 (step S306).

[0115] Figure 6 shows an example of target area data stored in the reading tendency DB20-1. This target area data consists of a content ID (identifier of the target content), page number, target area number (a number indicating the order in which stimuli are presented within the page), and target area position data (data indicating the position of the target area within the page). N is the number of the last page.

[0116] As shown in Figure 6, in the target content with content ID A1, page number 1 is assigned target area position data for target area numbers 1 to 5. For example, if target area number 1, the coordinates of the top left of the target area are (x1, y1), the width is w1, and the height is h1.

[0117] Note that the target area data shown in Figure 6 consists of a data set for the target content whose content ID is A1. In contrast, the target area data may consist of a data set for multiple target content items.

[0118] In this way, the target area setting unit 40 stores target area data, including the target area number and target area location data, for each page of the target content in the reading trend DB 20-1.

[0119] (User operation setting unit 11-1 / Trigger time setting unit 41) Next, the trigger time setting unit 41 provided in the user operation setting unit 11-1 will be described. Figure 4 is a flowchart showing an example of processing by the trigger time setting unit 41. The trigger time setting unit 41 takes sample content other than the target content as input and outputs the sample content page to the display device 2 for display (step S401).

[0120] In this case, the trigger time setting unit 41, similar to the target area setting unit 40, for example, receives sensor data (hand) from the acceleration sensor 4 and, when it determines that a page turn has occurred, outputs the next page to the display device 2 and displays it.

[0121] The trigger time setting unit 41 receives the target user's gaze data from the gaze measurement device 3, and stores the trajectory of the gaze point indicated by the gaze data as gaze trajectory data, and stores the target user's sample content gaze trajectory data in the reading tendency DB 20-1 (step S402).

[0122] The trigger time setting unit 41 reads the target user's sample content eye-tracking data from the reading tendency DB 20-1 (step S403). Then, based on the target user's sample content eye-tracking data, the trigger time setting unit 41 sets the target area using the same process as in step S304 shown in Figure 3. As a result, multiple target areas are set for the sample content.

[0123] The trigger time setting unit 41 calculates the time during which the gaze point remains (dwell time) for each of the multiple target areas based on the target user's sample content gaze trajectory data (step S404). The trigger time setting unit 41 then calculates the average value of the dwell times for the multiple target areas, sets this average value as the trigger time (step S405), and stores the trigger time in the reading tendency DB 20-1 (step S406).

[0124] In this way, the trigger time setting unit 41 stores the average dwell time obtained from the target user's eye-tracking data for the sample content as the target user's trigger time in the reading tendency DB 20-1.

[0125] (Stimulus setting section 12) Next, the stimulus setting unit 12 shown in Figure 1 will be described. Figure 7 is a block diagram showing an example configuration of the stimulus setting unit 12 in Example 1, and Figure 8 is a flowchart showing an example of processing by the stimulus setting unit 12 in Example 1. The example configuration and processing of the stimulus setting unit 12 shown in Figures 7 and 8 are also applicable to Example 2, which will be described later. The same applies to the stimulus generation rule DB22.

[0126] This stimulus setting unit 12 includes a display processing unit 42, a stimulus generation rule setting unit 43, and an audio data retrieval unit 44. The stimulus generation rule DB 22 stores the stimulus generation rules (data).

[0127] The display processing unit 42 inputs the target content (step S801), and the display processing unit 42 and the stimulus generation rule setting unit 43 read target area data from the reading tendency DB 20-1 (, 20-2) (step S802). Also, if the stimulus generation rule setting unit 43 has already set stimulus generation rules stored in the stimulus generation rule DB 22, it reads the stimulus generation rules from the stimulus generation rule DB 22 (step S803). Then, the display processing unit 42 inputs the stimulus generation rules from the stimulus generation rule setting unit 43.

[0128] The display processing unit 42 outputs and displays the page of the target content, as well as various data such as the target area number and target area position data of the page included in the target area data, to the display device 2 (step S804).

[0129] Furthermore, when the display processing unit 42 receives a stimulus generation rule that has already been set from the stimulus generation rule setting unit 43, it outputs and displays the page of the target content, as well as various data such as the target area number, target area position data, stimulus type, and type data included in the stimulus generation rule, to the display device 2 (step S804).

[0130] Figure 9 shows an example of the display screen of the display device 2 when setting stimulus generation rules. As shown in Figure 9, the display device 2 displays the file name of the target content image (page) "yakyu1.png", the manga panels on that page (consisting of 5 panels), the target regions TR1, TR2, ..., TR5 in each panel, and various data for each of the target regions TR1, TR2, ..., TR5. These various data correspond to the data of the stimulus generation rules shown in Figure 12, which will be described later.

[0131] In each frame, the target regions TR1, TR2, ..., TR5 are identified and displayed by the target region number and target region position data. The various data for each of the target regions TR1, TR2, ..., TR5 are: target region position data (a0, ..., d0, ... in Figure 9), stimulus type (a1, ..., d1, ...), type data (a2, ..., d2, ...), intensity (a3, ..., d3, ...), stimulus duration (a4, ..., d4, ...), and stimulus effect (start / end) (a5, ..., d5, ...).

[0132] Returning to Figures 7 and 8, the stimulus generation rule setting unit 43 sets (edits) rules for each target region by inputting various data (a0, a1, ..., a5, ..., d0, d1, ..., d5, ... in Figure 9) according to the operator's operation (such as drag-and-drop operations on the UI of the display screen shown in Figure 9) (step S805). Then, the stimulus generation rule setting unit 43 outputs the set stimulus generation rules to the display processing unit 42.

[0133] Specifically, the stimulus generation rule setting unit 43, in accordance with the operator's instructions, inputs various data for each target region, including target region position data (a0,···,d0,·), stimulus type (a1,···,d1,·), type data (a2,···,d2,·), intensity (a3,···,d3,·), stimulus duration (a4,···,d4,·), and stimulus effect (start / end) (a5,···,d5,·), and sets the rules for that page. Details of the various data will be explained later in Figure 12.

[0134] Furthermore, the stimulus generation rule setting unit 43 may be connected to a device such as the tactile stimulus presentation device 5A via the device control unit 31 shown in Figure 1, in order to allow editing while previewing the stimulus presentation, and the tactile stimulus presentation device 5A, etc., may be operated according to the set rules.

[0135] Furthermore, if the stimulus generation rule setting unit 43 sets "vibration" as the stimulus type in step S805 according to the operator's operation, the voice data search unit 44 uses the voice DB 21 to search for voice data as type data.

[0136] Figure 10 is a flowchart showing an example of processing by the audio data search unit 44. When the stimulus type is set to "vibration" by the stimulus generation rule setting unit 43, the audio data search unit 44 inputs the onomatopoeia of a pre-set search word within the frame corresponding to the target region (vibration target region) (step S1001).

[0137] Furthermore, the onomatopoeia detection unit, which is not shown in the stimulus setting unit 12 of Figure 7, may detect onomatopoeia within the frame corresponding to the target region by performing a predetermined character recognition process. In this case, the audio data retrieval unit 44 receives onomatopoeia input from the onomatopoeia detection unit.

[0138] After step S1001, the voice data retrieval unit 44 reads from the voice DB 21 the voice data and tactile data corresponding to the onomatopoeia entered in step S1001 (step S1002).

[0139] Figure 11 shows an example of the data structure of the voice database 21. This voice database 21 is composed of multiple data sets, each consisting of onomatopoeia, voice data, and tactile sensation (data). Figure 11 shows the file names of the voice data. The tactile sensation indicates how appropriate the voice data corresponding to the onomatopoeia is from the user's perspective.

[0140] In the audio DB21 shown in Figure 11, when the onomatopoeia is "byu," the corresponding audio data such as "byu1.mp3" and "byu2.mp3" are stored, and the tactile sensation of "byu1.mp3" is stored as "1," and the tactile sensation of "byu2.mp3" is stored as "2," etc.

[0141] Additionally, in the audio DB21, when the onomatopoeia is "Kan," the corresponding audio data such as "kan1.mp3" is set, and the tactile sensation of "kan1.mp3" is set to "1," etc.

[0142] When the target content is manga and the search is performed using onomatopoeia depicted in the manga, it would be ideal to obtain audio data that provides appropriate tactile stimuli. However, when audio data expressing onomatopoeia is input into a vibrator and used as tactile stimuli, it is not always possible to present the intended tactile stimuli.

[0143] Therefore, in order to present tactile stimuli in accordance with the user's intentions, the voice DB21 is configured to include a set of three data items: onomatopoeia, voice data, and tactile sensations.

[0144] This allows for the setting of appropriate audio data based on the degree of appropriateness of the tactile sensation data. In other words, it is possible to realize an authoring system that can set appropriate audio data to be used as input for tactile stimulus presentation using an audio DB21 that can be searched by onomatopoeia.

[0145] Furthermore, the voice DB 21 may be configured to include multiple data sets, each containing an onomatopoeia and voice data. In this case, the voice data search unit 44 reads the voice data corresponding to the onomatopoeia entered in step S1001 from the voice DB 21.

[0146] Returning to Figures 7 and 10, after step S1002, the audio data retrieval unit 44 displays the file name of the audio data on the display device 2 via the stimulus generation rule setting unit 43 and the display processing unit 42 according to the tactile sensation (step S1003).

[0147] Referring to the area marked aa in Figure 9, for example, if the search word is "byu," the search results will display "byu1.mp3," "byu2.mp3," etc., from top to bottom in the order of the numerical values ​​indicated by the tactile sensation. By operating the play button to the left of the file name in the search results, the audio of that file name will be played on a playback device not shown in Figure 1.

[0148] Furthermore, in step S1001, the voice data search unit 44 may generate voice data by synthesizing the search word when an onomatopoeic search word is entered, and may also assign a file name to the voice data. In this case, in step S1003, the voice data search unit 44 will also display the file name of the generated voice data on the display device 2.

[0149] This allows for the generation of unique audio data corresponding to onomatopoeia not present in the audio DB21, depending on the content, which can then be used to present vibration stimuli.

[0150] Returning to Figures 7 and 10, the audio data search unit 44 selects one of the audio data files (for example, "byu1.mp3", "byu2.mp3", etc.) displayed on the display device 2 according to the operator's instructions (step S1004).

[0151] In this case, the voice data retrieval unit 44 may select the voice data with the smallest numerical value indicating tactile sensation (the highest degree of appropriateness) from among the multiple voice data read from the voice DB 21 without requiring any intervention from the operator.

[0152] The audio data selected in this manner is then set by the stimulus generation rule setting unit 43 as the type data for when the stimulus type is "vibration".

[0153] In the example described above, the stimulus generation rule setting unit 43, in step S805 of Figure 8, sets the audio data selected by the audio data retrieval unit 44 as the type data when the stimulus type is "vibration," and sets its intensity according to the operator's input.

[0154] In contrast, the onomatopoeia detection unit, which is not shown in the stimulus setting unit 12 of Figure 7, may detect the size of the onomatopoeia characters for the frame corresponding to the target region by predetermined character recognition processing or the like.

[0155] In this case, the stimulus generation rule setting unit 43 receives the size of the onomatopoeia from the onomatopoeia detection unit and automatically sets the strength so that the strength level increases as the size increases and decreases as the size decreases.

[0156] Furthermore, in step S805 of Figure 8, if the stimulus type is "vibration", the stimulus generation rule setting unit 43 may also cause the onomatopoeia detection unit (not shown in the stimulus setting unit 12 of Figure 7) to automatically recognize the onomatopoeia and its size, and cause the voice data retrieval unit 44 to automatically select voice data using the voice DB 21.

[0157] This allows the system to automatically set the audio data and intensity of the stimulation type (vibration) without requiring any intervention from the operator, when the stimulation type is "vibration".

[0158] Returning to Figures 7 and 8, the stimulus generation rule setting unit 43 determines after step S805 whether the settings for all pages have been completed (step S806). If the stimulus generation rule setting unit 43 determines in step S806 that the settings for all pages have not been completed (step S806:N), it proceeds to step S804, where the display processing unit 42 performs the processing in step S804 for the next page, and the stimulus generation rule setting unit 43 performs the processing in step S805.

[0159] On the other hand, if the stimulus generation rule setting unit 43 determines in step S806 that the settings for all pages have been completed (step S806:Y), it stores the stimulus generation rules in the stimulus generation rule DB 22 (step S807).

[0160] Figure 12 shows an example of a stimulus generation rule stored in the stimulus generation rule DB22. A stimulus generation rule consists of a content ID (identifier of the target content), page number, target area number, target area location data, stimulus type, type data, intensity (level), stimulus duration, and stimulus effect (start / end). N is the number of the last page.

[0161] In the target content with content ID A1, page number 1 has target area position data set for target area numbers 1 to 5. For example, if target area number 1, the top-left coordinates of the target area are (x1, y1), the width is w1 and the height is h1, the stimulus type is "vibration", the type data is "byu2.mp3", the intensity is "4", the stimulus time is "until playback ends", and the stimulus effect is "none / none".

[0162] The types of stimuli include "vibration," "temperature," "wind," and "scent." The type data is, for example, audio data if the stimulus type is "vibration," hot or cold data indicating "hot" or "cold" if the stimulus type is "temperature," and data indicating the type of scent if the stimulus type is "scent." The intensity (level) is the magnitude if the stimulus type is "vibration" or "wind," the intensity of "hot" or "cold" in the "hot or cold data" if the stimulus type is "temperature," and the intensity of the scent if the stimulus type is "scent."

[0163] The stimulus duration is the period during which the stimulus is presented. For example, it can be set to "until playback ends" or "custom 2000ms" (a specific length of time). If "until playback ends" is set as the stimulus duration, for example, when the stimulus type is "vibration," it indicates that the stimulus will continue from the time the audio data starts playing until it finishes (the length of the audio data). If "custom 2000ms" is set as the stimulus duration, for example, when the stimulus type is "vibration," it indicates that the stimulus will continue until 2000ms have elapsed from the time the audio data starts playing.

[0164] In this way, by setting a specific duration for the stimulus, such as "until playback ends" or "custom 2000ms," it is possible to present the user with either an instantaneous or continuous stimulus.

[0165] The stimulus effect (start / stop) is set to whether or not to apply a fade-in effect when the stimulus presentation begins (fade or none), and whether or not to apply a fade-out effect when the stimulus presentation ends (fade or none).

[0166] This allows you to set the stimulus effect (start / stop) to "none / none" in the case of a sudden stimulus, for example. In this case, in step S1406 of Figure 14, which will be described later, stimulus information is generated and output at the moment the trigger input occurs to present the stimulus to the user. However, the effect of stopping the stimulus is often not necessary, and this can be accommodated.

[0167] Furthermore, in the case of continuous stimulation, for example, in step S1406 of Figure 14 (described later), after a trigger input occurs, such as when the gaze point remains within the target area, the stimulation continues, and stops when the gaze point remains outside the target area for a certain period of time or longer. In this case, since a sudden stop of stimulation would be unnatural, the stopping of the stimulation effect can be set to "fade," and by presenting a fade-out effect, the stimulation can be stopped without giving the user an unnatural impression.

[0168] Furthermore, the stimulus generation rule shown in Figure 12 consists of a data set for target users for the target content with content ID A1. In contrast, the stimulus generation rule may consist of a data set for target users for multiple target content, or it may consist of a data set for each of multiple target users for one or more target content.

[0169] In this way, the stimulus setting unit 12 stores a stimulus generation rule in the stimulus generation rule DB 22 for each page of the target content, which includes the target area number, target area location data, stimulus type, type data, etc.

[0170] (Stimulus information generation unit 30-1) Next, the stimulus information generation unit 30-1 in Example 1 shown in Figure 1 will be described in detail. Figure 13 is a block diagram showing an example configuration of the stimulus information generation unit 30-1 in Example 1, and Figure 14 is a flowchart showing an example of processing by the stimulus information generation unit 30-1 in Example 1.

[0171] This stimulus information generation unit 30-1 includes a display processing unit 50, a page turning determination unit 51, a gaze point target area determination unit 52, and a stimulus information processing unit 53.

[0172] The display processing unit 50 receives the target content as input (step S1401), and the gaze target area determination unit 52 reads the trigger time from the reading tendency DB 20-1 (step S1402). In addition, the gaze target area determination unit 52 and the stimulus information processing unit 53 read the stimulus generation rule from the stimulus generation rule DB 22 (step S1403).

[0173] The display processing unit 50 receives input from the page turn detection unit 51 indicating that a page turn has occurred and manages the current page by updating the page of the target content. The display processing unit 50 outputs the page of the target content to the display device 2 for display (step S1404). The display processing unit 50 outputs the page number of the page output to the display device 2 to the gaze target area detection unit 52.

[0174] The page-turning determination unit 51 and the gaze target area determination unit 52 receive gaze data from the gaze measurement device 3, and the page-turning determination unit 51 receives sensor data (hand) from the acceleration sensor 4 (step S1405).

[0175] The gaze target area determination unit 52 receives a page number from the display processing unit 50 and extracts all target area position data corresponding to that page number from the stimulus generation rule.

[0176] The gaze point target area determination unit 52 determines whether the gaze point indicated by the line of sight data is located within the target area indicated by the target area position data, and whether the time the gaze point has continuously been located within the target area has exceeded the trigger time (step S1406).

[0177] If the gaze point target area determination unit 52 determines in step S1406 that the conditions of the gaze point being within the target area and the trigger time having elapsed are met (step S1406: Y), it outputs data indicating that the conditions of step S1406 are met, as well as the page number and the target area number corresponding to the target area, to the stimulus information processing unit 53.

[0178] This allows us to determine when the trigger time has elapsed, confirming that the target user was indeed focusing on the target area, and excluding cases where the user was not focusing on the target area, such as when the point of focus simply crosses the target area.

[0179] When the stimulus information processing unit 53 receives data indicating that the conditions are met, as well as the page number and target area number, from the gaze point target area determination unit 52, it generates stimulus information corresponding to the input page number and target area number according to the stimulus generation rules (step S1407). The stimulus information processing unit 53 then outputs the stimulus information to the device control unit 31 (step S1408) and proceeds to step S1409.

[0180] As a result, the device control unit 31 controls devices such as the tactile stimulus presentation devices 5A, 5B, 5C and the olfactory stimulus presentation device 6 so that the stimuli indicated by the stimulus information are presented to the user.

[0181] Specifically, the stimulus information processing unit 53 extracts the stimulus type, type data, intensity, stimulus time, and stimulus effect corresponding to the input page number and target area number from the stimulus generation rule, and generates stimulus information consisting of the stimulus type, type data, intensity, stimulus time, and stimulus effect. For example, in step S1406, if conditions such as the gaze point being within the target area of ​​page number 1 and target area 1 are met, the stimulus information processing unit 53 extracts "vibration" as the stimulus type, "byu2.mp3" which is audio data as the type data, "4" as the intensity, "until playback ends" as the stimulus time, and "none / none" as the stimulus effect from the stimulus generation rule (see Figure 12), and generates stimulus information consisting of this data.

[0182] As a result, the device control unit 31 outputs stimulus information (control signal) to the tactile stimulus presentation device 5A, which reflects the following data: stimulus type: "vibration", type data: audio data "byu2.mp3", intensity: "4", stimulus time: "until playback ends", and stimulus effect: "none / none". The tactile stimulus presentation device 5A then vibrates its vibrator at an intensity of "4" linked to the audio data "byu2.mp3" for the duration indicated by the stimulus time, "until playback ends". In this case, fade-in and fade-out processing are not performed.

[0183] Here, for example, when using the Peltier element of the tactile stimulus presentation device 5B to present temperature, there is a problem of delay in temperature presentation, and a problem of time it takes for the target user to perceive the temperature change. The same applies when using the olfactory stimulus presentation device 6 to present scent.

[0184] To solve this problem, the stimulus information processing unit 53 may generate and output stimulus information for the stimulus type "temperature" or "fragrance" at the timing when it generates and outputs stimulus information corresponding to the target region of the previous target region number, or at a predetermined time before or after the previous timing.

[0185] In other words, in steps S1407 and S1408, the stimulus information processing unit 53 extracts the stimulus type corresponding to the target region of the next target region number from the stimulus generation rule at the timing of the target region number, and if the stimulus type is "temperature" or "fragrance", it generates and outputs the stimulus information.

[0186] As a result, when the stimulus type is "temperature" or "fragrance," the stimulus information is generated and output at the timing of the previous target region number, allowing the target user to receive the "temperature" or "fragrance" stimulus at the appropriate time or close to it. In other words, it solves problems such as presentation delays when the stimulus type is "temperature" or "fragrance."

[0187] On the other hand, if the gaze point target area determination unit 52 determines in step S1406 that the conditions of step S1406 are not met, that is, if it determines that the gaze point is not within the target area or that the trigger time has not elapsed (step S1406:N), it proceeds to step S1409.

[0188] Furthermore, in the process of determining the conditions in step S1406, the gaze point target area determination unit 52 determines whether the gaze point is moving within the page in the order indicated by the target area number while satisfying the conditions.

[0189] In other words, the gaze point target area determination unit 52 holds the target area number to be used to determine the conditions in step S1406. In step S1406, if it determines that the target area where the gaze point is staying is the target area of ​​the target area number, it determines the conditions in step S1406. If it determines that the target area is not the target area of ​​the target area number, it does not determine the conditions in step S1406 and waits until the gaze point stays in the target area of ​​the target area number.

[0190] This ensures that stimuli corresponding to the next target region are not presented until the stimuli in the order indicated by the target region numbers have been presented, thus allowing stimuli to be presented in the order of the target region numbers.

[0191] For example, even if the gaze point remains on target region number 3 after a stimulus corresponding to target region number 1 is presented, the stimulus corresponding to target region number 3 will not be presented. After a stimulus corresponding to target region number 1 is presented, a stimulus corresponding to target region number 2 will be presented.

[0192] The page-turning determination unit 51 moves from step S1408 or step S1406(N) to determine whether or not a page has been turned (step S1409).

[0193] Specifically, the page-turning determination unit 51 determines, for example, that a page-turning event occurred based on sensor data (hand) input from the acceleration sensor 4. If it determines that the hand has moved in a manner that suggests it is turning a page, it determines that a page-turning event occurred. If it determines that the hand has not moved in a manner that suggests it is turning a page, it determines that a page-turning event did not occur.

[0194] Furthermore, the page-turning determination unit 51 may determine that a page turn has occurred if it determines, based on the gaze data input from the gaze measurement device 3, that the point of gaze has moved from a predetermined area in the lower left of the page to a predetermined area in the upper right, and determine that a page turn has not occurred if it determines that the point of gaze has not moved from a predetermined area in the lower left of the page to a predetermined area in the upper right.

[0195] Furthermore, the page-turning determination unit 51 may determine that a page has been turned if it determines, based on sensor data (hand), that the hand has moved as if turning a page, and based on gaze data, that the point of gaze has moved from a predetermined area in the lower left of the page to a predetermined area in the upper right. The page-turning determination unit 51 determines that a page has not been turned if it determines that the hand has not moved as if turning a page, or if it determines that the point of gaze has not moved from a predetermined area in the lower left of the page to a predetermined area in the upper right.

[0196] Alternatively, the page-turning determination unit 51 may be configured to determine whether a page has been turned when a touch operation input is received on the screen of the display device 2.

[0197] If the page-turning detection unit 51 determines in step S1409 that no page has been turned (step S1409:N), it proceeds to step S1405 and inputs new gaze data and sensor data (hand).

[0198] Meanwhile, if the page-turning determination unit 51 determines in step S1409 that a page turn has occurred (step S1409: Y), it outputs "page turn occurred" to the display processing unit 50. The display processing unit 50 then receives the "page turn occurred" information from the page-turning determination unit 51 and determines whether or not processing of all pages (page number N) has been completed (step S1410).

[0199] If the display processing unit 50 determines in step S1410 that processing of all pages has not been completed (step S1410:N), it sets the next page (step S1411), proceeds to step S1404, and outputs and displays the page of the target content set in step S1411 to the display device 2. The display processing unit 50 outputs the new page number to the gaze target area determination unit 52.

[0200] On the other hand, if the display processing unit 50 determines in step S1410 that processing of all pages has been completed (step S1410:Y), it terminates the processing by the stimulus information generation unit 30-1.

[0201] In this way, the stimulus information generation unit 30-1 generates stimulus information corresponding to the target area in order of the target area number, according to the stimulus generation rules, for each page of the target content, when the target user's gaze remains in the target area and the trigger time has elapsed, and the stimulus is presented to the target user.

[0202] As described above, according to the stimulus information generation device 1-1 of Embodiment 1, the user operation setting unit 11-1 of the preprocessing unit 10 displays the page of the target content on the display device 2, inputs gaze data sequentially for each of a predetermined number of users, and sets the area where the gaze point lingers as a target area based on the gaze trajectory data indicated by the gaze data of the predetermined number of users, thereby generating target area data including multiple target area numbers and target area position data for each page, and stores the target area data in the reading tendency DB 20-1.

[0203] Furthermore, the user action setting unit 11-1 displays a sample content page on the display device 2, inputs the target user's gaze data, and sets the gaze trajectory data indicated by the gaze data as the target area where the gaze point dwells. The user action setting unit 11-1 then calculates the dwell time of the gaze point based on the gaze trajectory data, sets the average of the dwell times of multiple target areas as the trigger time, and stores the trigger time in the reading tendency DB 20-1.

[0204] The stimulus setting unit 12 reads target region data from the reading tendency DB 20-1 and, in accordance with the operator's instructions, sets (edits) the stimulus type, intensity, etc. for each of the multiple target regions indicated by the target region data to generate a stimulus generation rule, which is then stored in the stimulus generation rule DB 22.

[0205] The stimulus information generation unit 30-1 displays the page of the target content on the display device 2, inputs the gaze data of the target user viewing the page, reads target area data and trigger time from the reading tendency DB 20-1, and reads stimulus generation rules from the stimulus generation rule DB 22. When the stimulus information generation unit 30-1 determines that the gaze point indicated by the gaze data is within the target area of ​​the target area position data included in the target area data, and that the trigger time has elapsed, it extracts the stimulus type and intensity corresponding to the target area from the stimulus generation rules and generates stimulus information. The control signal for the stimulus information is output from the device control unit 31 to one of the tactile stimulus presentation devices 5A, 5B, 5C and the olfactory stimulus presentation device 6, etc.

[0206] Thus, in the case of static media content, when the target user's gaze point is within the target area and the trigger time has elapsed, stimulus information corresponding to that target area is generated. The gaze point is obtained from eye-tracking data that reflects the target user's natural movements, and the trigger time reflects the target user's unique reading tendencies regarding the dwell time of the gaze point.

[0207] In other words, the system can appropriately recognize the target user's unique reading habits and natural behaviors within the target content, and present appropriate stimuli at the appropriate time, in accordance with the user's unique pace of content progression.

[0208] Therefore, when users receive information from static content, a rich media experience can be provided by synchronizing with the progression of the content and presenting appropriate stimuli at the appropriate time.

[0209] [Example 2] Next, the stimulus information generation device 1-2 of Example 2 shown in Figure 1 will be described in detail. As mentioned above, the stimulus information generation device 1-2 of Example 2 realizes a second method that automatically presents stimuli in accordance with the user's reading tendencies (reading speed, eye movement speed, etc.).

[0210] Figure 15 is a block diagram showing an example configuration of the user operation setting unit 11-2 and the reading tendency DB 20-2 in Example 2.

[0211] This user action setting unit 11-2 includes a target area setting unit 60, a character count setting unit 61, a reading speed calculation unit 62, and a completion time calculation unit 63. The reading tendency DB 20-2 contains target content eye-tracking data, target area data, and character count C for each frame of the target content, acquired from each of a predetermined number of users. n , sample content eye-tracking data obtained from the target user, and the target user's reading speed (reading speed V C and movement speed V L ) and the time to reach each target area for the target user TT n This is stored.

[0212] (User operation setting unit 11-2 / Target area setting unit 60) The processing of the target area setting unit 60 provided in the user operation setting unit 11-2 in Embodiment 2 is the same as that of the target area setting unit 40 provided in the user operation setting unit 11-1 in Embodiment 1 shown in Figures 2 and 3, so a description is omitted here.

[0213] The target area setting unit 60 stores target area data, including the target area number and target area location data, for each page of the target content in the reading trend DB 20-2.

[0214] (User operation setting section 11-2 / Character count setting section 61) Next, the character count setting unit 61 provided in the user operation setting unit 11-2 will be described. The character count setting unit 61 receives the page of the target content from the target area setting unit 60, recognizes the characters in each frame, and calculates the number of characters C in each frame of the target content. n We seek.

[0215] In this case, the number of characters per frame is C. n This may be treated as pre-set data. The character count setting unit 61 sets the number of characters C for each frame of the target content. n This information is stored in the reading tendency database DB20-2.

[0216] Figure 16 shows the number of characters C per frame. n This is an example shown in the figure. The number of characters in each frame is C. n The data includes content ID (identifier of the target content), page number, frame number n (nth frame) (frame number on each page), and character count C. n It consists of (the number of characters present in the frame with frame number n), where N is the number of the last page.

[0217] In the target content with content ID A1, page number 1 is assigned a frame number n from 1 to 5. For frame numbers n=1 and 2, the number of characters C1 and C2 is 3; for frame numbers n=3 and 5, the number of characters C3 and C5 is 0; and for frame number n=4, the number of characters C4 is 4.

[0218] Furthermore, the number of characters C per frame shown in Figure 16 n This consists of data sets for the target content whose content ID is A1. In contrast, the number of characters per frame C n The data may consist of a set of data for multiple target contents.

[0219] In this way, the character count setting unit 61 sets the character count C for each frame for each of the pages that make up the target content. n This information is stored in the reading tendency DB20-2.

[0220] (User operation setting unit 11-2 / Reading speed calculation unit 62) Next, we will describe the reading speed calculation unit 62 provided in the user operation setting unit 11-2. Figure 17 is a flowchart showing an example of processing by the reading speed calculation unit 62.

[0221] The reading speed calculation unit 62 receives sample content other than the target content as input and outputs and displays the pages of the sample content to the display device 2 (step S1701).

[0222] In this case, the reading speed calculation unit 62, similar to the target area setting unit 60 and the target area setting unit 40 shown in Figure 2, for example, receives sensor data (hand) from the acceleration sensor 4 to determine if a page is being turned, and outputs the next page to the display device 2 for display.

[0223] The reading speed calculation unit 62 receives the target user's eye gaze data from the eye gaze measurement device 3, and stores the target user's sample content eye gaze trajectory data as eye gaze trajectory data using the trajectory of the point of fixation indicated by the eye gaze data in the reading tendency DB 20-2 (step S1702).

[0224] The reading speed calculation unit 62 reads sample content eye-tracking data for the target user from the reading tendency DB 20-2 (step S1703).

[0225] The reading speed calculation unit 62 calculates the movement time of the line of sight on the page based on the sample content gaze trajectory data of the page, for example, when there is a page turn on the page. Then, the reading speed calculation unit 62 divides the number of characters present on the page by the movement time to obtain the reading speed V, which is the speed at which the target user reads characters C (number of characters per second), and obtains the reading speed V of the target user C and stores it in the reading tendency DB 20-2 (step S1704).

[0226] Note that the number of characters present on the page may be obtained by adding the number of characters C for each frame stored in the reading tendency DB 20-2 n , may be obtained by recognizing characters on the page of the target content using an existing method, or may be set in advance. In addition, the reading speed calculation unit 62 may calculate the reading speed for all pages and obtain the average thereof as the reading speed V C .

[0227] The reading speed calculation unit 62 obtains a first retention area where the gaze point stays within a frame based on, for example, the sample content gaze trajectory data, and after the gaze point moves from the retention area, obtains a second retention area where the gaze point stays within the next frame. Then, based on the sample content gaze trajectory data, the reading speed calculation unit 62 obtains the movement time (inter-frame movement time) when the gaze point moves from the first retention area to the second retention area, and also obtains the distance (inter-frame distance) between (the center position of) the first retention area and (the center position of) the second retention area.

[0228] The reading speed calculation unit 62 divides the inter-frame distance by the inter-frame movement time to obtain the movement speed V of the gaze point between frames by the target user L (distance (mm) per second), and obtains the movement speed V of the target user L and stores it in the reading tendency DB 20-2 (step S1705).

[0229] Furthermore, the reading speed calculation unit 62 generates target area data of the sample content based on the sample content gaze trajectory data, and based on the target area position data included in the target area data and the sample content gaze trajectory data, obtains the inter-frame movement time when the gaze point moves from the first target area to the second target area, and obtains the inter-frame distance between (the center position of) the first target area and (the center position of) the second target area, thereby obtaining the movement speed V L may be calculated.

[0230] In addition, the reading speed calculation unit 62 may detect the center of a frame within a page, or detect the center of a drawn character or speech balloon, and treat the detected position as the center position of the aforementioned retention area.

[0231] In this way, the reading speed V, which is the reading speed acquired from the target user's gaze data for the sample content by the reading speed calculation unit 62 C and the movement speed V L are stored in the reading tendency DB 20-2.

[0232] (User operation setting unit 11-2 / Arrival time calculation unit 63) Next, the arrival time calculation unit 63 provided in the user operation setting unit 11-2 will be described. FIG. 18 is a flowchart showing a processing example of the arrival time calculation unit 63.

[0233] The arrival time calculation unit 63 acquires, from the reading tendency DB 20-2, target area data of the target content, the number of characters C for each frame of the target content n , the reading speed V of the target user C and the movement speed V L and reads them out (step S1801).

[0234] For the n-th frame, the arrival time calculation unit 63 calculates the number of characters C n to the reading speed V CBy dividing by this number, the time it takes to read the nth panel (the time from when you start reading the nth panel until you finish reading it) is calculated.

[0235] Furthermore, the arrival time calculation unit 63 calculates the distance between the nth frame and the (n+1)th frame (the distance between the target area in the nth frame and the target area in the (n+1)th frame) based on the target area position data included in the target area data, and calculates this distance using the movement speed V L By dividing by n, the movement time of the point of focus from frame n to frame n+1 is calculated. Note that if there is one target region for each frame, the target region number included in the target region data shown in Figure 6 corresponds to frame number n.

[0236] The arrival time calculation unit 63 calculates the time taken for the nth frame (the time from when the nth frame is read until the point of focus reaches the target area of ​​the next n+1th frame) T by adding the time taken to move to the nth frame to the time taken to read the nth frame using the following formula. n We find (step S1802).

number

[0237] (P xn ,P yn ) is the center position of the target region, obtained from the coordinates (x,y) of the top-left corner of the target region included in the target region position data of the nth frame (if there is one target region per frame, the target region number is n). (P xn+1 ,P yn+1 ) is the center position of the target region, obtained from the coordinates (x,y) of the top-left corner of the target region included in the target region position data of the (n+1)th frame (if there is one target region in each frame, the target region number is n+1).

[0238] Figure 19 illustrates an example of the processing of the arrival time calculation unit 63. The coordinates of the top left of the target region TRn in the nth frame included in the target region position data are (x n ,yn ) from the center position of the target region TRn (P xn ,P yn ) is calculated. Also, the top-left coordinate (x) of the target region TRn+1 in the n+1 frame included in the target region position data is calculated. n+1 ,y n+1 ) from the center position of the target region TRn+1 (P xn+1 ,P yn+1 ) is calculated. Then, using the above formula (1), the time taken for the nth frame (the time from when you start reading the nth frame until the point of focus reaches the target area of ​​the next (n+1)th frame) T is calculated. n This is calculated. Here, time T n This may include the trigger time.

[0239] Returning to Figures 15 and 18, the arrival time calculation unit 63 calculates the time T obtained in step S1802 using the following formula after step S1802. n By using this method, the time from when the page is turned until the target user's attention reaches the target area of ​​the nth frame is calculated as the arrival time TT. n Calculate as follows (Step S1803).

number

[0240] The arrival time calculation unit 63 calculates the arrival time TT for the target user. n Store this in the reading tendency DB20-2 (step S1804).

[0241] Figure 20 shows the arrival time TT. n This is an example shown in the figure. Arrival time TT n This information, along with the content ID (identifier of the target content), page number, frame number n (n-th frame), and target area number, is stored in the reading tendency DB20-2. N is the number of the last page. The example in Figure 20 shows the case where there is one target area in each frame.

[0242] As shown in Figure 20, in the target content with content ID A1, page number 1 is assigned frame numbers n from 1 to 5 and corresponding target area numbers from 1 to 5. For example, the arrival time TT1 corresponding to target area number 1 is set to 0.5 seconds, ..., and the arrival time TT5 corresponding to target area number 5 is set to 2.6 seconds.

[0243] Furthermore, the arrival time TT shown in Figure 20 is also shown. n This consists of data about target users for the target content whose content ID is A1. In contrast, the arrival time TT n This may consist of a data set for target users across multiple target content areas, or it may consist of a data set for each of the multiple target users across one or more target content areas.

[0244] In this way, the arrival time calculation unit 63 calculates the arrival time TT for each of the multiple target areas within a page, which is the time from when the page is turned until the page is reached. n This information is stored in the reading tendency database DB20-2.

[0245] Formulas (1) and (2) above represent the case where there is one target area in one panel. In contrast, if there are multiple target areas in one panel, or if the manga format is different, formulas (1) and (2) above should be modified and applied.

[0246] (Stimulus information generation unit 30-2) Next, the stimulus information generation unit 30-2 in Example 2 shown in Figure 1 will be described in detail. Figure 21 is a block diagram showing an example configuration of the stimulus information generation unit 30-2 in Example 2, Figure 22 is a flowchart showing an example of processing by the stimulus information generation unit 30-2 in Example 2, and Figure 23 is a flowchart showing a continuation of the processing example in Figure 22.

[0247] The stimulus information generating unit 30-2 includes a display processing unit 70, a page turning determination unit 71, a gaze point target area determination unit 72, a stimulus information processing unit 73, and an irregular motion determination unit 74.

[0248] The display processing unit 70 inputs target content (step S2201), and the gaze point target area determination unit 72 acquires the arrival time TT from the reading tendency DB 20-2 n and reads it (step S2202). Also, the gaze point target area determination unit 72 and the stimulus information processing unit 73 read a stimulus generation rule from the stimulus generation rule DB 22 (step S2203).

[0249] The display processing unit 70 manages the current page by inputting a notification of page turning from the page turning determination unit 71 and updating the page of the target content. The display processing unit 70 outputs the page of the target content to the display device 2 for display (step S2204). The display processing unit 70 outputs the page number of the page output to the display device 2 to the gaze point target area determination unit 72. When the gaze point target area determination unit 72 receives a new page number (the first page number at the start) from the display processing unit 70, it starts counting elapsed time t (step S2205).

[0250] The page turning determination unit 71 and the gaze point target area determination unit 72 input gaze data from the gaze measurement device 3, the page turning determination unit 71 inputs sensor data (hand) from the acceleration sensor 4, and the irregular motion determination unit 74 inputs sensor data (head) from the acceleration sensor 4-2 (step S2206).

[0251] The gaze point target area determination unit 72 obtains the arrival time TT read from the reading tendency DB 20-2 n and extracts the arrival time TT corresponding to the page number input from the display processing unit 50 n .

[0252] The gaze point target area determination unit 72 compares the elapsed time t with the arrival time TT n and compares whether the elapsed time t has reached the arrival time TT nDetermine whether or not the condition has been reached (step S2207).

[0253] Here, elapsed time t is the time since a new page was turned, and the arrival time TT n This is the set time from when a new page is turned until the target area (in this case, the frame) is reached. Therefore, the value of the elapsed time t is equal to the arrival time TT. n The smaller the value of , the further the target user's current focus is from the next target area they will reach, and the greater the value of elapsed time t is compared to the arrival time TT. n The closer the value is to (or the same as) the target user's current focus, the closer it is to the next target area they will reach. In other words, the time to reach TT n The elapsed time t allows us to predict the current focus of the target user on the next target area to be reached.

[0254] Furthermore, in step S2207, the gaze target area determination unit 72 makes determinations in the order of frame number n (in this case, target area number) on the page (n = 1, 2, ...). As mentioned above, this is to present stimuli in the order of the target area numbers.

[0255] In step S2207, the target area determination unit 72 determines that the elapsed time t is equal to the arrival time TT. n If it is determined that the condition has been reached (step S2207:Y), data indicating that the condition of step S2207 has been met, along with the page number and frame number n (here, assuming that there is one target region per frame, n is the target region number), is output to the stimulus information processing unit 73.

[0256] The stimulus information processing unit 73, upon receiving data indicating that a condition is satisfied, as well as the page number and target area number (n) input from the fixation point target area determination unit 72, generates stimulus information corresponding to the input page number and target area number (n) in accordance with a stimulus generation rule (step S2208). Then, the stimulus information processing unit 73 outputs the stimulus information to the device control unit 31 (step S2209), and proceeds to step S2214.

[0257] Thereby, the device control unit 31 controls devices such as the tactile stimulus presentation devices 5A, 5B, 5C and the olfactory stimulus presentation device 6 such that the stimulus indicated by the stimulus information is presented to the user.

[0258] Specifically, similarly to the stimulus information processing unit 53 shown in Embodiment 1 illustrated in FIG. 13, the stimulus information processing unit 73 extracts, from the stimulus generation rule, the stimulus type, type data, intensity, stimulus duration and stimulus effect corresponding to the input page number and target area number, and generates stimulus information including the stimulus type, type data, intensity, stimulus duration and stimulus effect.

[0259] Here, as in Embodiment 1, for example, when temperature is presented using the Peltier element of the tactile stimulus presentation device 5B, there are problems that temperature presentation is delayed, and that it takes time before the target user perceives a temperature change. This also applies to the case where a scent is presented using the olfactory stimulus presentation device 6.

[0260] In order to solve this problem, the delay caused by the time lag of temperature presentation and the like is considered in advance, and control for temperature presentation and the like is started before the timing at which the target user actually receives the stimulus such as temperature presentation.

[0261] Specifically, when the stimulus type is "temperature" or "scent", the fixation point target area determination unit 72 sets the arrival time TT n subtracts a preset delay time (a delay time set for each stimulus type) from the arrival time TT, and sets the subtraction result as a new arrival time TT nThen, in step S2207, the gaze target area determination unit 72 determines the elapsed time t and the new arrival time TT. n By comparing the elapsed time t with the new arrival time TT, the elapsed time t is obtained. n If it is determined that the condition has been reached, the same process as described above is performed.

[0262] Then, the stimulus information processing unit 73 processes the original arrival time TT. n Faster new arrival time TT n At that timing, if the stimulus type is "temperature" or "scent," stimulus information is generated and output.

[0263] This means that when the stimulus type is "temperature" or "scent," the stimulus information is recorded at the original arrival time TT. n Because it is generated and output at an earlier timing, the target user can receive the "temperature" or "scent" stimulus at the appropriate time. In other words, it can solve problems such as delays in presentation when the stimulus type is "temperature" or "scent".

[0264] On the other hand, in step S2207, the gaze target area determination unit 72 determines that the elapsed time t is equal to the arrival time TT. n If it is determined that the condition has not been reached (step S2207:N), that is, if it is determined that the conditions of step S2207 have not been met, the process proceeds to step S2210 in Figure 23 (α).

[0265] In step S2206, the irregular movement detection unit 74 determines whether there is an irregular movement related to the sudden head movement of the target user based on the sensor data (head) input from the acceleration sensor 4-2 (step S2210). This makes it possible to determine actions such as the target user rereading or temporarily interrupting reading.

[0266] Furthermore, in step S2210, the irregular movement determination unit 74 may input gaze data from the gaze measurement device 3 and determine, based on the gaze data, whether or not there is an irregular movement related to a sudden shift in the target user's gaze point. Alternatively, the irregular movement determination unit 74 may input video data from a camera (a camera that films the target user) not shown in Figure 1 and determine, based on the video data, whether or not there is an irregular movement by the target user. In this case as well, actions such as rereading or temporarily interrupting reading will be determined.

[0267] If the irregular movement determination unit 74 determines in step S2210 that an irregular movement has occurred (step S2210:Y), for example, if it determines that the reader has reread or temporarily interrupted reading, it outputs data indicating the presence of an irregular movement to the display processing unit 70, the page turning determination unit 71, the gaze target area determination unit 72, and the stimulus information processing unit 73.

[0268] When the display processing unit 70, page turning determination unit 71, gaze target area determination unit 72, and stimulus information processing unit 73 receive data from the irregular movement determination unit 74 indicating the presence of an irregular movement, they interrupt their respective processes, and the gaze target area determination unit 72 stops counting the elapsed time t (step S2211).

[0269] After step S2211, the irregular operation determination unit 74 determines whether or not there is a restart input according to the operation of the target user (step S2212). If the irregular operation determination unit 74 determines in step S2212 that there is no restart input (step S2212:N), it waits until there is a restart input.

[0270] On the other hand, if the irregular operation determination unit 74 determines in step S2212 that there is a restart input (step S2212:Y), it outputs data indicating the restart of processing to the display processing unit 70, the page turning determination unit 71, the gaze point target area determination unit 72, and the stimulus information processing unit 73.

[0271] When the display processing unit 70, page turning determination unit 71, gaze target area determination unit 72, and stimulus information processing unit 73 receive data from the irregular operation determination unit 74 indicating the resumption of processing, they each resume their respective processes. The gaze target area determination unit 72 then resumes counting the elapsed time t (step S2213), and the system proceeds to step S2214 in Figure 22 (β).

[0272] On the other hand, if the irregular operation determination unit 74 determines in step S2210 that there is no irregular operation (step S2210:N), it proceeds to step S2214 in Figure 22 (β).

[0273] The page-turning determination unit 71 moves from step S2209, step S2213, or step S2210(N) to determine whether or not a page has been turned (step S2214).

[0274] Specifically, the page-turning detection unit 71, similar to the page-turning detection unit 51 shown in Figure 13, determines whether or not a page has been turned based on sensor data (hand) input from, for example, the acceleration sensor 4.

[0275] If the page-turning detection unit 71 determines in step S2214 that no page has been turned (step S2214:N), it proceeds to step S2206 and inputs new gaze data and sensor data (hand, head).

[0276] Meanwhile, if the page-turning determination unit 71 determines in step S2214 that a page turn has occurred (step S2214: Y), it outputs "page turn occurred" to the display processing unit 70. The display processing unit 70 then determines whether or not processing of all pages (page number N) has been completed (step S2215).

[0277] If the display processing unit 70 determines in step S2215 that processing of all pages has not been completed (step S2215:N), it sets the next page, and the gaze target area determination unit 72 resets the elapsed time t (step S2216), and proceeds to step S2204. The display processing unit 70 then outputs and displays the page of the target content set in step S2216 to the display device 2. The display processing unit 70 outputs the new page number to the gaze target area determination unit 72.

[0278] This resets the elapsed time t each time a page is turned, allowing the starting point for automatic stimulus presentation to be reset to 0 and preventing the accumulation of discrepancies across pages. In other words, it eliminates the accumulation of discrepancies between the target user's actual gaze point and the gaze point that can be predicted according to the elapsed time t.

[0279] On the other hand, if the display processing unit 70 determines in step S2215 that processing of all pages has been completed (step S2215:Y), it terminates the processing by the stimulus information generation unit 30-2.

[0280] In this way, the stimulus information generation unit 30-2 generates the elapsed time t and the arrival time TT for each page of the target content. n Upon reaching the target region, stimulus information corresponding to the target area is generated according to the stimulus generation rules, and the stimulus is presented to the target user.

[0281] As described above, according to the stimulus information generation device 1-2 of Embodiment 2, the user operation setting unit 11-2 of the preprocessing unit 10, similar to Embodiment 1, displays the page of the target content on the display device 2 and generates target area data including multiple target area numbers and target area position data for each page based on the gaze trajectory data indicated by the gaze data of a predetermined number of users, and stores the target area data in the reading tendency DB 20-2.

[0282] Furthermore, the user operation setting unit 11-2 determines the number of characters per frame C through character recognition processing for each frame in the target content, or through manual setting processing. n This information is stored in the reading tendency database DB20-2.

[0283] Furthermore, the user action setting unit 11-2 displays the sample content page on the display device 2, acquires the target user's eye-tracking data, and retrieves the eye-tracking data and, for example, the number of characters per frame C. n Based on this, reading speed V C The reading tendency DB20-2 is then calculated and stored. Furthermore, the user action setting unit 11-2 determines the dwell area of ​​the gaze point within a frame based on the gaze trajectory data, and calculates the movement speed V from the dwell area in the first frame to the dwell area in the next second frame based on the movement time and distance. L This information is then retrieved and stored in the reading tendency database DB20-2.

[0284] Furthermore, the user operation setting unit 11-2 includes multiple target area position data and the number of characters C per frame in the target area data. n , reading speed V C and movement speed V L Based on this, for each page that makes up the target content, the time to reach the target area where the target user's attention is drawn is calculated. n This is calculated and stored in the reading tendency DB20-2.

[0285] The stimulus setting unit 12 generates stimulus generation rules and stores the stimulus generation rules in the stimulus generation rule DB 22, similar to the example in Example 1.

[0286] The stimulus information generation unit 30-2 displays the target content page on the display device 2, inputs the eye-tracking data of the target user viewing the page, and also obtains the arrival time TT from the reading tendency DB 20-2. n The system reads the data and retrieves the stimulus generation rule from the stimulus generation rule DB22. Then, the stimulus information generation unit 30-2 counts the elapsed time t since the page was turned, and the elapsed time t equals the arrival time TT. nWhen it is determined that the time to reach the target has been reached, the stimulus generation rule determines the time to reach that target TT. n The device reads out the type and intensity of the stimulus corresponding to the target area and generates stimulus information. The control signal for the stimulus information is output from the device control unit 31 to one of the tactile stimulus presentation devices 5A, 5B, 5C and the olfactory stimulus presentation device 6, etc.

[0287] In this case, the stimulus information generation unit 30-2 determines irregular movements related to sudden head movements of the target user based on sensor data (head) from the acceleration sensor 4-2, and if irregular movements are detected, it interrupts the processing of the stimulus information generation unit 30-2.

[0288] Thus, in the case of static media content, the elapsed time t from when the target user turns the page is defined as the arrival time TT. n When the point of focus reaches the target region, it is considered that the point of focus has reached the target region, and stimulus information corresponding to that target region is generated. Time to reach TT n This is the pre-calculated time from when a target user turns to a new page using sample content until they reach the target area, and it reflects the target user's unique reading habits.

[0289] In other words, the system can appropriately recognize the target user's unique reading habits and natural behaviors within the target content, and present appropriate stimuli at the appropriate time, in accordance with the user's unique pace of content progression.

[0290] Therefore, when users receive information from static content, a rich media experience can be provided by synchronizing with the progression of the content and presenting appropriate stimuli at the appropriate time.

[0291] Although the present invention has been described above with reference to Examples 1 and 2, the present invention is not limited to Examples 1 and 2, and can be modified in various ways without departing from the technical concept.

[0292] For example, in the above-mentioned embodiments 1 and 2, as shown in Figure 1, the stimulus information generation device 1 (1-1, 1-2) includes a pre-processing unit 10 equipped with a user operation setting unit 11 (11-1, 11-2) and a stimulus setting unit 12, a reading tendency DB 20 (20-1, 20-2), a voice DB 21, a stimulus generation rule DB 22, a stimulus information generation unit 30 (30-1, 30-2), and a device control unit 31.

[0293] In contrast, the stimulus information generation device 1 (1-1, 1-2) may be configured to include all components except the stimulus setting unit 12 and the voice DB 21, and a stimulus setting device not shown in Figure 1 may be configured to include the stimulus setting unit 12 and the voice DB 21. Alternatively, the stimulus setting device may consist only of the stimulus setting unit 12, and a voice DB server not shown in Figure 1 may be configured to include the voice DB 21.

[0294] Furthermore, a standard computer can be used as the hardware configuration for the stimulus information generation devices 1-1 and 1-2 according to Embodiments 1 and 2 of the present invention. The stimulus information generation devices 1-1 and 1-2 are composed of a computer equipped with a CPU, a volatile storage medium such as RAM, a non-volatile storage medium such as ROM, and an interface.

[0295] The functions of the user operation setting unit 11-1 and stimulus setting unit 12, reading tendency DB 20-1, voice DB 21, stimulus generation rule DB 22, stimulus information generation unit 30-1, and device control unit 31 of the pre-processing unit 10 provided in the stimulus information generation device 1-1 of Example 1 are each realized by having the CPU execute a program that describes these functions.

[0296] Furthermore, the functions of the user operation setting unit 11-2 and stimulus setting unit 12, reading tendency DB 20-2, voice DB 21, stimulus generation rule DB 22, stimulus information generation unit 30-2, and device control unit 31 of the pre-processing unit 10 of the stimulus information generation device 1-2 of Example 2 are each realized by having the CPU execute a program that describes these functions.

[0297] These programs are stored in the aforementioned storage medium and are read and executed by the CPU. These programs can also be stored and distributed on storage media such as magnetic disks (floppy disks, hard disks, etc.), optical disks (CD-ROMs, DVDs, etc.), and semiconductor memory, and can be transmitted and received via a network. [Explanation of Symbols]

[0298] 1 Stimulus information generation device 2 Display device 3. Eye-tracking device 4. Accelerometer 5. Tactile Stimulus Presentation Device 6. Olfactory Stimulus Presentation Device 10 Pre-processing 11 User Operation Setting Section 12 Stimulus setting section 20 Reading Trends Database 21 Voice Database 22 Stimulus Generation Rule Database 30 Stimulus information generation section 31 Device Control Unit 40,60 Target area setting section 41 Trigger time setting section 42, 50, 70 Display Processing Unit 43. Stimulus generation rule setting unit 44. Voice Data Retrieval Unit 51,71 Page turn detection unit 52,72 Target area determination unit 53,73 Stimulus Information Processing Unit 61 Character Count Setting Section 62 Reading Speed ​​Calculation Section 63 Arrival time calculation part 74 Irregular motion determination section N Last page number (x,y) Top-left coordinates of the target region w width of the target area h Height of the target area n Frame number TR Target Area Cn Number of characters in the nth frame V C reading speed V L movement speed T n Time (the time it takes to reach the target area of ​​the (n+1)th frame after starting to read the nth frame) TT n Time to reach (the time from when you start reading the first frame until you reach the target area of ​​the nth frame)

Claims

1. In a stimulus information generating device that generates stimulus information to present a stimulus to a user and outputs the stimulus information to a stimulus presentation device when the content progresses according to the user's actions in receiving information from the static content, The target area is defined as the region where the user's attention to the content lingers and which depicts the content's content; the user to whom the stimulus is presented is defined as the target user; and the content that progresses according to the target user's actions is defined as the target content. From the eye-tracking device that measures the gaze of the user, eye-tracking data is input from a predetermined number of users when they are viewing the target content. Based on the trajectories of the points of fixation indicated by the eye-tracking data of the predetermined number of users and the predetermined area, multiple target areas are determined, and the positions of the multiple target areas are set as multiple target area position data. The eye-tracking device inputs eye-tracking data from when the target user is viewing sample content other than the target content, and based on the trajectory of the gaze point indicated by the eye-tracking data, the dwell time of the gaze point is set as the trigger time. A user operation setting unit that stores the multiple target area position data and the trigger time in the first DB, A stimulus setting unit sets the type and intensity of stimulus corresponding to each of the multiple target regions and stores the stimulus type and intensity in a second database. The eye-tracking device inputs eye-gaze data when the target user is viewing the target content, reads the position data of the multiple target areas and the trigger time from the first DB, and reads the stimulus type and intensity for each of the multiple target areas from the second DB. A stimulus information generation unit determines whether the gaze point indicated by the gaze data is within the target area and whether the time the gaze point has been within the target area has elapsed the trigger time, and when it is determined that the conditions are met, generates stimulus information including the stimulus type and intensity corresponding to the target area, and outputs the stimulus information to the stimulus presentation device. A stimulus information generating device characterized by having the following features.

2. In the stimulus information generating device according to claim 1, The user operation setting unit is, For each of the predetermined number of users, the area where the gaze point lingers is determined based on the trajectory of the gaze point indicated by the user's gaze data. A stimulus information generating device characterized by determining a region that includes all areas where the gaze points linger for a predetermined number of users, and determining the predetermined region as the plurality of target regions, or averaging all areas where the gaze points linger, determined for each of the predetermined number of users, and determining the averaged area, and the predetermined region as the plurality of target regions, and storing the plurality of target region position data in the first DB.

3. In the stimulus information generating device according to claim 1, The user operation setting unit is, A stimulus information generating device characterized by calculating multiple dwell times in multiple regions where the gaze point remains, based on the gaze point indicated by the gaze data of the target user, calculating an average value from the multiple dwell times, setting the average value as the trigger time, and storing the trigger time in the first database.

4. In a stimulus information generating device that generates stimulus information to present a stimulus to a user and outputs the stimulus information to a stimulus presentation device when the content progresses according to the user's actions in receiving information from the static content, The target area is defined as the region where the user's attention to the content lingers and which depicts the content's content; the user to whom the stimulus is presented is defined as the target user; and the content that progresses according to the target user's actions is defined as the target content. From the eye-tracking device that measures the gaze of the user, eye-tracking data is input from a predetermined number of users as they view the target content. Based on the trajectories of the gaze points indicated by the eye-tracking data of the predetermined number of users and the predetermined area, multiple target areas are determined. The positions of the multiple target areas are set as multiple target area position data. The number of characters per frame in the target content is set. The eye-tracking device inputs eye-tracking data when the target user is viewing sample content other than the target content, calculates the reading speed of the target user when reading text based on the trajectory of the gaze point indicated by the eye-tracking data and the number of characters, and calculates the movement speed of the target user's gaze point based on the trajectory of the gaze point indicated by the eye-tracking data. A user operation setting unit calculates the arrival time for each of the multiple target area locations, the number of characters per frame, the reading speed, and the movement speed, for each of the multiple pages constituting the target content, and stores the arrival time in the first database. A stimulus setting unit sets the type and intensity of stimulus corresponding to each of the multiple target regions and stores the stimulus type and intensity in a second database. From the first DB, for each page of the target content, the arrival time for each of the multiple target areas included in that page is read, and from the second DB, the stimulus type and intensity for each of the multiple target areas are read. A stimulus information generation unit counts the elapsed time since each page was turned, determines the condition that the elapsed time has reached the arrival time, and when it is determined that the condition is met, generates stimulus information including the type and intensity of the stimulus corresponding to the target area of ​​the arrival time, and outputs the stimulus information to the stimulus presentation device. A stimulus information generating device characterized by having the following features.

5. In the stimulus information generating device according to claim 4, The user operation setting unit is, A stimulus information generating device characterized in that, when the gaze point indicated by the gaze data of the target user moves from a first target area to a second target area, the device determines the movement speed by dividing the distance between the first target area and the second target area by the time it takes to move from the first target area to the second target area.

6. In the stimulus information generating device according to claim 4, The user operation setting unit is, The reading speed is V C , the aforementioned moving speed is V L The number of characters in the nth frame is C n , the center position of the target region in the nth frame is (P xn ,P yn ), T is the time from when the target user starts reading the nth frame until the point of focus moves to the target area of ​​the next (n+1)th frame. n , the arrival time TT n as, The reading speed V C , the moving speed V L , the number of characters C in the n-th frame n , the center position (P xn , P yn ) of the target area in the n-th frame and the center position (P xn+1 , P yn+1 ) of the target area in the (n+1)-th frame, using the formula: Therefore, the aforementioned time T n Calculate the above time T n Using this, the formula: Therefore, the arrival time TT n A stimulus information generating device characterized by calculating [a certain value].

7. In the stimulus information generating device according to claim 1 or 4, Furthermore, it includes a voice database that stores multiple data sets, each consisting of onomatopoeia and voice data. The aforementioned stimulus setting unit is When vibration is set as the type of stimulus corresponding to the target region, the target region is set as a vibration target region. From the aforementioned audio database, audio data corresponding to onomatopoeia present in the frame including the vibration target region is selected, and the vibration and intensity of the stimulus type, along with the audio data, are stored in the second database. The aforementioned stimulus information generation unit, A stimulus information generating device characterized by reading vibration and intensity, which are the stimulus type, and audio data from the second DB, and when it is determined that the conditions are met for the vibration target region, generating stimulus information including vibration and intensity, which are the stimulus type, and audio data corresponding to the vibration target region, and outputting the stimulus information to the stimulus presentation device, in which the vibrator operates according to the intensity and audio data.

8. A computer comprising a stimulus information generation device that generates stimulus information to present stimuli to a user and outputs the stimulus information to a stimulus presentation device when the content progresses according to the user's actions in receiving information from static content, The target area is defined as the region where the user's attention to the content lingers and which depicts the content's content; the user to whom the stimulus is presented is defined as the target user; and the content that progresses according to the target user's actions is defined as the target content. From the eye-tracking device that measures the gaze of the user, eye-tracking data is input from a predetermined number of users when they are viewing the target content. Based on the trajectories of the points of fixation indicated by the eye-tracking data of the predetermined number of users and the predetermined area, multiple target areas are determined, and the positions of the multiple target areas are set as multiple target area position data. The eye-tracking device inputs eye-tracking data from when the target user is viewing sample content other than the target content, and based on the trajectory of the gaze point indicated by the eye-tracking data, the dwell time of the gaze point is set as the trigger time. A user operation setting unit that stores the multiple target area position data and the trigger time in the first database. A stimulus setting unit sets the type and intensity of stimulus corresponding to each of the multiple target regions and stores the stimulus type and intensity in a second database, and The eye-tracking device inputs eye-gaze data when the target user is viewing the target content, reads the position data of the multiple target areas and the trigger time from the first DB, and reads the stimulus type and intensity for each of the multiple target areas from the second DB. A program for functioning as a stimulus information generation unit that determines whether the gaze point indicated by the gaze data is within the target area and whether the time the gaze point has been within the target area has elapsed the trigger time, and when it is determined that the conditions are met, generates stimulus information including the stimulus type and intensity corresponding to the target area, and outputs the stimulus information to the stimulus presentation device.

9. A computer comprising a stimulus information generation device that generates stimulus information to present stimuli to a user and outputs the stimulus information to a stimulus presentation device when the content progresses according to the user's actions in receiving information from static content, The target area is defined as the region where the user's attention to the content lingers and which depicts the content's content; the user to whom the stimulus is presented is defined as the target user; and the content that progresses according to the target user's actions is defined as the target content. From the eye-tracking device that measures the gaze of the user, eye-tracking data is input from a predetermined number of users as they view the target content. Based on the trajectories of the gaze points indicated by the eye-tracking data of the predetermined number of users and the predetermined area, multiple target areas are determined. The positions of the multiple target areas are set as multiple target area position data. The number of characters per frame in the target content is set. The eye-tracking device inputs eye-tracking data when the target user is viewing sample content other than the target content, calculates the reading speed of the target user when reading text based on the trajectory of the gaze point indicated by the eye-tracking data and the number of characters, and calculates the movement speed of the target user's gaze point based on the trajectory of the gaze point indicated by the eye-tracking data. A user operation setting unit calculates the arrival time for each of the multiple target area locations, the number of characters per frame, the reading speed, and the movement speed, for each of the multiple pages constituting the target content, and stores the arrival time in the first database. A stimulus setting unit sets the type and intensity of stimulus corresponding to each of the multiple target regions and stores the stimulus type and intensity in a second database, and From the first DB, for each page of the target content, the arrival time for each of the multiple target areas included in that page is read, and from the second DB, the stimulus type and intensity for each of the multiple target areas are read. A program for functioning as a stimulus information generation unit that, for each page, counts the elapsed time since the page was turned, determines the condition for the elapsed time to reach the arrival time, and when it is determined that the condition is met, generates stimulus information including the type and intensity of the stimulus corresponding to the target area of ​​the arrival time, and outputs the stimulus information to the stimulus presentation device.

Images