Video display method, video display system, and program

Abstract

To provide video display which gives a user a sense of immersion and presence.SOLUTION: And the control device 11 that determines whether or not the state of the user viewing the displayed video has changed from the first state to the second state different from the first state based on the output from the sensor 20 that detects the user, and when the control device 11 determines that the state has changed to the second state, the display devices 40-1 and 40-2 change the display mode of the video from the first display mode to the second display mode different from the first display mode.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a video display method, a video display system, and a program. [Background technology]

[0002] In a system for displaying video, a technique is known in which the video changes in response to the user's movements on the display screen in order to give the user a sense of immersion and presence (see, for example, Patent Document 1). Specifically, this technique involves moving the camera position in a system that displays two-dimensional video from a camera position on the display screen, and making objects appear in the video in response to the user's position on the display screen. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-219820 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the above technology, the trigger for changing the image is the user's actions while viewing the image. In other words, unless the user moves, the step of giving the user a sense of immersion or realism, that is, the step of making an object appear in the image, does not proceed. In other words, unless the user consciously moves, the object does not appear, so in this sense, the sense of immersion is lacking and the sense of realism is reduced. [Means for solving the problem]

[0005] A video display method according to one embodiment of the present disclosure includes a sound generating device generating a sound; a control device determining whether a user's state has changed from a first state to a second state different from the first state based on an output from a sensor that detects the user; and, when the control device determines that the state has changed to the second state, a display device that displays a video changes the state of the video from the first display state to the second display state different from the first display state.

[0006] A video display system according to one embodiment of the present disclosure includes a sound generating device that generates sound, a display device that displays video, and a control device that determines whether the state of a user viewing video displayed on the display device has changed from a first state to a second state different from the first state based on output from a sensor that detects the user, and when the control device determines that the state has changed to the second state, changes the display mode of the video by the display device from the first display mode to the second display mode different from the first display mode.

[0007] A program according to one embodiment of the present disclosure causes a computer that controls a sound generating device that generates sound and a display device that displays video to instruct the sound generating device to generate the sound, determine whether the user's state has changed from a first state to a second state different from the first state based on output from a sensor that detects a user watching the video, and, if it is determined that the user has changed to the second state, change the display mode of the video by the display device from the first display mode to the second display mode different from the first display mode. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing a configuration of an image display system to which an image display method according to an embodiment is applied; [Figure 2] FIG. 2 is a functional block diagram constructed by a control device in the video display system. [Figure 3] FIG. 2 is a diagram showing the structure of audio-visual data used in the video display system. [Figure 4] FIG. 1 is a diagram illustrating an example of an arrangement of elements in a video display system. [Figure 5] 4 is a flowchart showing the operation of the control device. [Figure 6] 1A and 1B are diagrams mainly for explaining examples of video display in a video display system. [Figure 7] FIG. 10 is a diagram illustrating an example of an operation of the video display system. [Figure 8] FIG. 10 is a diagram illustrating an example of an operation of the video display system. [Figure 9] FIG. 10 is a diagram illustrating an example of an operation of the video display system. [Figure 10] FIG. 10 is a diagram illustrating an example of an operation of the video display system. DETAILED DESCRIPTION OF THE INVENTION

[0009] Display methods according to embodiments will be described below with reference to the drawings. In each drawing, the dimensions and scale of each part are appropriately different from the actual dimensions. Furthermore, the embodiments described below are preferred examples, and therefore various technically preferable limitations are applied. However, the scope of the present disclosure is not limited to these embodiments unless otherwise specified in the following description to the effect that the present disclosure is limited.

[0010] 1 is a block diagram showing the configuration of a video display system 1 that realizes a video display method according to an embodiment. The video display system 1 generates a sound (hereinafter referred to as a "specific sound") different from the background music at a certain timing while a user is watching a video together with background music, and changes the appearance of the video that has been displayed up to that point if the user notices the specific sound. The aspect of the image changes when the viewer notices something, giving the viewer the feeling that the change in the image was caused by their own actions, rather than being the result of the video creator's deliberate action.

[0011] As shown in the figure, the video display system 1 includes a computer 10, a sensor 20, a sound generating device 30, speakers 32-1 to 32-n, and display devices 40-1 and 40-2. The video display system 1 creates a video space using these elements. The computer 10 controls each unit in the video display system 1. The computer 10 includes a control device 11, a storage device 12, an input IF (interface) 13, and an output IF 14. These devices are electrically connected to each other via, for example, a bus Bs.

[0012] The control device 11 is composed of one or more processing circuits such as a CPU (Central Processing Unit), and comprehensively controls each element of the computer 10. Note that the control device 11 may be composed of circuits such as a DSP (Digital Signal Processor) and an ASIC (Application Specific Integrated Circuit) in addition to a CPU.

[0013] The storage device 12 is a single or multiple memories configured with known recording media such as magnetic recording media, semiconductor recording media, etc. The storage device 12 stores a control program 122 for the control device 11 to control each unit, and video and audio data 124 configured with video data and audio data. The storage device 12 may be configured by combining multiple types of recording media. The storage device 12 may be a portable recording medium that is detachable from the computer 10, or an external recording medium with which the computer 10 can communicate via a network, such as an online storage. The input IF13 is a device for inputting the photographic data Dt from the sensor 20. The output IF14 is a device for supplying sound data to the sound generating device 30, supplying a video signal Vd-1 to the display device 40-1, and supplying a video signal Vd-2 to the display device 40-2.

[0014] The sensor 20 is, for example, a photographing device that photographs the viewer viewing the video and sound produced by the video display system 1 and supplies photographing data Dt indicating the photographing results to the computer 10. For the sake of simplicity, in this embodiment, the sensor 20 is one photographing device, but it may be two or more sensors, or may be other sensors such as an infrared sensor, a sonic sensor, a range sensor, or a contact sensor, or a combination of these sensors may be used, or a sensor module combining these sensors may be used. The experiencer refers to a person who experiences / views the video and audio produced by the video display system 1. Hereinafter, for convenience, the experiencer will be referred to as a user.

[0015] The sound generating device 30 includes a sound source device 31 and speakers 32-1 to 32-n. The sound source device 31 applies a predetermined effect (processes) to the sound data supplied via the output IF 14, converts it into n-channel analog sound signals Ad-1 to Ad-n, amplifies them, and supplies them to the speakers 32-1 to 32-n in order. The effects that the sound source device 31 imparts to the sound data include a stereophonic effect (surround processing) that gives three-dimensional sound direction, distance, and spread to background sounds based on BGM data among the sound data, and an effect (panning processing) that fixes or moves the sound image position of a specific sound based on specific sound data.

[0016] The panning process requires user position information (strictly speaking, information indicating the direction and position of the user's head), which can be acquired by analyzing the image data Dt captured by the sensor 20. Hereinafter, the panning process will use the user position information acquired by analyzing the image data Dt.

[0017] The speakers 32-1 to 32-n are sound emitting devices that convert the acoustic signals Ad-1 to Ad-n in order into sounds, which are physical vibrations, and output the sounds. In Fig. 1, n speakers 32-1 to 32-n are shown, each provided for one channel. If the sound image position is not changed and a stereophonic effect is not provided, the number of speakers may be one.

[0018] The display device 40-1 is specifically a projector, and projects an image based on a video signal Vd-1 supplied from the computer 10. Similarly, the display device 40-2 projects an image based on a video signal Vd-2. For convenience, two display devices 40-1 and 40-2 are shown in the figure, but the number of display devices may be one, or three or more. Furthermore, the display devices 40-1 and 40-2 are not limited to projectors, and may be direct-view panel display devices.

[0019] FIG. 2 is a block diagram showing a functional configuration that is constructed when the control device 11 of the computer 10 executes the control program 122 stored in the storage device 12. As shown in FIG. As shown in the figure, the control device 11 functions as a processing control unit 112, a video processing unit 114, an audio processing unit 116, and a determination unit 118.

[0020] The processing control unit 112 starts reading out the audio-visual data 124 from the storage device 12 through a predetermined operation or the like, and also controls the video processing unit 114, the audio processing unit 116, and the determination unit 118.

[0021] The video processing unit 114 generates video signals Vd-1 and Vd-2 based on the video data in the audio and video data 124 read by the processing control unit 112 under the control of the processing control unit 112. The video signal Vd-1 generated by the video processing unit 114 is supplied to the display device 20-1 via the output IF14, and the video signal Vd-2 is supplied to the display device 20-2 via the output IF14.

[0022] The audio processing unit 116 detects whether the audio data in the audio-visual data 124 read by the processing control unit 112 indicates sound or silence, notifies the processing control unit 112 of the detection result, and outputs the audio data that indicates sound.

[0023] The determination unit 118 analyzes the photographed data Dt captured by the sensor 20 in accordance with instructions from the process control unit 112, and determines whether or not there has been a change in the user's behavior. Specifically, the user's behavior refers to the orientation of the face and body, facial expression, gaze direction, standing position, and the like. The speed of these changes over time may be taken into consideration when determining a change in behavior. Specifically, if the speed of change in behavior is less than a threshold, it may be determined that there has been no change in behavior.

[0024] 3 is a diagram showing an example of the audio-video data 124. In this example, the audio-video data and the video data that make up the audio-video data 124 are both track data in which the audio-video data is sampled at regular intervals and arranged in time series. Therefore, if the track data is read out at each sampling interval at point Rp, the audio-video data and the video data will be played back in sync.

[0025] In this embodiment, the display devices 20-1 and 20-2 display an image in which an object is superimposed on a background image. The image data includes background image data representing the background image and object data A to D representing each of the objects. The background image data is image data showing background images such as the sea, sky, and space. In this embodiment, underwater images including seaweed, rocks, bubbles, etc. are used as the background image. The object data A to D are data representing objects such as fish, birds, and animals that are superimposed on the background image, and the objects include real objects and virtual objects (CG), but do not include the background. In this embodiment, different schools of fish are used as the object data A to D.

[0026] The sound data includes background music data and specific sound data. BGM data is data that indicates music or environmental sounds that are played to create an atmosphere when displaying video. The specific sound data is data indicating a sound intended to give a certain kind of awareness to a user watching a video together with background music. The awareness sound is not limited to natural sounds such as rain, thunder, etc., but may also be electronic sounds such as noise.

[0027] When an operator or an external device issues an instruction to start display, the process control unit 112 starts reading out the track data, that is, the video and audio data 124. In detail, the process control unit 112 moves the point Rp to the right in the drawing at a constant speed, and continues to read out the video data and audio data at the readout point designated by the point Rp. The processing control unit 112 supplies the video data of the read video and audio data 124 to the video processing unit 114 and to the audio processing unit 126 .

[0028] The video processing unit 114 generates a video in which objects based on the object data A to D are superimposed on a background video based on background video data from the video data. However, the video processing unit 114 does not use all of the object data A to D read out in synchronization with the background video data, but rather superimposes any of the object data A to D on the background video when conditions described below are met to generate the video data Vd-1 and Vd-2.

[0029] Of the sound data, the specific sound data is sound data separated by data a, b, c, and d. In detail, data a, b, c, and d are data that are audible only during the periods of timing t1 to t2, t3 to t4, t5 to t6, and t7 to t8, respectively, and are silent during the other periods indicated by the dashed lines. Data a, b, c, and d may be unified as the same type of sound (e.g., the sound of bubbles), or may be different types of sounds (e.g., the sound of bubbles, the sound of waves, noise). It is preferable that the specific sound is a type of sound that is related to the object that will be displayed subsequently.

[0030] 4 is a diagram showing an example of the arrangement of each element in the video display system 1. Note that each element in the video display system 1 is provided in an audiovisual room 70, but in the figure, in order to clearly explain the example arrangement, some of the ceiling, walls, etc. in the audiovisual room 70 are omitted, and only speakers 32-1 and 32-2 of speakers 32-1 to 32-n are shown.

[0031] Display device 40-1 projects an image onto wall surface 70-1 in audiovisual room 70. Display device 40-2 projects an image onto wall surface 70-2, which is adjacent to wall surface 70-1, in audiovisual room 70. Sensor 20 is attached to wall surface 70-2, for example, and captures an image of a user (not shown in FIG. 4). The image may be, for example, a still image, a video, or a mixture of these. Speaker 32-1 is attached to, for example, wall surface 70-1, and speaker 32-2 is attached to, for example, wall surface 70-2. The sound emission direction of speakers 32-1 and 32-2 is, for example, toward the center of audiovisual room 70 in a plan view. Note that a plan view refers to looking from the ceiling of audiovisual room 70 to the floor. In this example, the image is projected onto 70-1 and 70-2, but the projection surface of the image may be all the wall surfaces of the audiovisual room 70, or may include the ceiling and floor.

[0032] Next, the operation of the video display system 1, specifically the operation of each part controlled by the control device 11 in the computer 10, will be described.

[0033] FIG. 5 is a flowchart showing the operation of each unit constructed in the control device 11. As described above, when an instruction to start display is given, the process control unit 112 starts reading out the audio-visual data 124 (step S11). The processing control unit 112 transfers the video data of the read video and audio data 124 to the video processing unit 114, and transfers the audio data to the video processing unit 114. The video processing unit 114 generates video signals Vd-1 and Vd-2 based on the transferred video data, and the audio processing unit 116 generates audio signals Ad-1 to Ad-n based on the transferred audio data.

[0034] The display device 40-1 projects an image based on the video signal Vd-1 onto the wall surface 70-1, and the display device 40-2 projects an image based on the video signal Vd-2 onto the wall surface 70-2. The speakers 32-1 to 32-n sequentially generate sounds based on the acoustic signals Ad-1 to Ad-n. 3, the specific sound data indicates silence immediately after starting to read out the audio and video data 124. Furthermore, the process control unit 112 causes the video processing unit 114 to hide the objects based on the object data A to D. Therefore, the display devices 40-1 and 40-2 display only the background image, and the speakers 32-1 to 32-n generate only the background music.

[0035] Next, the process control unit 112 causes the audio processing unit 116 to determine whether or not specific audio data from the audio data indicates the presence of audio (step S12). If the notification from the audio processing unit 116 indicates the absence of audio (if the determination result in step S12 is "No"), the process control unit 112 returns the process procedure to step S12. In other words, if the specific audio data indicates the absence of audio, the process procedure enters a standby state in step S12.

[0036] If the notification from the sound processing unit 116 indicates the presence of sound (if the determination result in step S12 is "Yes"), the process control unit 112 instructs the sound processing unit 116 to supply the sound-containing specific sound data together with the BGM data to the sound source device 31 (step S13). As a result, the speakers 32-1 to 32-n generate the specific sound together with the BGM. Note that the purpose of the specific sound is to give the user an awareness, so the initial volume is set low.

[0037] Furthermore, the processing control unit 112 instructs the sound source device 31 to position the sound image of the specific sound at a position different from the position in the direction in which the user's head is facing. For example, if the user is facing the wall surface 70-1, the processing control unit 112 instructs the sound source device 31 to position the sound image of the specific sound at the boundary between the wall surfaces 70-1 and 70-2. Therefore, the specific sound has directivity in the direction from the sound image position toward the user (which is an example of first directivity).

[0038] If the user notices the generation of a specific sound, they may change their behavior by turning their head or body in the direction of the sound, moving their eyes, etc. The process control unit 112 is notified by the determination unit 118 whether or not such a change in behavior has occurred. That is, the process control unit 112 determines whether or not there is a change in behavior based on the notification from the determination unit 118 (step S14).

[0039] If the notification from the determination unit 118 indicates that there is no change in the user's behavior (if the determination result in step S14 is "No"), the process control unit 112 instructs the sound processing unit 116 to change the mode of the specific sound (step S15). In this embodiment, the instruction to change the mode of the specific sound is an instruction to increase the volume of the specific sound. After this, the process control unit 112 returns the process procedure to step S14.

[0040] If the user's behavior does not change even when the processing procedure returns from step S15 to step S14, the processing procedure repeats steps S14 and S15. As a result, the volume of the specific sound gradually increases, and eventually the user notices the specific sound and changes their behavior. However, if the user's behavior does not change for a predetermined period of time even after the sound mode is changed, the generation of the specific sound may be stopped, and the processing procedure may return to step S12, and the system may wait until the next specific sound is generated, although this is not shown in the figure.

[0041] On the other hand, if the notification from the judgment unit 118 indicates that there has been a change in the user's behavior (if the judgment result in step S14 is "Yes"), the processing control unit 112 instructs the sound source device 31 to change the sound image position of the specific sound to a position different from the position in step S13 (step S16). For example, if the sound image position of the specific sound is set to the boundary between the wall surfaces 70-1 and 70-2 in step S13, the processing control unit 112 instructs the sound source device 31 to change the sound image position to the center position of the wall surface 70-2. As a result, the specific sound will have directivity (an example of second directivity) in the direction from the changed sound image position toward the user.

[0042] Next, the process control unit 112 instructs the video processing unit 114 to change the aspect of the video based on the video data (step S17). In this embodiment, the instruction to change the aspect of the video is to add or switch video based on object data. That is, if the video based on video data is only background video, the instruction is to display a new video based on object data superimposed on the background video. If a video based on some object data is already displayed superimposed on the background video, the instruction is to switch to a video based on another object data. The switch to the video based on object data may be instantaneous, or may be performed while overlapping the videos based on both object data, or may be performed by fading out the video based on the first object data and fading in the video based on the new object data. After step S17, the processing procedure returns to step S12.

[0043] The process control unit 112 terminates this display operation when the readout of the audiovisual data 124 reaches the end in time, or when an instruction to terminate the display is given by the operator or an external device.

[0044] Next, a specific operation of the video display system 1 will be described.

[0045] FIG. 6 is a diagram illustrating an example of an object that is superimposed on a background image in the video data. Since the background video data is read from the start point Sp to the end point Ep, the background video represented by the background video data is reproduced continuously without interruption. When the playback of the background video reaches timing t1, the specific sound data becomes a significant section of data a, and a specific sound is generated by the playback of data a. Here, if there is a change in the user's behavior at time t11 after time t1 when a specific sound based on data a is generated, the directionality of the specific sound is changed and an object based on object data A is displayed superimposed on the background image. The object is displayed in a fade-in manner, i.e., so that its appearance gradually appears. In this example, the specific sound stops at timing t2 when the reproduction of data a ends.

[0046] When the playback of the background video reaches timing t3, the specific sound data becomes a significant section of data b, and a specific sound is generated by the playback of data b. Here, if there is a change in the user's behavior at time t31 after time t3 when the specific sound based on data b is generated, the directionality of the specific sound is changed, and the object based on object data B is displayed superimposed on the background image, while the object based on object data A is erased. The object based on object data B is displayed with a fade-in effect, while the object based on object data A is displayed with a fade-out effect, that is, so that the object gradually disappears.

[0047] Similarly, when timing t5 is reached, a specific sound is generated by playing data c, and if there is a change in the user's behavior at a later timing t51, the directionality of the specific sound is changed, and an object based on object data C is displayed superimposed on the background image, while the object based on object data B is erased. Similarly, at time t7, a specific sound is generated by playing data d, and if the user's behavior changes at a later time t71, the directionality of the specific sound is changed and an object based on object data D is displayed superimposed on the background image.

[0048] 7 to 10 are diagrams for explaining specific operations in the video display system 1. FIG.

[0049] 7 is a diagram showing an example of a state after timing t11 but before timing t3 in FIG. 6, i.e., an example in which an object based on object data A is superimposed on a background image. In more detail, the example shows an underwater image displayed as a background image on walls 70-1 and 70-2, and this background image includes bubbles 91, rocks 92, and seaweed 93, with user U facing wall 70-1 in audiovisual room 70. Furthermore, on wall 70-1, a school of fish 80-A is displayed superimposed on the background image as an example of an object based on object data A.

[0050] 8 is a diagram showing the state immediately after a specific sound based on data b is generated, for example, at timing t3 in FIG. 6. For convenience, the sound image position of the specific sound is assumed to be the position of speaker 32-1. In this state, the specific sound has just been generated, and user U is not aware of the specific sound, so there is no change in his / her behavior.

[0051] 9 is a diagram showing a state immediately after the user U notices a specific sound based on data b and turns his / her body in the direction of the specific sound. Note that this diagram shows a state before the change in behavior is determined by the determination unit 118.

[0052] 10 is a diagram showing a state in which, due to the change in behavior, an object based on object data B is displayed. In detail, this is an example in which a school of fish 80-B, which is different from school 80-A, is displayed on wall surface 70-2 as an object based on object data B, superimposed on a background image. As described above, the group 80-A as objects based on the object data A then fades out.

[0053] As described above, according to the video display method of this embodiment, after the generation of a specific sound, the video changes in response to a change in the user's behavior, thereby providing a deep sense of immersion to the user watching the video and enhancing the sense of realism. That is, according to the video display method of this embodiment, the generation of a specific sound can change the user's behavior. Then, according to the video display method, the video changes when a change in the user's behavior is detected. Therefore, according to the video display method, the generation of a specific sound creates an opportunity for the user to pay attention to the video, thereby creating an opportunity for the user to actively interact with the video space. Conversely, the fact that the image changes in response to changes in the user's behavior means that the user can be immersed in the changes in the image without feeling that the image creator has manipulated them. Because the user does not sense the creator's manipulation, they will feel as if they have discovered the changes in the image themselves, and we can expect a new interaction between the user and the changes in the image. In addition, in this embodiment, the image changes in response to a change in the user's behavior, allowing the user to concentrate on the image. Therefore, it is expected to be applied to indirect treatment and calm-down for people with developmental disabilities who tend to be passive, and patients with mental illnesses who tend to think divergently. Furthermore, in this embodiment, if a user finds a causal relationship between a change in their own behavior and a change in the image, the user can infer that their own actions are influencing the image space, thereby further increasing the sense of immersion in the image space.

[0054] In the above-described embodiment, the specific sound is generated based on specific sound data that is played in synchronization with the video data, but the specific sound may also be generated according to a predetermined timetable, at regular intervals, or at random timing, for example.

[0055] In the embodiment, the object superimposed on the background image is faded out while a new object is faded in, triggered by a change in the user's behavior upon noticing the occurrence of a specific sound, but the object may be switched instantaneously. Also, the background image may be dynamically switched to another background image.

[0056] In the embodiment, it is assumed that there is only one user present in the audiovisual room 70, but if there are two or more users, for example, if the behavior of the majority of the users present changes, the judgment result of step S14 may be determined as "Yes."

[0057] The directivity of the specific sound may be configured to change rather than be fixed during the period in which the specific sound is generated. For example, the sound image of the specific sound may be localized on the north side of the audiovisual room 70 and then moved from north to east. During such movement, the user is likely to behave in a manner that follows the sound image from north to east. Therefore, the determination unit 118 can determine whether or not the user is aware of the movement of the specific sound by detecting the user's behavior that is moving from north to east.

[0058] In the embodiment, the image aspect is changed in response to a change in the user's behavior that occurs as a result of stimulating the user's hearing. The stimulus given to the user may be a sense other than hearing, such as vision. Specifically, an object may be displayed in the image, or an image different from that in other areas may be displayed in a certain area, and the image aspect may be changed in response to a change in the user's behavior that causes the user to notice the difference in the display.

[0059] Alternatively, tactile stimulation may be provided to the user. For example, multiple actuators may be provided on the floor surface, and one or more of the actuators may be vibrated, triggering a change in the image when the user notices the vibration. Alternatively, air waves such as wind, pressure waves, or vortex waves may be sent toward the user, triggering a change in the image when the user notices the air waves. The sense of smell may be used as a stimulus to the user. For example, a scented liquid may be sprayed on the user, and the image may be changed when the user's behavior changes as if they notice the scent.

[0060] A summary of this disclosure is provided below.

[0061] The image display method according to Supplementary Note 1 includes a sound generating device generating a sound; a control device determining whether a user's state has changed from a first state to a second state different from the first state based on an output from a sensor that detects the user; and, when the control device determines that the state has changed to the second state, a display device that displays an image changes the display mode of the image from the first display mode to the second display mode different from the first display mode. According to the video display method of Supplementary Note 1, the video changes from the first display mode to the second display mode when the user changes from the first state to the second state. Therefore, the change in the video occurs due to the user's own actions, not due to the intention of the video producer, and it is possible to immerse the user in watching the video.

[0062] Examples of a user changing from the first state to the second state include changes in the user's face or body orientation, gestures, facial expressions, line of sight, position, etc. Also, video refers to an image displayed by a display device, and includes still images and moving images. Furthermore, a change in the display mode of an image does not only refer to a change from a state in which some image is displayed to a state in which another image is displayed, but also to a change from a state in which nothing is displayed to a state in which a new image is displayed, or a change from a state in which some image is displayed to a state in which nothing is displayed.

[0063] The video display method according to Supplementary Note 2 is the method according to Supplementary Note 1, wherein the sound generating device generates the sound in a first sound manner during a first period, and generates the sound in a second sound manner different from the first sound manner during a second period in which the display device has changed to the second display manner. Examples of the sound manner include volume, pitch, and timbre, and a different manner means that one or more of these elements are different.

[0064] The video display method according to Supplementary Note 3 is the same as Supplementary Note 2, wherein the first sound aspect is that the sound has a first directivity, and the second sound aspect is that the sound has a second directivity different from the first directivity. According to the video display method according to Supplementary Note 3, the change in sound directivity can give a user a sense of awareness. Note that sound directivity refers to the direction in which the sound is strongly transmitted, and specifically, is perceived by the user as the position of the sound image generated by a sound generating device changes.

[0065] The video display method according to Supplementary Note 4 is similar to Supplementary Note 3, wherein the display device changes a display position of an object included in the video from a first position in the first period to a second position in the second period, and a second directivity of the sound in the second sound manner has a direction from the second position. According to the video display method according to Supplementary Note 4, the display position of the object included in the video changes according to the directivity of the sound. Note that the direction according to the second position refers to a direction starting from the second position or a direction ending at the second position.

[0066] The video display method of Supplementary Note 5 is such that, in Supplementary Note 3, the directionality of the sound in the first sound state is a first direction toward the user, and the directionality of the sound in the second sound state is a direction different from the first direction.

[0067] The video display method of Supplementary Note 6 is the same as that of Supplementary Note 1, wherein the display device displays video during a first period by playing it based on video data, and the sound generating device generates sound during the first period in synchronization with the playback based on the video data.

[0068] The video display system of Appendix 7 includes a sound generating device that generates sound, a display device that displays video, and a control device that determines whether the state of a user watching video displayed on the display device has changed from a first state to a second state different from the first state based on an output from a sensor that detects the user, and when the control device determines that the state has changed to the second state, it changes the display mode of the video by the display device from the first display mode to the second display mode different from the first display mode. According to the video display system of Appendix 7, as in Appendix 1, changes in the video occur due to the user's own actions, not due to the intention of the video producer, making it possible to immerse the user in watching the video.

[0069] The program according to Supplementary Note 8 causes a computer that controls a sound generating device that generates sound and a display device that displays video to instruct the sound generating device to generate the sound, determine whether the state of the user has changed from a first state to a second state different from the first state based on output from a sensor that detects a user watching the video, and, if it is determined that the state has changed to the second state, change the display mode of the video by the display device from the first display mode to the second display mode different from the first display mode. According to the program of Appendix 8, as in Appendix 1, changes in the image occur due to the user's own actions, not due to the intention of the video producer, making it possible to immerse the user in watching the image. [Explanation of symbols]

[0070] 1...video display system, 10...computer, 20...sensor, 30...sound generating device, 40-1, 40-2...display device

Claims

1. a sound generating device generating a sound; The control device determines whether the state of the user has changed from a first state to a second state different from the first state based on an output from a sensor that detects the user; When the control device determines that the state has changed to the second state, a display device that displays a video image changing a display mode of the video image from a first display mode to a second display mode different from the first display mode; A video display method including:

2. The sound generating device is generating the sound in a first sound manner during a first period; During a second period in which the display device has changed to the second display mode, the sound is generated in a second sound mode different from the first sound mode.

2. The image display method according to claim 1.

3. the first sound aspect is that the sound has a first directivity; The second sound aspect has a second directivity of the sound that is different from the first directivity. The image display method according to claim 2.

4. The display device includes: changing a display position of an object included in the video from a first position in the first period to a second position in the second period; A second directivity of the sound in the second sound manner has a direction from the second position. The image display method according to claim 3.

5. The directionality of the sound in the first sound manner is a first direction toward the user, The directionality of the sound in the second sound mode is a direction different from the first direction. The image display method according to claim 3.

6. The display device includes: displaying the video in the first period by playing it back based on the video data; The sound generating device is The sound during the first period is generated in synchronization with the reproduction based on the video data.

2. The image display method according to claim 1.

7. a sound generating device that generates a sound; a display device for displaying an image; a control device that determines whether a state of a user viewing an image displayed on the display device has changed from a first state to a second state different from the first state based on an output from a sensor that detects the user; and Including, When the control device determines that the state has changed to the second state, changing a display mode of the image on the display device from a first display mode to a second display mode different from the first display mode; Video display system.

8. a sound generating device that generates a sound; a display device that displays an image; The computer that controls instructing the sound generating device to generate the sound; determining whether a state of the user has changed from a first state to a second state different from the first state based on an output from a sensor that detects the user viewing the video; When it is determined that the state has changed to the second state, changing a display mode of the image by the display device from a first display mode to a second display mode different from the first display mode; A program that executes the following.

Images