Electronic device, method and computer program

Abstract

An electronic device comprising circuitry configured to control, based on video data, a displaying of a video by a display and control, based on vestibular stimulus data, a generating of a vestibular stimulus, corresponding to a content of the video, by a vestibular stimulator wherein the generating of the vestibular stimulus and the displaying of the video are synchronized.

Description

[0001] ELECTRONIC DEVICE, METHOD AND COMPUTER PROGRAM

[0002] TECHNICAL FIELD

[0003] The present disclosure generally pertains to a device, a method and a computer program.

[0004] TECHNICAL BACKGROUND

[0005] Known modes of displaying a video to a user comprise displaying video data in conjunction with auditory information. There is a need for additional sensory input in order to make viewing of a video more enjoyable to a user.

[0006] There are devices that include stimulation of a vestibular system to provide, for example, stimuli to accompany visual stimuli, as described in patent document 1. Moreover, there are devices that stimulate a patient’s vestibular system in order to augment or control a patient's respiratory function, open the patient's airway, induce sleep, and / or counteract vertigo, such as described in patent document 2. The are furthermore systems and methods for game playing using vestibular stimulation that include detecting motions associated with the user by a feedback sensor device and providing motion information from the feedback sensor device to a game device, such as described in patent document 3.

[0007] LIST OF REFERENCES

[0008] Patent document 1 : United States Patent Nr. 11458313 Bl

[0009] Patent document 2: United States Patent Application Nr. 20080275513 Al

[0010] Patent document 3: United States Patent Application Nr. 20100113150 Al

[0011] SUMMARY

[0012] According to a first aspect, the present disclosure provides an electronic device according to claim 1. According to a second aspect, the present disclosure provides a method according to claim 15. According to a third aspect, the present disclosure provides a computer program according to claim 20. Further aspects are set forth in the dependent claims, the drawings and the following description.

[0013] BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Embodiments are explained by way of example with respect to the accompanying drawings, in which: Fig. l is a schematic illustration of the function of a vestibular video enhancement device; and

[0015] Fig. 2 is a schematic illustration of a timed sequence of video frames with associated timed vestibular stimulation instructions; and

[0016] Fig. 3a to Fig. 3c are schematic illustrations of different embodiments of a stimulation vector included in vestibular stimulation instructions; and

[0017] Fig. 4 is a schematic illustration of a timed sequence of video frames with associated timed subsequences of vestibular stimulation instructions; and

[0018] Fig. 5 is a schematic illustration of a vestibular video enhancement device comprising the vestibular stimulator; and

[0019] Fig. 6 is a schematic illustration of a vestibular video enhancement device comprising the vestibular stimulator and a display; and

[0020] Fig. 7 is a schematic illustration of a vestibular video enhancement device configured to generate the vestibular stimulation data; and

[0021] Fig. 8 is a flow diagram of a method for generating vestibular stimulation data; and

[0022] Fig. 9 is a symbolic illustration of two frames of a video on a basis of which vestibular stimulation data may be generated; and

[0023] Fig. 10a and Fig. 10b are symbolic illustrations of a method for generating vestibular stimulation data for a predetermined entity; and

[0024] Fig. 11 is a symbolic illustration of two frames of a video on a basis of which vestibular stimulation data may be generated for a predetermined entity visible in the video; and

[0025] Fig. 12 is a flow diagram of a method for synchronously displaying a video and generating a vestibular stimulus based on a received data stream; and

[0026] Fig. 13 is a flow diagram of a method for synchronously displaying a video, generating vestibular stimulus data and generating a vestibular stimulus based on a received video data stream; and

[0027] Fig. 14 is an illustration of transmission of data from a distribution device to the vestibular video enhancement device; and

[0028] Fig. 15 is a flow diagram of a method for obtaining a calibration result; and Fig. 16a and Fig. 16b are illustrations of a vestibular stimulator generating a vestibular stimulus to stimulate a vestibular system of a user; and

[0029] Fig. 17 is an illustration of a method for generating the vestibular stimulus based on vestibular stimulus data and user preference information; and

[0030] Fig. 18 is a flow diagram of a method for generating video data and vestibular stimulus data using an inertial measurement unit and a camera and streaming the video data and vestibular stimulus data to the vestibular video enhancement device; and

[0031] Fig. 19 is a flow diagram of a method for generating video data and vestibular stimulus data using an inertial measurement unit and a camera and storing, then distributing, the video data and vestibular stimulus data to the vestibular video enhancement device; and

[0032] Fig. 20 is a flow diagram of a method for generating video data using a camera and generating vestibular stimulus data from the video data by the vestibular video enhancement device; and

[0033] Fig. 21a, Fig. 21b and Fig. 21c are respectively illustrations of the vestibular video enhancement device as a head-mounted device, a headset and a plurality of head-mounted casings; and

[0034] Fig. 22 is a schematic illustration of the circuitry of the vestibular video enhancement device; and

[0035] Fig. 23 and Fig. 24 illustrate anatomical background.

[0036] DETAILED DESCRIPTION OF EMBODIMENTS

[0037] Before a detailed description of the embodiments under reference of Fig. 1 is given, general explanations are made.

[0038] An electronic device according to the present disclosure comprises circuitry configured to control, based on video data, a displaying of a video by a display; and control, based on vestibular stimulus data, a generating of a vestibular stimulus, corresponding to a content of the video, by a vestibular stimulator; wherein the generating of the vestibular stimulus and the displaying of the video are synchronized.

[0039] The electronic device may be called a vestibular video enhancement device. The electronic device may be any mobile or stationary data processing device that is capable of performing control of another electronic device, specifically control of a video display.

[0040] Circuitry is, for example, a programmable microprocessor, a computer or the like. Control may be performed by the electronic device issuing digital control signals to the display and the vestibular stimulator via a data transmission means. The data transmission means may be a wire or a wireless data transmission means, such as a radio-based transmitter.

[0041] Video data is any data or data structure that carries information on a times sequence of images that, when displayed by a digital video display, results in a video being displayed.

[0042] A display is any electronic apparatus that is capable of displaying a video based on digital video data, for example, a computer monitor, a digital TV, a head-mounted display, an augmented reality device, a mobile phone display and the like.

[0043] A vestibular stimulus is a physical stimulus that is perceivable by a human vestibular system. Known methods for artificially generating a vestibular stimulus include galvanic stimulation using direct current stimulation of the vestibular organ and acoustic stimulation.

[0044] Vestibular stimulation allows to artificially generate a perception of motion. This is performed using Galvanic Vestibular Stimulation (GVS), which consists in electrically stimulating the vestibular organ. By placing electrodes in various configurations, it is possible to give motion perception along all three axes of rotation (roll pitch and yaw, and intermediate axes through interpolation.

[0045] Other stimulation modes are also possible (vibration, magnetic, auditive).

[0046] The vestibular stimulator may be an external apparatus that is not part of the electronic device but controlled by the external device in the manner of a peripheral.

[0047] The vestibular stimulus corresponds to a content of the video. A content of the video may, for example, be a movement of the camera or a movement of an object visible in the video.

[0048] In other words, the vestibular stimulus corresponds to what a user perceives as the content of the video, not to the act of watching the video.

[0049] The term “synchronized” is to be understood that the control for the displaying of the video and the generating of the vestibular stimulus is initiated at the same time and terminated at the same time and that for each time that the video is displayed, a corresponding vestibular stimulus is generated.

[0050] There are embodiments wherein the vestibular stimulus data comprises a timed sequence of vestibular stimulation instructions; and wherein the video data, comprising timed sequences of frames, is associated with the vestibular stimulation instructions on a frame-by-frame basis. Inertial information may be information directly representing a numerical value of the acceleration. Alternatively, inertial information may be information that allow deriving a numerical value of the acceleration. For example, information on a rotational velocity of a solid body allows derivation of numerical values for the acceleration using known principles of mechanics. Alternatively, acceleration information may be information on a movement that may be mapped onto an experience of acceleration through the vestibular organ. For example, information of a rotation by itself may be inertial information.

[0051] Inertial information is information that is output in the shape of an adequate digital data structure by, for example, an inertial measurement unit.

[0052] A mapping of inertial data to vestibular stimulus data may be effected as follows.

[0053] Inertial data comprises information on a rotation about a certain space axis. For example, rotation about a z-axis, rotation about a y-axis and rotation about an x-axis, wherein said axes are linearly independent axes in a Cartesian coordinate system. Each rotation about a certain axis may be called a rotation component. The information on the rotation also comprises information about a velocity of the rotation. The velocity of the rotation is, for purposes of illustration, given in terms of a basic rotation unit (BRU). The basic rotation unit may, for example, be 10 deg / s or 5 deg / s.

[0054] The information of each rotation component is then mapped to a stimulus in a corresponding direction.

[0055] The mapping of the direction may be accomplished according to the following table:

[0056] The mapping of the velocity to the strength may be accomplished according to the following equation:

[0057] Strength = A x velocity Here A is an adjustment factor that may be chosen at the point of implementation. The adjustment factor A may have any real value, for example, the adjustment factor A may be 1.0 or 0.1 or 0.01 or 5.0. However, the value of the adjustment factor A is set with a condition that no value of the velocity will result in a vestibular stimulus higher than 3 mA. The adjustment factor A may also be computed based on user preference information as described hereinbelow.

[0058] Vestibular stimulus instructions, as the vestibular stimulus data, may be encoded directly with the video by adding an additional channel on the video and audio. Alternatively, inertial forces, as the vestibular stimulus data, may be encoded directly with the video by adding an additional channel on the video and audio.

[0059] Vestibular stimulation instructions instruct the generation of the vestibular stimulus. Therefore, the vestibular stimulus may be a time-dependent vestibular stimulus. The vestibular stimulation instructions may be called inertial force data.

[0060] This way, the generated vestibular stimuli are perfectly synchronized with the video and audio footage.

[0061] There are embodiments wherein each one of the frames of the video data is associated with a timed sub-sequence of the vestibular stimulation instructions.

[0062] Therefore, a frequency of vestibular stimuli may be higher than the frame rate of the video, improving perception and making it possible to prevent abrupt transitions of the vestibular stimuli between video frames.

[0063] For compression needs, various existing lossless or lossy compression techniques may be used. If using lossy compression, the inertial data may be post-processed (“cleaned up”) to remove high-frequency movements which would be difficult to reproduce on the vestibular stimulation device, and that might result in an unpleasant experience to the user.

[0064] There are embodiments wherein each one of the vestibular stimulation instructions comprises a stimulation vector that indicates a space-direction of the vestibular stimulus to be generated.

[0065] Alternatively, the space vector may indicate a direction of rotation.

[0066] There are embodiments wherein the stimulation vector further indicates a strength of the vestibular stimulus to be generated.

[0067] Alternatively, the space vector may further indicate a magnitude of rotation.

[0068] There are embodiments wherein the circuitry is further configured to generate, based on the content of the video, the vestibular stimulus data. This allows omission of capturing inertial data using, for example, an inertial measurement unit. Further, a user may choose to experience inertial forces as experienced by an entity shown in the video in a third-person perspective. For example, the user may wish to experience inertial forces experienced by a played at a football match or inertial forces imparted on a ball at a football match

[0069] There are embodiments wherein the circuitry is further configured to generate the vestibular stimulus data based on the content of the video by determining, from the content of the video, using a physics engine simulating a virtual model of the content of the video, inertial data of a predetermined entity, and generating the vestibular stimulus data based on the inertial data; wherein the predetermined entity is an entity visible in the video or the camera that recorded the video.

[0070] The virtual model may be any digital representation of a physical scenario. The physics engine is a computer program capable of simulating a time-evolution of a physical system under conditions of pre-set physical constraints.

[0071] Known physics engines include the “Advanced Simulation Library”, “Newton Game Dynamics” or “Open Game Dynamics”. Other physics engines are known to the skilled person.

[0072] The mapping of the content of the video may be performed using a neural network or the like.

[0073] There are embodiments wherein the circuitry is further configured to generate the vestibular stimulus data based on a movement of an entity visible in the video as the predetermined entity in the video in a frame of reference of the surrounding of the predetermined entity, or generate the vestibular stimulus data based on a movement of the camera that recorded the video as the predetermined entity in a frame of reference of the surrounding of the camera that recorded the video.

[0074] The frame of reference surrounding the predetermined entity is the frame of reference wherein the predetermined entity or the camera that captured the video is moving and Earth is stationary. The predetermined entity is a person or an object.

[0075] There are embodiments wherein the circuitry is further configured to receive, from an external distribution device, the video data; wherein the receiving of the video data; the generating of the vestibular stimulus data, the generating of the vestibular stimulus and the displaying of the video are synchronized. Capturing of the video would be the same as for regular videos with an addition of one or multiple inertial measurement units (IMUs). The inertial measurement units may be located on the camera, for example a video camera used by a professional video creator, a mobile phone camera or an action camera. Alternatively, an actor may be provided with an inertial measurement unit, ideally on the head close to ears to ensure as close experience as the playback would be. Or an object if it is the focus of the experience - e.g. a football during a football match. For animated footage, the inertial forces may be recorded in realistic simulated environment using the physics engine.

[0076] The inertial forces might be recorded in case of recorded videos or livestreamed in case of livestreaming video.

[0077] There are embodiments wherein the circuitry is further configured to receive, from an external distribution device, the video data and the vestibular stimulus data; wherein the receiving of the video data and the vestibular stimulus data, the generating of the vestibular stimulus and the displaying of the video are synchronized.

[0078] There are embodiments wherein the circuitry is configured to further generate the vestibular stimulus based on user preference information.

[0079] User preference information may be a calibration or a setting of a maximum or minimum strength.

[0080] The stimulation may be adapted to the user according to their needs. Alternatively, a feedback loop of the stimulation and observation of its effect may be utilized. The inertial measurement unit could be used to estimate the extent of stimulation effect experienced by the user.

[0081] There are embodiments further comprising the vestibular stimulator.

[0082] There are embodiments wherein the vestibular stimulator is a galvanic vestibular stimulator, comprising two or more electrodes, that are worn on a head of a user.

[0083] There are embodiments wherein the electronic device is any one of a headset, a head-mounted wearable device, a virtual reality device or an augmented reality device.

[0084] A vestibular stimulation method comprises: control, based on video data, a displaying of a video by a display; and control, based on vestibular stimulus data, a generating of a vestibular stimulus, corresponding to a content of the video, by a vestibular stimulator; wherein the generating of the vestibular stimulus and the displaying of the video are synchronized. There are embodiments of the method further comprising: receive, from an external distribution device, the video data and the vestibular stimulus data; wherein the receiving of the video data and the vestibular stimulus data, the generating of the vestibular stimulus and the displaying of the video are synchronized.

[0085] The external distribution device is any electronic apparatus capable of storing digital data and transferring said data to another electronic apparatus, such as the electronic device according to the present disclosure.

[0086] Thus, as video data is received, vestibular stimulus instructions are generated and the vestibular stimulus is generated simultaneously

[0087] There are embodiments of the method further comprising: receive, from an external distribution device, the video data and generate, based on the content of the video, the vestibular stimulus data; wherein the receiving of the video data, the generating of the vestibular stimulus data, the generating of the vestibular stimulus and the displaying of the video are synchronized.

[0088] There are embodiments of the method further comprising: record, using a camera, the video data; and synchronously record, using an inertial measurement unit in a same frame as the camera, inertial data; and generate, based on the inertial data, the vestibular stimulus data; and store the video data and the vestibular stimulus data in the external distribution device.

[0089] There are embodiments of the method further comprising: record, using a camera, the video data; and generate the vestibular stimulus data by determining, from the video data, using a physics engine, inertial data, and generating the vestibular stimulus data based on the inertial data.

[0090] A computer program, if executed by a computer, causes the computer to control, based on video data, a displaying of a video by a display; and control, based on a vestibular stimulus data, a generating of a vestibular stimulus, corresponding to a content of the video, by a vestibular stimulator; wherein the generating of the vestibular stimulus and the displaying of the video are synchronized.

[0091] The methods as described herein are also implemented in some embodiments as a computer program causing a computer and / or a processor to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer- readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed. Prior to specific discussions of embodiments, the anatomic background of the present disclosure is briefly discussed with reference to Fig. 23 and Fig. 24.

[0092] Fig. 23, illustrating anatomical background, illustrates the vestibular system 10, located in a head 11 of a human being stimulated, affecting the thalamus 550 the cerebellum 530 and higher brain functions 550 (illustrative). The consciously perceptible sensation 540 evokes a feeling of either being linearly accelerated in a given direction or rotating about a given axis, even while the user, or - more precisely - the user’s vestibular system 10 remains motionless. The vestibular system 10 is located inside the human skull in the area of the ears on either side.

[0093] Fig. 24, further illustrating anatomical background, shows as a mode of natural stimulation of the vestibular system. The vestibular system 10 is located in two distinct areas on both sides of the human skull. It includes three so-called semicircular canals 501, arranged in three linearly independent spatial planes. Each semicircular canal 501 is formed by channel -like tubes 250 in a ring shape. Each semicircular canal contains a fluid 520. If the vestibular system 10 (and, by extension, the user’s head 11) is rotated about an axis, by inertia, the fluid 520 is displaced with respect to the tube 555. Set in the wall 555 of the tube 550 are a series of hairs 510, that are deflected by the displaced fluid 520. This deflection leads to a nerve signal that is interpreted by the nervous system of the user as (in the case of the semicircular canals 501) rotation about an axis. Linear acceleration is sensed by stimulation of the otolithic organs (not shown).

[0094] Returning to the figures, Fig. l is a schematic illustration of the function of a vestibular video enhancement device 1-1.

[0095] The vestibular video enhancement device 1-1 comprises a control unit 5. The device 1-1 receives video data 100 and vestibular stimulus data 200. The video data 100 and the vestibular stimulus data 200 are stored, for example, in the control, unit 5. The video data 100 and the vestibular stimulus data 200 are further processed by the control unit 5. The device 1-1, using the control unit 5, controls a display 190 such that the video data 100 is displayed as a video 101. The device 1-1, using the control unit 5, further control a vestibular stimulator 290 such that a vestibular stimulus 201 is generated by the vestibular stimulator based on the vestibular stimulus data 200.

[0096] In the present embodiment, the video data 100 and the vestibular stimulus data 200 are received from an external source.

[0097] According to the present embodiment, the display 190 and the vestibular stimulator 290 are provided as electronic apparatuses separate from the vestibular video enhancement device 1-1 The display 190 and the vestibular stimulator are connected to the device 1-1 by means of a data bus. The data bus may, according to a first non-restrictive example, be a physical data bus, such as the Universal Serial Bus (USB). The data bus may, according to a second non-restrictive example, be a wireless data transmission means, such as Bluetooth, of the Bluetooth Special Interest Group. In general, any known method of transmitting digital data from one electronic apparatus to another electronic apparatus may be utilized as the data bus.

[0098] The vestibular stimulus 201 is generated to be synchronous with the displaying of the video 101 such that a content 102 of the video 101 that is visible to a user observing the video and stimulated by the vestibular stimulus 201 is associated by the user with the content 102 of the video 101.

[0099] The control unit 5 controls the display 190 by outputting a control signal to the display 190. The control unit 5 may also control the display 190 by transmitting the video data 100, as the control signal, to the display.

[0100] The control unit 5 controls the vestibular stimulator 290 by outputting a control signal to the vestibular stimulator 290.

[0101] The control unit 5 may be provided as circuitry or as a logical computing unit executed by circuitry. The control unit 5 may also be a virtual machine or the like.

[0102] The displaying of the video 101 by the display 190 and the generating of the vestibular stimulus 201 be the vestibular stimulator 290 are synchronized by the control unit 5 outputting synchronized control signals.

[0103] Fig. 2 is a schematic illustration of a timed sequence of video frames Fl . . .FN with associated timed vestibular stimulation instructions II . . .IN.

[0104] The video frames Fl . . .FN are comprised in a time-ordered sequence in the video data 100. The vestibular stimulation instructions II .. .IN are comprised in a time-ordered sequence in the vestibular stimulates data 200.

[0105] Note, however, that the video frames may be displayed in a time-reversed sequence, such as when the user requests a rewind of the video.

[0106] The video frame Fl represents a first frame of the video 101 to be displayed and the video frame FN represents an n-th, for example a last, frame of the video 101 to be displayed. Note that the first video frame Fl to be displayed may, but is not required to, be the first frame of a video data file. Likewise the last video frame FN to be displayed may, but is not required to, be the last frame of a video data file.

[0107] In Fig. 2, for example, the first video frame Fl is associated with a first vestibular stimulation instruction II, a second video frame F2 is associated with a second vestibular stimulation instruction 12 and so on. The n-th video frame FN is associated with an n-th vestibular stimulation instruction IN. The n-th video frame may or may not be the last frame of a video data file.

[0108] The vestibular stimulation data 200 may be seen as an additional data track associated with the video data 100, similar to an audio track in a conventional video file.

[0109] The vestibular stimulation instructions II . . .IN indicate a vestibular stimulus 201 to be generated by the vestibular stimulator 290. Each vestibular stimulation instruction II . . .IN thus indicates a vestibular stimulus 201 to be generated by the vestibular stimulator while the corresponding frame Fl . . .FN is displayed by the display 190.

[0110] Note that the video frames Fl . . .FN and vestibular stimulation instructions are ordered in a timed sequence, as indicated by the arrow labeled “time” and the video 101 is displayed based on the video frames and the vestibular stimulus 201 is generated based on the vestibular stimulation instructions in order of time.

[0111] The sequence of frames Fl . . .FN is displayed with a predetermined framerate to generate the video. The framerate may, for example, be 60 Hz, but any framerate usually considered for displaying video data may be chosen.

[0112] Displaying of the video 101 based on the video data 100 comprising the video frames Fl . . .FN is controlled by the control unit 5 as described hereinabove.

[0113] Generating of the vestibular stimulus 201 based on the vestibular stimulus data 200 comprising the vestibular stimulus instructions II .. .IN is controlled by the control unit 5 as described hereinabove.

[0114] In the present embodiment, the vestibular stimulus 201 is output, based on the vestibular stimulus data 200, at the same output rate as the framerate of the video 101. However, there are other embodiments where the output rate of the vestibular stimulus 201 is higher than the framerate of the video 101.

[0115] Fig. 3a to Fig. 3c are schematic illustrations of different embodiments of a stimulation vectors V included in vestibular stimulation instructions II . . .IN. The stimulation vector V indicates a direction of a vestibular stimulus 201 to be generated. The direction of the stimulation vector V is, for example, a direction with respect to a visual axis of a user of the device 1-1. For example, the visual axis is an axis indicated a direction that a user’s eyes are oriented towards if the user’s eyes are oriented straight ahead with respect to the head of the user.

[0116] Note that, instead of the stimulation vector V indicating a direction of the vestibular stimulus, the stimulation vector V may indicate inertial forces instead, such that the video data 100, instead of directions of a vestibular stimulus to be generated, is instead encoded with inertial forces. In this case, the inertial forces are mapped to the directions of the vestibular stimulus by the control unit 5 while performing control.

[0117] Fig. 3a specifically is a symbolic representation of a stimulation vector, here shown as an arrow. The arrow is shown as a two-dimensional arrow, each dimension indicating a, axis in space.

[0118] Each vestibular stimulus instruction II . . .13 may comprise a stimulation vector V. Note that the three vestibular stimulus instruction II . . .13 are merely representative for any number of vestibular stimulation instructions II . . .IN comprised in vestibular stimulus data.

[0119] The stimulation vector may be a three-dimensional vector, each dimension indicating a space axis of three-dimensional space in Cartesian coordinates. For example, the space axes may be oriented downwards to upwards, left to right and backward to forward.

[0120] The stimulation vector V may also be four-dimensional, with the fourth dimension indicating a strength of the vestibular stimulus 201 to be generate. The fourth dimension may also indicate a time-coordinate.

[0121] A strength of the stimulus may also be indicated by a length of the stimulation vector V. However, the stimulation vector may be normalized to a desired value, for example unity.

[0122] Any vector of any orientation in two dimensional space, three-dimensional space or fourdimensional space may of any length may be the stimulation vector V.

[0123] Fig. 3b illustrates an embodiment of the stimulation vector in a mathematical representation in three-dimensional space. The vestibular stimulus instruction II shown in Fig. 3b is a generalized example representation of a mathematical vector that may be the stimulation vector V. The stimulation vector V comprises a triple of coordinate values xl, yl and zl. The coordinate values xl yl and zl may be coordinates in a Cartesian coordinate system or in a curvilinear system, such as spherical or cylindrical coordinates may be provided within the meaning of the present disclosure.

[0124] In the following examples, a Cartesian system is assumed. Any curvilinear system may be provided by implementing a known coordinate transformation.

[0125] Each of the coordinate values xl, yl and zl may have positive or negative values. A numerical value of the coordinate value may correspond to a stimulation in units of a basic vestibular stimulation strength unit along that particular axis. A basic vestibular stimulation strength unit may correspond to stimulation with a particular strength.

[0126] For example, if the vestibular stimulator is, as described hereinabove, a galvanic stimulator, a vestibular stimulation strength unit may, for illustrative purposes, correspond to a vestibular stimulus of 0.3 mA. However, the vestibular stimulation strength unit may correspond to any suitable numerical value. For example, the vestibular stimulation strength unit may correspond to a vestibular stimulus of 0.01 mA or 1 mA or any number between or below said values.

[0127] The stimulation instructions 12 and 13 shown in Fig. 3b comprise particular examples of the stimulation vector V. In the stimulation instruction II, the coordinate values of the stimulation vector V take, for example, the values xl=-l, yl=0 and zl=2. In the stimulation instruction 13, the coordinate values of the stimulation vector V take, for example, the values xl=2, yl=3 and zl=l.

[0128] Negative values of the coordinate values xl, yl and zl may indicate a vestibular stimulus that should be generated in a direction opposite to a vestibular stimulus associated with positive values.

[0129] For example, a positive value of the coordinate value xl may indicate a stimulus to be generated left to right, as seen along the visual axis of the user and a negative value of the coordinate value xl may indicate a stimulus to be generated right to left, as seen along the visual axis of the user.

[0130] Correspondingly, a positive value of the coordinate value yl may indicate a stimulus to be generated back to front, as seen along the visual axis of the user and a negative value of the coordinate value yl may indicate a stimulus to be generated front to back, as seen along the visual axis of the user.

[0131] Note that a stimulus generated left to right may induce a feeling of rotation about the z-axis, a stimulus generated front to back may induce a feeling of rotation about the x-axis and a stimulus generated downward to upward may induce a feeling of rotation about the y-axis. The stimulus is generated such that the desired feeling of rotation is induced.

[0132] Moreover, a positive value of the coordinate value zl may indicate a stimulus to be generated below to above, as seen along the visual axis of the user and a negative value of the coordinate value zl may indicate a stimulus to be generated above to below, as seen along the visual axis of the user.

[0133] For example, in the stimulation instruction 13, the vestibular stimulus may cause a vestibular stimulus of strength xl=2, corresponding to two vestibular stimulation strength units and thus a vestibular stimulus of 0.6 mA, to be generated from left to right as seen along the visual axis of the user.

[0134] Fig. 3c illustrates an embodiment of the stimulation vector in a mathematical representation in four-dimensional space.

[0135] The vestibular stimulus instruction II in Fig. 3c, like the vestibular stimulus instruction II in Fig. 3b, is a generalized example representation of a mathematical vector that may be the stimulation vector V. The stimulation vector V comprises a quadruple of coordinate values xl, yl, zl and f. Each of the coordinate values xl, yl, zl and f may have positive or negative values. The coordinate f may indicate, for example, a total strength of the stimulus. The coordinate values xl yl and zl may be coordinates in a Cartesian coordinate system or in a curvilinear system, such as spherical or cylindrical coordinates may be provided within the meaning of the present disclosure.

[0136] In the following examples, a Cartesian system is assumed. Any curvilinear system may be provided by implementing a known coordinate transformation.

[0137] As in Fig. 3b, a numerical value of the coordinate values shown in Fig. 3c may correspond to a stimulation in units of a basic vestibular stimulation strength unit along that particular axis. A basic vestibular stimulation strength unit may correspond to stimulation with a particular strength.

[0138] A strength of the stimulus may also be indicated by a length of a three-dimensional vector defined by the coordinate values xl, yl and zl. However, length of a three-dimensional vector defined by the coordinate values xl, yl and zl may be normalized to a desired value, for example unity. In this case, the strength of the stimulus may correspond to the value of the coordinate value f. Fig. 4 is a schematic illustration of a timed sequence of video frames Fl . . .FN with associated timed sub-sequences IS1 . . .ISN of vestibular stimulation instructions IS 1 - 1 . . .ISN-2.

[0139] As in Fig. 2, each video frame Fl . . .FN is associated with a vestibular stimulus instruction

[0140] Il . . .IN.. However, in contrast, in the present embodiment each vestibular stimulation instruction comprises a sub-sequence of vestibular stimulation instructions IS 1 - 1 . . .ISN-2. Each subsequence IS1 . . .ISN includes a plurality of vestibular stimulation instructions IS1-1 . . .ISN-2. A number of vestibular stimulation instructions IS 1-1 . . .ISN-2 may be two, as shown, but may also be more, such as three, or ten, or any natural number.

[0141] The vestibular stimulation instructions IS 1 - 1 . . .ISN-2 comprised in a sub-sequence IS1 . . .ISN associated with a video frame Fl . . F2 are output at a higher output rate that the framerate of the video 101. For example, if the framerate of the video is 60 Hz, then, in the embodiment shown in Fig. 4, the output rate of the vestibular stimulus 201 may be 120 Hz.

[0142] By associating a plurality of vestibular stimulation instructions IS 1-1 . . .ISN-2 with a single video frame Fl . . .FN, abrupt changes in the generated vestibular stimulus, possibly leading to discomfort in the user, may be avoided.

[0143] Fig. 5 is a schematic illustration of a vestibular video enhancement device 1-2 comprising the vestibular stimulator 290.

[0144] In the embodiment shown in Fig. 5, the device 1-2, in addition to the control unit 5 also comprises the vestibular stimulator 190. The vestibular stimulator 290 is connected to the control unit 5 using a suitable data transmission means. For example, the vestibular stimulator 290 may be hardwired to the control unit 5. However, a data bus as described with respect to Fig. 1 may also be utilized.

[0145] As in the embodiment of Fig. 1, the control unit 5 controls the display 190 by providing a control signal and control the vestibular stimulator 290 by providing a synchronized control signal.

[0146] Fig. 6 is a schematic illustration of the vestibular video enhancement device 1-3 further comprising the vestibular stimulator 290 and the display 190.

[0147] In the present embodiment, the device 1-3 may, for example be configured as a virtual reality headset.

[0148] The display 190 is connected to the control unit 5 using a suitable data transmission means. For example, the display 190 may be hardwired to the control unit 5. However, a data bus as described with respect to Fig. 1 may also be utilized. As in the embodiment of Fig. 1, the control unit 5 controls the display 190 by providing a control signal and control the vestibular stimulator 290 by providing a synchronized control signal.

[0149] Fig. 7 is a schematic illustration of a vestibular video enhancement device 1-4 configured to generate the vestibular stimulation data 200.

[0150] In addition to the device 1-2 shown, for example, in Fig. 5, the device 1-4 according to the present embodiment further comprises a vestibular stimulus data generator 6.

[0151] Unlike in the device 1-2 of Fig. 5, the device 1-4 of Fig. 7 receives only video data 100 from an external source.

[0152] Otherwise the device 1-4 according to Fig. 7 functions like the device 1-2 of Fig. 5.

[0153] The vestibular stimulus data generator 6 analyses the video data to generate vestibular stimulus data 200 corresponding to a content 102 of the video 101. The analysis of the video data may comprise using object recognition and a physics engine, as described hereinabove.

[0154] The vestibular stimulus data generator 6 may be provided as circuitry separate from the control unit 5. The vestibular stimulus data generator 6 may also be provided as circuitry or as a logical computing unit executed by circuitry, such as the control unit 5. The vestibular stimulus data generator 6 may also be a virtual machine or the like.

[0155] The vestibular stimulus data generator 6 provides the vestibular stimulus data 200 to the control unit 5, whereupon the control unit 5 control the displaying of the video 101 by the display 190 and the generating of the vestibular stimulus 201 by the vestibular stimulator 290.

[0156] Note that, according to the present disclosure, the devices 1 according to Fig. 1, Fig. 5 and Fig. 6 may likewise be provided to include the vestibular stimulus data generator 6. Furthermore, the devices 1 of Fig. 1, Fig. 5, Fig. 6 and Fig. 7 may be combined in a single aggregate device.

[0157] Fig. 8 is a flow diagram of a method for generating vestibular stimulation data 200 from video data 100. This method is, executed by the vestibular stimulus data generator 6 of Fig. 7, for example.

[0158] The method starts with a set of video frames Fl . . .FN constituting the video data 100. In a recognition step S81, a predetermined entity is recognized with respect to the set of video frames Fl . . .FN. In a subsequent movement determination step S82, a movement of the predetermined entity in the set of video frames Fl . . .FN is determined. In a subsequent virtual model mapping step S83, the movement of the predetermined entity is mapped to a virtual model. In a subsequent physics modelling step S84, an acceleration affecting the predetermined entity in each video frame Fl . . .FN is calculated using a physics engine. The acceleration is stored as inertial data.

[0159] After mapping the inertial data to vestibular stimulus data, the method ends.

[0160] In vestibular stimulus mapping step S85, the inertial data is mapped to a corresponding vestibular stimulus vector V included vestibular stimulus data.

[0161] Note that the set of video frames Fl . . .FN may comprise all video frames Fl . . .FN constituting a video 101 or a small subset of video frames Fl . . .FN. The method provides for each video frame Fl . . .FN to be associated with a vestibular stimulus instruction II . . .IN.

[0162] As movement in a video 101 is spread over a plurality of video frames Fl . . .FN, it is insufficient to analyze a single frame Fl . . .FN to determine movement of the predetermined entity, and subsequently acceleration and vestibular stimulus instructions II . . .IN, for that video frame Fl . . .FN. Generating vestibular stimulus instructions II .. .IN for a specific video frame Fl . . .FN may comprise analysis of a set of video frames Fl . . .FN adjacent to the specific video frame F1...FN.

[0163] According to a first non-restrictive example, the vestibular stimulus instruction 12 associated with video frame F2 may be generated based on analysis of video frame Fl in addition to video frame F2.

[0164] According to a second non-restrictive example, the vestibular stimulus instruction 12 associated with video frame F2 may be generated based on analysis of video frames Fl and F3 in addition to video frame F2.

[0165] The predetermined entity may be an entity visible in the video 101. The predetermined entity may also be a camera that captured the video 101.

[0166] The predetermined entity may be chosen by the user of the device, by providing user input, or by an automated process.

[0167] Recognizing the predetermined entity with respect to the set of video frames Fl . . .FN should be understood to recognize where, in relation to an observer located in the same frame of reference as the camera that captured the video 101 the predetermined entity is located.

[0168] If the predetermined entity is the camera, then the predetermined entity is located at the same position as the observer. The camera will never be visible in the video 101. Thus, no further recognition is required to recognize the predetermined entity with respect to the set of video frames Fl . . .FN. However, if the camera is the predetermined object, object recognition may be employed to determine the acceleration of the camera.

[0169] If the predetermined entity is not the camera, then the predetermined entity is located not at the same position as the observer. In this case, the predetermined entity may be visible or become visible in the video 101. In this case, an object recognition step and an object localization step to recognize and localize the predetermined entity with respect to the set of video frames Fl . . .FN may be provided. The object localization analyses the video 101 to determine a relative position of the predetermined object with respect to the observer, i.e. the camera.

[0170] Both object recognition and localization for any purpose required by the present disclosure may utilize a neural network, machine learning or similar, as known to the skilled person.

[0171] Note that the movement of the predetermined entity refers to a movement of the entity in its frame of reference at the location and time that the video was captured, specifically a frame of reference, wherein the Earth is stationary and the entity is moving. The movement of the predetermined entity does not refer to a movement of the predetermined entity with respect to a position of the predetermined entity on the display 190 that the video 101 displayed on.

[0172] Fig. 9 is a symbolic illustration of two frames F1,F2 of a video 101 on a basis of which vestibular stimulation data 200 may be generated using the method of Fig. 8.

[0173] The first frame Fl is assumed to be captured at a time tl. The second frame F2 is assumed to be captured at a second time t2. The first image frame Fl includes visible objects Cl . . ,C4. In the second frame F2 the same objects Cl ... C4 are visible, but their positions in the frame have changed. Analysis of this change can be used to determine inertial data, as will be shown hereinbelow.

[0174] Fig. 10a and Fig. 10b are symbolic illustrations of a method for generating vestibular stimulation data 200 for a predetermined entity 15. Fig. 10a shows the same video frames F1,F2 as Fig. 9.

[0175] Fig. 10b is a symbolic illustration of the same scene as shown in Fig. 9, but shown in a lateral direction, including a camera 15 which captured the video frames Fl and F2 shown in Fig. 10a. The camera 15 is, in the present example, the predetermined entity 15.

[0176] Thus, the goal of the example shown in Fig. 10a and Fig. 10b is to generate vestibular stimulation data 200 based on the movement of the camera 15.

[0177] The method proceeds as follows: The camera 15 is recognized as the predetermined entity with respect to the set of video frames F1,F2.

[0178] In a first video frame, for example the first video frame Fl as shown in Fig. 10a, object recognition is performed to recognize an object that is visible in the video frame Fl. Preferentially, the object to be recognized is one of a list of objects that are likely to be stationary with respect to a surrounding of the camera 15 while being imaged by the camera 15. Object recognition may utilize object recognition software as known to the skilled person. In the example shown in Fig. 10a, the stationary object is assumed to be a tree Cl. Alternatively, the stationary object may be a cloud C2 or a house (not shown) or any other object that meets the criterion stated above.

[0179] A position P1(X1,Y1) of the stationary object Cl is determined in the image. The time tl of the first video frame Fl is stored. The position is a position in the two-dimensional plane of the image.

[0180] In a second video frame, for example the second video frame F2 as shown in Fig. 10a, object recognition is likewise performed to recognize the same object Cl as in the first video frame Fl.

[0181] A position P2(X2,Y2) of the stationary object Cl is determined in the image. The time t2 of the first video frame F2 is stored.

[0182] Based on optical characteristics of the camera 15, as known a priori, and estimates on a parallax of the object Cl, a rotation angle P of the camera 15 is determined.

[0183] The inertial data is then calculated by dividing the rotation angle P by the difference between the time tl of the first frame Fl and the time t2 of the second frame F2.

[0184] Fig. 11 is a symbolic illustration of two frames of a video 101 on a basis of which vestibular stimulation data 200 may be generated for a predetermined entity 15 visible in the video.

[0185] Here, unlike in Fig. 10a and 10b, the predetermined entity 15, for example a car 15, is an entity visible in the video 101.

[0186] Note that the predetermined entity is not limited to a car 15, but may be any object, animal or person that may be visible in a video 101.

[0187] Here, object recognition is employed to determine a first position, given by the coordinate values OXI and OY1 of the car 15 at a first time tl in the first frame Fl. Here, object recognition is employed to determine a second position, given by the coordinate values OX2 and OY2 of the car 15 at a second time t2 in the second frame F2. The first position and the second position are mapped into a virtual model. Using a physics engine, the acceleration affecting the car 15 between the first frame Fl and the second frame F2 is calculated.

[0188] In the present example, the acceleration may indicate a strong rotation about the z-axis as well as a weaker rotation about the y-axis and a medium rotation about the z-axis.

[0189] The inertial data is then mapped to a vestibular stimulus vector V.

[0190] Fig. 12 is a flow diagram of a method for synchronously displaying a video 101 and generating a vestibular stimulus 201 based on a received data stream. Note that the data stream comprises the video data 100 and the vestibular stimulus data 200 as described with respect to Fig. 2 or Fig. 4 above.

[0191] The method comprises a start of the method, a reception step S 121 , a control step S122, a decision step S123 and an end of the method.

[0192] The method may start once the control unit 5 receives a first packet of a data stream. In the reception step S 121 , a fraction of a data stream is received as data. The data stream may be provided by an external device.

[0193] It is to be noted that the data stream may only comprise a fraction of video data 100 and a corresponding fraction of vestibular stimulus data 200.

[0194] In the control step S122, output (i.e. a displaying) of the video 101 is controlled based on the video data 100 and, synchronously, output (i.e. a generating) of the vestibular stimulus 201 is controlled based on the vestibular stimulus data 200.

[0195] For example, the control unit 5 issues a control signal to the display 190 such that one video frame Fl . . .FN, e.g. the video frame Fl, of the video data 100 is displayed by the display 190. Synchronously, the control unit 5 issues a control signal to the vestibular stimulator 290 such that a vestibular stimulus corresponding to one vestibular stimulation instruction II . . .IN, e.g. the vestibular stimulation instruction II corresponding to the video frame Fl, is generated by the vestibular stimulator 290.

[0196] Synchronous control in the control step S122 may be effected by issuing the control signals in a cadence determined by a system clock of the control unit. Alternatively, in the reception step, the control unit may check whether each received video frame Fl . . .FN corresponds to a received vestibular stimulation instruction and then generate the control signals immediately for each tuple of received video frame Fl . . .FN and vestibular stimulation instruction II . . .IN. In the decision step S123, it is determined whether the data stream is over. For example, this may be checked by determining whether a stream end header has been received. However, the end of the data stream may be determined by any other suitable method known to the skilled person.

[0197] If it is determined in the decision step S123 that the data stream is over (“yes” in Fig. 12), the method ends. If it is determined in the decision step S123 that the data stream is not over (“no” in Fig. 12), the method reverts to the reception step S121 in which another fraction of the data stream is received.

[0198] Note that the data stream received in the reception step S121 may encompass an entire data file corresponding to a video 101 encoded as video data 100 and associated vestibular stimulus data 200.

[0199] The present method may, for example, be executed by the device 1 shown in Fig. 1, Fig. 5 and Fig. 6.

[0200] Fig. 13 is a flow diagram of a method for synchronously displaying a video 101, generating vestibular stimulus data 200 and generating a vestibular stimulus 201 based on a received video data 100 stream.

[0201] In contrast to the method shown in Fig. 12, in the method shown in Fig. 13 only video data 100 is received. The vestibular stimulus data 200 is generated based on the video data 100.

[0202] The present method comprises a start of the method, a reception step S131, a generation step S132 a control step S133, a decision step S134 and an end of the method.

[0203] The reception step S131 proceeds analogously to the reception step S121 of Fig. 12. However, only video data 100, e.g. video frames Fl . . .FN, are received.

[0204] In the generation step SI 32, based on the video data 100, corresponding vestibular stimulus data 200 is generated. The generation of vestibular stimulus data 200 uses the methods and techniques described with reference to Fig. 8 to Fig. 11 hereinabove.

[0205] Said methods of Fig. 8 to Fig. 11 require a plurality of frames to be used to generate vestibular stimulus instructions II .. .IN for a single video frame Fl . . .FN. In order to make a sufficient number of video frames Fl . . .FN available, the control unit may store one or more previously received video frames Fl . . .FN.

[0206] Once the vestibular stimulus data is generated in the generation step SI 32, the method proceeds analogously to the method described in Fig. 12, with the control step S133 of Fig. 13 corresponding to the control step S122 of Fig. 12 and the decision step S134 in Fig. 13 corresponding to the decision step S123 of Fig. 13.

[0207] Fig. 14 is an illustration of transmission of data from a distribution device 2 to the vestibular video enhancement device 1.

[0208] The distribution device 2 may provide the data stream of Fig. 12 and Fig. 13. The distribution device 2 may be a physical data processing device or a physical data storage device. The distribution device 2 may be a virtual resource, such as a virtual machine, a cloud storage device or the like.

[0209] The distribution device 2 is configured to be able to send data, such as video data 100 or vestibular stimulus data 200 to the device 1. The data may be distributed through a data link 50. The data link 50 may be an internet connection or a physical data bus or a logical data bus or a virtual data bus.

[0210] Fig. 15 is a flow diagram of a method for obtaining a calibration result 32. The calibration result 32 maps an intended stimulus to a strength of an applied stimulus. The intended stimulus may be a stimulus of a predetermined strength in a certain direction.

[0211] The calibration result is an example of user preference information. One purpose of user preference information is to prevent a vestibular stimulus 201 overwhelming a user and causing discomfort. Another purpose of user preference information is to prevent a vestibular stimulus 201 being generated at a strength that is too low to be felt by the user, thus decreasing enjoyment.

[0212] The method comprises a setting step S 151 , an application step S152, a user input step S153, an evaluation step SI 54, a conditional decreasing step SI 55, a conditional increasing step SI 56, and a storing step SI 57 leading to the generation of a calibration result 32.

[0213] This method may be used in a distinct calibration mode of the device 1. This calibration mode may be requested by the user through a user input command or may be entered automatically the first time the device 1 is activated by the user.

[0214] In the setting step SI 51 an intended strength of the stimulus is set. In the application step SI 52, the stimulus is applied with the set strength, i.e. a vestibular stimulus 201 is generated with the strength set in the setting step S 151. In the user input step S153, the user is asked to provide feedback on the strength of the effect the vestibular stimulus 201 has on the user’ s vestibular system 10. The feedback may be provided in the shape of a user input. For example, the user may be asked, if a low stimulus is applied, via an acoustic signal or a display prompt “Do you feel a stimulus?” or the like. If the user answers “no” then the stimulus 201 is too low to be felt.

[0215] Alternatively the user may be asked, if a high stimulus 201 is applied, via an acoustic signal or a display prompt “Do you feel comfortable?” or the like. If the user answers “no” then the stimulus 201 is too high to be comfortable. Corresponding questions may be asked with corresponding stimuli 201 to establish whether the stimulus 201 is too high, too low or comfortable. The aim is to arrive at a stimulation level that is both felt sufficiently strongly by the user but still comfortable.

[0216] In the evaluation step 154, the user input is evaluated. If the stimulation level is too high (“high” in Fig. 15), then the conditional decreasing step SI 55 follows the evaluation step SI 54. In the conditional decreasing step S155, the strength of the stimulus 201 is decreased. Following the conditional decreasing step SI 55, the application step SI 52 is repeated.

[0217] If the stimulation level is found to be too low in the evaluation step SI 54 (“low” in Fig. 15), then the conditional increasing step SI 56 follows the evaluation step SI 54. In the conditional increasing step SI 56, the strength of the stimulus 201 is increased. Following the conditional increasing step SI 56, the application step SI 52 is repeated.

[0218] If the stimulation level is found to be comfortable, i.e. neither too high nor too low (“comfortable” in Fig. 15), the thus retrieved stimulation strength is saved in the storing step SI 57. This may happen after a single iteration of the sequence of the application step SI 52, user input step SI 53 and evaluation step SI 54, or after a plurality of iterations of said sequence.

[0219] The stored result maps the intended stimulation set in the setting step SI 51 to the strength of the stimulus as applied during the most recent iteration. The result is the calibration result 32.

[0220] Note that the user preference information does not necessarily comprise the calibration result 32. Alternatively, user preference information may be input by the user. For example, the user may set an upper limit for the strength of the stimulus 201 or a lower limit for the strength of the stimulus 201.

[0221] Again alternatively, the calibration result 32 may be obtained by applying a stimulus 201 as described and then sensing, with an IMU, for example, whether a swaying motion of the user follows the application of the stimulus 201. Fig. 16a and 16b are illustrations of a vestibular stimulator 290 generating a vestibular stimulus 201 to stimulate a vestibular system 10 of a user.

[0222] Fig. 16a and Fig. 16b specifically each show a symbolic illustration of a head 11 of the user in a view from the front. The user’s vestibular system 10 is comprised by a left vestibular organ 10L and a right vestibular organ 10R. The vestibular stimulator 290 is shown as a galvanic vestibular stimulator 290 that comprises two distinct components. The first distinct component is the left component 290L of the vestibular stimulator 290. The second distinct component is the right component 290R of the vestibular stimulator 290. Each of the left component 290L and the right component 290R comprises an electrode that is configured to generate an electric current flowing from one of the components to the other of the components.

[0223] The left component 290L is positioned on a left side of the user’s head 11, for example, in a region of the left temple. The right component 290R is positioned on a right side of the user’s head 11, for example, in a region of the right temple.

[0224] Fig. 16a shows the components 290L,290R generating a left vestibular stimulus 201L, i.e. a vestibular stimulus 201 directed from the right component 290R, acting as an anode, to the left component 290L, acting as a cathode.

[0225] Fig. 16b shows the components 290L,290R generating a right vestibular stimulus 201R, i.e. a vestibular stimulus 201 directed from the left component 290L, acting as an anode, to the right component 290R, acting as a cathode. The vestibular stimulus may have a certain strength, as described hereinabove.

[0226] The vestibular stimulator 290 may comprise more components than the right component 290R and the left component 290L. The left and the right components 290L,290R may be positioned different from the shown arrangement.

[0227] Furthermore, for example, an upper component may be positioned on a forehead of the user and a lower component may be located at a chin of the user. A front and a back component may likewise be provided.

[0228] Note that the visual axis of the user, in the depiction of Fig. 16a and Fig. 16b would be oriented perpendicular to the plane of the figure.

[0229] Fig. 17 is an illustration of a method for generating the vestibular stimulus 201 based on vestibular stimulus data 200 and user preference information 300.

[0230] The user preference information 300 comprises, for example, the calibration result 32 of Fig. 15. Application of the user preference information is explained, for example, with reference to the first video frame Fl as shown in Fig. 17.

[0231] The first video frame Fl is associated with the first vestibular stimulation instruction II. The first vestibular stimulation instruction comprises, for example, an instruction to generate a stimulus 201 with a strength 3 (in units of a vestibular stimulation strength unit as discussed in Fig. 3) indicating a left rotation. This instruction is carried by the stimulation vector V as described hereinabove with reference to Fig. 3. The left rotation may be generated, for example, by a left stimulus 20 IL.

[0232] The user preference information 300 may include a map of a left stimulus 201L to be decreased in strength by 1 vestibular stimulation strength unit. This causes the vestibular stimulus 201 to be applied according to the vestibular stimulation instruction II with a strength 2 in the direction provided by the vestibular stimulation instruction II.

[0233] The application of the stimuli 201 according to the other frames proceeds accordingly. For example, in the second video frame F2, associated with the second vestibular stimulation instruction 12, a stimulus 201R in a right direction is applied with a strength 2.

[0234] In a further example, in the third video frame F3, associated with the third vestibular stimulation instruction 13, a stimulus 201F in a forward direction is applied with a strength 2. In a further example, in the n-th video frame FN, associated with the n-th vestibular stimulation instruction IN, a stimulus 201B in a backward direction is applied with a strength 8.

[0235] Fig. 18 is a flow diagram of a method for generating video data 100 and vestibular stimulus data 200 using an inertial measurement unit and a camera and streaming the video data 100 and vestibular stimulus data 200 to the vestibular video enhancement device 1-1, 1-2, 1-3.

[0236] The video data 100 and the vestibular stimulus data 200 may be streamed by the distribution device 2 of Fig. 14, which may comprise the camera and the inertial measurement unit.

[0237] The method of Fig. 18 comprises a data capturing step S 181 , a subsequent generation step SI 82, a subsequent encoding step S183, a subsequent streaming step S184 and a control step S185.

[0238] In the capturing step SI 81, video data 100 and inertial data is simultaneously captured. During the capturing, the camera capturing the video data 100 and the inertial measurement unit are in a same frame. This means that the video data 100 and the inertial data are captured simultaneously, but the video data 100 may be captured by the camera and the inertial measurement unit may positioned on an entity visible in the video 101. Alternatively, the camera and the inertial measurement unit may be moving in a same frame of reference.

[0239] Moving in the same frame of reference means that the inertial measurements unit is stationary with respect to the camera as the camera moves and vice versa. For example, the camera and the inertial measurement unit may be provided as parts of a common apparatus or be mounted in a common casing.

[0240] In the generation step SI 82, based on the inertial data, vestibular stimulation instructions II . . .IN are generated as described above. The generation step SI 82 results in video data 100 and corresponding vestibular stimulus data 200 as shown, for example, in Fig. 2 and Fig. 4 above.

[0241] In the encoding step SI 83, the video data 100 and vestibular stimulation data 200 are encoded in a suitable file format. Note that this step may be omitted. It is sufficient in the senso of the present disclosure to provide the video data 100 and vestibular stimulation data 200 in a sufficient data format, for example as two separate files, provided that the generation of the vestibular stimulus 201 can be synchronized to the content of the video.

[0242] The preceding steps (SI 81, SI 82, SI 83) are not necessarily executed by the vestibular video enhancement device 1-1, 1-2, 1-3.

[0243] In the streaming step 184, the video data 100 and vestibular stimulation data 200 is transferred to the vestibular video enhancement device 1-1, 1-2, 1-3. The vestibular video enhancement device 1-1, 1-2, 1-3 may store the received data before continuing to the next step.

[0244] In the control step 185, displaying of the video 101 based on the video data 100 and generating the vestibular stimulus 201 based on the vestibular stimulus data 200 is controlled by the vestibular video enhancement device 1-1, 1-2, 1-3.

[0245] The streaming step S184 and the control step S185 correspond to the method shown in Fig. 12 above.

[0246] Fig. 19 is a flow diagram of a method for generating video data 100 and vestibular stimulus data 200 using an inertial measurement unit and a camera and storing, then distributing, the video data 100 and vestibular stimulus data 200 to the vestibular video enhancement device 1-1, 1-2,1- 3.

[0247] The method of Fig. 19 is an alternative embodiment of the method of Fig. 18. The embodiment of Fig. 19 comprises a data capturing step S 191 , corresponding to the data capturing step 181 of Fig. 18, a subsequent generation step SI 92, corresponding to the generation step SI 82 of Fig. 18 and a subsequent encoding step S 183, corresponding the encoding step S183 of Fig. 18. Unlike the embodiment of Fig. 18, the embodiment of Fig. 19 comprises, after the encoding step S193, a storing step SI 94. In the storing step 194, the encoded video data 100 and vestibular stimulus data 200 is stored on a suitable digital storage medium. The digital storage medium may be a hard drive, a virtual storage resource or the like.

[0248] In a subsequent distribution step SI 95, the encoded video data 100 and vestibular stimulus data 200 is distributed in its entirety, as shown in Fig. 14, to the vestibular video enhancement device 1-1, 1-2, 1-3 and stored therein. The control step S196 corresponds to the control step S185 of Fig. 18. The distribution step S195 and the control step S196 correspond to the method shown in Fig. 12 above.

[0249] Note that the method shown in Fig. 19 may, instead of generating vestibular stimulus data 200 based on inertial data provided by an inertial measurement unit, provide for the vestibular stimulus data 200 to be generated according to the method of Fig. 8. Generation of the vestibular stimulus data 200 based on the method of Fig. 8 may, for example, be provided before the encoding step SI 94.

[0250] Fig. 20 is a flow diagram of a method for generating video data 100 using a camera and generating vestibular stimulus data 200 from the video data 100 by the vestibular video enhancement device 1-4. Here, the vestibular video enhancement device 1-4 is only provided with video data 100.

[0251] The method of Fig. 20 comprises a data capturing step S201, a subsequent encoding step S202, a subsequent first storing step S203, a subsequent distribution step S204 and a generation step S205, a subsequent second storing step 206 and a subsequent control step.

[0252] In the capturing step S201, video data 100 is captured. In the encoding step 202, the video data is encoded in a suitable video data format. In the first storing step 203, the encoded video data 100 is stored in a suitable digital storage medium. In the distribution step 204, the encoded video data 100 is distributed, as shown in Fig. 14, to the vestibular video enhancement device 1-4.

[0253] In the generation step S205, based on the video data 100, vestibular stimulation instructions

[0254] Il .. .IN are generated as described above in Fig. 8 by the vestibular video enhancement device 1- 4. The generation step S205 results in video data 100 and corresponding vestibular stimulus data 200 as shown, for example, in Fig. 2 and Fig. 4 above.

[0255] In the second storing step S206, the encoded video data 100 is stored in a suitable digital storage medium. Note that the second storing step S206 may be omitted. The control of displaying the video 101 and the control of generating the vestibular stimulus 201 proceed according to the method shown in Fig. 13.

[0256] The control step S207 corresponds to the control step SI 85 of Fig. 18 and to the control step S196 of Fig. 19.

[0257] Fig. 21a is a symbolic illustration of the vestibular video enhancement device 1 as an augmented reality device 21 worn on the head 11 of the user. Fig. 21b is a symbolic illustration of the vestibular video enhancement device 1 as a headset 22 worn on the head 11 of the user. Fig. 21c is an illustration of the vestibular video enhancement device 1 as or a plurality of head-mounted casings 20 or wearables 20 worn on the head 11 of the user.

[0258] In particular, the vestibular video enhancement device 1 may be provided in a configuration of two or more casings 20, each comprising one component 290R,290L of the vestibular stimulator 290. The control unit 5 may be comprised in one, some or all of the provided casings.

[0259] Fig. 22 shows a general configuration of circuitry 1200 according to the present disclosure. The circuitry may represent, for example, the control unit 5. The circuitry 1200 can include a CPU 1201, interacting with storage 1202. The storage 1202 can, for example, be a solid state disk (SSD). The device 1200 can further include a read-only-memory (RAM) 1203 interacting with the CPU 1201. The device can include a Bluetooth transceiver and decoder 1204 and an antenna and circuitry configured to interface with a wireless local area network (WLAN) 1205. The circuitry 1200 contains the vestibular stimulator interface 1213, which interfaces with the vestibular stimulator 290 and allows the CPU 1201 to control generating the vestibular stimulus 201 according to the embodiments described hereinabove. The circuitry 1200 can further include a loudspeaker array (not shown) capable of producing audible signals and a user interface 1212. The user interface 1212 may be used to acquire user input as required, for example, by the method of Fig. 15. The circuitry can further include a sensor array 1211 capable of sensing a user reaction. The circuitry further comprises a video control interface 1210, which interfaces with the display 190 and allows the CPU to control displaying of the video 101.

[0260] All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software.

[0261] In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure.

[0262] Note that the present technology can also be configured as described below.

[0263] (1) An electronic device 1 comprising circuitry configured to control, based on video data 100, a displaying of a video 101 by a display 190; and control, based on vestibular stimulus data 200, a generating of a vestibular stimulus 201, corresponding to a content 102 of the video 101, by a vestibular stimulator 290; wherein the generating of the vestibular stimulus 201 and the displaying of the video 101 are synchronized.

[0264] (2) The electronic device 1 according to (1), wherein the vestibular stimulus data 200 comprises a timed sequence of vestibular stimulation instructions II . . .IN; and wherein the video data 100, comprising timed sequences of frames Fl . . .FN, is associated with the vestibular stimulation instructions II .. .IN on a frame-by-frame basis.

[0265] (3) The electronic device according to (2), wherein each one of the frames Fl . . .FN of the video data 100 is associated with a timed sub-sequence IS1 . . .ISN of the vestibular stimulation instructions I1-1...IN-2.

[0266] (4) The electronic device 1 according to any of (2) to (3), wherein each one of the vestibular stimulation instructions II . . .IN comprises a stimulation vector V that indicates a space-direction of the vestibular stimulus 201 to be generated.

[0267] (5) The electronic device according to (4), wherein the stimulation vector V further indicates a strength of the vestibular stimulus 201 to be generated.

[0268] (6) The electronic device according to any of (1) to (5), wherein the circuitry is further configured to generate, based on the content 102 of the video 101, the vestibular stimulus data 200.

[0269] (7) The electronic device according to (6), wherein the circuitry is further configured to generate the vestibular stimulus data based on the content of the video by determining, from the content of the video, using a physics engine simulating a virtual model of the content of the video, inertial data of a predetermined entity, and generating the vestibular stimulus data based on the inertial data; wherein the predetermined entity 15 is an entity visible in the video or the camera that recorded the video.

[0270] (8) The electronic device 1 according to (7), wherein the circuitry is further configured to generate the vestibular stimulus data 200 based on a movement of an entity visible in the video 101 as the predetermined entity 15 in the video 101 in a frame of reference of the surrounding of the predetermined entity 15, or generate the vestibular stimulus data 200 based on a movement of the camera that recorded the video as the predetermined entity 15 in a frame of reference of the surrounding of the camera that recorded the video.

[0271] (9) The electronic device 1 according to any of (1) to (8), wherein the circuitry is further configured to receive, from an external distribution device 2, the video data 100; wherein the receiving of the video data 100; the generating of the vestibular stimulus data 200, the generating of the vestibular stimulus 201 and the displaying of the video 101 are synchronized.

[0272] (10) The electronic device 1 according to any of (1) to (9), wherein the circuitry is further configured to receive, from an external distribution device 2, the video data 100 and the vestibular stimulus data 200; wherein the receiving of the video data 200 and the vestibular stimulus data 200, the generating of the vestibular stimulus 201 and the displaying of the video 101 are synchronized.

[0273] (11) The electronic device 1 according to any of (1) to (10), wherein the circuitry is configured to further generate the vestibular stimulus 201 based on user preference information 300.

[0274] (12) The electronic device 1 according to any of (1) to (11), further comprising the vestibular stimulator 290.

[0275] (13) The electronic device 1 according to (12), wherein the vestibular stimulator 290 is a galvanic vestibular stimulator 290, comprising two or more electrodes 290R,290L, that are worn on a head 11 of a user.

[0276] (14) The electronic device according to any of (1) to (13), wherein the electronic device is any one of a headset 22, a head-mounted wearable device 20, a virtual reality device or an augmented reality device 21.

[0277] (15) A vestibular stimulation method comprising: control, based on video data 101, a displaying of a video 101 by a display 190; and control, based on vestibular stimulus data 200, a generating of a vestibular stimulus 201, corresponding to a content 102 of the video 101, by a vestibular stimulator 290; wherein the generating of the vestibular stimulus 201 and the displaying of the video 101 are synchronized.

[0278] (16) The method according to (15), further comprising: receive, from an external distribution device 2, the video data 100 and the vestibular stimulus data 200; wherein the receiving of the video data 100 and the vestibular stimulus data 200, the generating of the vestibular stimulus 201 and the displaying of the video 101 are synchronized.

[0279] (17) The method according to any of (15) to (16), further comprising: receive, from an external distribution device 2, the video data 100 and generate, based on the content 102 of the video 101, the vestibular stimulus data 200; wherein the receiving of the video data 100, the generating of the vestibular stimulus data 200, the generating of the vestibular stimulus 201 and the displaying of the video 101 are synchronized.

[0280] (18) The method according to any of (15) to (17), further comprising: record, using a camera 15, the video data 100; and synchronously record, using an inertial measurement unit in a same as the camera 15, inertial data; and generate, based on the inertial data, the vestibular stimulus data 200; and store the video data 100 and the vestibular stimulus data 200 in the external distribution device 2.

[0281] (19) The method according to any of (15) to (18), further comprising: record, using a camera 15, the video data 100; and generate the vestibular stimulus data 200 by determining, from the video data 100, using a physics engine, inertial data, and generating the vestibular stimulus data 200 based on the inertial data.

[0282] (20) A computer program comprising program code causing a computer to perform the method according to anyone of (15) to (19), when being carried out on a computer.

[0283] (21) A non-transitory computer-readable recording medium that stores therein a computer program product, which, when executed by a processor, causes the method according to anyone of (15) to (19) to be performed.

[0284] LIST OF REFERENCE SIGNS

[0285] 1 vestibular video enhancement device

[0286] 2 external distribution device

[0287] 11 head of user

[0288] 15 predetermined entity

[0289] 20 head-mounted wearable device

[0290] 21 augmented reality device

[0291] 22 headset

[0292] 100 video data 101 video

[0293] 102 content of the video

[0294] 190 display

[0295] 200 vestibular stimulus data 201 vestibular stimulus

[0296] 290 vestibular stimulator

[0297] 300 user preference information

[0298] Il . . .INvestibular stimulation instructions

[0299] Fl ... FN video frames IS 1 . . . ISN sub-sequences of vestibular stimulation instructions

[0300] V stimulation vector

Claims

CLAIMS1. An electronic device comprising circuitry configured to control, based on video data, a displaying of a video by a display; and control, based on vestibular stimulus data, a generating of a vestibular stimulus, corresponding to a content of the video, by a vestibular stimulator; wherein the generating of the vestibular stimulus and the displaying of the video are synchronized.

2. The electronic device according to claim 1, wherein the vestibular stimulus data comprises a timed sequence of vestibular stimulation instructions; and wherein the video data, comprising timed sequences of frames, is associated with the vestibular stimulation instructions on a frame-by-frame basis.

3. The electronic device according to claim 2, wherein each one of the frames of the video data is associated with a timed sub-sequence of the vestibular stimulation instructions.

4. The electronic device according to claim 2, wherein each one of the vestibular stimulation instructions comprises a stimulation vector that indicates a space-direction of the vestibular stimulus to be generated.

5. The electronic device according to claim 2, wherein the stimulation vector further indicates a strength of the vestibular stimulus to be generated.

6. The electronic device according to claim 1, wherein the circuitry is further configured to generate, based on the content of the video, the vestibular stimulus data.

7. The electronic device according to claim 6, wherein the circuitry is further configured to generate the vestibular stimulus data based on the content of the video by determining, from the content of the video, using a physics engine simulating a virtual model of the content of the video, inertial data of a predetermined entity, and generating the vestibular stimulus data based on the inertial data; wherein the predetermined entity is an entity visible in the video or the camera that recorded the video.

8. The electronic device according to claim 7, wherein the circuitry is further configured to generate the vestibular stimulus data based on a movement of an entity visible in the video as the predetermined entity in the video in a frame of reference of the surrounding of the predetermined entity, or generate the vestibular stimulus data based on a movement of the camera that recorded the video as the predetermined entity in a frame of reference of the surrounding of the camera that recorded the video.

9. The electronic device according to claim 6, wherein the circuitry is further configured to receive, from an external distribution device, the video data; wherein the receiving of the video data; the generating of the vestibular stimulus data, the generating of the vestibular stimulus and the displaying of the video are synchronized.

10. The electronic device according to claim 1, wherein the circuitry is further configured to receive, from an external distribution device, the video data and the vestibular stimulus data; wherein the receiving of the video data and the vestibular stimulus data, the generating of the vestibular stimulus and the displaying of the video are synchronized.

11. The electronic device according to claim 1, wherein the circuitry is configured to further generate the vestibular stimulus based on user preference information.

12. The electronic device according to claim 1, further comprising the vestibular stimulator.

13. The electronic device according to claim 12, wherein the vestibular stimulator is a galvanic vestibular stimulator, comprising two or more electrodes, that are worn on a head of a user.

14. The electronic device according to claim 1, wherein the electronic device is any one of a headset, a head-mounted wearable device, a virtual reality device or an augmented reality device.

15. A vestibular stimulation method comprising: control, based on video data, a displaying of a video by a display; and control, based on vestibular stimulus data, a generating of a vestibular stimulus, corresponding to a content of the video, by a vestibular stimulator; wherein the generating of the vestibular stimulus and the displaying of the video are synchronized.

16. The method according to claim 15, further comprising:receive, from an external distribution device, the video data and the vestibular stimulus data; wherein the receiving of the video data and the vestibular stimulus data, the generating of the vestibular stimulus and the displaying of the video are synchronized.

17. The method according to claim 15, further comprising: receive, from an external distribution device, the video data and generate, based on the content of the video, the vestibular stimulus data; wherein the receiving of the video data, the generating of the vestibular stimulus data, the generating of the vestibular stimulus and the displaying of the video are synchronized.

18. The method according to claim 16, further comprising: record, using a camera, the video data; and synchronously record, using an inertial measurement unit in a same as the camera, inertial data; and generate, based on the inertial data, the vestibular stimulus data; and store the video data and the vestibular stimulus data in the external distribution device.

19. The method according to claim 15, further comprising: record, using a camera, the video data; and generate the vestibular stimulus data by determining, from the video data, using a physics engine, inertial data, and generating the vestibular stimulus data based on the inertial data.

20. A computer program that, if executed by a computer, causes the computer to control, based on video data, a displaying of a video by a display; and control, based on a vestibular stimulus data, a generating of a vestibular stimulus, corresponding to a content of the video, by a vestibular stimulator; wherein the generating of the vestibular stimulus and the displaying of the video are synchronized.

Images