Apparatus and method for audiovisual presentation

Abstract

An apparatus comprises a receiver (101) that receives audiovisual data representing a scene. A source (105, 107) provides a vehicle motion signal indicative of motion of the vehicle and a relative user motion signal indicative of motion of a user relative to the vehicle. A predictor (109) generates a predicted relative user motion signal by applying the prediction model to the vehicle motion signal. A residual signal generator (111) generates a residual user motion signal indicating a residual difference between the predicted relative user motion and the received relative user motion. A view gesture determiner (113) determines view gestures having different dependencies on the predicted relative user motion signal and the residual user motion signal. A renderer (103) renders an audiovisual signal for the view gesture in accordance with the audiovisual data. The method may provide an enhanced user experience, which may compensate for or include the effects of user motion caused by vehicle motion.

Description

Technical Field

[0001] The present invention relates to an apparatus and method for audiovisual presentation, and in particular, but not exclusively, to presenting audiovisual signals for an extended reality application to a user experiencing vehicle motion. Background Art

[0002] In recent years, the variety and range of image and video applications have increased dramatically with the continued development and introduction of new services and ways to utilize and consume video.

[0003] For example, an increasingly popular service is to provide image sequences in such a way that the viewer can actively and dynamically interact with the system to change the parameters of the presentation. A very attractive feature in many applications is the ability to change the viewer's effective viewing position and viewing direction, for example allowing the viewer to move and look around the presented scene.

[0004] Such a feature may specifically allow providing a user with a virtual reality experience. This may allow the user, for example, to move (relatively) freely in a virtual environment and to dynamically change his position and where he is looking. Typically, such virtual reality (VR) applications are based on a three-dimensional model of a scene, wherein the model is dynamically evaluated to provide a specific requested view. This approach is well known from, for example, gaming applications for computers and consoles, such as in the category of first-person shooter games. Other examples include augmented reality (AR) or mixed reality (MR) applications. Such applications are often referred to as extended reality (XR) applications.

[0005] An important feature in many, for example, XR applications is to determine user movement in the real world and adjust the audiovisual representation of virtual features to reflect the user movement.

[0006] Detection of user movement can include what is known as outside-in and inside-out tracking. Outside-in VR tracking uses a camera or other sensor placed in a stationary position and oriented toward a tracked object (e.g., a headset) that is free to move within a predetermined area covered by the sensor.

[0007] Inside-out tracking distinguishes itself from outside-in tracking, where sensors are attached to the object (e.g., integrated into a headset). Image data, as captured by multiple camera sensors, is used to reconstruct 3D features in the visual surrounding world. The headset is systematically positioned relative to the world, assuming that the world is static. Beside the cameras, other sensors, such as accelerometers, can be used to improve the accuracy and robustness of the headset pose estimate.

[0008] When consuming A / V content in a stationary environment with a VR headset or headphones, head rotation and optionally translation (e.g., 3DoF or 6DoF (degrees of freedom)) are continuously measured (e.g., using one or more cameras). The measured motion is fed back to the rendering process so that the user's motion relative to the real world is taken into account. For example, when the user turns his head to the left, the relevant object will turn to the right. When the user moves slightly to the side, the relevant object is rendered, enabling the user to view from the side, and even surround the object to view previously occluded objects. An important assumption for the correct operation of such a system is that the world objects used as reference to determine the user's movement do not themselves move (referred to as independent motion in the context of visual structure from motion algorithms). For example, if the world reference system would move but not move the user, the user would observe a moving virtual scene when the user is not moving. This would be an unwanted effect. Therefore, inside-out tracking needs to be robust to independently moving objects in the real-world scene surrounding the user.

[0009] It has been proposed to provide XR applications and services that are not only suitable for consumption in a mobile environment, but can also be adapted to reflect or compensate for movement. However, this provides many challenges and difficulties, and in particular the preferred operation and experience may depend on the specific expectations and preferences of the individual application.

[0010] In some cases, when a user is traveling in a mobile environment (e.g., a train or car), it may be desirable not to classify the vehicle as an independently moving object. For example, visual features inside the vehicle may be considered to define a reference system, and the motion of the world outside the vehicle (relative to the vehicle) may be ignored.

[0011] If the vehicle is travelling at a purely constant speed, this may be relatively easy to achieve. However, once the vehicle changes speed or vibrates, it may introduce motion differences between the occupants and the vehicle's internal reference system (typically determined by the vehicle's visual appearance and geometry), which may lead to complex relationships that may significantly complicate the operation of the application. Typically, such motion differences may be picked up by inside-out tracking devices and used to render the virtual scene. Such effects may result in a limited and / or suboptimal user experience, and may, for example, result in a rendering that is not fully consistent with the perceived motion.

[0012] Furthermore, operations and algorithms that take into account such more complex motions are often suboptimal in terms of complexity, accuracy, usage of computational resources, etc.

[0013] Therefore, improved methods for audiovisual presentation suitable for users experiencing vehicle motion would be advantageous. In particular, methods allowing improved operation, increased flexibility, improved user experience, reduced complexity, facilitated implementation, improved presentation quality, improved and / or facilitated presentation, improved and / or facilitated adaptation to user and vehicle motion and / or improved performance and / or operation would be advantageous. Summary of the invention

[0014] Accordingly, the Invention seeks to preferably mitigate, alleviate or eliminate one or more of the above mentioned disadvantages singly or in any combination.

[0015] According to an aspect of the present invention, there is provided an apparatus for audio-visual presentation, the apparatus comprising: a receiver arranged to receive audio-visual data representing a scene; a first source providing a vehicle motion signal indicating the motion of a vehicle; a second source providing a relative user motion signal indicating the motion of a user relative to the vehicle; a predictor arranged to generate a predicted relative user motion signal by applying a prediction model to the vehicle motion signal; a residual signal generator arranged to generate a residual user motion signal indicating at least a component of a difference between the relative user motion signal and the predicted relative user motion signal; a view pose determiner arranged to determine a view pose based on the residual user motion signal and the predicted relative user motion signal, the dependence of the view pose on the residual user motion signal being different from the dependence of the view pose on the predicted relative user motion signal; and a renderer arranged to render an audio-visual signal for the view pose along with the audio-visual data.

[0016] The invention may allow, in many applications and scenarios, improved generation of audiovisual signals for users subject to vehicular motion. It may provide improved user perception of a scene in many scenarios and applications and may, for example, provide a more desirable user experience. The method may specifically allow differentiated adaptation of the presented audiovisual experience to different types of motion. For example, it may allow a user experience that is less affected by vehicular motion and may be perceived as more similar to the experience in a scenario where the user is not subject to vehicular motion.

[0017] In some scenarios, the method may, for example, provide more flexible and / or improved compensation for the effects of vehicle movement. For example, it may provide an experience in which the presented audiovisual signals may be compensated or adjusted according to the vehicle movement, such that, for example, motion-induced sickness (which is often caused by conflicts between sensory inputs from different senses, such as balance and vision) or discomfort may be reduced, while still allowing the user's intended movement to be determined and accounted for in the presentation.

[0018] For example, it may allow for adaptation of a VR or XR experience where some user motion may be reflected in the audiovisual cues (images and / or sounds) provided to the user, while some compensation for vehicle motion and its impact on the user may be compensated for.

[0019] In many scenarios, efficient and / or low complexity and / or resource demanding operations may be achieved.

[0020] The method can be used, for example, as a specific example, for a situation where a user may be traveling in a vehicle while trying to use a VR application that presents a 3D image to the user. The 3D image can show a view of a 3D scene, and when the user moves his head, the view of the scene can change accordingly. For example, if the user turns his head to the left, the 3D image is updated to show the view of the scene that the person in the scene will see by turning his head to the left. However, if the user is in a moving vehicle, his head movement may reflect different components. One is an intentional movement actively performed by the user, such as moving his head to the side. However, another component is the movement of the user's head caused by the movement of the car. For example, crossing a speed bump may cause the head of a passenger in a car to make a sudden vertical movement relative to the car. This movement is involuntary and can be simply determined by the mechanical dynamics of the car (seat suspension, etc.) and the user's body (e.g., the elasticity provided by the user's neck). Therefore, the movement of the vehicle may have an impact on the user, which causes the user's head to move relative to the vehicle. If such a movement is detected by the VR headset and the image is adjusted accordingly, the user may experience an unreal virtual world experience. The apparatus for audio-visual presentation may provide improved performance and an improved user experience in many such / similar scenarios and situations. The apparatus may be arranged to distinguish between movements caused intentionally by the user (e.g. by turning his head) or by the movement of the vehicle, without the user intending to make such head movements. The presentation of the scene and in particular the view pose of the user in the scene may then be reflected as having different dependencies on intentional / voluntary movements and movements that are involuntary / unintentional and occur simply as a function of the movement of the vehicle.

[0021] In many embodiments, the view pose determiner may be arranged to determine the view pose signal from the residual user motion signal and the predicted relative user motion signal, the dependence of the view pose on the residual user motion signal being different from the dependence of the view pose on the predicted relative user motion signal. The renderer may be arranged to render an audio-visual signal of (a view pose of) the view pose signal from the audio-visual data.

[0022] The user movement may specifically indicate movement of the user's head (or possibly) eyes. The movement may be a time derivative of a gesture. A signal may be one or more values ​​having a time aspect / correlated with time / which may vary over time.

[0023] According to an optional feature of the invention, the prediction model comprises temporal filtering of the vehicle motion signal.

[0024] In many embodiments this may provide particularly advantageous operation and / or presentation. It may allow predictions that are particularly suitable for adjusting the presentation of the audiovisual signal to reflect the user's movement for a user experiencing vehicle movement. The temporal filtering may be a high pass filtering of the vehicle motion signal.

[0025] According to an optional feature of the invention, the predictor is arranged to apply temporal filtering to the relative user motion signal to generate a filtered relative user motion signal, and to predict the predicted relative user motion signal from the filtered relative user motion signal.

[0026] In many embodiments, this may provide particularly advantageous operation and / or presentation. It may allow for more accurate prediction of relative user motion resulting, for example, directly from vehicle motion.

[0027] According to an optional feature of the invention, the predictive model comprises a biomechanical model.

[0028] In many embodiments, this may provide particularly advantageous operation and / or presentation. It may allow for more accurate prediction of relative user motion resulting, for example, directly from vehicle motion.

[0029] According to an optional feature of the invention, the view pose determiner is arranged to apply a different weighting to the predicted relative user motion signal than to the residual user motion signal.

[0030] This may provide particularly advantageous effects in many embodiments and scenarios, and may in particular provide improved adaptation and differentiation of different movements.

[0031] According to an optional feature of the invention, the view pose determiner is arranged to apply a different temporal filtering to the predicted relative user motion signal than to the residual user motion signal.

[0032] This may provide particularly advantageous effects in many embodiments and scenarios, and may in particular provide improved adaptation and differentiation of different movements.

[0033] According to an optional feature of the invention, the view pose determiner is arranged to determine a first view pose contribution from the predicted relative user motion signal and a second view pose contribution from the residual user motion signal, and to generate the view pose by combining the first view pose contribution and the second view pose contribution.

[0034] This can provide advantageous operation in many scenarios.

[0035] According to an optional feature of the invention, the view pose determiner is arranged to extract a first motion component from the predicted relative user motion signal by attenuating temporal frequency, and to determine the view pose to include a contribution from the first motion component.

[0036] This can provide advantageous operation in many scenarios.

[0037] According to an optional feature of the invention, the view pose determiner is arranged to detect a user gesture motion component in the residual user motion signal, and to extract a first motion component from the residual user motion signal based on the user gesture motion component, and to determine the view pose to include a contribution from the user gesture motion component.

[0038] This can provide advantageous operation in many scenarios.

[0039] According to an optional feature of the invention, the predictor is arranged to determine the predicted relative user motion signal in response to a correlation of the relative user motion signal and the vehicle motion signal.

[0040] This can provide advantageous operation in many scenarios.

[0041] According to an optional feature of the invention, at least one of the predicted relative user motion signal and the residual user motion signal indicates multiple posture components, and the view posture determiner is arranged to determine a view posture having different dependencies on at least a first posture component and a second posture component of the multiple posture components.

[0042] This can provide advantageous operation in many scenarios.

[0043] According to an optional feature of the invention, the receiver is arranged to receive processing data indicative of processing of at least one of the predicted relative user motion signal and the residual user motion signal; and the view pose determiner is arranged to determine the view pose in accordance with processing instructions of the processing data.

[0044] This can provide advantageous operation in many scenarios.

[0045] According to an optional feature of the invention, the predictor is arranged to generate the predicted relative user motion signal from the residual user motion signal.

[0046] In many embodiments, this may provide particularly advantageous operation and / or presentation. It may allow for more accurate prediction of relative user motion resulting, for example, directly from vehicle motion.

[0047] According to one aspect of the present invention, a method for audio-visual presentation is provided, the method comprising: receiving audio-visual data representing a scene; providing a vehicle motion signal indicating the movement of a vehicle; providing a relative user motion signal indicating the movement of a user relative to the vehicle; generating a predicted relative user motion signal by applying a prediction model to the vehicle motion signal; generating a residual user motion signal indicating at least a component of the difference between the relative user motion signal and the predicted relative user motion signal; determining a view pose based on the residual user motion signal and the predicted relative user motion signal, the dependence of the view pose on the residual user motion signal being different from the dependence of the view pose on the predicted relative user motion signal; and presenting an audio-visual signal for the view pose based on the audio-visual data.

[0048] These and other aspects, features and advantages of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which

[0050] Figure 1 illustrates an example of a presentation apparatus for generating an audio-visual signal representing a scene for a user experiencing vehicle motion according to some embodiments of the present invention;

[0051] Figure 2 illustrates an example of a user's motion during a vehicle turn; and

[0052] Figure 3 The diagram shows the Figure 1 An example of a processor of a device. DETAILED DESCRIPTION

[0053] The following description will focus on examples of audiovisual presentations of extended reality applications for users experiencing vehicular motion, such as, for example, users in cars, trucks, airplanes, ships, trains, etc. The audiovisual presentation may be a presentation of audio and / or video, and may specifically be a presentation of an audiovisual signal that includes both audio and video data.

[0054] Figure 1 An example of a presentation apparatus arranged to generate and present an audiovisual signal representing a scene as perceived from a given view pose and, in effect, from different varying view poses is illustrated. In particular, the presentation apparatus may be arranged to receive (directly or indirectly) motion input indicative of a change in user pose as a function of time, to determine a view pose signal (a time sequence of view poses), and to present / generate audio and / or video data representing the scene from the view pose.

[0055] In the art, the terms placement and pose are used as common terms for position and / or direction / orientation. For example, the combination of the position and direction / orientation of an object, camera, head or view can be referred to as a pose or placement. Therefore, a placement or pose indication can include six values / components / degrees of freedom (6DoF), wherein each value / component generally describes an individual property of the position / position or orientation / orientation of the corresponding object. Of course, in many cases, for example, if one or more components are considered to be fixed or unrelated (for example, if all objects are considered to be at the same height and have a horizontal orientation, four components can provide a complete representation of the pose of the object), then the placement or pose can be considered or represented with fewer components. In the following, the term pose is used to refer to a position and / or orientation that can be represented by one to six values ​​(corresponding to the maximum possible degrees of freedom). The term pose can be replaced by the term placement. The term pose can be replaced by the term position and / or orientation. The term pose can be replaced by the term position and orientation (if the pose provides information on both position and orientation), the term position (if the pose provides information on (possibly only) position) or the orientation (if the pose provides information on (possibly only) orientation).

[0056] A motion may be a sequence of postures, and in particular a temporal sequence of posture changes / variations. A motion may represent one or more components of a posture. A motion may be a sequence / variation of orientations and / or positions. A motion (of an object) may be a change of orientation and / or position (of an object) with respect to time. A motion (of an object) may be a change of posture (of an object) with respect to time.

[0057] The rendering device is arranged to receive audiovisual data representing a scene and to render an audiovisual signal from the data reflecting an audio and / or visual perception of the scene from a view posture of the scene. It may proceed to generate an audiovisual signal representing the scene from a view posture that depends on the movement of a user in and relative to the vehicle. The device may seek to distinguish between movements that are intentionally / voluntarily caused by the user and movements that are involuntary and caused by the movement of the vehicle. The view posture for which the audiovisual signal is generated may then depend differently on voluntary / intentional / known movements and involuntary / unknown movements that are caused by the movement of the vehicle without the user intentionally causing the movement. This approach may provide a significantly improved user experience for VR applications used, for example, by a user in a vehicle.

[0058] In some embodiments, the audiovisual data may be audio data, and the generated audiovisual signal may be an audio signal providing an auditory representation of the scene from the view gesture. In some embodiments, the audiovisual data may be visual / video data, and the generated audiovisual signal may be a visual / video signal providing a visual representation of the scene from the view gesture. In many embodiments, the presentation apparatus may be arranged to generate an audiovisual signal comprising audio and visual data representing the visual and audio perception of the scene from the view gesture (and thus the received audiovisual data representing the scene may be video and audio data).

[0059] The presentation device comprises a receiver 101, which is arranged to receive audio-visual data representing a scene. The receiver 101 can provide a three-dimensional image and / or audio data, which provides a representation of the three-dimensional scene. Hereinafter, for the sake of brevity, the audio-visual data received by the receiver 101 will also be referred to as scene data.

[0060] The receiver 101 may, for example, include a storage device or memory that stores scene data and from which the scene data may be retrieved. In other embodiments, the image data source 101 may receive or retrieve audiovisual data from any external (or other internal source). In many embodiments, the receiver 101 may, for example, receive from a video and / or audio capture system that includes a camera and microphone (array) that captures a real-world scene (e.g., in real time).

[0061] Scene data can use suitable image or common video format / representation and / or suitable three-dimensional audio format / representation to provide suitable representation of scene.In some embodiments, receiver 101 can receive scene data from different sources and / or in different formats, and it can generate suitable data for presentation, such as by converting or processing received scene data.For example, image and depth can be received from remote camera, and can be processed to generate video representation according to given format.In some embodiments, receiver 101 can be arranged to generate three-dimensional image data by evaluating the model of scene.

[0062] The scene data is provided according to a suitable three-dimensional data format / representation. The representation may, for example, include multi-view and depth, multi-layer (multi-plane, multi-spherical), mesh model and / or point cloud representation. Furthermore, in the described examples, the scene data may specifically be video data including a temporal component. Similarly, the scene data may include a suitable three-dimensional audio representation, such as a multi-channel representation, a spatial audio representation based on audio objects, etc.

[0063] The receiver 101 is coupled to a renderer 103, which is arranged to generate an audio-visual signal providing a representation / view / audio from a view gesture for a scene. In many embodiments, the renderer 103 is arranged to generate an audio-visual signal comprising both audio and one or more images. In particular, in many XR applications, a continuous series of images and audio will be generated representing the scene from different view images as the user changes gestures.

[0064] It should be appreciated that those skilled in the art will be aware of many different techniques, algorithms and methods for generating such audio and / or images / video, and that any suitable method of renderer 103 may be used without departing from the invention.

[0065] From the perspective of an image, the operation of the renderer 103 will be described below with reference to the generation of a single image. However, it should be understood that in many embodiments, the image may be part of a sequence of images, and in particular may be a frame of a video sequence. In fact, the described method may be applied to generate multiple and typically all frames / images of an output video sequence.

[0066] It should be understood that a stereoscopic video sequence can often be generated that includes a video sequence for the right eye and a video sequence for the left eye. Thus, if the image is presented to the user, for example, via an AR / VR headset, it will appear as if the three-dimensional scene is seen from the viewing posture. In another example, the image is presented to the user using a tablet with a tilt sensor, and a monoscopic video sequence can be generated. In yet another example, multiple images are woven or tiled for presentation on an autostereoscopic display.

[0067] The renderer 103 may perform appropriate image synthesis / generation operations to generate a view image of a particular representation / format of the three-dimensional image data (or indeed in some embodiments the three-dimensional image data may be converted into a different format to generate the view image).

[0068] For example, for a multi-view + depth representation, the renderer 103 may typically be arranged to perform a view shift or projection of the received multi-view image based on the depth information. This will typically include techniques such as offsetting pixels (changing pixel positions to reflect the appropriate disparity corresponding to the disparity change), de-occlusion (typically based on padding from other images), combining pixels from different images, etc., as will be known to those skilled in the art.

[0069] It will be appreciated that many algorithms and methods are known for synthesizing images from different three-dimensional image data formats and representations, and that the renderer 103 may use any suitable method.

[0070] Examples of suitable view synthesis algorithms can be found, for example, in:

[0071] "A review on image-based rendering" (Yuan HANG, Guo-Ping ANG, VirtualReality&Intelligent Hardware, Vol1, Issue1, February2019, Pages39-54, httpspstheli.idoi.org10.3724 / SP.J.2096-5796.2018.0004)

[0072] "A Review of Image-Based Rendering Techniques" (Shum; Kang, Proceedings of SPIE-The International Society for Optical Engineering4067:2-13, May2000, DOI:10.1137 / 12.386541)

[0073] Or, for example from the Wikipedia article on 3D rendering:

[0074] https: / / en.wikipedia.org / wiki / 3D_rendering

[0075] Similarly, from an audio perspective, many different spatial audio rendering methods are known, including, for example, surround sound algorithms, room / head related transfer function (HRTF) based algorithms, etc. Examples can be found, for example, in:

[0076] "Sound Rendering" (Takala; Tapio; James, Hahn.SIGGRAPHComput.Graph.Vol.26.pp.211-220.doi:10.1145 / 133994.134063.ISBN978-0897914796.)

[0077] "Techniques for Low Cost Spatial Audio by Burgess; David A, Proceedings of the 5th Annual ACM Symposium on User Interface Software andTechnology" (pp.53-59, doi:10.1145 / 142621.142628.)

[0078] "Psychoacoustic Music Sound Field Synthesis"(Ziemer,Tim,CurrentResearch in Systematic Musicology.Vol.7.Cham:Springer.p.287.doi:10.1007 / 978-3-030-23033-3.)

[0079] exist Figure 1 In an example of , the rendering device is arranged to dynamically modify and update the view gesture for which the renderer 103 renders images and / or sounds to reflect the user motion / movement. The rendering device is also arranged to provide an audio-visual output signal suitable for the user in the vehicle, and it can provide an improved user experience and operation for such a scenario. In particular, the rendering device is arranged to take into account multiple movements when determining the view gesture for which the audio-visual signal is generated, rather than determining the view gesture for which the audio-visual signal is presented based on a single gesture / motion.

[0080] The presentation device comprises a first motion source 105 arranged to provide a vehicle motion signal indicative of a vehicle motion.The first vehicle motion signal may provide information about changes in at least one position or orientation component of the vehicle over time.

[0081] In some embodiments, the first motion source 105 may be coupled to or include one or more sensors of the vehicle (attached or otherwise ensured to be co-located with the vehicle), from which the motion of the vehicle can be determined. For example, the first motion source 105 may include or be coupled to a satellite navigation receiver, which may continuously provide the position and possible orientation of the vehicle. As another example, the first motion source 105 may be located on the vehicle and include an inertial guidance system, according to which the posture can be determined based on the measured acceleration.

[0082] The vehicle motion signal may reflect the posture of the vehicle as a function of time. The vehicle motion signal may indicate the current position and / or orientation of the vehicle, and may reflect how these change over time. The vehicle motion signal may be a digital signal comprising a time-sampled motion signal, wherein each sample reflects one or more coordinates of a posture (including one or more position coordinates and / or orientation coordinates). For example, in some embodiments, the vehicle motion signal may include time samples, wherein each time sample includes one, more, or all of the forward / backward, left / right, up / down, pitch, yaw, and roll posture values.

[0083] In some embodiments, the first motion source 105 may receive the vehicle motion signal directly from an external source (eg, part of a vehicle) and may provide the signal to other functions of the presentation device either directly or after appropriate processing.

[0084] The presentation device comprises a second motion source 107 arranged to provide a relative user motion signal indicative of the motion of the user relative to the vehicle.

[0085] In some embodiments, the second motion source 107 may be coupled to or include one or more sensors that detect a user and derive an indication of the user's movement relative to the vehicle. For example, the second motion source 107 may include or be coupled to a set of cameras that monitor the interior of the vehicle. The user may be detected in the image, and from this information, the position and / or orientation of the user (or, for example, only the user's head / face) may be determined. As another example, a distance sensor (e.g., based on infrared or ultrasonic signals) may be positioned in the vehicle so that the distance from the distance sensor to the user may be determined, and thereby the position may be detected. As another example, the user may wear one or more sensors that detect signals from a suitable transmitting element (e.g., an ultrasonic transmitter) mounted in the vehicle, thereby allowing the distance to the radiator to be determined, and thereby the viewer's posture relative to the vehicle may be determined.

[0086] The relative user motion signal may reflect the user's posture relative to the vehicle as a function of time. The relative user motion signal may indicate the current position and / or orientation of the user's posture relative to the vehicle, and may reflect how these change over time. The relative user motion signal may be a digital signal comprising a time-sampled motion signal, wherein each sample reflects one or more coordinates of the posture (including one or more position coordinates and / or orientation coordinates). For example, in some embodiments, the relative user motion signal may include time samples, wherein each time sample includes one, more, or all of the forward / backward, left / right, up / down, pitch, yaw, and roll posture values.

[0087] It will be appreciated that similar properties apply (as appropriate) to other gesture or motion signals processed by the rendering device, including those derived from vehicle motion signals and / or relative user motion signals.

[0088] In some embodiments, the second motion source 107 may receive the relative user motion signal directly from an external source (eg, part of a vehicle) and may provide the signal to other functions of the presentation device either directly or after appropriate processing.

[0089] Thus, in this method, the motion signal for which the audiovisual signal is generated is not based on a single posture measurement for the user, but rather two different motions are considered, namely the motion of the vehicle and the motion of the user relative to the vehicle.

[0090] The rendering apparatus is further arranged to split the relative user motion signal into different components, which components may then be processed differently such that their relative influence / contribution to the renderer 103's determination of the view pose is different.

[0091] The first motion source 105 is coupled to a predictor 109 which is arranged to predict a relative user motion signal by applying a prediction model to the vehicle motion signal. The prediction model is applied to the received vehicle motion signal in order to predict / estimate from the signal what the relative user motion (relative to the vehicle) will be according to the prediction model.

[0092] The predictor 109 may therefore generate a predicted relative user motion signal that reflects relative user motion that may be expected to be caused by vehicle motion. It may generate a prediction of relative user motion that reflects relative user motion caused by vehicle motion, and in particular, it may reflect relative user motion associated with vehicle motion. The predicted relative user motion may reflect involuntary motion of the user caused by the impact of vehicle motion on the user.

[0093] The predictor 109 is also coupled to a residual signal generator 111, which generates a residual user motion signal reflecting at least a component of the difference between the relative user motion signal and the predicted relative user motion signal. The residual signal generator 111 is therefore also connected to the second motion source 107, from which the residual signal generator 111 receives the relative user motion signal. In many embodiments, the residual user motion signal is simply generated as the relative user motion signal by subtracting the predicted relative user motion signal. For example, a gesture-by-gesture component subtraction can be performed for each moment (i.e., for each sample).

[0094] Therefore, the presentation device is arranged to generate two motion signals, namely a predicted relative user motion signal which may indicate an estimated / predicted user motion resulting from the vehicle motion and a residual user motion signal which reflects the remaining user motion (i.e., user motion which may be considered not to be directly resulting from the vehicle motion). The predicted relative user motion signal may indicate an expected / relevant / involuntary motion of the user, while the residual user motion signal may indicate an unintended / irrelevant / intentional / voluntary user motion.

[0095] The predictor 109 and the residual signal generator 111 are coupled to a view pose determiner 113 arranged to determine one or more view poses. The view pose determiner 113 is coupled to the renderer 103, which is fed the view poses from which it proceeds to generate an audiovisual signal, and in particular it may generate an audiovisual signal comprising a sequence of view images for varying view poses and / or spatial audio of the scene as perceived by the user at the view pose(s).

[0096] The view pose determiner 113 may be arranged to generate the view pose signal by repeatedly generating the view pose and including it in the view pose signal. For example, for each sample / time instant, a new view pose may be determined and fed to the renderer 103 to generate samples (images and / or audio) of the audiovisual signal.

[0097] The view pose determiner 113 is arranged to determine the view pose based on the predicted relative user motion signal and the residual user motion signal. The two signals may specifically be combined. However, the dependency of the view pose on the two signals is different. The contribution from the predicted relative user motion signal and the contribution from the residual user motion signal will be different, and typically scaling and / or filtering is different between the two signals.

[0098] As a specific example, in many embodiments, the two signals may be filtered and / or scaled before being added together. For example, the predicted relative user motion signal may be filtered to attenuate higher frequencies and may be scaled before being added to the residual user motion signal. This approach will result in the view gesture closely following the residual user motion signal, but with only a reduced contribution of primarily only lower frequencies from the predicted relative user motion signal. As a result, the view gesture is determined to primarily follow user movements that do not directly follow vehicle motion, but are instead the result of other and typically intentional movements (e.g., movement caused by the user turning their head).

[0099] The predicted relative user motion signal may reflect motion strictly caused by the vehicle towards the user and may correspond directly to vehicle motion. Depending on the prediction model, the predicted relative user motion signal may also include motion resulting from unintentional compensation by the user. For example, when the vehicle makes a sharp turn, the user may be bobbed up and down or pushed sideways due to a bumpy road.

[0100] In addition, the residual user motion signal may reflect motion resulting from active compensation or even the user's expectations. For example, it may reflect user motion caused by the user's expectations when approaching a sharp turn or traffic bump in the road or when approaching a traffic light. As a result, the user typically tenses his muscles. When the expected timing is good and accurate enough, the user can fully compensate for the motion that would have been introduced by the vehicle. Note that this component may not be too dominant when the user is wearing a VR headset, but there may be different triggers, such as screaming or a fellow passenger squeezing your hand. In addition, while the user may be surprised by the start of a car turning sharply on the road, it is natural to compensate and lean in the opposite direction of the force vector introduced by the car. In addition, the residual user motion signal may reflect the user's voluntary movements, such as the user turning his head to look around.

[0101] In the method, the movement of the user's head relative to the world coordinate system can be segmented / decomposed / divided into at least two components, namely one corresponding to the predicted relative movement and one reflecting the remaining part of the movement. Subsequently, a separate compensation of the at least two components can be performed to determine a view pose that can then be used, for example, to generate a 3D virtual audio-visual user representation.

[0102] As another example, the predicted relative motion is compensated only in cases where it corresponds to a large motion (such as a car making a sharp turn). In all other cases, the predicted relative motion is not compensated in the calculation of the user's gesture.

[0103] As yet another example, the predicted relative motion is compensated according to what is happening in the virtual scene (virtual content).

[0104] As yet another example, separate compensation depends on whether the user is viewing a purely virtual scene or an augmented scene. In the latter case, the user will see overlay graphics content superimposed on the interior of the car or on the road, depending on whether the user is looking inside or outside the car. Each of these requires different compensation for separate items.

[0105] Thus, by considering specific multiple motion components of the user's movement and having different dependencies of the view pose on these components, enhanced effects and a more appropriate and flexible user experience may be achieved.

[0106] In different embodiments, the predictor 109 may use different methods and prediction models to determine the predicted relative user motion signal based on the vehicle motion signal.

[0107] In some embodiments, the prediction model may include or incorporate temporal filtering of the vehicle motion signal.The prediction model may specifically attenuate some frequencies relative to other frequencies.

[0108] The filtering is temporal as opposed to spatial filtering and therefore affects the temporal properties / variations of the vehicle motion signal. In the time domain, a (prediction) filter may be a mathematical operation where the future value of a discrete time signal is estimated as a linear function of previous samples. In practice, this linear function may represent a filter with a specific frequency transfer function where some frequencies are attenuated relative to other frequencies.

[0109] For example, some very low frequencies of vehicle motion usually tend to cause corresponding user motion. For example, when a car turns or accelerates, the user will usually follow the motion of the car very closely. If the car passes over a speed bump, it will cause an up and down motion, and for lower frequencies, the passengers sitting in the car will usually follow the up and down motion. Therefore, for such low-frequency movement, user motion will tend to follow the vehicle motion, and therefore the vehicle motion will tend to have a relatively low impact on the relative user motion relative to the vehicle. However, for higher frequencies, the attenuation of seats, human bodies, etc. may cause the vehicle motion to not be transferred to the user motion. Therefore, for these higher frequencies, the vehicle motion can be directly converted into relative user motion (for example, according to the applied coordinate system, the sign is reversed). In this example, the predictor 109 can apply a prediction model in the form of a high-pass filter accordingly. The resulting high-pass vehicle motion signal can be used as a predicted relative user motion signal.

[0110] As another example, if a vehicle is driving through a slow but long curve on a road, the user will experience a constant force on one side of the vehicle body. However, the relative user motion will indicate that the user is facing the force and not moving at all. In this case, motion compensation is not needed.

[0111] In some embodiments, the predictor 109 may further be arranged to apply temporal filtering to the relative user motion signal to generate a filtered relative user motion signal.The prediction of the predicted relative user motion signal may then also be responsive to the filtered relative user motion signal.

[0112] Thus, in some embodiments, the prediction model may be based on inputs of both the vehicle motion signal and the relative user motion signal, and specifically on filtered versions thereof.

[0113] Specifically, if the vehicle only provides inaccurate posture (position and rotation) measurements, this measurement can be made more precise by observing the relative user motion signal. The latter is usually measured by a camera in a headset, and the camera observes changes in features about the car's interior. By calculating the correlation between the predictions made via the relative user motion and the car's sensors, we can segment out which parts of the relative motion over time are mainly determined by the car. For those highly correlated segments over time, the changes in the car's posture parameters can be made more precise by assuming that the user motion is only determined by the car (and therefore the correlation should be 1).

[0114] In some embodiments, the predictive model may include a biomechanical model. The biomechanical model may provide a mechanical model of the human body and may be, for example, a model that shows how movement of a person's torso or seating area causes movement of a user's head.

[0115] A suitable model could be, for example, a vertical bar of length between 0.6-1.0 m, with the mass of a typical human upper body representing a person sitting in the back seat. Assuming the vehicle is travelling along a bend in the road, the curvature of the bend, the speed of the car and the mass of the upper body will determine the forces that will be generated on the upper body. Assuming the person's muscles are completely relaxed, the person's upper body will tip over until the person notices that he is falling. At least for the first unintended part of the bend (the beginning), the tipping action will follow a simple model of a tipping rod of length and mass. Once the person feels the action of the rod (usually within one second), the person will try to compensate via muscle action and will try to correct or maintain an upright sitting position.

[0116] As an example, the biomechanical model may be used to reflect additional motion of the user (and specifically, in many embodiments, the user's head) that is not directly present in the vehicle motion signal itself, but is instead directly and unintentionally generated by the vehicle movement. For example, a car turning a corner on a road may result in continued movement of a human head, and therefore continued movement of a headset, but that movement may be one that counteracts the turn, and therefore may be different from the vehicle motion signal. For example, a user may tend to turn his head or torso inward to compensate for the centrifugal force experienced by the user relative to the vehicle.

[0117] An example of relative head movement can be represented by Figure 2 Shown, Figure 2 Examples of three different situations during a vehicle turn are shown.

[0118] Initially, the car is moving straight ahead, and the passenger / user is sitting upright (a). Thus, in this case, the movement of the user's head can remain substantially fixed relative to the vehicle itself.

[0119] The car may then begin to turn, and during the turn itself, the passenger / user is swung to one side by the force of the turning car (b). Thus, the physical force created by the turning of the car (relative to the "centrifugal force" of the car) pushes the user to the side of the car that is on the outside of the turn. Figure 2 In the example of , the user's head may be tilted / pivoted by an angle α towards the outside of the car. Thus, merely starting the rotation already results in the user's head being tilted relative to the car, without the user voluntarily deciding to make such a relative movement.

[0120] After the turn is completed, the car is traveling straight again, and when the physical force toward the outside of the car is no longer present, the user's head will return to the same upright position as before the turn (c).

[0121] Therefore, the user's head moves a lot relative to the car during the turn. This movement is caused by the motion of the car, but is different from that motion. The behavior can be modeled by a biomechanical model.

[0122] In some embodiments, the predictor 109 may be arranged to determine the predicted relative user motion signal based on the correlation of the relative user motion signal and the vehicle motion signal. In some embodiments, the prediction model may include the correlation of the relative user motion signal and the vehicle motion signal. The prediction model may be a linear prediction model that performs the correlation between the relative user motion signal and the vehicle motion signal.

[0123] In more detail, to generate an experimental prediction model based on correlations, a passenger can be instructed to sit in a relaxed manner in the back seat of a car with a headset on which all external automatic visual signals are blocked. The car is then driven through a number of specific road bends at different speeds, and an accelerometer installed in the passenger's headset can measure the stippling effect as a function of the passenger's biophysical parameters (total length, upper body length, mass, etc.). After sufficient measurements have been made, a linear or nonlinear pattern can be fitted, which in the future can be used for real-time predictions.

[0124] In some embodiments, the predictor 109 may be arranged to generate a predicted relative user motion signal from the residual user motion signal.

[0125] For example, the prediction model may be a feedback or loop model in which one or more parameters may be adapted to reduce the level or amplitude of the residual user motion signal. In some embodiments, the residual user motion signal may be fed back to the predictor 109, which may proceed to modify the parameters of the model that generates the predicted relative user motion signal to reduce the amplitude or level of the residual user motion signal. This will typically result in maximization (or at least increase) of the level of the predicted relative user motion signal. Therefore, such feedback operations may tend to provide improved predictions and more accurate determinations of motions that may be directly generated by vehicle motion. Therefore, effectively, the residual user motion signal may be used as an error signal for adjusting the prediction model.

[0126] It will be appreciated that different methods for adjusting the prediction based on the error signal are known. For example, the prediction model may consist of a linear prediction filter for which the coefficients are adjusted based on a linear adaptive algorithm based on least mean squares (LMS) (e.g. without a predetermined range).

[0127] As another example, at regular intervals the predictions are reset based on the error model.

[0128] In different embodiments, the view pose determiner 113 may be arranged to apply different methods to determine the view pose(s), and the reliance on predicted relative user motion and residual user motion signals may depend on the specific preferences and requirements in each individual embodiment.

[0129] In many embodiments, view pose determiner 113 may process the predicted relative user motion signal to generate a first motion component. Similarly, view pose determiner 113 may process the residual user motion signal to generate a second motion component. The processing of the two motion signals may be different. Each motion component may describe a motion and may be represented as an absolute pose in a scene coordinate system, or may be a relative pose relative to, for example, a fixed pose and / or a sequence of poses (such as a predetermined route through a scene). View pose determiner 113 may then combine the first motion component and the second motion component into a view pose signal that includes the view pose for which rendering is to be performed.

[0130] Thus, the view pose determiner 113 may generate a first view pose contribution from the predicted relative user motion signal represented by the first motion component. It may also generate a second view pose contribution from the residual user motion signal represented by the second motion component. It may then proceed to generate a view pose for which rendering is performed as a combination of these individual contributions.

[0131] The combination may simply be a sum of the contributions / components, or may more generally be a weighted sum in many embodiments. For example, the relative poses of the first component and the second component may be added together to form a single view pose offset signal, which may, for example, be added to the fixed pose and / or varying pose signals (particularly, as a function of time). In some embodiments, more complex combinations may be performed, such as those including the introduction of a time offset between the signals, etc.

[0132] The view pose determiner 113 is arranged to apply different processing to the predicted relative user motion signal and the residual user motion signal when generating the view pose (signal). Thus, the impact of different types of motion on the resulting view pose can be distinguished and individually adjusted to provide the desired effect and user experience, and in particular to compensate for involuntary user motion caused by, for example, vehicle motion.

[0133] In many embodiments, the view pose determiner 113 may be arranged to apply a different weight to the predicted relative user motion signal than to the relative user motion signal when determining the view pose. The treatment (or equivalently combined weighting) of one motion relative to the other may include different scaling / gains for the two motions.

[0134] For example, in some embodiments, the scaling or gain of the predicted relative user motion signal may be significantly reduced relative to the scaling or gain of the residual user motion signal. Thus, an effect may be provided in which involuntary movement caused by the vehicle motion signal may be significantly attenuated relative to the voluntary movement of the user. This may, for example, provide an improved VR experience to the user because the effects of being in the vehicle may be mitigated or reduced.

[0135] In some embodiments the view pose determiner 113 may be arranged to apply a different temporal filtering to the predicted relative user motion signal than to the relative user motion signal.The first and second motion components may, for example, be generated to have different frequencies attenuated.

[0136] Specifically, in some embodiments, the view gesture determiner 113 may be arranged to extract a first motion component from the predicted relative user motion signal by attenuating the temporal frequency of the predicted relative user motion signal. For example, really low frequencies may be filtered out, resulting in motion corresponding to, for example, slow accelerations being reduced or removed. For example, low frequency motions of a vehicle, such as long turns (0.1 Hz), may be attenuated because the user will not perceive them. Similarly, high frequency motions, such as the shaking of a car (glt; 10 Hz), may be filtered out completely, because this may be annoying to the user and when such motion is not transferred from the physical world to the virtual world, it will not cause motion induced nausea. For medium frequency motions generated by the vehicle, no filtering may be performed, resulting in the motion being presented to the user. Furthermore, no filtering may be applied to the residual user motion signal, resulting in the view gesture following this motion completely.

[0137] As a result, an experience can be provided where the virtual user's view follows the user's voluntary movements perfectly but does not follow slow accelerations or turns or high frequency shakes of the car. However, the mid-range frequencies of the vehicle's motion signals may still be reflected in the view pose, so the user's perceived view of the scene may still include some motion corresponding to the car's motion. If this motion is not reflected in the view presented to the user, it may lead to motion sickness / nausea (which is often caused by conflicts between sensory inputs from different senses, such as balance and vision).

[0138] Therefore, this approach can provide a significantly improved user experience.

[0139] Of course, in other applications, other approaches may be used. For example, some filtering of the mid-range may be applied, e.g., depending on the VR experience provided (e.g., the story presented) or the user's sensitivity to nausea in certain frequency bands.

[0140] As another example, a VR experience may be designed to include some or all slow turns, and filter coefficients corresponding to slow turns may be filtered or not filtered based on the state of the VR experience.

[0141] In some embodiments, the determination of the second motion component may include extracting portions of the motion from the residual user motion signal. For example, it may be desirable to include some specific motions in the view pose changes, while it may still be undesirable to include some motions that cannot be directly predicted from the vehicle motion signal in the presentation to the user.

[0142] Specifically, in some embodiments, the view pose determiner 113 may be arranged to detect a user gesture motion component in the residual user motion signal. It may then proceed to extract a first motion component from the residual user motion signal by at least partially removing the user gesture motion component. Specifically, a second motion component may be generated from the residual user motion signal by subtracting the gesture motion component.

[0143] The user gesture motion may specifically be a predetermined user motion. For example, the view pose determiner 113 may store parameters of a plurality of predetermined user motions, wherein each of them is described by a set of parameters that may vary within a range. The view pose determiner 113 may then correlate the residual user motion signal with these predetermined user motions, while varying the parameters to achieve as close a fit as possible. If the correlation is above a suitable level, the view pose determiner 113 may consider that the residual user motion signal includes the user performing such a predetermined user motion. It may then proceed to subtract the predetermined user motion (for the identified parameters) from the residual user motion signal to generate a second motion component, which is then combined with the first motion component.

[0144] Thus, in this method, specific predetermined / gesture movements of the user may be detected and removed from the user's movements included in the view pose movement.

[0145] In some embodiments, the view pose determiner 113 may be arranged to detect the presence of a predetermined user movement / user gesture by compensating the residual user motion signal for a movement believed to be caused by the vehicle motion signal. For example, the correlation of the predetermined user movement with the residual user motion signal may be compensated by a motion signal before the residual user motion signal, which motion signal reflects the influence of the specific user movement on the vehicle motion. For example, the compensation may be based on a predicted relative user motion signal.

[0146] As a specific example, residual user motion is presented to a gesture detector and tracker. If a gesture is detected, then the movement corresponding to the gesture is tracked. For example, if the gesture involves movement of two arms, the trajectory of the nodes of the arms is estimated by the tracker. This movement is then converted back into a representation that matches the residual user motion. The signal can therefore be a component of the residual user motion associated with the gesture, and the difference signal (residual of the residual) can be the other component.

[0147] In such an example, the second motion component may be a predetermined user motion / gesture motion, for example, a motion related to the movement of a limb making the gesture.

[0148] This approach can be advantageous in many scenarios. For example, gestures made by a user in a car will be affected by the movement of the car. For gesture control in the virtual world, it can be advantageous to correct these gestures to subtract the motion caused by the car.

[0149] In many embodiments, motion and gestures may be multi-dimensional and include, for example, three position coordinates and often three orientation coordinates, providing a full 6DoF experience. However, in some embodiments, gestures / motions may be represented by fewer dimensions, such as, for example, only three or even two spatial position coordinates.

[0150] In scenarios where multiple components may be present, the dependencies of different posture components may be different when determining the view pose. Therefore, the view pose determiner 113 may be arranged to determine a view pose with different dependencies on at least a first posture component and a second posture component. In many embodiments, this may be the case where both the first posture component and the second posture component are position components or both are orientation components.

[0151] For example, the dependency in the sideways direction of the user may be different than in the up / down direction. This may result in different differences between the predicted user motion and the residual user motion depending on whether it is related to the user's motion in the up / down direction in the seat or to the sideways motion in the seat. For example, up / down motion in a car is usually easier to translate into the virtual scene, and transferring this component to the view pose helps with realism. Sideways motion is more likely to cause nausea and needs to be filtered more.

[0152] In some embodiments, the rendering apparatus may further be arranged to receive processing data indicative of processing of the predicted relative user motion signal and / or the residual user motion signal. The processing data may specifically define dependencies between the motion signals and the view pose. The view pose determiner 113 may then adjust its processing to determine the view pose according to the processing instructions of the processing data.

[0153] The processing data may specifically be provided in a message, which message may be part of a received audiovisual signal, which audiovisual signal further comprises audiovisual data for a scene.For example, the data may be received in a bitstream from a remote server.

[0154] As an example, data may be packaged with audio-visual signals to optimize the user's experience while in a moving vehicle. For example, for each gesture component, such a processing message may indicate how it should be filtered. For example, a processing message may contain:

[0155] Each movement component (e.g., involuntary, voluntary, intended, etc.)

[0156] Optionally, each axis

[0157] Optionally, each device class

[0158] Only a subset of the components need be present in the message.

[0159] The filter values ​​may be attenuation factors, but may also include cut-off frequencies, or may even be FIR or IIR filter coefficients.

[0160] For example, a message providing the following processed data indicating different types of gesture coordinates and decay percentages of motion components may be provided:

[0161] Comp.1 Comp.2 Comp.3 Comp.4 X 10％ 30％ 50％ 100％ Y 10％ 10％ 80％ 100％ Z 0％ 50％ 60％ 100％ yaw 50％ 50％ 25％ 100％ Pitch 50％ 100％ 100％ 50％ scroll 50％ 0％ 0％ 0％

[0162] Such a message may also include information about how each scriptable motion event should be converted into motion in the virtual scene motion response. Only a subset of motion events must be present in the message. The response may be a mapping or include a filter value. The filter value may be an attenuation factor, as indicated in the table below, but may also include a cutoff frequency or may even include an IIR filter coefficient.

[0163] sit Look left Looking Down Action 1 30％ 50％ Action 2 10％ Action 3 100％ 50％ 60％

[0164] In the following, a specific example of a method that may be used will be described. The description will be considered as an example in which the rendering is of a view image rendered from a view gesture generated by the view gesture determiner 113 .

[0165] In this example, the view pose used to render the audio-visual signal and (virtual scene objects) is bound to a common world coordinate system. In the following specific examples, the view pose will be referred to as a virtual camera (generating an image to reflect the image that would be captured by the virtual camera in the view pose).

[0166] Therefore, mapping scene object points to the virtual camera can be expressed as a transformation to the common world system followed by a transformation to the virtual camera:

[0167]

[0168] In this equation, M o is the so-called model matrix that places virtual scene objects in the virtual world coordinate system. This matrix changes when objects move through virtual space. The matrix T w Transform points from the virtual world to the real world.

[0169] Matrix T wTypically generated by an initialization step and kept constant during operation. Under user control, calibration, reset, or setup can be performed. For example, at initialization, T can be defined by actively placing visual markers in the scene or by automatically detecting visual features and placing virtual axes somewhere relative to their positions. w Cameras, depth sensors, and / or other sensors may be used to assist with this initial setup.

[0170] The view matrix V can capture the position of the user's eyes in the real world space, and the view matrix V represents the user's movement. The view matrix V is essentially the pose of the human head or eyes. Since virtual reality headsets are usually stereo, we have different view matrices for the left and right eyes. The view matrix is ​​usually estimated from sensors (cameras, depth sensors, etc.) mounted in and on the headset (inside-out tracking).

[0171] When the vehicle changes motion, this results in different torques [m·s] acting on different body parts. Since the human body is flexible, and the distance from the car seat to the human head is large, and a typical human head is heavy, the moment of inertia (angular mass) will also be large. In the absence of early muscle anticipation (based on visual input), a small vehicle motion may result in a large head motion relative to the vehicle (the world). This may be an example of an undesired movement caused by vehicle motion.

[0172] Therefore, any vehicle movement that causes a difference in movement between the headset and the vehicle, i.e., relative user motion, will cause a change in V as observed by the sensors in the headset. The measured 4x4 view matrix is ​​the user pose and contains the user's view of the world scene (rotation and translation), and can be decomposed into:

[0173] V≡T 车辆 V 预期

[0174] Among them, the matrix V 预期 The view may reflect the desired / intended view of the VR content being presented to the viewer. This may be the pose that the user expects to achieve by controlling the head pose using the neck muscles and other body muscles. However, vehicle motion introduces unintended rotations and translations, and thus the VR content presented by the VR player is never achieved. 预期 represents the expected pose. The joint rotation and translation can be obtained by another 4x4 transformation matrix T 车辆 Note that in T 车辆 =I (identity matrix), vehicle movement / vibration has no effect on the expected posture.

[0175] Thus, in this example, the matrix V corresponds to a combination of expected and unexpected movements and can be represented as acting on the expected movement V 预期 The post-multiplication transformation matrix T 车辆 The matrix is ​​time-varying, and in this method, the matrix T 车辆 is determined by prediction and corresponds to the predicted relative user motion signal. 预期 is determined as the residual signal, ie the relative user motion signal.

[0176] The system may proceed to determine the view pose to, for example, compensate for vehicle movement / vibration. It may be particularly desirable to render the virtual world using the following equation:

[0177]

[0178] in,

[0179]

[0180] In this approach, the predictor 109 may specifically seek to estimate the vehicle transformation matrix T 车辆 , so that its affine inverse may be calculated and used by the view pose determiner 113 to determine the compensation view matrix V 补偿 The view pose represented.

[0181] As a specific example, the matrix T corresponding to the predicted relative user motion signal may be determined by filtering the view matrix V 车辆 , where V is a view matrix that may correspond to the vehicle motion signal combined with the relative user motion signal.

[0182] Although the vehicle suspension system is very effective, high temporal frequencies of vehicle movement may still occur. Therefore, we can use T 车辆 is extracted as the high-pass component of V, as measured by the inside-out tracking sensor inside the headset. 车辆 The predicted relative user motion signal represented by V can be predicted as a high-pass component of V representing the vehicle motion signal. In this case, the vehicle-induced high-frequency component of the pose, as measured from the headset camera relative to the car interior, is directly used to estimate the component caused by the car motion.

[0183] For completeness, we specify how a 4×4 homogeneous transformation matrix consisting of a 3×3 rotation matrix and a 3×1 translation vector can be processed. As the determination of T 车辆 For the specific example of , we can first split the 4x4 view matrix into the orientation quaternion and the translation vector:

[0184] q=quat(V)

[0185] t=(V 14 ,V 24 ,V 34 ) t

[0186] Note that there are alternative representations of rotation, and they may be preferred in some cases. To measure temporal rotational and translational changes, the differential rotation and differential translation can be calculated as:

[0187]

[0188] t Δ =t k -t k-1

[0189] where k and k-1 correspond to two discrete time steps, e.g. 1 / 30 second apart. We then use a recursive filter to output a filtered rotation that ignores instantaneous large rotations assumed to be introduced by vehicle motion:

[0190]

[0191] in, is the filtered rotation, |2cos -1 (q Δ,w )| is the magnitude of the instantaneous rotation change angle [radians], and t 旋转 is the threshold above which we do not want to apply the instantaneous rotation to the headset view matrix.

[0192] Similarly, the translation vector is filtered as:

[0193]

[0194] in, is the filtered translation, and Δ 平移 is the threshold above which we do not apply the instantaneous translation to the headset view matrix.

[0195] Now put T 车辆 Calculated as:

[0196]

[0197] T 车辆 =T 平移 T 旋转 .

[0198] The above process therefore assumes that we can derive the effect of vehicle motion on headset motion from the high frequency motion components measured by the sensors in the headset relative to the interior (visual) features of the car.

[0199] In the following, we cancel out the high-frequency effect caused by this vehicle. Therefore, it can be determined that Finally, the rendering mapping can be performed according to the following formula (where V 补偿 Reflecting the view pose determined by view pose determiner 113):

[0200]

[0201] The data signal generating means and the presenting means may be specifically implemented in one or more appropriately programmed processors. Examples of suitable processors are provided below.

[0202] Figure 3 is a block diagram illustrating an example processor 300 according to an embodiment of the present disclosure. The processor 300 may be used to implement one or more processors that implement Figure 1 The processor 300 may be any suitable processor type, including but not limited to a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable array (FPGA) (where the FPGA has been programmed to form a processor), a graphics processing unit (GPU), an application specific integrated circuit (ASIC) (where the ASIC has been designed to form a processor), or a combination thereof.

[0203] Processor 300 may include one or more cores 302. Core 302 may include one or more arithmetic logic units (ALUs) 304. In some embodiments, core 302 may include a floating point logic unit (FPLU) 306 and / or a digital signal processing unit (DSPU) 308 in addition to or in lieu of ALU 304.

[0204] The processor 300 may include one or more registers 312 that are communicatively coupled to the core 302. The registers 312 may be implemented using dedicated logic gates (e.g., flip-flops) and / or any memory technology. In some embodiments, the registers 312 may be implemented using static memory. The registers may provide data, instructions, and addresses to the core 302.

[0205] In some embodiments, the processor 300 may include one or more levels of cache memory 310 communicatively coupled to the core 302. The cache memory 310 may provide computer readable instructions for execution to the core 302. The cache memory 310 may provide data for processing by the core 302. In some embodiments, the computer readable instructions may have been provided to the cache memory 310 by a local memory (e.g., a local memory attached to the external bus 316). The cache memory 310 may be implemented using any suitable cache memory type, for example, a metal oxide semiconductor (MOS) memory, such as a static random access memory (SRAM), a dynamic random access memory (DRAM), and / or any other suitable memory technology.

[0206] The processor 300 may include a controller 314 that may control inputs to the processor 300 from other processors and / or components included in the system and / or outputs from the processor 300 to other processors and / or components included in the system. The controller 314 may control data paths in the ALU 304, FPLU 306, and / or DSPU 308. The controller 314 may be implemented as one or more state machines, data paths, and / or dedicated control logic. The gates of the controller 314 may be implemented as independent gates, FPGAs, ASICs, or any other suitable technology.

[0207] Registers 312 and cache 310 may communicate with controller 314 and core 302 via internal connections 320A, 320B, 320C, and 320D. The internal connections may be implemented as buses, multiplexers, crossbar switches, and / or any other suitable connection technology.

[0208] Input and output for processor 300 may be provided via bus 316, which may include one or more conductive lines. Bus 316 may be communicatively coupled to one or more components of processor 300, such as controller 314, cache 310, and / or registers 312. Bus 316 may be coupled to one or more components of the system.

[0209] The bus 316 may be coupled to one or more external memories. The external memory may include a read-only memory (ROM) 332. The ROM 332 may be a mask ROM, an electronically programmable read-only memory (EPROM), or any other suitable technology. The external memory may include a random access memory (RAM) 333. The RAM 333 may be a static RAM, a battery-powered static RAM, a dynamic RAM (DRAM), or any other suitable technology. The external memory may include an electrically erasable programmable read-only memory (EEPROM) 335. The external memory may include a flash memory 334. The external memory may include a magnetic storage device, such as a disk 336. In some embodiments, the external memory may be included in the system.

[0210] The present invention can be implemented in any suitable form including any combination of hardware, software, firmware or these projects.Optionally, the present invention can be implemented at least in part as computer software running on one or more data processors and / or digital signal processors.The elements and parts of an embodiment of the present invention can be implemented physically, functionally and logically in any suitable manner.In fact, the function can be implemented in a single unit, in a plurality of units, or be implemented as the part of other functional units.So, the present invention can be implemented in a single unit, or can be distributed between different units, circuits and processors physically and functionally.

[0211] The term "according to" or depending on may be exemplified by the terms "in response to" or "as a function of" or "based on".

[0212] Although the present invention has been described in conjunction with some embodiments, the present invention is not intended to be limited to the specific forms set forth herein. On the contrary, the scope of the present invention is limited only by the claims. In addition, although features may be described in conjunction with specific embodiments, those skilled in the art will recognize that the various features of the described embodiments may be combined according to the present invention. In the claims, the term comprising does not exclude the presence of other elements or steps. The method(s) of a claim excludes methods for performing such mental actions.

[0213] In addition, although listed separately, multiple modules, elements, circuits or method steps can be implemented by, for example, a single circuit, unit or processor. In addition, although individual features can be included in different claims, these features may also be advantageously combined, and the inclusion of features in different claims does not mean that the combination of features is not feasible and / or advantageous. Moreover, the inclusion of features in a category of claims does not mean that the features are limited to this category, but indicates that the features are equally applicable to other categories of claims where appropriate. In addition, the order of features in the claims does not mean that the features must work in any particular order, and in particular, the order of the various steps in the method claims does not mean that the steps must be performed in this order. On the contrary, the steps can be performed in any suitable order. In addition, singular references do not exclude multiple. Therefore, references to "one", "one", "first", "second", etc. do not exclude multiple. The figure marks provided in the claims are only used to clarify the examples and should not be interpreted as limiting the scope of the claims in any way.

Claims

1. A device for audio-visual presentation, the device comprising: a receiver (101) arranged to receive audiovisual data representing a scene; a first source (105) providing a vehicle motion signal indicative of the motion of the vehicle; a second source (107) providing a relative user motion signal indicative of a user's motion relative to the vehicle; a predictor (109) arranged to generate a predicted relative user motion signal by applying a prediction model to the vehicle motion signal; a residual signal generator (111) arranged to generate a residual user motion signal indicative of at least a component of a difference between the relative user motion signal and the predicted relative user motion signal; a view pose determiner (113) arranged to determine a view pose from the residual user motion signal and the predicted relative user motion signal, the view pose having a dependency on the residual user motion signal different from a dependency of the view pose on the predicted relative user motion signal; as well as A renderer (103) is arranged to render an audiovisual signal for the view gesture based on the audiovisual data.

2. The device according to claim 1, wherein: The prediction model includes temporal filtering of the vehicle motion signal.

3. The device according to claim 2, wherein: The predictor (109) is arranged to: apply temporal filtering to the relative user motion signal to generate a filtered relative user motion signal; and predict the predicted relative user motion signal from the filtered relative user motion signal.

4. An apparatus according to any preceding claim, wherein: The predictive model includes a biomechanical model.

5. The device according to any one of the preceding claims, wherein The view pose determiner (113) is arranged to apply a different weighting to the predicted relative user motion signal than to the residual user motion signal.

6. The device according to any one of the preceding claims, wherein The view pose determiner (113) is arranged to apply a different temporal filtering to the predicted relative user motion signal than to the residual user motion signal.

7. An apparatus according to any preceding claim, wherein: The view pose determiner (113) is arranged to: determine a first view pose contribution from the predicted relative user motion signal and determine a second view pose contribution from the residual user motion signal; And generating the view pose by combining the first view pose contribution and the second view pose contribution.

8. An apparatus according to any preceding claim, wherein: The view pose determiner (113) is arranged to: extract a first motion component from the predicted relative user motion signal by attenuating temporal frequencies; and determine the view pose to include a contribution from the first motion component.

9. An apparatus according to any preceding claim, wherein: The view posture determiner (113) is arranged to: detect a user gesture motion component in the residual user motion signal; and extract a first motion component from the residual user motion signal based on the user gesture motion component; and determine the view posture to include a contribution from the user gesture motion component.

10. An apparatus according to any preceding claim, wherein: The predictor (109) is arranged to determine the predicted relative user motion signal in response to a correlation of the relative user motion signal with the vehicle motion signal.

11. An apparatus according to any preceding claim, wherein: At least one of the predicted relative user motion signal and the residual user motion signal indicates multiple posture components, and the view posture determiner is arranged to determine the view posture, which has different dependencies on at least a first posture component and a second posture component of the multiple posture components.

12. An apparatus according to any preceding claim, wherein: The receiver (101) is arranged to: receive processing data indicating processing of at least one of the predicted relative user motion signal and the residual user motion signal; and the view pose determiner (113) is arranged to determine the view pose according to processing instructions of the processing data.

13. An apparatus according to any preceding claim, wherein: The predictor (109) is arranged to generate the predicted relative user motion signal based on the residual user motion signal.

14. An audio-visual presentation method, the method comprising: receiving audiovisual data representing a scene; providing a vehicle motion signal indicative of the motion of the vehicle; providing a relative user motion signal indicative of a user's motion relative to the vehicle; generating a predicted relative user motion signal by applying a prediction model to the vehicle motion signal; generating a residual user motion signal indicating at least a component of a difference between the relative user motion signal and the predicted relative user motion signal; determining a view pose based on the residual user motion signal and the predicted relative user motion signal, the view pose having a dependency on the residual user motion signal that is different from a dependency of the view pose on the predicted relative user motion signal; and An audiovisual signal for the view gesture is presented based on the audiovisual data.

15. A computer program product comprising computer program code means adapted to perform all the steps of claim 14 when said program is run on a computer.

Images