Apparatus and method for generating an audio signal - Patents.com

Abstract

The device includes a first receiver 201 that receives a real-time image sequence of depth and / or vision data from an image sensor. A second receiver 209 receives metadata indicative of a set of audio objects and associations of the audio objects with material properties. An image generator 203 generates an output image sequence including image objects corresponding to virtual scene objects of the real-world scene, and a detector 205 detects interactions of the virtual scene objects with real-world scene objects of the real-world scene. An estimator 207 determines material properties of the real-world scene objects according to the image data of the sequence of image frames, and a selector 211 selects an audio object according to the material properties and the material features associated with the set of audio objects. An output circuit 213 generates an output audio signal including the first audio object.

Description

[Technical Field]

[0001] The present invention relates to an apparatus and method for generating audio signals for interaction between virtual and real world scene objects, and particularly, but not exclusively, to an apparatus and method for generating audio signals for augmented reality applications. [Background technology]

[0002] In recent years, the variety and range of experiences based on audiovisual content has expanded significantly, and new services and methods for using and consuming such content are continually being developed and introduced. In particular, many spatial and interactive services, applications, and experiences are being developed to provide users with more complex and immersive experiences.

[0003] Examples of such applications include virtual reality (VR), augmented reality (AR), and mixed reality (MR) applications, which are rapidly becoming mainstream, with numerous solutions for the consumer market. Several standards are also under development by a number of standards bodies, which are actively working to standardize various aspects of VR / AR / MR systems, including streaming, broadcasting, and rendering.

[0004] While VR applications tend to provide a user experience corresponding to the user being in a different world / environment / scene, AR (including mixed reality MR) applications tend to provide a user experience corresponding to the user being in an environment that is a real local environment but with additional information or virtual objects or information added. Thus, VR applications tend to provide synthetically generated worlds / scenes that are fully immersive, while AR applications tend to provide partially synthetic worlds / scenes that are overlaid on the real scene in which the user is physically present. However, these terms are often used interchangeably and there is a high degree of overlap. In the following, the term augmented reality / AR will be used to refer to both augmented reality and mixed reality (and some variations may be referred to as virtual reality).

[0005] For example, augmented reality services and applications using handheld devices are becoming increasingly popular, and software APIs (application programming interfaces) and toolkits, such as ARKit (developed by Apple) and ARCore (developed by Google), have been introduced to support augmented reality applications on smartphones and tablets. These applications use the device's built-in camera and other sensors to generate real-time images of the environment along with virtual graphics that overlay the presented image. For example, an application might generate a live video feed along with graphic objects that overlay the live video. These graphic objects can be used, for example, to place virtual objects so that they are perceived as being present in a real-world scene.

[0006] As another example, headsets and glasses are being developed that allow the user to view real-world scenes directly through AR glasses, but which can also generate images that the user sees through the glasses, which can be used to present virtual images that are perceived as part of the real-world scene being viewed. Motion sensors are used to track head movements, and the presented virtual objects are adapted accordingly to give the impression that the virtual objects are real-world objects seen in the real world.

[0007] These approaches are known as pass-through and see-through, respectively, and both can offer new and exciting user experiences.

[0008] In addition to the visual rendering, some AR applications can provide a corresponding audio experience. Furthermore, it has been proposed to provide audio corresponding to virtual objects in addition to the visual experience. For example, if the virtual object is a noise-making object, the visual representation of the object is supplemented by the generation of a corresponding sound. In some cases, sounds may also be generated to reflect the actions of the virtual object.

[0009] Typically, sounds are generated as default sounds by rendering pre-recorded audio clips at appropriate times. In some cases, sounds are adapted to reflect the current environment, for example by adapting perceived reverberation depending on the current environment, or processed so that they are perceived to arrive from a position that corresponds to the perceived location of a virtual object in the real world. Often, such positioning is achieved through binaural processing to generate appropriate headphone audio output.

[0010] However, while such approaches may provide interesting applications and user experiences in many embodiments, traditional approaches tend to be suboptimal, difficult to implement, and / or tend to provide suboptimal performance and user experiences.

[0011] As such, improved approaches would be advantageous, particularly approaches for generating audio signals that allow for improved operation, increased flexibility, reduced complexity, easier implementation, a better audio experience, improved audio quality, reduced computational burden, improved suitability and performance for mixed reality / augmented reality applications, a better user immersion, and / or improved performance and operation. Summary of the Invention [Problem to be solved by the invention]

[0012] SUMMARY OF THE INVENTION Accordingly, the Invention seeks to preferably mitigate, alleviate or eliminate one or more of the above mentioned disadvantages singly or in any combination. [Means for solving the problem]

[0013] According to one aspect of the present invention, there is provided an apparatus for generating an output audio signal, the apparatus including: a first receiver that receives a real-time image sequence of a real-world scene from an image sensor, the real-time image sequence including a sequence of image frames, each image frame including at least one of visual image data and depth image data; a second receiver that receives a set of audio objects and metadata for the audio objects of the set of audio objects, the metadata indicating associations between the audio objects of the set of audio objects and material features; an image generator that generates an output image sequence including image objects corresponding to virtual scene objects of the real-world scene; a detector that detects interactions between the virtual scene objects and real-world scene objects of the real-world scene in response to detecting proximity of the virtual scene objects to the real-world scene objects; an estimator that determines material properties of the real-world scene objects in response to the sequence image data of the image frames; a selector that selects a first audio object of the set of audio objects in response to the material properties and the material features associated with the audio object of the set of audio objects; and an output circuit that generates an output audio signal including the first audio object.

[0014] The present invention, in many embodiments, provides an improved user experience, and in particular, in many embodiments, an improved, more immersive augmented reality experience. This approach is achieved, in many embodiments, while keeping complexity and implementation low. This approach is well suited to AR systems where audio and supporting metadata are provided from a remote server. This approach facilitates and / or supports systems where a centralized server provides support for a large number of remote clients, substantially simplifying implementation of the remote clients. This approach supports centralized generation and management of audio to enhance AR applications and experiences.

[0015] Audio objects are audio clips / fragments / etc. and are presented in any suitable manner. In many embodiments, each audio object represents sound over a time interval. In many embodiments, this time interval is less than 5, 10, or 20 seconds for any one audio object.

[0016] Material features are the material characteristics of real-world objects.

[0017] The output image sequence includes visual images.

[0018] Interactions are detected in a 3D coordinate system that represents the real-world scene. Interactions are proximity / collision / contact between virtual scene objects and real-world scene objects.

[0019] The device is an augmented reality device. The device is a device that provides an output image sequence and an output audio signal for an augmented reality application, which presents virtual scene objects within a real-world scene.

[0020] According to an optional feature of the invention, the estimator is configured to determine an interaction image region in at least one image frame of the sequence of image frames, the interaction image region being an image region of the at least one image frame in which an interaction occurs, and to determine material properties of the scene object in response to image data of the interaction image region.

[0021] This provides a particularly efficient and advantageous material property estimation in many embodiments, and in particular allows for more accurate material property estimation in many embodiments, which in turn improves the user experience.

[0022] According to an optional feature of the invention, the second receiver receives the metadata from a remote server.

[0023] This approach provides a particularly efficient application where audio is generated and managed remotely, and possibly centrally, while being effectively adapted locally to the current conditions as appropriate.

[0024] According to an optional feature of the invention, the metadata of at least some of the audio objects includes an indication of an association of at least some of the audio objects with material characteristics of real-world scene objects and an indication of an association of at least some of the audio objects with material characteristics of virtual scene objects, and the selector selects a first audio object according to the material properties and material characteristics of the real-world objects associated with the set of audio objects, and according to the material properties and material characteristics of the virtual scene objects and the material characteristics of the virtual scene objects associated with the set of audio objects.

[0025] This results in improved performance in many embodiments, and in particular improved adaptation to specific interactions, often resulting in a more immersive user experience.

[0026] According to an optional feature of the invention, the selector selects the first audio object in response to a dynamic characteristic of the virtual scene object.

[0027] This results in improved performance in many embodiments, and in particular improved adaptation to specific interactions, often resulting in a more immersive user experience.

[0028] According to an optional feature of the invention, a detector determines a characteristic of the interaction, and a selector selects the first audio object in response to the characteristic of the interaction.

[0029] This results in improved performance in many embodiments, and in particular improved adaptation to specific interactions, often resulting in a more immersive user experience.

[0030] According to an optional feature of the invention, the characteristic of the interaction is at least one characteristic selected from the group of: a speed of the interaction, a force of a collision between the virtual scene object and the real-world scene object, an elasticity of a collision between the virtual scene object and the real-world scene object, a duration of the interaction, and a direction of movement of the virtual scene object relative to the real-world scene object.

[0031] According to an optional feature of the invention, a selector selects the first audio object depending on an orientation of the virtual object relative to the real-world scene object.

[0032] This results in improved performance in many embodiments, and in particular improved adaptation to specific interactions, often resulting in a more immersive user experience.

[0033] According to an optional feature of the invention, the estimator determines a match indication of the real-world scene object to at least a first category of the plurality of object categories, and determines a material property as a function of the match indication and a material property associated with the object category.

[0034] This provides a particularly advantageous, and often low complexity, determination of material properties, which, however, remains highly accurate. In many embodiments, categorization / classification is advantageously achieved using neural networks.

[0035] According to an optional feature of the invention, the apparatus further includes an audio receiver for receiving an audio signal of real-time audio captured in a real-world scene, and the estimator determines a match indication in response to the audio signal.

[0036] Such an approach, in many embodiments, substantially improves the accuracy of material property estimation, resulting in improved overall performance.

[0037] According to an optional feature of the invention, the selector selects the first audio object as a default audio object if no audio object is found that satisfies the selection criteria.

[0038] According to an optional feature of the invention, at least one image frame includes depth image data, and the estimator determines material properties of the real-world scene object in response to detecting that at least a portion of an image region of the at least one image frame representing the real-world scene object has a confidence level for the depth image data that does not exceed a threshold.

[0039] In some embodiments, the image frame includes visual image data and depth image data, and the estimator determines that the real-world scene object has a metallic component in response to detecting that, for at least a portion of the image region, the brightness of the visual image data exceeds a threshold and the confidence level of the depth image data does not exceed a threshold.

[0040] According to another aspect of the present invention, there is provided a method for generating an output audio signal, the method including: receiving a real-time image sequence of a real-world scene from an image sensor, the real-time image sequence including a sequence of image frames, each image frame including at least one of visual image data and depth image data, receiving a set of audio objects and metadata for the audio objects of the set of audio objects, the metadata indicating associations between the audio objects of the set of audio objects and material features, generating an output image sequence including image objects corresponding to virtual scene objects of the real-world scene, detecting interactions between the virtual scene objects and real-world scene objects of the real-world scene in response to detecting proximity of the virtual scene objects to the real-world scene objects, determining material properties of the real-world scene objects in response to the image data of the sequence of image frames, selecting a first audio object of the set of audio objects in response to the material properties and the material features associated with the audio object of the set of audio objects, and generating an output audio signal including the first audio object.

[0041] The method may include the steps of displaying the output image sequence and rendering the output audio signal.

[0042] These and other aspects, features and advantages of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. [Brief explanation of the drawings]

[0043] Embodiments of the invention will now be described, by way of example only, with reference to the drawings in which:

[0044] [Figure 1] FIG. 1 shows an example of elements of an augmented reality system. [Figure 2] FIG. 2 illustrates an example of an audio device for generating an output audio signal in accordance with some embodiments of the present invention. [Figure 3] FIG. 3 shows an example image of a real-world scene with virtual scene objects. [Figure 4] FIG. 4 illustrates an example approach for generating an output audio signal according to some embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0045] The following description focuses on generating audio signals that complement the generation of images of virtual objects in a real-world scene as part of an augmented reality application, but it will be appreciated that the principles and concepts described may be used in many other applications and embodiments.

[0046] Augmented reality experiences, which can present virtual information and objects to complement real-world environments, are becoming increasingly popular, and services are being developed to meet this demand.

[0047] In many approaches, the AR application is provided locally to the viewer, such as by a standalone device that does not use or even have access to a remote AR server. However, in other applications, the AR application may be based on data received from a remote or central server. For example, a remote central server may provide audio or graphical data to the AR device, which is then processed locally to generate the desired AR experience.

[0048] 1 shows an example of an AR system in which a remote AR client device 101 communicates with an AR server 103 over a network 105, such as the Internet. The server 103 is configured to support multiple client devices 101 simultaneously.

[0049] For example, the AR server 103 supports an augmented experience by transmitting data to the client device 101 that defines elements and objects in the virtual environment. This data specifically describes the visual features and geometric characteristics of numerous virtual objects, which the client device 101 uses to generate overlay graphics that can be presented to the user. In some embodiments, the data also includes various information that can be presented to the user. Additionally, the server 103 provides the client device 103 with audio data that can be used to locally generate virtual sounds / audio that can further enhance the user experience, particularly the sense of immersion.

[0050] Figure 2 illustrates a device according to some embodiments of the invention, specifically client device 101 of Figure 1, and will be described with reference to such embodiments.

[0051] The apparatus includes a first receiver 201 that receives image data from one or more image sensors. The image data from the image sensors specifically includes a real-time image sequence of frames / images of a real-world scene.

[0052] In many embodiments, a frame comprises visual image data from a visual image camera. The frame / image data may include a real-time sequence of visual images in which each pixel represents the intensity of light received from the pixel's view direction. For example, each pixel includes a set of light brightness / intensity values ​​for a (possibly weighted) interval of the visible spectrum. For example, pixel values ​​in a visual image represent one or more brightness levels, such as a set of brightness / intensity values ​​for color channels. For example, the image is an RGB image. In many embodiments, the visual image is a color image and / or an RGB image (and in some embodiments, references to visual images are replaced with references to color images and / or RGB images).

[0053] Alternatively or additionally, the image sensor may be a depth sensor that provides a real-time sequence of depth images / frames. In such depth images, each pixel value represents the depth / distance to an object in the pixel's view direction. For example, each pixel value may be a disparity value or a depth value. A depth image is also called a depth map.

[0054] Thus, in different embodiments, the frames / images received from the image sensor may be, for example, visual / color images, infrared images, depth images, radar images, multispectral images, sonar images, phase images, phase contrast images, intensity images, coherence magnitude images, and / or confidence images, etc. The sequence of images / frames is thus a two-dimensional structure of pixels containing values ​​that represent properties of the real world in the pixel's view direction.

[0055] In many embodiments, the device processes both visual and depth images, and the first receiver 201 is capable of receiving both a sequence of real-time visual images of a real-world scene and a sequence of real-time depth images of the real-world scene. In many embodiments, the images are combined visual and depth images, and the first receiver 201 receives a sequence of real-time depth and visual images of the real-world scene.

[0056] Many different sensors and approaches for generating visual and depth images are known, and it will be appreciated that any suitable approach and image sensor may be used. For example, a conventional video camera may be used as an image sensor for generating visual images. For example, depth images may be generated using a dedicated depth camera, such as an infrared range camera, or may be generated by disparity estimation, such as based on two visual cameras with a known physical offset.

[0057] The image sensors capture a real-world scene, and the received images include a capture of this real-world scene. For example, in many AR applications, the received images are images of a real-world scene corresponding to the user's environment. In particular, a user wears an AR headset or glasses that include one or more image sensors that capture the real-world scene in the direction the viewer is looking.

[0058] The device further includes an image generator 203 that generates and outputs a visual image sequence including image objects that represent the virtual scene objects. The output visual image sequence is presented to a user using an appropriate display, such as a display (or displays) of an AR headset or AR glasses. Thus, when presented with the output visual image sequence, the user perceives the image objects as corresponding to virtual objects present in the real-world scene.

[0059] The image generator 203 adapts the image objects (orientation within the image, position within the image, view direction, etc.) in response to changes in the user's pose (position and / or orientation), for example, as provided by data received from appropriate motion sensors, so that the image objects are presented in a way that gives the impression that the virtual objects are present in the viewed real-world scene.

[0060] In some embodiments, the generated output visual image sequence includes only the virtual object (or virtual objects, if there are multiple), and is presented using, for example, see-through glasses, allowing the real-world scene to be seen through the glasses, such an approach being known as an AR see-through application.

[0061] In other embodiments, the output visual image sequence is generated to also include a representation of the real-world scene captured by the image sensor. Thus, in such an approach, the real-world scene is also (or only) viewable through the generated images. Such an approach is known as an AR pass-through application.

[0062] It will be appreciated that many different approaches are known for generating such AR images and for adapting and modifying the generated image objects to give the impression that the virtual objects are present in a real-world scene. For the sake of brevity, such features will not be described in detail. It will be appreciated that any suitable approach may be used without diverging from the present invention. For example, algorithms and approaches from developed AR toolkits, such as the ARKit and ARCore processes, may be used.

[0063] Thus, the image generator 203 generates image objects that allow the virtual scene object to be perceived as having been added to the real-world scene. In many embodiments, the virtual scene object is movable relative to at least one real-world scene object, and typically the virtual scene object is movable relative to the entire real-world scene. For example, the virtual scene object may move in response to user actions, changes in properties of the real-world scene, predetermined movements to create a predetermined effect, etc.

[0064] The device further includes a detector 205 that detects interactions between virtual scene objects and real-world scene objects in the real-world scene. Specifically, the detector 205 detects collisions or contacts between virtual scene objects and real-world scene objects. Other possible interactions may include, for example, detecting proximity between a virtual scene object and a real-world scene object, or the duration of contact, which is the time interval during which a virtual object and a real-world object are certain to have contacted each other. Another example of an interaction is when it rains in the real world and raindrops fall on a virtual object. Another example is when air currents occur in the real world and the air flows toward a virtual object.

[0065] Thus, detector 205 detects an interaction between a virtual scene object and a real-world scene object of a real-world scene in response to detecting proximity between the virtual scene object and the real-world scene object, which may be based on any suitable proximity criteria.

[0066] Detecting proximity between a virtual scene object and a real-world scene object is detecting that the distance between the position of the real-world scene object and the position of the virtual scene object satisfies a proximity criterion. Detecting proximity between a virtual scene object and a real-world scene object is detecting that the distance between the position of the real-world scene object and the position of the virtual scene object is less than a threshold. The positions and distances are determined in the scene coordinate system of the real-world scene. A virtual scene object is an object that is presented / displayed as if it were a real-world scene object present in a real-world scene.

[0067] In some embodiments, detector 205 determines a (first) position of a real-world scene object in a (scene) coordinate system (for the scene) and a (second) position of a virtual scene object in said coordinate system. Detector 205 detects an interaction in response to, and in particular if, the (first and second) positions satisfy a proximity criterion. Specifically, detector 205 detects an interaction in response to, and in particular if, a distance (according to any suitable distance measure) between the (first and second) positions satisfies a distance proximity criterion, and in particular if the distance is less than a threshold.

[0068] The image generator generates image objects to represent the virtual scene object as an object with a given pose / spatial presence in a real-world scene / scene coordinate system. Proximity is determined according to a position indicating this pose / spatial presence in the real-world scene / scene coordinate system and a position of the real-world scene object in the real-world scene / scene coordinate system. Proximity is determined according to a position of this virtual scene object / spatial presence in the real-world scene / scene coordinate system and a position of the real-world scene object in the real-world scene / scene coordinate system.

[0069] Different approaches to detecting interactions are used in different embodiments depending on the desired operation and performance of the particular embodiment, and various approaches are described below.

[0070] The detector 205 is coupled to an estimator 207 that determines the material properties of the detected scene objects with which the virtual scene objects interact. The material properties indicate the material / substance / thing that the real-time object is made of (including its material composition if the object contains various materials). Material properties may be generated, for example, to indicate whether the real-world object is made of one of several categories (e.g., whether it is made of wood, cloth, metal, plastic, etc.).

[0071] The estimator 207 determines the material properties in response to image data received from the image sensor. That is, the material properties are determined in response to data of a sequence of image frames received from the sensor. In many embodiments, the sequence of image frames includes both depth image data and visual image data, and the estimator 207 estimates the material properties in response to both the depth image data and the visual image data. In other embodiments, the estimator 207 considers only visual data or only depth data.

[0072] It will be appreciated that different algorithms and approaches may be used to estimate material properties in different embodiments, and that in many embodiments, even highly inaccurate and unreliable estimates of material properties may be useful and provide an improved user experience.

[0073] In a low-complexity implementation, the estimator 207 may, for example, apply color-based segmentation to the received visual image to determine material properties and determine image objects that are likely to match the real-time image object. The average color may then be compared to a set of predefined categories to find the best match. The material properties of the real-time image object may be set to the material properties associated with the closest category. As a simple example, if a real-world image object is determined to be primarily brown, it may be inferred to be made of wood; if it is primarily silver, it may be inferred to be metal; if it is primarily bright primary colors, it may be inferred to be plastic, etc.

[0074] It will be appreciated that in most embodiments, more complex and more accurate approaches to estimating material properties may be used, and further examples are provided below.

[0075] The device further includes a second receiver 209 that receives a number of audio objects and audio object metadata. An audio object is a sound / audio clip / audio fragment, specifically an audio signal of limited duration. It will be understood that an audio object may use any representation and format to represent a sound. Thus, an audio object corresponds to a sound that is selected by the device and rendered to provide an audio output when various actions occur.

[0076] Specifically, the device includes a selector 211 that selects an audio object from among the received audio objects when an interaction between a virtual scene object and a real-world audio object is detected. The selected audio object is provided to an output circuit 213 that generates an output audio signal that includes the first audio object. Thus, the output circuit 213 renders the selected audio object by generating an output audio signal that includes a rendering of the audio object.

[0077] It will be appreciated that, depending on the format of the audio object, the output circuitry 213 may in some cases simply generate an output signal as the audio of the audio object, or in some embodiments may generate an output audio signal to include decoding of the audio object or to include mixing between various audio components (ambient sounds, narrator audio, etc.) that are combined with the audio object. It will also be appreciated that the output circuitry 213 may include digital-to-analog conversion and amplification of the analog signal in embodiments in which the output circuitry 213 generates an analog output audio signal, for example to drive speakers directly.

[0078] The output circuitry 213 may include rendering of audio objects, for example, by binaural processing using Head-Related Impulse Responses (HRIRs), Head-Related Transfer Functions (HRTFs), or Binaural Room Impulse Responses (BRIRs), or rendering to speaker configurations, for example, by Vector-Based Amplitude Panning (VBAP), or with additional acoustic stimuli, for example, occlusion, diffraction, reflections, reverberation, source range, etc. In particular, the rendering is configured such that the sound of the audio object is perceived as originating from the detected interaction location.

[0079] In some embodiments, the output circuitry 213 includes timing adaptations, such as variable delays, to control the timing of the rendering of selected audio objects. For example, delays may be included between the time of "visual" contact / interaction and the time of rendering the sound. The delays may be adjusted based on, for example, the speed of sound or the distance between the virtual object and the observer, so that, for example, the generated sound is perceived simultaneously with the visual interaction. In some embodiments, such timing adjustments may be performed in other parts of the device other than the output circuitry 213.

[0080] The metadata received by the second receiver 209 includes metadata indicating associations between at least some of the audio objects and material characteristics. For example, the metadata indicates, for each audio object, the material of the associated real-world object. For example, the metadata indicates that one audio object is associated with a real-world object made of wood, another audio object is associated with a real-world object made of metal, and another audio object is associated with a real-world object made of plastic.

[0081] As will be described in more detail below, in many embodiments, the metadata may include other additional associations, such as associations between audio objects and virtual scene objects, associations between audio objects and material characteristics of virtual scene objects, and types of interactions.

[0082] In response to the detector 205 detecting an interaction between a virtual scene object and a real-world scene object, the selector 211 selects an audio object based on the estimated material properties of the real-world scene object and metadata indicating the association of the audio object with the material characteristics.

[0083] In a low complexity embodiment, the selector 211 simply selects the audio object that matches the estimated material property and the associated material feature, for example, if the estimated property is "wood," then the audio object associated with "wood" is selected and a corresponding output audio signal is generated.

[0084] In many embodiments, the device can improve the user experience, and in particular provide a more immersive and natural sound experience in which the perceived audio more closely reflects the interaction between virtual scene objects and real-world objects. For example, rather than simply accompanying interactions (such as collisions) between virtual scene objects and real-world objects with uniform audio, the device can provide audio output that more closely matches the user's visual perception, providing a closer correspondence between the visual and audio inputs perceived by the user. This is achieved by the device adapting and generating audio that more closely matches the interaction.

[0085] The system uses, for example, multiple recorded or synthesized audio objects / clips / fragments provided for virtual scene objects. Each audio fragment may model an interaction with a particular class of real-world object that may occur in the scene. At runtime, the device uses its built-in camera to, for example, classify the surface on which the virtual object is placed, and based on this classification, plays the audio fragment appropriate for the particular interaction.

[0086] In some embodiments, the second receiver 209 receives the audio objects and / or metadata from an internal source, but in many embodiments, the second receiver 209 receives the metadata, and often the audio objects, from a remote server (particularly server 103 of FIG. 1).

[0087] For example, the remote server may maintain a large library of audio objects corresponding to various possible interactions between objects, and may also maintain metadata including data defining the material properties of the objects involved in the interaction, the characteristics of the interaction, etc.

[0088] Upon application initialization or at a repetition rate, the second receiver 209 may retrieve metadata for all audio objects. When an interaction is detected by the detector 205, the estimator 207 estimates the material properties of the real-world scene objects, and the selector 211 evaluates the metadata to find matching audio objects. The second receiver 209 is then controlled to retrieve the selected audio objects from the server 103. Upon receipt, the audio device generates an audio signal including the received audio objects.

[0089] In some embodiments, such an approach may be too slow for real-time operation, and in some embodiments, the second receiver 209 retrieves a subset or all of the audio objects from the remote server 103 and stores them locally for faster retrieval. For example, in some embodiments, an analysis of the real-world scene is performed at startup or at regular intervals, with the device detecting as many objects as possible. The device then retrieves all audio objects corresponding to detected objects or materials corresponding to objects in the real-world scene, as well as virtual scene objects that are active or nearby near the user's location. In other embodiments, the user provides input for a real-world scene (e.g., living room, sports arena, etc.) at initialization, and the device retrieves a set of audio objects stored on the device as possible objects in that environment.

[0090] This approach therefore provides an efficient approach for enabling a central server to interact with remote devices to provide audio support for AR applications. It will be appreciated that the central server described above may also be represented by a relatively local server, such as a 5G edge server.

[0091] In this approach, virtual scene objects are presented to the user in a way that makes them appear to exist in the real-world scene, and are therefore associated with the spatial (usually three-dimensional) geometry / shape of the real-world scene. Detector 205 detects an interaction as detecting that the distance between a virtual scene object and a real-world scene object in three-dimensional real-world space is less than a given threshold, specifically, that they are touching each other. In some cases, additional considerations are taken into account, such as the speed at which the objects are moving toward each other.

[0092] For example, some toolkits developed to support AR applications offer 3D planar surface detection functionality, where planar regions are detected through tracking feature points over time and the specific motions followed by clusters of feature points over time.

[0093] In some embodiments, detector 205 detects interactions by tracking the shortest distance between any vertex in the bounding mesh of the virtual graphics scene object and any detected plane.

number

[0094] If the distance is less than a threshold, an interaction is detected to have occurred. Specifically, if the distance is less than a small threshold, or reaches zero, for example, a collision is detected to have occurred.

[0095] In some embodiments, the minimum distance from a vertex to a plane is a minimum threshold distance Δ collision The instantaneous object velocity v(t) decreases within a given minimum threshold v min When this is exceeded, an interaction, specifically a collision, is detected to have occurred. (d(t)≦Δ collision )Λ(v(t)>v min )

[0096] The reason behind the velocity condition is that when the object velocity is low it tends to be difficult in practice to determine whether a collision has actually occurred, since the virtual object may stop just in front of the detected surface or not. Useful parameter values ​​are, for example, Δ collision = 5 mm, v min =0.05m / s.

[0097] The estimator 207 then determines material properties of the real-world scene objects where an interaction is detected to have occurred. In many embodiments, the detection is based on the estimator 207 determining an interaction image region in at least one input image frame. This interaction image region is the image region of the image frame where the interaction occurred. Thus, the estimator 207 determines a two-dimensional region in the image frame that includes the contact point between the objects detected by the detector 205. For example, a projection of the contact point on the two-dimensional image frame is determined, and a surrounding area is identified.

[0098] In some embodiments, for example, the image region is a predetermined shape centered around the projected contact point. In other embodiments, the image region may be adaptive to image characteristics. For example, the image region may be identified as a region around the projected contact point that meets similarity criteria, such as visual characteristics or depth, such that the visual characteristics or depth do not deviate from the characteristics of the projected contact point by more than a predetermined amount.

[0099] In this case, the estimator 207 determines the material properties of the scene object according to the image data of the interactive image region, for example, the color or texture variations are compared with the corresponding properties of a predetermined set of classes, and the material properties are set to the properties of the closest matching class.

[0100] In many embodiments, the material property is determined by comparing the image property determined for the image data with a plurality of image property references. Each image property reference is associated with a material property value. The image property is determined as a material property value corresponding to / associated with a matching image property reference among the plurality of image property references. The matching image property reference is determined as the image property and the matching image property reference satisfy a matching criterion. The image property may be any or all of brightness, color, and / or texture.

[0101] Specifically, once a collision is detected, the interaction area in the image is determined by first finding the closest mesh point x using the camera view matrix and the camera projection matrix. i Then, the resulting 2D image points (u i ,v i ) to extract (u i ,v i 3 shows an example image frame in which a virtual scene object 301 in the shape of a ball interacts with a real-world scene object 303 in the shape of a real-world table, resulting in an image region 305 being detected.

[0102] In many embodiments, the estimator 207 determines a match indication for a real-world scene object for at least a first category / class of a plurality of object categories / classes. For example, the estimator 207 has a number of characteristics stored for image regions that correspond to particular materials, such as color, texture variations, depth variations, etc. The estimator 207 determines the corresponding characteristics for the image regions and compares these to the stored characteristics for all categories. A match indication is determined to reflect how well the characteristics match, and the category is selected as the category for which the match indication indicates the highest match.

[0103] The material properties of the real-world scene object are then selected as the stored properties of the selected category or class. Thus, each category / class is associated with one or more material properties, e.g., simply an indication of the material the object is made of. The material properties of the real-world scene object are then set to those stored in the selected category.

[0104] It will be appreciated that in many embodiments, the estimator 207 may use substantially more complex approaches to determine the material properties of real-world scene objects, and in particular to identify the class or category that is believed to most closely match the properties of the image region corresponding to the image region.

[0105] Indeed, in many embodiments, the estimator 207 includes a neural network that estimates material properties of real-world scene objects based on image data of the interactive image region. Neural networks are used, among other things, to identify categories or classes of real-world scene objects.

[0106] In many embodiments, the extracted (image) interaction region is much smaller in size than the full image: the full image is often 2K or 4K in size, while the interaction sub-image has a fixed small size, such as 256x256 pixels.

[0107] For example, a 256x256 pixel color image is represented as a tensor of dimensions 3x256x256 and fed into a trained classifier neural network. The neural network outputs a vector, providing the probability that the image belongs to a given class, given a set of predefined material classes.

[0108] In many embodiments, the following object (surface) classes may be particularly advantageous: Carpeted floor (short height variations in material) The floor is carpeted (the high height variations in the material suppress impact noise) The ground is stone The ground is wood The table surface is glass The table surface is wood wall Couch cushion Plastic surfaces of household objects (cookers, keyboards, laptop computers)

[0109] Each of these classes has specific color or texture statistics, or is distinguished by the geometry of its boundaries (e.g., tables and cushions).

[0110] As a training dataset, images from different object surfaces are captured and manually annotated using ground truth class labels.

[0111] In neural network architecture, an option is to use a 2D convolutional layer as the basic block, doubling the number of channels, followed by a 2D average pooling layer (stride=2) with rectified linear as the activation. Each block reduces the spatial resolution of the tensor by a factor of 2, but doubles the number of channels. This approach continues to combine these blocks until the tensor has a resolution N channels × 1 × 1. Then, two fully connected layers are added, followed by a sigmoid activation function.

[0112] If the maximum value of all class probabilities is below a given threshold (0.5 seems to be a suitable value in many embodiments), a default (neutral) interaction sound is played. That is, if the estimator 207 cannot determine the material of a real-world scene object accurately enough, a default audio object is selected. In all other cases, an audio object / sound file is selected by the selector 211. Specifically, the audio object corresponding to the material determined by categorization to have the highest probability is selected.

[0113] In some embodiments, the device may further include an audio receiver 215 that receives an audio signal of real-time audio captured in a real-world scene, and the estimator 207 may determine the match indication in response to the audio signal.

[0114] Therefore, material classification is aided by audio-based classification, which uses microphones to record sounds within a physical space. When interactions occur between physical objects, the resulting sounds provide valuable information for material classification or more specific properties. This can be used to better classify materials or provide further material attributes (solid / hollow, thin / thick, tension / flex, large / small).

[0115] For example, the present and recent past environmental sounds S env The audio is captured over a short time window (e.g., 10 seconds) and fed to a material classifier. For example, a user holding a smartphone, wearing an AR headset, or other person is walking through a room, and the sound of their shoes on the ground provides information about the ground's material. The material classifier is generated by collecting images and sounds of various materials. The more advanced neural network described above combines snippets of audio fragments with captured images and uses them as input for the trained network.

[0116] As a specific example, in some embodiments, the determination of the material property is made in response to the depth image data. Specifically, the estimator can determine the material property of the real-world scene object in response to detecting that at least a portion of an image region of at least one image frame representing the real-world scene object has a confidence level for the depth image data that does not exceed a threshold.

[0117] Depth estimates are often generated along with confidence data that indicates the reliability of the generated depth value / estimate. For example, in the case of disparity estimation, a confidence level is determined to reflect how closely the image regions that form the basis of the disparity estimate match within the image. As another example, in the case of a depth camera based on infrared light reflection, a confidence level is generated to reflect the amount of infrared light received by a given pixel. If only a small amount of light is received, the distance estimation / distance measurement process will not be as accurate as if a large amount of light were received. In some cases, each pixel in a depth image / map includes both an estimated depth value and a confidence value / level that indicates the reliability of the pixel's depth value.

[0118] In some embodiments, this depth confidence data is taken into account when determining material properties. For example, some materials are known to have lower confidence levels compared to other materials. For example, some materials have highly variable local visual appearance, resulting in less reliable disparity estimation.

[0119] As another example, depth estimation based on reflected light emitted from the sensor, especially reflected infrared light, can be significantly less reliable for some materials due to a significant reduction in the amount of light reflected back to the sensor. This is the case, for example, for materials that are non-reflective but highly absorbent of infrared light. This is also the case, particularly for materials that exhibit specular reflection (e.g., metal objects). In this case, only a small amount of infrared light is reflected back to the transmitter and sensor. Note that for active depth sensors based on time-of-flight or structured light, the infrared light sensor (e.g., CMOS) is often co-located with the optical transmitter / emitter. At the same time, the visual light received from a metal surface can be quite substantial due to the reflection of ambient light by the surface. In many embodiments, for example, the estimator 207 detects significantly brighter regions with lower depth confidence in certain parts of the interaction image region. Such regions may indicate highly reflective but dispersive surfaces, such as metal surfaces.

[0120] In some embodiments, the image frame includes visual image data and depth image data, and the estimator 207 determines that the real-world scene object has a metallic component in response to detecting that, for at least a portion of the image region, the brightness of the visual image data exceeds a threshold and the confidence level of the depth image data does not exceed a threshold.

[0121] As previously described, the metadata is received from a remote source and includes an association between an audio object and the material properties of an object (specifically, a real-world scene object). However, in some embodiments, the metadata may include an association to the material properties of two objects, and may actually distinguish between a virtual scene object and a real-world scene object. In such cases, the selector 211 may select an audio object based on both the estimated material properties of the real-world scene object and the material properties of the virtual scene object. Because the virtual scene object is a virtual object generated by a device, the material from which it is perceived to be made is typically known. Therefore, in this case, the audio object is selected to match both properties as closely as possible.

[0122] Thus, in some embodiments, the metadata of at least some of the audio objects includes an indication of an association of at least some of the audio objects with material characteristics of real-world scene objects and an indication of an association of at least some of the audio objects with material characteristics of virtual scene objects. In such a system, the selector 211 selects audio objects according to the material properties and material characteristics of the real-world objects associated with the set of audio objects, and according to the material properties and material characteristics of the virtual scene objects associated with the set of audio objects.

[0123] As an example, the remote server 103 provides a two-dimensional structure that associates audio objects with material properties. An example of such a structure is the following two-dimensional lookup table: [Table 1]

[0124] In other embodiments, the metadata provides associations to other characteristics, and the selection of audio objects takes such additional characteristics into account.

[0125] For example, in many embodiments, dynamic characteristics of the virtual scene objects are further considered. In such embodiments, the metadata indicates that different audio objects are associated with different states / characteristics of the virtual scene objects. The states / characteristics of the virtual scene objects change dynamically. Then, when finding a matching audio object, the selection is made by further considering the current state / characteristics.

[0126] For example, a deflated virtual ball makes a different sound than an inflated virtual ball. In some embodiments, the ball virtual scene object is included in the lookup table twice, once when deflated and once when inflated, each associated with a different audio object / sound effect.

[0127] Further attributes of the virtual object may influence the selection or generation of an appropriate sound effect, for example, different sound effects may occur depending on the orientation of the virtual object.

[0128] A virtual scene object may have different types of surfaces, such as a cube with different surface properties on different sides, in which case the orientation of the virtual scene object determines which surfaces will collide with the real-world scene object, and audio object selection takes this orientation into account.

[0129] As a specific example, the lookup table contains different entries for different orientations, and thus selecting an audio object involves selecting the entry that corresponds to the current orientation. Virtual scene objects can be represented by meshes, with each face or face group associated with a range of object orientations, virtual material properties, or a subset of audio objects.

[0130] Finally, further attributes of real-world scene objects can be detected and used to select or generate sound effects: an open door sounds different from a closed door; a glass sounds different if it contains liquid;

[0131] In some embodiments, the selection of the audio object may further depend on the characteristics of the interaction, in particular the type of interaction. Thus, in some embodiments, the detector 205 determines the characteristics of the interaction and the selector 211 selects the first audio object depending on the characteristics of the interaction.

[0132] Thus, in some embodiments, attributes of the interaction itself may influence the selection or generation of sound effects. Physical collisions may be nonlinear with the intensity of the collision, so that the speed or strength of the interaction will cause different sounds (e.g., a ball bouncing, glass breaking at a certain collision intensity). Other examples include the direction of the collision or non-collision interactions such as a virtual object sliding / rubbing on a physical surface.

[0133] By way of example, interaction characteristics that may be further considered in the selection of an audio object may be one or more of the following:

[0134] Interaction speed:For example, the detector 205 determines the relative velocity of a virtual scene object and a real-world scene object and selects an audio object based on this. For example, a ball hitting a floor slowly has a different sound than a ball hitting the floor at a much higher velocity. For example, a lookup table may have different entries for different velocity ranges. Each entry is associated with an audio object representing a recording of a ball hitting the appropriate flooring material at a different velocity. As an example, interaction detection may include a velocity vector being compared to the orientation of a physical surface, and optionally its behavior immediately after the interaction to enable detection of the interaction type. For example, if the object's velocity vector is perpendicular to and points toward the physical surface, the interaction is considered an "impact" or "stick" interaction type. A velocity vector along the object's surface results in a "slide" interaction type. These different interaction types are associated with different audio objects.

[0135] Between virtual scene objects and real-world scene objects Collision Force For example, the force is determined as a function of the current velocity and the estimated or assumed weight of the virtual scene object and / or the real-world scene object, with different force impulses resulting in different sounds and different audio objects being provided or selected based on the force.

[0136] Virtual scene objects and real-world scene objects Collision Elasticity For example, a highly inflated ball will sound different from a partially deflated ball when hitting the floor. The different elasticity of the balls will result in very different sounds. In particular, in some embodiments, different audio objects are provided for different elasticity of the virtual scene object, and the metadata may include data for distinguishing the audio objects based on such elasticity.

[0137] Interaction duration:For example, the device may include or communicate with a physics engine that predicts an updated trajectory of a virtual scene object after an interaction with a real-world object is detected. From the predicted trajectory, the duration of the interaction is derived due to the momentum, inertia, direction, or elasticity of the interaction or the virtual and real-world materials involved in the interaction. A long duration of the interaction results in an audio object with a long or nominal duration, while a shorter duration results in an audio object with a shorter duration or a shorter rendering of the nominal audio object. In other examples, the interaction results from the user interacting with the virtual content, such as dragging the virtual scene object through the real-world environment. This interaction continues as long as the user keeps the virtual scene object in proximity to or in (virtual) contact with the real-world object.

[0138] Virtual scene objects versus real-world scene objects Direction of movement For example, sliding a virtual object along a real-world wooden surface while touching it will generally produce a higher-pitched sound than locally hitting the real-world wooden surface in a direction parallel to the surface normal.

[0139] In many embodiments, the metadata is provided in the form of a look-up table. The look-up table may be multi-dimensional depending on the number of characteristics that are / should be considered when selecting an audio object. For example, a look-up table may be provided with the following dimensions: Dimension 1: Virtual Materials Dimension 2: Detected real-world materials Dimension 3: Type of interaction (impact / sliding / bouncing / attachment / detachment, etc.) Dimension 4: Speed ​​of interaction

[0140] In some embodiments, a metadata table is provided that essentially associates two material types with each sound effect. However, there may be many sound effects for the same material and even interaction type (Material 1, Material 2, and Interaction Type). Based on this information, they may all be suitable to represent a particular interaction. One approach is to always choose the first one, but the experience may be more convincing if the sound effect is selected randomly. For repetitive interactions, small audio differences can make them sound less artificial. The selection of sound clips can even vary based on interaction attributes not covered by the library metadata (such as speed, direction of impact, or the nth bounce of the ball).

[0141] A specific approach of the apparatus of Fig. 2 is shown in the flow diagram of Fig. 4. In this embodiment, the current video image I is analyzed to predict image regions where interactions are likely to occur. This analysis is performed based on 3D position information of the virtual scene objects / graphics objects G. Image portions of the resulting spatial image region R where the virtual scene objects are likely to collide with real-world objects of the real environment are fed to a material classifier (estimator 207) which determines a material class M. Finally, depending on the material class M and the virtual scene objects / graphics objects G, a sound is generated.

[0142] Surface material classification based solely on visual information is likely sufficient for many practical implementations and applications. Furthermore, if no matching category or sound is identified, the default interaction sounds of the virtual scene object G can be used, rather than those specific to the real-world object's material. Optionally, current and recent past environmental sounds S env can be input into the material classifier.

[0143] For example, this approach enables an application to begin playing a pre-recorded sound as soon as a collision between a virtual scene object and a real-world scene object occurs. Rather than specifying every possible real-world material interaction sound for each virtual object, each virtual object is given material properties and a single lookup table is used as the basis for selecting the appropriate sound. Even if the material classifier does not generate a sufficiently high probability for any of the predefined classes, an "unknown" class can be used to generate a nearly accurate sound. For example, a virtual rubber ball will make more or less the same sound even if it hits different real-world surfaces.

[0144] The AR standard provides a syntax for bitstreams that carry audio data, visual data, and metadata for virtual elements to augment a particular physical area. The standard allows users to interact with these virtual elements in their physical environment. To augment the audio of these interactions, the standard provides a means to send a number of sound effect clips to the decoder. These sound effect clips are not rendered to the user unless triggered.

[0145] The provided bitstream includes metadata that indicates a multi-dimensional lookup table. This lookup table assigns every sound effect clip to its entries. Each dimension of the table corresponds to an aspect of an interaction between a virtual element and a physical element. If an interaction has aspects that jointly identify a particular table entry that are present in this table (e.g., as detected by classification and / or logic algorithms), the sound effect clip associated with this table entry is played at the location of the interaction between the physical element and the virtual element.

[0146] It will be understood that, for clarity, the above description describes embodiments of the invention with reference to various functional circuits, units, and processors. However, it will be apparent that functionality may be distributed among various functional circuits, units, or processors as appropriate without detracting from the invention. For example, functionality described as being performed by separate processors or controllers may be performed by the same processor or controller. Thus, references to specific functional units or circuits are to be seen merely as references to suitable means for providing the described functionality, rather than to any strict logical or physical structure or organization.

[0147] The invention can be implemented in any suitable form including hardware, software, firmware or any combination of these. The invention may optionally be implemented at least partly as computer software running on one or more data processors and / or digital signal processors. The elements and components of embodiments of the invention may be physically, functionally, and logically implemented in any suitable way. Indeed, functionality may be implemented in a single unit, in several units or as part of other functional units. Thus, the invention may be implemented in a single unit or may be physically and functionally distributed between different units, circuits, and processors.

[0148] While the present invention has been described in connection with some embodiments, it is not intended to be limited to the specific form set forth herein. Rather, the scope of the present invention is limited only by the appended claims. Moreover, while certain features may appear to be described in connection with particular embodiments, those skilled in the art will recognize that various features of the described embodiments may be combined in accordance with the present invention. In the claims, the term "comprising" does not exclude the presence of other elements or steps.

[0149] Furthermore, although individually listed, a plurality of means, elements, circuits, or method steps may be implemented by, for example, a single circuit, unit, or processor. Moreover, although individual features may be included in different claims, these features may also be advantageously combined, and the inclusion of features in various claims does not imply that such combinations are not feasible and / or advantageous. Furthermore, the inclusion of a feature in one category of claims does not imply a limitation of this category, but rather indicates that the feature may be applied to other claim categories as well, where appropriate. Furthermore, the order of features in the claims does not imply a particular order in which the features must function, and in particular the order of individual steps in method claims does not imply that the steps must be performed in this order. Rather, steps may be performed in any suitable order. Furthermore, a reference in the singular does not exclude a reference in the plural. Thus, a reference to "first," "second," etc. does not exclude a reference in the plural. Reference signs in the claims are provided solely as examples for clarity, and these examples should not be construed as limiting the scope of the claims in any way.

[0150] A computer program product is provided comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method of claim 13.

[0151] Embodiments: [Claim 1] 1. An apparatus for generating an output audio signal, comprising: a first receiver (201) for receiving a real-time image sequence of a real-world scene from an image sensor, the real-time image sequence including a sequence of image frames, each image frame including at least one of visual image data and depth image data; a second receiver (209) for receiving a set of audio objects and metadata of the audio objects of said set of audio objects, said metadata indicating associations of the audio objects of said set of audio objects with material features; an image generator (203) for generating an output image sequence including image objects corresponding to virtual scene objects of said real-world scene; a detector (205) for detecting interactions between the virtual scene objects and real-world scene objects of the real-world scene; an estimator (207) for determining material properties of said real-world scene object in response to image data of said sequence of image frames; a selector (211) for selecting a first audio object of said set of audio objects in response to said material properties and said material features associated with an audio object of said set of audio objects; an output circuit (213) for generating the output audio signal including the first audio object; 1. An apparatus comprising: [Claim 2] The estimator (207) is determining an interaction image area in at least one image frame of said sequence of image frames, said interaction image area being an image area of ​​said at least one image frame in which said interaction occurs; determining the material properties of the scene object in response to image data of the interactive image region; The apparatus of claim 1 , [Claim 3] 3. The device according to claim 1, wherein the second receiver (209) receives the metadata from a remote server. [Claim 4] 4. The apparatus of claim 1, wherein the metadata of at least some of the audio objects includes an indication of an association between at least some of the audio objects and material characteristics of real-world scene objects and an indication of an association between at least some of the audio objects and material characteristics of virtual scene objects, and wherein the selector (211) selects the first audio object according to the material properties and material characteristics of real-world objects associated with the set of audio objects, and according to the material properties of the virtual scene objects and material characteristics of virtual scene objects associated with the set of audio objects. [Claim 5] 5. The apparatus according to claim 1, wherein the selector (211) selects the first audio object depending on a dynamic property of the virtual scene object. [Claim 6] 6. The apparatus of claim 1, wherein the detector (205) determines a characteristic of the interaction, and the selector (211) selects the first audio object depending on the characteristic of the interaction. [Claim 7] The above properties of the above interactions are: The speed of the above interactions, the force of a collision between the virtual scene object and the real-world scene object; the elasticity of a collision between the virtual scene object and the real-world scene object; the duration of said interaction, and a direction of movement of the virtual scene object relative to the real-world scene object; 7. The device of claim 6, wherein the at least one characteristic is selected from the group consisting of: [Claim 8] 8. The device of claim 1, wherein the selector (211) selects the first audio object depending on an orientation of the virtual object relative to the real-world scene object. [Claim 9] 9. The apparatus of claim 1, wherein the estimator determines a match indication of the real-world scene object for at least a first category among a plurality of object categories, and determines the material property depending on the match indication and a material property associated with the object category. [Claim 10] 10. The apparatus of claim 9, further comprising an audio receiver (215) for receiving an audio signal of real-time audio captured in the real-world scene, wherein the estimator determines the match indication in response to the audio signal. [Claim 11] 11. The apparatus of claim 1, wherein the selector (211) selects the first audio object as a default audio object if no audio object satisfying a selection criterion is found. [Claim 12] 12. The apparatus of claim 1, wherein at least one image frame includes depth image data, and wherein the estimator (207) determines the material property of the real-world scene object in response to detecting that at least a portion of an image region of the at least one image frame representing the real-world scene object has a confidence level for the depth image data that does not exceed a threshold. [Claim 13] 1. A method for generating an output audio signal, comprising: receiving a real-time image sequence of a real-world scene from an image sensor, the real-time image sequence comprising a sequence of image frames, each image frame comprising at least one of visual image data and depth image data; receiving a set of audio objects and metadata for the audio objects of the set of audio objects, the metadata indicating associations between the audio objects of the set of audio objects and material features; generating an output image sequence including image objects corresponding to virtual scene objects of said real-world scene; detecting an interaction between the virtual scene object and a real-world scene object of the real-world scene; determining material properties of said real-world scene object in response to image data of said sequence of image frames; selecting a first audio object of said set of audio objects in response to said material properties and said material features associated with audio objects of said set of audio objects; generating the output audio signal including the first audio object; A method comprising: [Claim 14] 14. A computer program product comprising computer program code means adapted to perform all the steps of claim 13 when said program is run on a computer. [Claim 15] 1. An audio signal comprising: a set of audio objects; and metadata for said audio objects, said metadata indicating associations between the audio objects and material characteristics of real-world objects.

Claims

1. 1. An apparatus for generating an output audio signal, the apparatus comprising: a first receiver for receiving a real-time image sequence of a real-world scene from an image sensor, the real-time image sequence including a sequence of image frames, each image frame including at least one of visual image data and depth image data; a second receiver for receiving a set of audio objects and metadata of the audio objects of the set of audio objects, the metadata indicating associations of the audio objects of the set of audio objects with material features; an image generator for generating an output image sequence including image objects corresponding to virtual scene objects of the real world scene; a detector for detecting an interaction between the virtual scene object and a real-world scene object of the real-world scene in a three-dimensional coordinate system representing the real-world scene in response to detecting proximity of the virtual scene object to the real-world scene object; an estimator for determining material properties of the real-world scene object based on image data contained within an image frame of the sequence of image frames; a selector for selecting a first audio object of the set of audio objects in response to the material characteristics and the material features associated with an audio object of the set of audio objects; an output circuit for generating the output audio signal including the first audio object; a circuit for presenting the output image sequence to a user; 13. An apparatus comprising:

2. The estimator may be determining an interaction image area in at least one image frame of the sequence of image frames, the interaction image area being an image area of ​​the at least one image frame in which the interaction occurs; determining the material property of the real-world scene object in response to image data of the interactive image region; The apparatus of claim 1 , further comprising:

3. The apparatus of claim 1 or 2, wherein the second receiver receives the metadata from a remote server.

4. 4. The apparatus of claim 1, wherein the metadata of at least some of the audio objects includes an indication of an association of the at least some of the audio objects with the material characteristics of the real-world scene objects and an indication of an association of the at least some of the audio objects with the material characteristics of the virtual scene objects, and the selector selects the first audio object in response to the material characteristics and the material characteristics of the real-world scene objects associated with the set of audio objects, and in response to the material characteristics of the virtual scene objects and the material characteristics of the virtual scene objects associated with the set of audio objects.

5. The apparatus of claim 1 , wherein the selector selects the first audio object depending on dynamic characteristics of the virtual scene object.

6. The apparatus of claim 1 , wherein the detector determines a characteristic of the interaction, and the selector selects the first audio object in response to the characteristic of the interaction.

7. The characteristic of the interaction is the rate of said interaction; the force of a collision between the virtual scene object and the real-world scene object; the elasticity of collisions between the virtual scene object and the real-world scene object; the duration of said interaction, and a direction of movement of the virtual scene object relative to the real-world scene object; The apparatus of claim 6, wherein the at least one characteristic is selected from the group consisting of:

8. The apparatus of claim 1 , wherein the selector selects the first audio object depending on an orientation of the virtual scene object relative to the real-world scene object.

9. 9. The apparatus of claim 1 , wherein the estimator determines a match indication of the real-world scene object for at least a first category among a plurality of object categories, and determines the material property depending on the match indication and the material property associated with the object category.

10. The apparatus of claim 9 , further comprising an audio receiver for receiving an audio signal of real-time audio captured in the real-world scene, the estimator determining the match indication in response to the audio signal.

11. 11. The apparatus of claim 1, wherein the selector is adapted to select the first audio object as a default audio object if no audio object satisfying a selection criterion is found.

12. 12. The apparatus of claim 1, wherein at least one image frame includes depth image data, and the estimator determines the material property of the real-world scene object in response to detecting that at least a portion of an image region of the at least one image frame representing the real-world scene object has a confidence level for the depth image data that does not exceed a threshold.

13. 1. A method for generating an output audio signal, the method comprising: receiving a real-time image sequence of a real-world scene from an image sensor, the real-time image sequence including a sequence of image frames, each image frame including at least one of visual image data and depth image data; receiving a set of audio objects and metadata for the audio objects of the set of audio objects, the metadata indicating associations between the audio objects of the set of audio objects and material features; generating an output image sequence including image objects corresponding to virtual scene objects of the real world scene; detecting, in a three-dimensional coordinate system representing the real-world scene, an interaction between the virtual scene object and a real-world scene object of the real-world scene in response to detecting proximity between the virtual scene object and the real-world scene object; determining material properties of the real-world scene object based on image data contained within an image frame of the sequence of image frames; selecting a first audio object of the set of audio objects in response to the material properties and the material features associated with an audio object of the set of audio objects; generating the output audio signal including the first audio object; presenting said output image sequence to a user; A method comprising:

14. A computer program comprising computer program code means for performing all the steps of the method according to claim 13 when said computer program code means is executed on a computer.

Images