Apparatus and method for playing a spatially extended audio source or for generating a bitstream from a spatially extended audio source - Patents.com

Abstract

An apparatus for reproducing a spatially extended sound source having a position and geometry defined in space, the apparatus comprising: an interface (100) for receiving a listener position; a projector (120) for calculating a projection onto a projection plane of a two-dimensional or three-dimensional convex hull associated with the spatially extended sound source using the listener position, information about the geometry of the spatially extended sound source, and information about the position of the spatially extended sound source; a sound position calculator (140) for calculating at least two sound source positions for the spatially extended sound source using the projection plane; and a renderer (160) for rendering the at least two sound sources at said positions to obtain a reproduction of the spatially extended sound source having two or more output signals, the renderer (160) configured to use different sound signals for different positions, the different sound signals being associated with the spatially extended sound source.

Description

[Technical field]

[0001] The present invention relates to audio signal processing, in particular to the encoding or decoding or reproduction of spatially extended sound sources. [Background technology]

[0002] The reproduction of sound sources for several speakers or headphones has been studied for a long time. The simplest way to reproduce sound sources for such a setup is to render them as point sources, i.e. very (ideally, infinitely) small. However, this theoretical concept has difficulty in modeling existing physical sound sources in a realistic way. For example, a grand piano has a large vibrating wooden closure with many spatially distributed strings inside, and therefore sounds perceptually much larger than a point source (especially when the listener (and microphone) is close to the grand piano). Many real-world sound sources have a significant size ("spatial spread"), such as musical instruments, machines, orchestras or choirs, or ambient sounds (water drops).

[0003] Correct / realistic reproduction of such sound sources is the goal of many sound reproduction methods, whether it be binaural reproduction using headphones (i.e. using the so-called Head-Related Transfer Function HRTF or Binaural Room Impulse Response BRIR) or using traditional loudspeaker setups ranging from two loudspeakers ("stereo") to many loudspeakers arranged in a horizontal plane ("surround sound") and many loudspeakers surrounding the listener in all three dimensions ("3D audio").

[0004] The object of the present invention is to provide a concept for encoding or reproducing spatially extended sound sources, possibly with complex geometric shapes.

[0005] 2D sound source width This section describes methods involved in rendering augmented sound sources on a 2D surface as seen from the listener's perspective, e.g., at a specific azimuth range with 0 degrees elevation (as in traditional stereo / surround sound) or at a specific range of azimuth and elevation angles (when the user's movement, i.e., head rotation in pitch / yaw / roll axes, has three degrees of freedom ["3DoF"], as in 3D audio or virtual reality).

[0006] Increasing the apparent width of an audio object panned between two or more loudspeakers (generating a so-called phantom image or phantom source) can be achieved by decreasing the correlation of the participating channel signals (Blauert, (2001), s. 241-257). As the correlation decreases, the spread of the phantom source increases until it covers the entire range between the loudspeakers for correlation values ​​close to zero (and not too wide an opening angle).

[0007] Decorrelated versions of the source signal can be obtained by deriving and applying appropriate decorrelation filters. Lauridsen (Lauridsen, 1954) proposed to add / subtract a time-delayed and scaled version of the source signal to itself to obtain two decorrelated versions of the signal. More complex approaches were proposed, for example, by Kendall (Kendall, 1995), who iteratively derived a pair of decorrelated all-pass filters based on a combination of random number sequences. Faller et al. proposed an appropriate decorrelation filter (“diffuser”) in (Baumgarte & Faller, 2003). Zotter et al. also derived a pair of filters in which frequency-dependent phase or amplitude differences are used to realize the extension of phantom sources (Zotter & Frank, 2013). Furthermore, (Alary, Politis, & Vaelimaeki, 2017) proposed a decorrelation filter based on velvet noise, which was further optimized by (Schlecht, Alary, Vaelimaeki, & Habets, 2018).

[0008] In addition to reducing the correlation of the corresponding channel signals of phantom sources, the source width can also be increased by increasing the number of phantom sources due to audio objects. In (Pulkki, 1999), the source width is controlled by panning the same source signal in (slightly) different directions. This method was originally proposed to stabilize the perceived phantom source spread of VBAP-panning (Pulkki, 1997) source signals when moving in a sound scene. This has the advantage that, depending on the source direction, the rendered source may be played by two or more loudspeakers, which may result in undesirable changes in the perceived source width.

[0009] DirAC for virtual worlds (Pulkki, Laitinen, & Erkut, 2009) is an extension of the traditional Directional Audio Coding (DirAC) (Pulkki, 2007) approach for speech synthesis in virtual worlds. To render spatial coverage, directional sound components of a sound source are panned randomly within a certain range around the original direction of the sound source, and the pan direction varies with time and frequency.

[0010] A similar approach was pursued in (Pihlajamaeki, Santala, & Pulkki, 2014), where spatial coverage was achieved by randomly distributing frequency bands of the source signal in different spatial directions, a method that aims to create a spatially distributed and surrounded sound arriving equally from all directions, rather than controlling the exact degree of coverage.

[0011] Verron et al. achieved spatial coverage of sources without the use of panned correlated signals by synthesizing multiple incoherent versions of the source signal, distributing them evenly on a circle around the listener, and mixing between them. The number and gain of simultaneously active sources determines the strength of the coverage effect. This method was implemented as a spatial extension to a synthesizer for environmental sounds.

[0012] 3D sound source width This section describes a suitable method for rendering augmented sound sources in 3D space, i.e. in a stereoscopic manner as required for virtual reality with six degrees of freedom (“6DoF”). This means six degrees of freedom of user movement, i.e. head rotation in pitch / yaw / roll axes, plus three translational directions x / y / z.

[0013] Potard et al. extended the concept of source extent as a one-dimensional parameter of a sound source (i.e. the width between two loudspeakers) by studying the perception of the shape of the sound source (Potard, 2003). They generated multiple incoherent point sources by applying a (time-varying) decorrelation technique to the original source signal and then placing the incoherent sources at different spatial positions and giving them a three-dimensional extent (Potard & Burnett, 2004).

[0014] In MPEG-4 Advanced AudioBIFS (Schmidt & Schroeder, 2004), volumetric objects / shapes (shucks, boxes, ellipsoids and cylinders) are filled with several evenly distributed, uncorrelated sound sources to evoke a three-dimensional source spread.

[0015] To increase and control source spread using Ambisonics, Schmele et al. (Schmele & Sayin, 2018) propose a mixture of decreasing Ambisonics orders of the input signal, which essentially increases the apparent source width and distributes uncorrelated copies of the source signal around the listening space.

[0016] Another approach was brought by Zotter et al., who adopted the principles proposed in (Zotter & Frank, 2013) for Ambisonics (i.e., deriving filter pairs that introduce frequency-dependent phase and magnitude differences to achieve source width in a stereo playback setup) (Zotter F. , Frank, Kronlachner, & Choi, 2014).

[0017] A common drawback of panning-based approaches (e.g., (Pulkki, 1997) (Pulkki, 1999) (Pulkki, 2007) (Pulkki, Laitinen, & Erkut, 2009)) is their dependency on the listener position. Any deviation from the sweet spot will cause the spatial image to collapse to the loudspeaker closest to the listener. This severely limits their application in virtual and augmented reality situations with six degrees of freedom (6DoF), where the listener is assumed to be free to move around. Furthermore, the distribution of time-frequency bins in DirAC-based approaches (e.g., (Pulkki, 2007) (Pulkki, Laitinen, & Erkut, 2009)) does not always guarantee a proper rendering of the spatial extent of the phantom source. Moreover, it typically significantly degrades the quality of the source signal.

[0018] Decorrelation of the source signals is typically achieved by one of the following methods: i) deriving filter pairs with complementary magnitudes (e.g., (Lauridsen, 1954)), ii) using all filters with constant magnitude but (randomly) scrambled phase (e.g., (Kendall, 1995) (Potard & Burnett, 2004)), or iii) randomly distributing the time-frequency bins of the source signals in space (e.g., (Pihlajamaeki, Santala, & Pulkki, 2014)).

[0019] Every approach has its own implications: Complementary filtering of the source signal according to i) typically leads to an altered perceived sound quality of decorrelated signals. All-path filtering as in ii) preserves the sound quality of the source signal, but the scrambled phase disrupts the original phase relationships, causing severe time dispersion and smearing artifacts, especially for transient signals. Spatially dispersing time-frequency bins has been proven effective for some signals, but also modifies the perceived sound quality of the signal. Moreover, it has been shown to have a high degree of signal dependence and introduces severe artifacts for instantaneous signals.

[0020] Adding volumetric shapes with multiple uncorrelated versions of the source signal, as proposed in Advanced AudioBEFS ((Schmidt & Schroeder, 2004) (Potard, 2003) (Potard & Burnett, 2004)), assumes the availability of a large number of filters that generate mutually uncorrelated output signals (typically 10 or more point sources per volumetric shape are used). However, finding such filters is not a trivial task and becomes more difficult the more such filters are needed. Furthermore, if the source signals are not completely uncorrelated and the listener moves around such a shape, e.g. in a (virtual reality) scenario, the distance of the individual sources to the listener corresponds to different delays of the source signals and their superposition at the listener's ears results in position-dependent comb filtering that potentially introduces unpleasant non-stationary coloration of the source signals.

[0021] In (Schmele & Sayin, 2018), we showed that using an Ambisonics-based technique to control source width by lowering the Ambisonic order only has an audible effect for transitions from the 2nd to the 1st or 0th order. Moreover, these transitions are not only perceived as source broadening, but are frequently also perceived as phantom source movement. Additional decorrelated versions of the source signal can help stabilize the perception of apparent source width, but also introduce comb filtering effects that modify the sound quality of the phantom sources. Summary of the Invention [Problem to be solved by the invention]

[0022] It is an object of the present invention to provide an improved concept for reproducing a spatially extended audio source or for generating a bitstream from a spatially extended audio source.

[0023] The object of the invention is achieved by an apparatus for reproducing a spatially extended audio source according to claim 1, by an apparatus for generating a bitstream according to claim 27, by a method for reproducing a spatially extended audio source according to claim 35, by a method for generating a bitstream according to claim 36, by a bitstream according to claim 41 or by a computer program according to claim 47. [Means for solving the problem]

[0024] The present invention allows the reproduction of spatially extended sound sources, in particular two-dimensional or three-dimensional sound sources associated with the spatially extended sound sources using the listener position. HaruThis is based on the finding that it is possible to achieve this by calculating a projection of the spatially extended sound source onto a projection plane of the spatially extended sound source, which projection is used to calculate at least two sound source positions for the spatially extended sound source, which at least two sound sources are rendered at said positions to obtain a reproduction of the spatially extended sound source, where the rendering results in two or more output signals, using different sound signals for the different positions, but which are all associated with the same spatially extended sound source.

[0025] On the one hand, a high quality two- or three-dimensional audio reproduction is obtained since the time-varying relative positions between the spatially extended sound sources and the (virtual) listener position are taken into account. On the other hand, the spatially extended sound sources are efficiently represented by a geometry regarding the perceived sound source spread and a number of at least two sound sources such as surrounding point sources, which can be easily implemented by renderers known in the art. In particular, simple renderers in the art always exist at that position and render a sound source at a specific position for a specific output format or speaker setup. For example, two sound sources calculated by a sound position calculator at specific positions may be rendered to these positions, for example by amplitude panning.

[0026] For example, if a sound location is between the left and left surround in a 5.1 output format, and another sound source is between the right and right surround in the output format, the amplitude panning method performed by the renderer will result in the left and left surround channels for one sound source being fairly similar signals, and the right and right surround for the corresponding other sound source being fairly similar signals, so that the user perceives the sound source coming from the location calculated by the sound location calculator. However, due to the fact that all four signals are ultimately related and associated with the spatially extended sound source, the user does not simply perceive two phantom sources associated with the location calculated by the sound location calculator, but the listener perceives a single spatially extended sound source.

[0027] The device for reproducing spatially extended sound sources having a geometrically defined position in space includes an interface, a projector, a sound position calculator and a renderer. The invention makes it possible to take into account enhanced sound situations occurring, for example, in a piano. A piano is a large device and up until now, the sound of the piano may have been rendered as coming from a single point sound source. However, this does not fully represent the true acoustic characteristics of the piano. According to the invention, a piano as an example of a spatially extended sound source is represented by at least two sound signals, where one sound signal can be recorded by a microphone placed close to the left part of the piano, i.e. close to the bass strings, while the other sound source can be recorded by a different second microphone placed close to the right part of the piano, i.e. close to the treble strings that generate the treble sounds. Naturally, both microphones will record different sounds from each other due to the reflection situation in the piano and the fact that the bass strings are closer to the left microphone than the right microphone and vice versa. On the other hand, however, the signals of both microphones will have a significant amount of similar tonal content that ultimately constitutes the distinctive sound of the piano.

[0028] According to the invention, a bitstream representing a spatially extended sound source, such as a piano, is generated by recording a signal by also recording geometry information of the spatially extended sound source, optionally also by recording position information related to different microphone positions (or, in general, two different positions associated with two different sound sources) or by providing a description of the perceived geometry of the (piano) sound associated with a spatially extended sound source, such as a piano, in order to reflect the listener position relative to the sound source, i.e. so that the listener can "walk around" in a virtual or augmented reality or any other sound scene. Haruis calculated using the listener position and the positions of at least two sound sources are calculated using the projection plane, where in particular the preferred embodiment relates to the placement of the sound sources at points around the projection plane.

[0029] A realistic representation of an exemplary piano sound in two or three dimensional context is made possible by reducing indirect calculations and indirect rendering, such that, for example, if a listener is closer to the left side of a sound source such as a piano, the sound perceived by the listener will be different from the sound that would occur if the user was closer to the right side of the sound source such as a piano or behind the sound source such as a piano.

[0030] In view of the above, the inventive concept is unique in that it provides a way to characterize spatially extended sound sources at the encoder side, making it possible to use spatially extended sound sources for true two-dimensional or three-dimensional settings within a sound reproduction context. Furthermore, the use of listener position within a highly flexible description of spatially extended sound sources allows for the creation of two-dimensional or three-dimensional sound reproduction contexts using listener position. Haru This is made possible in an efficient manner by calculating a projection of at least two sound sources onto a projection plane of the at least two sound sources for the spatially extended sound source, where sound positions of the at least two sound sources are calculated using the projection plane, and the at least two sound sources are rendered at the positions calculated by the sound position calculator to obtain a reproduction of the spatially extended sound source with two or more output signals on headphones or a multichannel output signal of two or more channels in a stereo reproduction setup or a reproduction setup having more than two channels, such as 5, 7 or more channels.

[0031] Compared to prior art methods of filling 3D volumes with sound by placing many different point sound sources in every part of the filled volume, projection eliminates the need to model many sound sources. HaruSince we only need to fill the projections of the two-dimensional space, the number of point sources employed can be significantly reduced. Moreover, it is possible to simply have one source at the left edge of the spatially extended source and one source at the right edge of the spatially extended source. Haru The number of required point sources can be further reduced by modeling only the above sources. Both reduction steps are based on two psychoacoustic findings: 1. In contrast to the azimuth (and elevation) of a sound source, its distance is less reliably perceived, so projecting the original volume onto a plane perpendicular to the listener does not significantly change perception (but it may reduce the number of point sources required for rendering). 2. Two uncorrelated sounds placed side by side as point sources tend to perceptually fill the space between them with sound.

[0032] Furthermore, the encoder side is flexible in that it not only allows the characterization of a single spatially extended sound source, but the bitstream generated as a representation can preferably contain all data for two or more spatially extended sound sources relative to a single coordinate system for their geometry information and position. On the decoder side, the playback can be done not only for a single spatially extended sound source, but for several spatially extended sound sources, but the projector calculates the projection for each sound source using the (virtual) listener position. Furthermore, the sound position calculator calculates at least two sound source positions for each spatially extended sound source, and the renderer can render all the calculated sound sources for each spatially extended sound source, for example by adding two or more output signals from each spatially extended sound source, signal-by-signal or channel-by-channel, and providing the added channels to corresponding headphones for binaural playback, or to corresponding speakers in a speaker-related playback setup, or alternatively to a storage that stores the (combined) two or more output signals for later use or transmission.

[0033] On the generator or encoder side, a bitstream is generated using an apparatus for generating a bitstream representing a compressed description of a spatially extended sound source, where the apparatus includes a sound provider for providing one or more different sound signals for the spatially extended sound source, and an output data former generates a bitstream representing a compressed sound scene, the bitstream preferably including one or more different sound signals in a compressed manner, such as compressed by a bitrate compression encoder, for example an MP3, AAC, USAC or MPEG-H encoder. Furthermore, the output data former is preferably configured to incorporate into the bitstream any individual position information for each sound signal of the two or more different sound signals, in case there are two or more different sound signals, indicating the position of the corresponding sound signal with respect to the information on the geometry of the spatially extended sound source. That is, the first signal is the signal recorded in the left part of the piano in the above example, and the signal recorded in the right part of the piano.

[0034] However, alternatively, the position information does not have to relate to the geometry of the spatially extended sound source, but can also relate to a general coordinate origin, although it is preferred to have a relationship to the geometry of the spatially extended sound source.

[0035] Furthermore, the device for generating a compressed bitstream also includes a geometry provider for calculating information on the geometry of the spatially extended sound sources, and the output data former is configured to introduce information on the geometry, individual position information for each sound signal, in addition to at least two sound signals, such as sound signals recorded by a microphone, into the bitstream. However, the sound provider does not necessarily have to pick up microphone signals, but can also generate sound signals on the encoder side, possibly using a decorrelation process. At the same time, only a few sound signals or only a single sound signal can be sent for the spatially extended sound signals, and the remaining sound signals can be generated on the playback side using a decorrelation process. This is preferably signaled by bitstream elements in the bitstream so that the playback device can determine how many sound signals are available and how many sound signals should be derived on the decoder side, such as by signal synthesis or correlation processing, especially in the sound position calculator, so that the sound playback device always knows how many sound signals are included per spatially extended sound source.

[0036] In this embodiment, the reproducer writes a bitstream element into the bitstream indicating the number of sound signals contained in the spatially extended sound source, and on the decoder side the sound reproducer derives the bitstream element from the bitstream, reads the bitstream element and determines based on the bitstream element how many signals to calculate, preferably for surrounding point sound sources or auxiliary sound sources located between the surrounding sound sources, based on at least one received sound signal in the bitstream.

[0037] Next, preferred embodiments of the present invention will be described with reference to the accompanying drawings. [Brief description of the drawings]

[0038] [Figure 1] FIG. 1 is a schematic block diagram of a preferred embodiment of the playback side. [Diagram 2]FIG. 2 shows a spherical spatially extended sound source with a different number of surrounding point sources. [Diagram 3] FIG. 3 shows an ellipsoidal spatially extended source with several surrounding point sources. [Figure 4] FIG. 4 shows a linear spatially extended sound source with different ways of being positioned at the location of the surrounding point sources. [Diagram 5] FIG. 5 shows a rectangular parallelepiped spatially extended sound source with different ways to arrange the surrounding point sound sources. [Figure 6] FIG. 6 shows a spherical spatially extended sound source at different distances. [Figure 7] FIG. 7 shows a piano-shaped spatially extended sound source within an approximate parametric ellipsoid shape. [Figure 8] FIG. 8 shows a piano-shaped spatially extended source with three surrounding point sources located on the extreme points of the projected convex hull. [Figure 9] FIG. 9 shows a preferred implementation of an apparatus or method for reproducing spatially extended sound sources. [Figure 10] FIG. 10 shows a preferred implementation of an apparatus or method for generating a bitstream representing a compressed description for a spatially extended sound source. [Figure 11] FIG. 11 illustrates a preferred implementation of a bitstream generated by the apparatus or method illustrated in FIG.

[0039] Fig. 9 shows a preferred implementation of an apparatus for playing a spatially extended sound source having a defined position and geometry in a space. The apparatus includes an interface 100, a projector 120, a sound position calculator 140, and a renderer 160. The interface is configured to receive a listener position. The projector 120 also uses the listener position received by the interface 100 in the space, information about the geometry of the spatially extended sound source, and information about the position of the spatially extended sound source to generate a two-dimensional or three-dimensional sound associated with the spatially extended sound source. Haru The audio decoder 190 is configured to calculate a projection of the spatially extended sound source on a projection plane of the audio source. Preferably, the defined position of the spatially extended sound source in the space and also the geometry of the spatially extended sound source in the space are received via a bitstream arriving at a bitstream demultiplexer or scene parser 180 for playing the spatially extended sound source. The bitstream demultiplexer 180 extracts the information of the geometry of the spatially extended sound source from the bitstream and provides this information to the projector. Furthermore, the bitstream demultiplexer also extracts the position of the spatially extended sound source from the bitstream and forwards this information to the projector. Preferably, the bitstream also includes position information for at least two different sound sources, and preferably the bitstream demultiplexer extracts from the bitstream a compressed representation of the at least two sound sources, which are restored / decoded by a decoder as the audio decoder 190. The decoded at least two sound sources are finally transferred to the renderer 160 , which renders the at least two sound sources at the positions provided by the sound position calculator 140 .

[0040] Although Fig. 9 shows a bitstream-related playback device with a bitstream demultiplexer 180 and an audio decoder 190, playback can also take place in situations different from the encoder / decoder scenario. For example, defined positions and geometries in space may already exist in the playback device, such as virtual or augmented reality scenes, where data is generated on the fly and consumed on the fly. The bitstream demultiplexer 180 and audio decoder 190 are not actually required, and the information of the spatially extended sound source geometry and the spatially extended sound source position is available without extraction from the bitstream. Furthermore, the position information relating the positions of at least two sound sources to the information of the spatially extended sound source geometry may be fixedly agreed upon in advance, and therefore does not need to be transmitted from the encoder to the decoder, or alternatively, this data is generated again on the fly.

[0041] Therefore, it should be noted that in the embodiment, only the position information is provided, and even in the case of more than one sound source signal, this information does not need to be transmitted. For example, a decoder or playback device can always obtain the first sound source signal in the bitstream as a sound source on the projection located on the left. Similarly, the second sound source signal in the bitstream can be obtained as a sound source on the projection located on the right.

[0042] Furthermore, the sound position calculator calculates the positions of at least two sound sources relative to the spatially extended sound sources using the projection plane, but the at least two sound sources do not necessarily have to be received from the bitstream. Instead, only a single sound source of the at least two sound sources can be received via the bitstream and the other sound source, and therefore the other positions or position information can also be actually generated on the playback side, only if there is no need to transmit such information from the bitstream generator to the playback device. However, in other embodiments, all this information can be transmitted, and more than one or two sound signals can be transmitted in the bitstream if the bitrate requirements are not strict, and the audio decoder 190 decodes two, three or more sound signals representing the at least two sound sources whose positions are calculated by the sound position calculator 140.

[0043] Fig. 10 shows the encoder side of this scenario when playback is applied within an encoder / decoder application. Fig. 10 shows an apparatus for generating a bitstream representing a compressed description of a spatially extended sound source. In particular, a sound provider 200 and an output data former 240 are provided. In this implementation, the spatially extended sound source is represented by a compressed description with one or more different sound signals, and the output data former generates a bitstream representing a compressed sound scene, where the bitstream includes at least one or more different sound signals and geometry information related to the spatially extended sound source. This represents the situation described with respect to Fig. 9, where all other information such as the location of the spatially extended sound source (see the dotted arrow of block 120 of Fig. 9) is freely selectable by the user on the playback side. Thus, a unique description of the spatially extended sound source with at least one or more different sound signals for this spatially extended sound source is provided, where these sound signals are simply signals of point sources.

[0044] Furthermore, the apparatus for generating includes a geometry provider 220 for computing and otherwise providing information regarding the geometry of the spatially extended sound source. Other ways of providing the geometry information different from the computation include receiving a user input such as a drawing manually drafted by a user or any other information provided by a user, for example speech, a tone, a gesture or any other user action. In addition to one or more different sound signals, the information regarding the geometry is embedded in the bitstream.

[0045] Additionally, information on individual position information for each sound signal of the one or more different sound signals is also incorporated into the bitstream and / or position information for spatially extended sound sources is also incorporated into the bitstream. The position information of the sound source can be separated from the geometry information or can be included in the geometry information. In the first case, the geometry information can be given in relation to the position information. In the second case, the geometry information can include, for example, a sphere, a center point in coordinates and a radius or diameter. For a box-shaped spatially extended sound source, eight or at least one corner point can be given in absolute coordinates.

[0046] The position information for each of the one or more different sound signals is preferably related to the geometric information of the spatially extended sound source. Alternatively, however, absolute position information related to the same coordinate system in which the position or geometric information of the spatially extended sound source is given is also useful, alternatively the geometric information may be given in an absolute coordinate system with absolute coordinates, rather than in a relative manner. However, providing this data in a relative manner not related to a common coordinate system allows the user to position the spatially extended sound source in her or his own playback setup, as indicated by the dotted line towards the projector 120 in FIG. 9.

[0047] In another embodiment, the sound provider 200 of FIG. 10 is configured to provide at least two different sound signals for a spatially extended sound source and the output data former is configured to generate a bitstream such that the bitstream includes the at least two different sound signals, preferably in an encoded format, and optionally individual position information of each of the at least two different sound signals in terms of absolute coordinates or the geometry of the spatially extended sound source.

[0048] In one embodiment, the sound provider is configured to perform recordings of natural sound sources at individual microphone positions or orientations, or to derive a sound signal from a single basis signal or multiple basis signals by one or more decorrelation filters, for example as described with respect to items 164 and 166 in Figure 1. The basis signal used by the generator may be the same or different from the basis signal provided at the playback site or transmitted from the generator to the playback device.

[0049] In another embodiment, the geometry provider 220 is configured to derive a parametric or polygonal description from the geometry of the spatially extended sound source, and the output data former is configured to incorporate this parametric or polygonal description into the bitstream.

[0050] Furthermore, the output data former is in a preferred embodiment configured to incorporate a bitstream element into the bitstream, where this bitstream element indicates the number of at least one different sound signal for the spatially extended sound source contained in the bitstream or contained in the encoded audio signal associated with the bitstream, where the number is greater than or equal to 1. The bitstream generated by the output data former does not have to be a complete bitstream with audio waveform data on the one hand and metadata on the other hand. Instead, there can be only a separate metadata bitstream, which contains for example a bitstream field for the number of sound signals of each spatially extended sound source, geometry information for the spatially extended sound sources and, in one embodiment, also position information for the spatially extended sound sources, and optionally position information for each sound signal and each spatially extended sound source, geometry information for the spatially extended sound sources and, in one embodiment, also position information for the spatially extended sound sources. The waveform audio signals, typically available in compressed form, are transmitted by separate data streams or separate transmission channels to a playback device, which receives the encoded metadata from one audio source and the (encoded) waveform signal from a different audio source.

[0051] Furthermore, the embodiment of the bitstream generator includes a controller 250. The controller 250 is configured to control the sound provider 200 regarding the number of sound signals provided by the sound provider. According to this method, the controller 250 is also provided with bitstream element information to the output data former 240, which is shown with hatched lines showing additional features. The output data former is controlled by the controller 250 and introduces specific information regarding the number of sound signals into the bitstream elements as provided by the sound provider 200. Preferably, the number of sound signals is controlled such that the output bitstream containing the encoded audio sound signals meets the external bitrate requirements. If the allowed bitrate is high, the sound provider can provide more sound signals than if the allowed bitrate is small. In an extreme case, the sound provider can provide only a single sound signal for a spatially extended sound source when the bitrate requirements are strict.

[0052] The playback device reads the correspondingly configured bitstream elements and starts to synthesize, in the renderer 160, the corresponding number of other sound signals using the decoder side and the transmitted sound signals, until finally the required number of ambient point sound sources and optionally auxiliary sound sources are generated.

[0053] However, if the bitrate requirements are not so strict, the controller 250 can control the sound provider to provide a number of different sound signals, e.g. recorded by a corresponding number of microphones or one microphone orientation, and no or little de-correlation processing is required on the playback side, and finally a better playback quality can be obtained by the playback device due to reduced or no de-correlation processing on the playback side. The trade-off between bitrate on the one hand and quality on the other hand is preferably obtained via a function of the bitstream element indicating the number of sound signals per spatially extended sound source.

[0054] Figure 11 shows a preferred embodiment of a bitstream generated by the bitstream generator shown in Figure 10. The bitstream includes a second spatially extended audio source 401, for example denoted as SESS2, with corresponding data.

[0055] Further, Fig. 11 shows detailed data for each spatially extended sound source with respect to spatially extended sound source number 1. In the example of Fig. 11, two sound signals are for the spatially extended sound source that are generated in a bitstream generator from microphone output data taken from microphones placed at two different locations of the spatially extended sound source, for example. The first sound signal is sound signal 1, indicated by 301, and the second sound signal is sound signal 2, indicated by 302, and both sound signals are preferably encoded via an audio encoder for bitrate compression. Furthermore, item 311 represents a bitstream element indicating the number of sound signals for spatially extended sound source 1, controlled, for example, by the controller 250 of Fig. 10.

[0056] Geometry information of the spatially extended sound source is incorporated as shown in block 331. Item 301 preferably indicates any position information for the sound signal in relation to the geometry information, such as for the piano example, for sound signal 1, "close to the bass strings" and for sound signal 2, "close to the treble strings", as shown in 302. The geometry information may for example be a parametric or polygonal representation of a piano model, which is different from, for example, a grand piano or a (small) piano. Item 341 further indicates any data regarding the position for the spatially extended sound source in the space. As stated, this position information 341 is not necessary if the user provides the position information as shown by the dotted line directed towards the projector in FIG. 9. However, even if the position information 341 is included in the bitstream, the user can replace or change the position information by user interaction.

[0057] In the following, a preferred embodiment of the present invention will be described, which relates to rendering spatially extended sound sources in 6DoF VR / AR (Virtual Reality / Augmented Reality).

[0058] A preferred embodiment of the present invention relates to a method, device or computer program designed to enhance the reproduction of spatially extended sound sources (SESS). In particular, the method or device embodiments of the present invention take into account the time-varying relative position between the spatially extended sound source and the virtual listener position. In other words, the method or device embodiments of the present invention allow the auditory sound source width to match the spatial extent of the represented sound object for a listener at any relative position. Thus, the method or device embodiments of the present invention are applied in particular to six degree of freedom (6DoF) virtual, mixed and augmented reality applications where spatially extended sound sources complement the traditionally employed point sound sources.

[0059] An embodiment of the inventive method or device renders a spatially extended sound source by using several surrounding point sound sources provided with (preferably significantly) uncorrelated signals. In contrast to other methods, the position of these surrounding point sound sources depends on the position of the listener relative to the spatially extended sound source. Figure 1 shows an overview block diagram of a spatially extended sound source renderer according to an embodiment of the inventive method or device.

[0060] The key components of the block diagram are: 1. Listener Position: This block provides the instantaneous position of the listener as measured, for example, by a virtual reality tracking system. The block can be implemented as a detector 100 for detecting or an interface 100 for receiving the listener position. 2. Spatially extended sound source position and geometry: This block provides spatially extended sound source position and geometry data for rendering, for example, as part of a virtual reality scene representation. 3. Projection and Convex Hull Calculation: This block 120 calculates the convex hull of the spatially extended geometry of the sound source and then projects it in a direction towards the listener position (e.g., "image plane", see below). Alternatively, the same functionality can be achieved by first projecting the geometry in a direction towards the listener position and then calculating the convex hull. 4. Surrounding point source positions: This block 140 calculates the positions of the surrounding point sound sources used from the convex hull projection data calculated by the previous block. The calculation may take into account the listener position and the proximity / distance of the listener (see below). The output is the positions of the n surrounding point sound sources. 5. Renderer Core: The renderer core 162 renders the sounds of n ambient point sources into the mind by positioning them at identified target positions. It may be, for example, a binaural renderer using head-related transfer functions or a renderer for loudspeaker playback (e.g., vector-based amplitude panning). The renderer core generates l loudspeaker or headphone output signals from k input audio basis signals (e.g., decorrelated signals of musical instrument recordings) and m≧(nk) additional decorrelated audio signals. 6. Source Basis Signals: This block 164 is the input for k basis audio signals that are (sufficiently) uncorrelated with each other and represent the sound sources to be rendered (e.g. mono-k=1- or stereo-k=2-recordings of musical instruments). The k basis audio signals can for example be obtained from a bitstream received from a generator on the decoder side (see for example elements 301, 302 in Fig. 11) or provided to the playback site from an external source. 7. Decorrelator: This optional block 166 generates the additional decorrelated audio signals needed to render n surrounding point sources. 8. Signal Outputs: The renderer provides l output signals for loudspeaker (e.g. n=5.1) or binaural (typically n=2) rendering.

[0061] Figure 1 shows a block diagram overview of an embodiment of a method or apparatus of the present invention. The dashed lines indicate the transmission of metadata such as geometry and position. The solid lines indicate the transmission of audio, where k, l and m indicate a number of audio channels. A renderer core 162 receives k+m audio signals and n (<=k+m) position data. Blocks 162, 164, 166 together form one embodiment of a general renderer 160.

[0062] The location of the peripheral point sound sources depends on the geometry of the spatially extended sound sources, especially in spatial extent, and on the relative position of the listener with respect to the spatially extended sound sources. In particular, the peripheral point sound sources may be located on the projection plane of the projection of the convex hull of the spatially extended sound sources. The projection plane may have the image plane, i.e. a plane perpendicular to the line from the listener to the spatially extended sound sources, or a sphere around the listener's head. The projection plane may be located at any small distance from the center of the listener's head. Alternatively, the projection convex hull of the spatially extended sound sources can be calculated from the azimuth and elevation angles, which are a subset of the relative spherical coordinates from the spatial location of the listener's head. In the following exemplary embodiment, the projection plane is preferred due to its more intuitive properties. In the implementation of the calculation of the projected convex hull, the angle representation is preferred due to its simpler formalization and lower computational complexity. It should be noted that both projections of the convex hull of the spatially extended sound sources are identical to the convex hull of the projected spatially extended sound source geometry. That is, the calculation and projection of the convex hull onto the image plane can be used in either order.

[0063] The positions of the surrounding point sound sources may be located on the projection of the convex hull of the spatially extended sound source in a variety of ways, including the following: ● Those Haru They can be evenly spaced around the projection. ● Those Haru It can be placed at the extreme points of the projection. ● Those HaruThey can be placed at the horizontal and / or vertical extreme points of the projection (see figure in the Examples section).

[0064] In addition to the surrounding point sources, other auxiliary point sources can be used to generate an enhanced sense of acoustic filling, at the cost of additional computational complexity. Furthermore, the projected convex hull may be modified before placing the surrounding point sources. For example, the projected convex hull may be contracted towards the centroid of the projected convex hull. Such a contracted projected convex hull may take into account the additional spatial extent of each surrounding point source introduced by the rendering method. The transformation of the convex hull may further distinguish between scaling in the horizontal and vertical directions.

[0065] When the listener position relative to the spatially extended sound source changes, the projection of the spatially extended sound source onto the projection plane changes accordingly. Similarly, the positions of the surrounding point sound sources change accordingly. The positions of the surrounding point sound sources are preferably selected to change smoothly with the continuous motion of the spatially extended sound source and the listener. Furthermore, when the geometry of the spatially extended sound source is changed, the projected convex hull changes. This includes a rotation of the geometry of the spatially extended sound source in 3D space, which changes the projected convex hull. The rotation of the geometry is equal to the angular displacement of the listener position relative to the spatially extended sound source, as referenced in a comprehensive manner as the relative position of the listener and the spatially extended sound source. For example, the circular movement of the listener around a spherical spatially extended sound source is represented by rotating the position of the point sound source around the center of gravity. Similarly, a rotation of a spatially extended sound source with a stationary listener results in the same change in the position of the surrounding point sound sources.

[0066] The spatial spread produced by embodiments of the method or apparatus of the present invention is essentially correctly reproduced for any distance between the spatially extended sound source and the listener. Of course, as the user moves closer to the spatially extended sound source, the opening angle between the surrounding point sound sources increases, in a manner suitable for modeling physical reality.

[0067] While the angular placement of the surrounding point sound sources is uniquely determined by their position on the projected convex hull on the projection plane, the distance of the surrounding point sound sources may further be selected in various ways: All surrounding point sound sources have a distance equal to the total distance of the spatially extended sound source, defined for example via the center of gravity of the spatially extended sound source relative to the listener's head. The distance of each surrounding point source is determined by the backprojection of the projected convex hull position onto the spatially extended source geometry, such that the projection of the surrounding point sources onto the projection plane is at the same location. The backprojection of the surrounding point sources from the convex hull onto the spatially extended source is not necessarily uniquely determined, and additional projection rules must be applied (see the Example section). • Rendering surrounding point sources does not require distance characteristics, but if only their relative angular placement in azimuth and elevation is required, the distance of the surrounding point sources may not be determined at all.

[0068] To specify the geometric shape / convex hull of the spatially extended sound source, approximations are used (and possibly transmitted to the renderer or renderer core), including simplified 1D, e.g., lines, curves; 2D, e.g., ellipses, rectangles, polygons; or 3D shapes, e.g., ellipsoids, cuboids, and polyhedra. The geometry of the spatially extended sound source or the corresponding approximation shape, respectively, can be described in various ways: Parametric description, i.e. the formulation of a geometry through a mathematical expression that accepts additional parameters. For example, an ellipsoid shape in 3D can be described by an implicit function on a Cartesian coordinate system, where the additional parameters are the extensions of the principal axes in all three directions. Further parameters can include 3D rotations, transformation functions of the ellipsoid surface. A description of polygons, i.e. a collection of basic geometric shapes such as lines, triangles, squares, tetrahedrons and cuboids. Basic polygons and polyhedra can also be connected into more complex geometries.

[0069] The ambient point source signals are derived from the spatially extended sound source basis signals. The basis signals can be obtained in various ways, such as: 1) recording of natural sound sources at single or multiple microphone positions and orientations (e.g. recording of piano sounds as shown in the example); 2) synthesis of artificial sound sources (e.g. synthesis of sounds with varying parameters); 3) combination of any audio signals (e.g. various mechanical sounds of a car such as engine, tires, doors, etc.). Furthermore, additional ambient point source signals may be artificially generated from the basis signals by multiple decorrelation filters (see previous section).

[0070] Certain application scenarios focus on compact and interoperable storage / transmission of 6DoF VR / AR content. In this case, the whole chain consists of three steps: 1. Authoring / encoding the desired spatially enhanced audio source into a bitstream 2. Transmitting / storing the generated bitstream. According to the invention, the bitstream contains, among other elements, a description of the spatially extended sound source geometry (parametric or polygonal) and the associated sound source basis signal, such as a mono or stereo piano recording. The waveform may be compressed using a perceptual audio coding algorithm such as mp3 or MPEG-2 / 4 Advanced Audio Coding (AAC) (see item 260 in FIG. 10). 3. Decoding / rendering the spatially enhanced audio source based on the transmitted bitstream as described above.

[0071] In addition to the core methods described above, several options for further processing exist:

[0072] Option 1 - Dynamic selection of the number and location of surrounding point sound sources The number of surrounding point sources can be varied depending on the distance of the listener to the spatially extended sound source. As an example, when the spatially extended sound source and the listener are far away from each other, the opening angle (aperture) of the projected convex hull is small, and therefore a smaller number of surrounding point sources can be advantageously selected, saving computational and memory complexity. In the extreme case, all surrounding point sources are reduced to a single remaining point source. To ensure that interference between the base signal and the derived signal does not degrade the audio quality of the resulting surrounding point source signal, a suitable downmixing technique can be applied. Similar techniques can also be applied when the spatially extended sound source is close to the listener position, when the geometry of the spatially extended sound source is highly irregular depending on the relative viewpoint of the listener. For example, the geometry of the spatially extended sound source, which is a line of finite length, may degenerate towards a single point on the projection plane. In general, when the angular range of the surrounding point sources on the projected convex hull is narrow, the spatially extended sound source can be represented by fewer surrounding point sources. In the extreme case, all surrounding point sources are reduced to a single remaining point source.

[0073] Option 2 - Spread compensation Since each surrounding point source exhibits spatial extension towards the outside of the convex hull projection, the width of the perceived auditory image of a rendered spatially extended source is somewhat larger than the convex hull used for rendering. To reconcile this with the desired target geometry, there are two possibilities: 1. Compensation during authoring: During content authoring, the additional spread of the rendering method is taken into account. Specifically, during content authoring, a somewhat smaller spatially extended geometry of the sound source is selected so that the actual rendered size is as desired. This can be checked by monitoring the effect of the renderer or renderer core in the authoring environment (e.g., playback studio). In this case, the transmitted bitstream and the renderer or renderer core use a reduced target geometry compared to the target size. 2. Compensation during rendering: A renderer or renderer core for spatially extended sources may be able to recognize the additional perceptual widening by the rendering method and therefore be able to compensate for this effect. As a simple example, the geometry used for rendering may be adjusted to accommodate the placement of surrounding point sources before being applied. It can be reduced by a constant factor a<1.0 (e.g. a=0.9), or ○The constant opening angle alpha can be reduced by 5 degrees. In this case, the transmitted bitstream contains the final target size of the spatially extended source geometry.

[0074] A combination of these approaches is also feasible.

[0075] Option 3 - Generate waveforms of ambient point sources Furthermore, to model spatially extended sound sources with geometry that depends on the sound contribution, such as a piano with bass sounds on the left and vice versa on the right, real signals to provide ambient point sound sources can be generated from the recorded audio signals by taking into account the user position relative to the spatially extended sound source. Example: The sound of an upright piano is characterized by its acoustic behavior. This is also modeled by (at least) two audio basis signals, one near the bottom end of the piano keyboard ("bass"), and one near the top end of the keyboard ("treble"). These basis signals can be obtained by the use of suitable microphones when recording the piano sound, and are sent to the 6DoF renderer or renderer core, ensuring that there is sufficient correlation between them.

[0076] The signals of ambient point sources are then derived from these basis signals by considering the user position relative to the spatially extended sources. When the user faces the piano from the front (keyboard) side, the two ambient point sound sources are far apart near the left and right ends of the piano keyboard. In this case, the basis signals for the lower keys can be fed directly to the left ambient point source, and the basis signals for the higher keys can be used directly to drive the right ambient point source. ● When the listener walks around the piano to the right by about 90 degrees, the two surrounding point sources are panned very close to each other, since the projection of the piano volume model (e.g. an ellipse) becomes smaller when viewed from the side. If the basis signals continue to be used to directly drive the surrounding point source signals, one surrounding point source will contain mainly high tones, while the other one will convey mostly low tones. Since this is undesirable from a physical point of view, the rendering can be improved by rotating the two basis signals that form the surrounding point source signals by a Givens rotation, by the same angle as the user's movement relative to the piano's center of gravity. In this way, both signals contain signals of similar spectral content and are still uncorrelated (assuming that the basis signals are uncorrelated).

[0077] Option 4 - Post-processing of rendered spatially extended sound sources The real signals can be pre- or post-processed to take into account position- and direction-dependent effects, e.g. the directional patterns of spatially extended sound sources. In other words, as mentioned above, all sounds emanating from spatially extended sound sources can be modified to exhibit, for example, direction-dependent sound radiation patterns. In the case of piano signals, this may mean that radiation towards the rear of the piano has less high-frequency content than radiation towards the front of the piano. Furthermore, the pre- and post-processing of the signals of the surrounding point sound sources may be tailored separately for each of the surrounding point sound sources. For example, the directional patterns can be selected differently for each of the surrounding point sound sources. In the given example of a spatially extended sound source representing a piano, the directional patterns of the low and high key ranges may be similar as mentioned above, while additional signals such as pedaling noise have a more omnidirectional directional pattern.

[0078] Some advantages of the preferred embodiment are summarized below.

[0079] It has lower computational complexity compared to completely filling the interior of a spatially extended source with point sources (as used, for example, in Advanced AudioBIFS). Lower probability of destructive interference between signals of point sources ●Compact size of bitstream information (approximation of geometric shapes, one or more waveforms) ●Enabling the use of legacy recordings produced for music consumption (e.g. stereo recordings of a piano) for the purposes of VR / AR rendering.

[0080] Next, various practical implementation examples are presented: Spherical spatially extended sound source Ellipsoidal spatially extended sound source Linear spatially extended sound source A rectangular, spatially extended sound source Distance-dependent surrounding point sound sources ● Spatially extended piano-shaped sound source

[0081] As described in the embodiments of the method or device of the present invention, the above various methods for determining the position of a surrounding point sound source can be applied. The following examples show some separated methods in specific cases. In the complete implementation of the embodiments of the method or device of the present invention, the various methods can be appropriately combined, taking into account the computational complexity, application purpose, audio quality and ease of implementation.

[0082] The spatially extended sound source geometry is shown as a green surface mesh. Note that the mesh visualization does not imply that the spatially extended sound source geometry is described by a polygonal method, and in fact may be generated from a parametric specification. The listener position is indicated by a blue triangle. In the following example, the screen is chosen as the projection surface and is depicted as a transparent grey surface showing a finite subset of the projection surface. The projected geometry of the spatially extended sound source onto the projection surface is shown with the same surface mesh in green. The surrounding point sources on the projected convex hull are shown as red cross symbols on the projection surface. The back-projected surrounding point sources onto the spatially extended sound source geometry are shown as red dots. The corresponding surrounding point sources on the projected convex hull and the back-projected surrounding point sources onto the spatially extended sound source geometry are connected by red lines to help identify the visual correspondence. The positions of all relevant objects are shown in a Cartesian coordinate system with units in meters. The choice of the illustrated coordinate system does not imply that the associated calculations are performed in Cartesian coordinates.

[0083] The first example in Figure 2 considers a spherical spatially extended sound source. The spherical spatially extended sound source has a fixed size and a fixed position relative to the listener. Three different sets of surrounding point sources are selected on the projected convex hull: three, five, and eight. All three sets of surrounding point sources are selected with uniform distances on the curve of the convex hull. The offset positions of the surrounding point sources on the curve of the convex hull are intentionally chosen so that the horizontal extent of the spatially extended sound source geometry is well represented.

[0084] FIG. 2 shows a spherical spatially extended source with different numbers of point sources uniformly distributed on the convex hull: 3 (top), 5 (middle), and 8 (bottom).

[0085] The next example in Fig. 3 considers an ellipsoidal spatially extended sound source. An ellipsoidal spatially extended sound source has a fixed shape, position and rotation in 3D space. In this example, four surrounding point sound sources are chosen. Three different ways of determining the positions of the surrounding point sound sources are illustrated: a) Two surrounding point sources are placed at the two horizontal extreme points and two surrounding point sources are placed at the two vertical extreme points. Meanwhile, the positioning of the extreme points is simple and usually adequate. This example shows that the method may generate positions of surrounding point sources that are relatively close to each other. b) All four surrounding point sources are uniformly positioned on the projected convex hull. The offsets of the surrounding point source positions are chosen such that the topmost surrounding point source coincides with the position of the topmost surrounding point source in a). It can be seen that the choice of the offset of the surrounding point source positions significantly affects the representation of the geometry via the surrounding point sources. c) All four surrounding point sources are uniformly positioned on the contracted projected convex hull. The offset positions of the surrounding point sources are equal to the offset positions selected in b). The contraction motion of the projected convex hull is preformed with a direction-independent stretch factor towards the centroid of the projected convex hull.

[0086] Figure 3 shows a spatially extended source of an ellipsoid with four surrounding point sources based on three different methods of determining the locations of the surrounding point sources: a / top) horizontal and vertical extreme points, b / middle) uniformly spaced points on a convex hull, c / bottom) uniformly spaced points on a reduced convex hull.

[0087] The next example in Fig. 4 considers a linear spatially extended sound source. While the previous example considers a volumetric spatially extended sound source geometry, this example shows that the spatially extended sound source geometry can be chosen as a one-dimensional object in 3D space. Subfigure a) shows two surrounding point sound sources placed on the extreme points of the finite linear spatially extended sound source geometry. b) Two surrounding point sound sources are placed on the extreme points of the finite linear spatially extended sound source geometry, and one additional point sound source is placed at the center of the line. As described in the embodiments of the method or device of the present invention, placing additional point sound sources in the spatially extended sound source geometry can help to fill in the large gaps for large spatially extended sound source geometries. c) The same linear spatially extended sound source geometry as in a) and b) is considered, but the relative angle towards the listener is changed so that the projected length of the linear geometry is significantly smaller. As described in the embodiments of the method or apparatus of the present invention above, the reduced size of the projected convex hull can be represented by a reduced number of surrounding point sources, in this particular example by a single surrounding point source placed at the center of the linear geometry.

[0088] Figure 4 shows a line-shaped spatially extended source with three different ways to place the location of the surrounding point sources: a / top) two extreme points on the projected convex hull; b / middle) two extreme points on the projected convex hull with an additional point source at the center of the line; c / bottom) one surrounding point source at the center of the convex hull where the projected convex hull of the rotated line is too small to allow more than one surrounding point source.

[0089] The next example in Fig. 5 considers a rectangular parallelepiped spatially extended sound source. The rectangular parallelepiped spatially extended sound source has a fixed size and a fixed position, but the relative position of the listener changes. Subfigures a) and b) show different ways of placing four surrounding point sources on the projected convex hull. The positions of the backprojected surrounding point sources are uniquely determined by a selection on the projected convex hull. c) shows four surrounding point sources that do not have well-separated backprojected positions. Instead, the distances of the surrounding point source positions are selected to be equal to the distance of the centroid of the spatially extended sound source geometry.

[0090] Figure 5 shows a rectangular parallelepiped spatially extended source with three different ways to place the surrounding point sources: a / top) two surrounding point sources on the horizontal axis and two surrounding point sources on the vertical axis; b / middle) two surrounding point sources on the horizontal extreme points of the projected convex hull and two surrounding point sources on the vertical extreme points of the projected convex hull; c / bottom) backprojected surrounding point sources whose distance is chosen equal to the distance of the centroid of the spatially extended source geometry.

[0091] The next example in FIG. 6 considers spherical spatially extended sound sources of fixed size and shape, but at three different distances relative to the listener position. The surrounding point sound sources are uniformly located on the convex hull curve. The number of surrounding point sound sources is dynamically determined from the length of the convex hull curve and the minimum distance between the possible surrounding point sound source positions: a) spherical spatially extended sound sources at close distance, such that four surrounding point sound sources are selected on the projected convex hull; b) spherical spatially extended sound sources at medium distance, such that three surrounding point sound sources are selected on the projected convex hull; a) spherical spatially extended sound sources at far distance, such that only two surrounding point sound sources are selected on the projected convex hull. As described in the embodiments of the method or apparatus of the present invention above, the number of surrounding point sound sources may be determined from the spread expressed in spherical angular coordinates.

[0092] Figure 6 shows spherical spatially extended sound sources of equal size but at different distances: a / top) four surrounding point sources uniformly positioned on the projected convex hull at close distance; b / middle) three surrounding point sources uniformly positioned on the projected convex hull at mid distance; c / bottom) two surrounding point sources uniformly positioned on the projected convex hull at far distance.

[0093] The final example in Figures 7 and 8 considers a piano-shaped spatially extended sound source placed in a virtual world. The user wears a head-mounted display (HMD) and headphones. A virtual reality scene is presented to the user consisting of an open world canvas and a 3D upright piano model standing on a floor in a free movement area (see Figure 7). The open world canvas is a spherical static image projected onto the user's surrounding sphere. In this particular case, the open world canvas shows a blue sky with white clouds. The user can walk around, see and hear the piano from various angles. In this scene, the piano is rendered either as a single point source placed at the center of gravity or as a spatially extended sound source with three surrounding point sources on the projected convex hull (see Figure 8). Rendering tests show very good realism of the surrounding point source rendering method even by rendering as a single point source.

[0094] To simplify the calculation of the surrounding point source positions, the piano geometry is abstracted into the shape of an ellipsoid with similar dimensions, see Figure 7. Furthermore, two alternative point sources are placed at the left and right extreme points on the same line, while a third alternative point remains at the far north, see Figure 8. This arrangement ensures proper horizontal source width from all angles, at highly reduced computational cost.

[0095] FIG. 7 shows a piano-shaped spatially extended sound source (shown in green) with an approximate parametric ellipsoid shape (shown in red mesh).

[0096] Figure 8 shows a piano-shaped spatially extended sound source with three surrounding point sources located on the vertical extreme points of the projected convex hull and on the vertical vertices of the projected convex hull, although for better visualization the surrounding point sources are located on a stretched projected convex hull.

[0097] Next, specific features of embodiments of the present invention are provided. The characteristics of the presented embodiments are as follows: ● To fill the perceived acoustic space of a spatially extended sound source, preferably its entire interior is not filled with uncorrelated point sources (surrounding point sources), but only its periphery if it faces the listener (e.g. "projection of the convex hull of the spatially extended sound source towards the listener"). In particular, this means that the positions of the surrounding point sources are not attached to the geometry of the spatially extended sound source, but are dynamically calculated taking into account the relative position of the spatially extended sound source with respect to the listener position. Dynamic calculation of surrounding point sound sources (number and position) ● An approximation of the spatially extended shape of the sound source is used (for scenarios using compressed representations: transmitted as part of the bitstream).

[0098] The application of the described technology can be part of the standard for audio 6DoF VR / AR. In this context, we have a classic encoding / bitstream / decoder (+renderer) scenario: In the encoder, the shape of the spatially extended sound source characterizes the spatially extended sound source. Mono signal, or ○ A stereo signal (preferably sufficiently uncorrelated), or ○ More recorded signals (preferably well uncorrelated) These waveforms can be coded at low bit rates. At the decoder / renderer, the shape and corresponding waveform of the spatially extended sound source are extracted from the bitstream and used to render the spatially extended sound source as described above.

[0099] It should be noted that depending on the embodiment used, and as an alternative to the described embodiment, the interface can be implemented as an actual tracker or detector for detecting the listener position. However, the listening position is typically received from an external tracker device and provided to the playback device via the interface. However, the interface can only represent a data input to output data from an external tracker, or it can represent the tracker itself.

[0100] Furthermore, as outlined above, additional auxiliary sound sources may be required between the surrounding sound sources.

[0101] Furthermore, it has been found that the left and right ambient sound sources and any horizontally (relative to the listener) spaced auxiliary sound sources are more important to the perceptual impression than the vertically spaced ambient sound sources, i.e., the ambient sound sources above the top and bottom spatially extended sources. For example, in cases of resource scarcity, it is preferable to use at least the horizontally spaced ambient sound sources (and any auxiliary sound sources) since the vertically spaced ambient sound sources can be omitted in order to conserve processing resources.

[0102] Furthermore, as outlined, the bitstream generator can be implemented to generate a bitstream with only one sound signal for the spatially extended sound source, and the remaining sound signals are generated at the decoder side or playback side by decorrelation. If there is only a single signal and the whole space is filled equally with this single signal, then any position information is not necessary. However, in such a situation, it is beneficial to have at least additional information about the geometry of the spatially extended sound source calculated by a geometry information calculator such as that shown in 220 of Fig. 10.

[0103] It should be mentioned here that all alternatives or aspects as described above, and all aspects defined by the independent claims in the following claims, can be used individually, i.e. without any other alternatives or objectives than the intended alternatives, objectives or independent claims. However, in other embodiments, two or more alternatives or aspects or independent claims can be combined with each other, and in other embodiments, all aspects, or alternatives and all independent claims can be combined with each other.

[0104] The inventive encoded sound field description can be stored on a digital or non-transitory storage medium or can be transmitted over a transmission medium, such as a wireless transmission medium or a wired transmission medium such as the Internet.

[0105] Although some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where a block or device corresponds to a method step or a function of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block, item or function of a corresponding apparatus.

[0106] Depending on specific implementation requirements, embodiments of the present invention can be implemented in hardware or in software. Implementation can be carried out using a digital storage medium, such as a floppy disk, DVD, CD, ROM, PROM, EPROM, EEPROM or flash memory, having electronically readable control signals stored thereon and cooperating (or capable of cooperating) with a programmable computer system such that the respective methods are performed.

[0107] Some embodiments of the present invention comprise a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to cause one of the methods described herein to be performed.

[0108] In general, embodiments of the present invention may be implemented as a computer program product with program code operable to perform one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine readable carrier.

[0109] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier or a non-transitory storage medium.

[0110] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0111] A further embodiment of the inventive method is therefore a data carrier (or digital storage medium or computer readable medium) comprising, recorded thereon, a computer program for performing one of the methods described herein.

[0112] A further embodiment of the inventive method is therefore a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals can for example be arranged to be transmitted over a data communication connection, for example the Internet.

[0113] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0114] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0115] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0116] The above-described embodiments are merely illustrative of the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to those skilled in the art. It is therefore intended that the present invention be limited only by the scope of the impending claims and not by the specific details expressed by the description and illustration of the embodiments herein.

[0117] References Alary, B., Politis, A., & Vaelimaeki, V. (2017). Velvet Noise Decorrelator. Baumgarte, F., & Faller, C. (2003). Binaural Cue Coding-Part I: Psychoacoustic Fundamentals and Design Principles. Speech and Audio Processing, IEEE Transactions on, 11(6), S. 509-519. Blauert, J. (2001). Spatial hearing (3 Ausg.). Cambridge; Mass: MIT Press. Faller, C., & Baumgarte, F. (2003). Binaural Cue Coding-Part II: Schemes and Applications. Speech and Audio Processing, IEEE Transactions on, 11(6), S. 520-531. Kendall, G. S. (1995). The Decorrelation of Audio Signals and Its Impact on Spatial Imagery. Computer Music Journal, 19(4), S. p 71-87. Lauridsen, H. (1954). Experiments Concerning Different Kinds of Room-Acoustics Recording. Ingenioren, 47. Pihlajamaeki, T., Santala, O., & Pulkki, V. (2014). Synthesis of Spatially Extended Virtual Source with Time-Frequency Decomposition of Mono Signals. Journal of the Audio Engineering Society, 62(7 / 8), S. 467-484. Potard, G. (2003). A study on sound source apparent shape and wideness. Potard, G., & Burnett, I. (2004). Decorrelation Techniques for the Rendering of Apparent Sound Source Width in 3D Audio Displays. Pulkki, V. (1997). Virtual Sound Source Positioning Using Vector Base Amplitude Panning. Journal of the Audio Engineering Society, 45(6), S. 456-466. Pulkki, V. (1999). Uniform spreading of amplitude panned virtual sources. Pulkki, V. (2007). Spatial Sound Reproduction with Directional Audio Coding. J. Audio Eng. Soc, 55(6), S. 503-516. Pulkki, V., Laitinen, M.-V., & Erkut, C. (2009). Efficient Spatial Sound Synthesis for Virtual Worlds. Schlecht, S. J., Alary, B., Vaelimaeki, V., & Habets, E. A. (2018). Optimized Velvet-Noise Decorrelator. Schmele, T., & Sayin, U. (2018). Controlling the Apparent Source Size in Ambisonics Unisng Decorrelation Filters. Schmidt, J., & Schroeder, E. F. (2004). New and Advanced Features for Audio Presentation in the MPEG-4 Standard. Verron, C., Aramaki, M., Kronland-Martinet, R., & Pallone, G. (2010). A 3-D Immersive Synthesizer for Environmental Sounds. Audio, Speech, and Language Processing, IEEE Transactions on, title=A Backward-Compatible Multichannel Audio Codec, 18(6), S. 1550-1561. Zotter, F., & Frank, M. (2013). Efficient Phantom Source Widening. Archives of Acoustics, 38(1), S. 27-37. Zotter, F., Frank, M., Kronlachner, M., & Choi, J.-W. (2014). Efficient Phantom Source Widening and Diffuseness in Ambisonics.

Claims

1. 1. An apparatus for reproducing spatially extended sound sources having defined positions and geometries in a space, comprising: an interface (100) for receiving a listener position; a projector (120) for calculating a projection onto a projection plane of a two-dimensional or three-dimensional hull associated with the spatially extended sound source using the listener position, information about the geometry of the spatially extended sound source (331) and information about the position of the spatially extended sound source (341); a sound location calculator (140) for calculating at least two sound source locations for said spatially extended sound source using said projection plane; a renderer (160) for rendering the at least two sound sources at their positions to obtain a reproduction of the spatially extended sound source having two or more output signals, the renderer (160) being configured to use different sound signals for different positions of the at least two sound sources, the different sound signals being associated with the spatially extended sound source; Including, the device is configured to receive a scene description, the scene description comprising information about the position (341) and about the defined geometry of the spatially extended sound source, as well as at least one fundamental sound signal (301, 302) associated with the spatially extended sound source, the apparatus further comprises a scene description parser (180) for parsing the scene description to extract information about the position (341), information about the defined geometry (331) and the at least one underlying sound signal (301, 302), or The scene description includes, for the spatially extended sound source, at least two fundamental sound signals (301, 302) and position information (321) of each of the at least two fundamental sound signals (301, 302) relative to information (331) regarding the geometry of the spatially extended sound source, and the sound position calculator (140) is configured to use the position information (321) of the at least two fundamental sound signals (301, 302) when calculating the position of the at least two sound sources using the projection plane.

2. The apparatus of claim 1 , wherein the detector is configured to detect an instantaneous listener position in the space using a tracking system.

3. The apparatus of claim 1 , wherein the interface (100) is configured to use location data input via the interface (100).

4. the projector (120) is configured to calculate the hull of the spatially extended sound source using information about the geometry (331) of the spatially extended sound source and to project the hull in a direction towards the listener position to obtain the projection of the two-dimensional or three-dimensional hull onto the projection surface; or 4. The apparatus of claim 1, wherein the projector (120) is configured to project a geometry of the spatially extended sound source defined by information (331) about the geometry of the spatially extended sound source in a direction towards the listener position and to calculate the hull of the projected geometry to obtain the projection of the two-dimensional or three-dimensional hull onto the projection plane.

5. The apparatus of claim 1 , wherein the sound position calculator (140) is configured to calculate the positions of the at least two sound sources in the space from hull projection data and the listener position.

6. the sound location calculator (140) is configured to calculate the locations such that the at least two sound sources are ambient sound sources and are located on the projection surface; or 6. The apparatus of claim 1, wherein the sound position calculator (140) is configured to calculate a position of one of the plurality of ambient sound sources such that the position of the one ambient sound source is located to the right of the projection surface relative to the listener position and / or to the left of the projection surface relative to the listener position and / or to the top of the projection surface relative to the listener position and / or to the bottom of the projection surface relative to the listener position.

7. The renderer (160) - rendering the at least two sound sources using a panning motion dependent on the position of the at least two sound sources to obtain speaker signals for a default speaker setup; or Rendering the at least two sound sources using a binaural rendering operation using head-related transfer functions depending on the positions of the at least two sound sources to obtain a headphone signal.

7. An apparatus according to any one of claims 1 to 6, configured to:

8. a first number of fundamental sound signals (301, 302) are associated with said spatially extended sound source, said first number being equal to or greater than one, said first number of fundamental sound signals (301, 302) being associated with the same spatially extended sound source; the sound location calculator (140) determines a second number of sound sources to be used in the rendering of the spatially extended sound source, the second number being greater than one; 8. The apparatus of claim 1, wherein the renderer (160) includes one or more decorrelators (166) for generating a decorrelated signal from one or more of the first number of fundamental sound signals (164, 301, 302), the second number being greater than the first number.

9. the interface (100) is configured to receive a time-varying value of the listener's position in the space; The projector (120) is configured to calculate a time-varying projection in the space; the sound location calculator (140) is configured to calculate a time-varying number of sound sources or the time-varying locations of sound sources; 9. The apparatus of claim 1, wherein the renderer (160) is configured to render a time-varying number of the sound sources or the at least two sound sources at the time-varying positions in the space.

10. The projector (120) comprises:

10. Apparatus according to any one of the preceding claims, configured to calculate the projection as an image plane perpendicular to a listener's line of sight at the listener position.

11. The projector (120) comprises:

10. Apparatus according to any one of the preceding claims, arranged to calculate the projection as a sphere around a listener's head at the listener position.

12. The projector (120) comprises: calculating the projection as the projection plane located at a predefined distance from the centre of a listener's head at the listener position; or calculating the projection of the hull of the spatially extended sound source from azimuth and elevation angles derived from spherical coordinates relative to a spatial location of a listener's head at the listener position, where the hull is a convex hull; 10. Apparatus according to any one of claims 1 to 9, configured to:

13. 13. The apparatus of claim 1, wherein the sound position calculator (140) is configured to calculate positions of the at least two sound sources such that the positions are uniformly distributed around the projection of the hull, or such that the positions are located at extreme or peripheral points of the projection of the hull, or such that the positions are located at horizontal or vertical extreme or peripheral points of the projection of the hull.

14. 14. The apparatus of claim 1, wherein the sound location calculator (140) is configured to determine locations for auxiliary sound sources located between the locations for the surrounding sound sources in addition to the locations for the surrounding sound sources.

15. 15. The apparatus of claim 1, wherein the projector (120) is configured to additionally contract the projection of the hull towards a center of gravity of the hull.

16. the sound position calculator (140) is configured to calculate at least one additional auxiliary sound source to be located on the projection plane between a left ambient sound source and a right ambient sound source relative to the listener position, or 16. The apparatus of claim 1, wherein the sound position calculator (140) is configured to calculate at least one additional auxiliary sound source to be located on the projection plane between a left ambient sound source and a right ambient sound source relative to the listener position, with a single additional auxiliary sound source positioned midway between the left ambient sound source and the right ambient sound source, or two or more additional auxiliary sound sources positioned equidistantly between the left ambient sound source and the right ambient sound source.

17. 17. The device according to any one of claims 1 to 16, wherein the sound position calculator (140) is configured to perform a rotation of the positions of the at least two sound sources of the spatially extended sound source, preferably around the center of gravity of the projection, when a circular movement of the listener position around the spatially extended sound source is received via the interface or when a rotation of the spatially extended sound source with respect to a fixed listener position is received via the interface.

18. 18. The apparatus of claim 1, wherein the renderer (160) is configured to receive, for each sound source, an divergence angle depending on a distance between the listener position and the sound source, and to render the sound source depending on the divergence angle.

19. The sound location calculator (140) For each sound source, determining a distance equal to the distance of the spatially extended sound source to the listener position; or determining the distance of each sound source by backprojecting the position of the sound source onto the projection of the spatially extended sound source onto the geometry; It is configured as follows: The apparatus of claim 1 , wherein the renderer (160) is configured to render the at least two sound sources using information about the distances.

20. The geometric information (331) is defined as a one-dimensional line or a one-dimensional curve, a two-dimensional area, or a three-dimensional object, or The device according to any one of claims 1 to 19, wherein the information about the geometry (331) is defined as a parametric description or a polygon description or a parametric representation of the polygon description.

21. 21. The apparatus of claim 1, wherein the sound position calculator (140) is configured to determine a number of sound sources depending on a distance from the listener position to the spatially extended sound sources, the number of sound sources being larger when the distance between the listener position and the spatially extended sound sources is small compared to a smaller number when the distance is large.

22. configured to receive information regarding the widening introduced by the spatially extended sound source; 22. The apparatus of claim 1, wherein the projector (120) is configured to use information about the extent and apply a contraction operation to the hull or the projection to at least partially compensate for the extent.

23. 23. The apparatus of claim 1, wherein the renderer is configured to render the at least two sound sources by combining basis signals associated with the spatially extended sound sources to obtain a rotated basis signal when the positions of the at least two sound sources are identical to one another within a defined tolerance range, and to render the rotated basis signal at the positions of the at least two sound sources.

24. The apparatus of claim 1 , wherein the renderer (160) is configured to perform pre-processing or post-processing when generating the at least two sound sources according to direction-dependent characteristics.

25. 25. The device according to claim 1, wherein the spatially extended sound source has as the information on the geometry (331) the information that the spatially extended sound source is a spherical, ellipsoidal, linear, cuboid or piano-shaped spatially extended sound source.

26. receiving a bitstream representing a compressed description of the spatially extended sound source, the bitstream including a bitstream element (311) indicating a first number of distinct sound signals for the spatially extended sound source included in the bitstream or in an encoded audio signal received by the device, the number being equal to or greater than one; reading the bitstream elements (311) to extract the first number of distinct sound signals for the spatially extended sound source contained in the bitstream or the encoded audio signal; It is configured as follows: the sound location calculator (140) determines a second number of sound sources to be used in the rendering of the spatially extended sound source, the second number being greater than one; 26. The apparatus of claim 1, wherein the renderer (160) is configured to generate (164, 166) a third number of one or more decorrelated signals depending on the first number extracted from the bitstream, the third number being derived from a difference between the second number and the third number.

27. 1. An apparatus for generating a bitstream representing a compressed description of a spatially extended sound source, the apparatus comprising: A sound provider (200) for providing at least two different sound signals (301, 302) for said spatially extended sound source, comprising: The sound provider (200) Performing recordings of natural sound sources at a single microphone position or orientation or at multiple microphone positions or orientations; or Deriving a sound signal from a single basis signal or from multiple basis signals by one or more decorrelation filters A sound provider (200) configured to: a geometry provider (220) for computing information (331) on the geometry of said spatially extended sound source; an output data former (240) for generating a bitstream representing said condensed description, said bitstream comprising said at least two different sound signals (301, 302) and said information on the geometry (331), as well as individual position information (321) for each sound signal of said at least two different sound signals (301, 302), said individual position information (321) indicating a position of a corresponding sound source relative to said information on the geometry (331) of said spatially extended sound sources; 13. An apparatus comprising:

28. 28. The apparatus of claim 27, wherein the apparatus is configured to include in a bitstream information regarding the position of the spatially extended sound source in space (341).

29. the sound provider (200) is configured to bit-rate compress the at least two different sound signals (301, 302) using an audio signal encoder (260) to obtain at least two bit-rate compressed different sound signals (301, 302); 29. Apparatus according to claim 27 or 28, wherein the output data former (240) is configured to use the bit-rate compressed at least two different sound signals (301, 302) for the spatially extended sound source.

30. 30. The apparatus of claim 27, wherein the geometry provider (220) is configured to derive a parametric description or a polygonal description from the geometry of the spatially extended sound source, and the output data former (240) is configured to incorporate the parametric description or the polygonal description or a parametric representation of the polygonal description into the bitstream as information (331) relating to the geometry.

31. 31. The apparatus according to claim 27, wherein the output data former (240) is configured to incorporate into the bitstream a bitstream element (311) indicating the number of the at least two different sound signals (301, 302) for the spatially extended sound source contained in the bitstream or contained in an encoded audio signal associated with the bitstream, said number being 2 or more.

32. 1. A method for reproducing a spatially extended sound source having a defined position and geometry in a space, comprising: receiving a listener position; calculating a projection onto a projection plane of a two-dimensional or three-dimensional hull associated with the spatially extended sound source using the listener position, information about the geometry of the spatially extended sound source (331), and information about the position of the spatially extended sound source (341); - calculating at least two sound source positions for the spatially extended sound source using the projection plane; - rendering the at least two sound sources at their positions to obtain a reproduction of the spatially extended sound source having two or more output signals, the rendering step comprising using different sound signals for different positions of the at least two sound sources, the different sound signals being associated with the spatially extended sound source; Including, The method comprises the step of receiving a scene description, the scene description comprising information on the position (341) and on the defined geometry of the spatially extended sound source, as well as at least one fundamental sound signal (301, 302) associated with the spatially extended sound source, The method further comprises the step of analysing the scene description to extract information about the position (341), information about the defined geometry (331) and the at least one fundamental sound signal (301, 302), or The method, wherein the scene description includes at least two fundamental sound signals (301, 302) for the spatially extended sound source and position information (321) of each of the at least two fundamental sound signals (301, 302) relative to information (331) regarding the geometry of the spatially extended sound source, and the calculating step includes a step of using the position information (321) of the at least two fundamental sound signals (301, 302) when calculating the position of the at least two sound sources using the projection plane.

33. 1. A method for generating a bitstream representing a compressed description of a spatially extended sound source, comprising the steps of: providing at least two different sound signals (301, 302) for said spatially extended sound source; And, The providing step includes: - performing a recording of a natural sound source at a single microphone position or orientation or at multiple microphone positions or orientations; or deriving said at least two different sound signals (301, 302) from a single base signal or multiple base signals by one or more decorrelation filters, Providing at least two different sound signals (301, 302); providing information (331) regarding the geometry of said spatially extended sound source; generating said bitstream representing said compressed description, said bitstream comprising information on said at least two different sound signals (301, 302) and on the geometry of said spatially extended sound sources (331), as well as individual position information (321) for each sound signal (301, 302) of said at least two different sound signals, said individual position information (321) indicating a position of a corresponding sound source relative to the information on the geometry of said spatially extended sound sources (331); The method includes:

34. 34. The method of claim 33, configured to include in the bitstream information (341) about the position of the spatially extended sound source in space.

35. 35. The method according to claim 33 or 34, wherein the step of generating the bitstream comprises incorporating in the bitstream a bitstream element (311) indicating the number of the at least two different sound signals (301, 302) for the spatially extended sound source contained in the bitstream or contained in an encoded audio signal associated with the bitstream, said number being 2 or more.

36. A computer program comprising instructions for causing a computer to carry out the method of any one of claims 32 to 35 when the computer program is executed by a computer or processor.

Images