Particle selection for particle synthesis

Abstract

A method for rendering audio corresponding to an audio recording, the audio recording being divided into multiple particles. The method includes selecting particles from the multiple particles and using the selected particles to render the audio. Selecting particles from the multiple particles includes: i) a) selecting a set of particle groups, wherein each particle group included in the set of particle groups is assigned a weight, and selecting a particle group from the set of particle groups based on the assigned weights, and b) selecting particles from the selected particle groups; or ii) a) assigning a final weight to each particle in a set of candidate particles, and b) selecting a particle from the set of candidate particles based on the assigned final weights.

Description

Technical Field

[0001] Examples related to particle synthesis are disclosed. Background Technology

[0002] Audio rendering is a process for presenting audio (e.g., audio within an extended reality (XR) scene, such as a virtual reality (VR), augmented reality (AR), or mixed reality (MR) scene) to give the listener the impression that the sound originates from a physical source located at a certain location within the scene. This presentation can be done via headphone speakers or other loudspeakers. If the presentation is done via headphone speakers, the processing used is called binaural rendering and utilizes spatial cues of human spatial hearing, which allow the direction from which the sound originates to be determined. These cues involve interaural time delay (ITD), interaural sound level difference (ILD), and / or spectral difference.

[0003] Procedural audio refers to the creation of sounds in real time in response to live input. As an example, consider the sound of a car engine in a virtual space, where the sound varies based on the car's speed or acceleration. This mechanism is commonly used in video games to provide a better user experience. It is reasonable to assume that for XR (e.g., AR or VR) use cases, many sounds will benefit from being dynamically generated so that they can react in real time to changes in the scene. For example, the sound produced when a user touches a surface or operates an engine. In reference [1], the sound is simulated based on the position and orientation of a virtual object (e.g., a sword, axe, or wand). There are a fundamental tone and overtones. Both are modulated to change the pitch, timbre, and amplitude to convey the speed of movement of the virtual object. Live input can come from the user via sensors (e.g., a manual controller or headphones), and it can be control data generated in real time by some software process such as physics simulation, or predefined automated data. Regardless of how the input data is generated, the audio renderer needs to process the incoming data and generate sounds in real time in response to that data.

[0004] There are many different approaches to procedural audio (see, for example, reference [2]), including synthetic sound synthesis using audio processing modules, machine learning methods trained on real recordings, and cascade synthesis methods that utilize original recordings and rearrange segments of those recordings to produce variations. Because this type of cascade synthesis method directly utilizes real recordings, the resulting audio sounds very natural, thus enhancing the user experience.

[0005] Particle synthesis is a type of cascade synthesis in which sound recordings are divided into segments called “particles”. (See, for example, reference [3]). By carefully selecting segments (particles) during rendering, reasonably dynamic sounds can be produced.

[0006] The particle synthesis process comprises two main steps: (1) particle extraction and (2) particle synthesis. Particle extraction refers to the extraction of relevant particles from a long, raw recording. The extraction method depends on the type of sound source and the desired features to be extracted. Particle synthesis refers to the technique of selecting an appropriate particle order; this selection of order can also be based on real-time user input.

[0007] Many sound design tools support particle extraction and synthesis, such as Audiokinectic Wwise's Soundseed particles, LogicPro's Alchemy, or FMOD's AudioMotors. These tools allow sound designers to extract particles manually or semi-automatically, and allow for other simple manipulations. Designers can select control parameters such as particle length, amplitude envelope of each particle, or shape. In the case of AudioMotors, a dedicated automatic particle extraction tool is provided for motor sounds.

[0008] Particle extraction can be performed manually by the sound designer or in a data-driven manner by identifying relevant features of the audio for segmentation. Relevant features include, for example, pitch period in the case of a sound with pitch, spectral energy at a given frequency, Mel-frequency cepstral coefficients (MFCC), and local maxima of the amplitude envelope.

[0009] There are many methods for particle synthesis. One common method is to randomly select particles and perform an overlap and addition (OLA) operation. This method is not suitable for all types of sound sources and cannot capture the temporal correlation between adjacent particles.

[0010] Corpus-based cascade synthesis (CBCS) methods are based on selecting particles from a corpus of particles sampled from a database of heterogeneous particles. These methods utilize descriptors associated with sound segments to organize the corpus and perform a search within the descriptor space to select the next particle. Note that the concept of a descriptor is not limited to features of the audio signal (see, for example, reference [4]). When a direct mapping between the desired effect and features in the audio signal is not possible, the user can use perceptual descriptors to annotate the particles.

[0011] The descriptor space is multidimensional, with its dimension equal to the number of descriptors. By utilizing the weighted Euclidean distance between the target descriptor location (e.g., a point or region) in the descriptor space (hereinafter referred to as "target descriptor coordinates") and the particle location in the descriptor space, the search for appropriate particles is performed in a computationally efficient manner. Reference [5] proposes a warping function for distance measurement to better select the particle set and also avoid repeating previously rendered particles.

[0012] To perform an efficient search in the descriptor space, a kD-tree search is used. Particles are selected that are the k nearest neighbors of the target descriptor coordinates or within a radius "r" of the target descriptor coordinates. In reference [6], the corpus is organized into regions so that particles from different regions are not selected when using the k nearest neighbor search.

[0013] The software CATERPILLAR (see reference [7]) performs cascade synthesis in an offline setting (which provides a series of target descriptors). The program uses the Viterbi algorithm to identify the particle sequence to match the target descriptor. The cost function is a combination of the distance to the target descriptor coordinates and the cascade cost based on the similarity of consecutive particles.

[0014] On the other hand, CataRT is a real-time system, so it randomly selects subsequent particles from a set of particles that are either the k nearest neighbors or the radii centered on the target descriptor coordinates (see, for example, reference [8]).

[0015] To make the transitions of the particle-synthesized sound smoother, reference [9] uses feature descriptors such as pitch, loudness, spectral centroid, fundamental frequency, periodicity, and autocorrelation coefficient with hysteresis 1. Feature descriptors are computed for each particle, and the correlation between these feature descriptors is captured using a Gaussian mixture model (GMM) (based on which particles are sampled for synthesis).

[0016] There are also several works that construct Markov models to generate or synthesize sound textures by modeling or assigning the transition probabilities between adjacent particles, such as reference

[10] . Although records are used to estimate model parameters, unlike particle synthesis, sound synthesis does not directly use records.

[0017] Reference

[11] discusses particle synthesis, in which the next particle is selected based on the feature descriptor of the current particle to maintain timbre continuity. A kD-tree search is performed to select the candidate particle that is closest to the current particle in Euclidean distance from the current particle in the feature descriptor space. Summary of the Invention

[0018] Several challenges exist. For example, in existing systems, the method for selecting particles (e.g., choosing the k nearest neighbors of the target descriptor coordinates in the descriptor space) is fixed rather than adaptive, which can lead to spurious output when the neighborhood is not compact and the particle database consists of clusters of similar sizes without directional orientation. While in some systems (e.g., CataRT) the user can adaptively select the neighborhood size, this is still limited to 3D; beyond 3D, visualization is impossible, so the user cannot adjust the neighborhood size. Furthermore, particles near the target descriptor coordinates are uniformly and randomly selected without distinguishing between the most likely and least likely particles to be played. Therefore, conventional systems may fail to produce natural and smooth output, especially if the user input does not linger at the target descriptor coordinates long enough. In such cases, it is desirable to render the most representative particles closest to the target descriptor coordinates, rather than random selection. Furthermore, while some systems (e.g., CATERPILLAR) use the Viterbi algorithm to find the next particle that minimizes the cost function describing the discontinuities introduced by the cascading of that next particle, this optimization, while useful for minimizing discontinuities, does not avoid unnatural sound effects due to the repetition of identical particles or particle patterns. Moreover, in many cases, it would be beneficial to be able to control the probabilities of individual particles and groups of particles and to be able to update these probabilities in real time.

[0019] Therefore, in one aspect, a method is provided for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of particles. The method includes selecting particles from the plurality of particles. The method also includes using the selected particles to render the audio. Selecting particles from the plurality of particles includes: i) a) selecting a particle group from a set of particle groups including a first particle group and a second particle group, wherein each particle group included in the set of particle groups is assigned a weight, and selecting a particle group from the set of particle groups based on the assigned weights, and b) selecting particles from the selected particle group; or ii) a) assigning a final weight to each particle in a set of candidate particles including a first particle and a second particle, wherein assigning a final weight to each particle in the set of candidate particles includes assigning a first final weight to the first particle and assigning a second final weight to the second particle, and b) selecting a particle from the set of candidate particles based on the assigned final weights, wherein the first final weight assigned to the first particle is a function of a weight associated with the particle group to which the first particle belongs.

[0020] On the other hand, an apparatus is provided configured to perform a method for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of particles. The method includes selecting particles from the plurality of particles. The method also includes using the selected particles to render the audio. Selecting particles from the plurality of particles includes: i) a) selecting a group of particles from a set of particle groups including a first group of particles and a second group of particles, wherein each particle group in the set of particle groups is assigned a weight, and selecting a particle group from the set of particle groups based on the assigned weights; and b) selecting particles from the selected particle groups; or ii) a) assigning a final weight to each particle in a set of candidate particles including a first particle and a second particle, wherein assigning a final weight to each particle in the set of candidate particles includes: assigning a first final weight to a first particle and assigning a second final weight to a second particle, and b) selecting a particle from the set of candidate particles based on the assigned final weights, wherein the first final weight assigned to the first particle is a function of a weight associated with the particle group to which the first particle belongs. The apparatus may include a memory and processing circuitry coupled to the memory.

[0021] In another aspect, a computer program including instructions is provided that, when executed by the processing circuitry of a device, causes the device to perform any of the methods disclosed herein. In one embodiment, a carrier including a computer program is provided, wherein the carrier is one of an electrical signal, an optical signal, a radio signal, and a computer-readable storage medium.

[0022] The advantage of embodiments that select one or more particles based on target descriptor coordinates is that they facilitate particle synthesis that generates particles through natural evolution. By using weighted selection (i.e., a selection process that uses weights assigned to particles or groups of particles), precise control of sound and a combination of natural variations can be achieved. The solution has low complexity, enabling the rendering of numerous granular sound sources even on devices with limited computing power. Low real-time complexity also allows the solution to produce latency-free, reactive, and responsive sound.

[0023] The advantage of defining particle groups and assigning weights to each particle group is that the complexity burden is shifted from the database creator to the rendering algorithm. These embodiments allow the database creator to test many different weights by rendering and changing the pre-assigned weights. This involves less overhead compared to changing the model and re-estimating parameters. These embodiments provide an efficient way to update particle weights that is easy to use and requires a small bit rate when sending updates from devices not co-located with the renderer. Attached Figure Description

[0024] The accompanying drawings, which are included herein and form part of this specification, illustrate various embodiments.

[0025] Figure 1 A system according to an embodiment is shown.

[0026] Figure 2A An example two-dimensional descriptor space is shown.

[0027] Figure 2B An example two-dimensional descriptor space is shown.

[0028] Figure 3A An example two-dimensional descriptor space is shown.

[0029] Figure 3B The process for determining the optimal k value for a given particle, according to an embodiment, is illustrated.

[0030] Figure 4A An example of the descriptor trajectory of the original record is shown.

[0031] Figure 4B Examples of the original record's descriptor trajectory and the generated descriptor trajectory are shown.

[0032] Figure 4C Example sets of descriptor trajectories for the original records and the generated descriptor trajectories are shown.

[0033] Figure 5 This is a flowchart illustrating a process according to an embodiment.

[0034] Figure 6 This is a flowchart illustrating a process according to an embodiment.

[0035] Figure 7A and Figure 7B A system according to some embodiments is shown.

[0036] Figure 8 This is a block diagram of an apparatus according to some embodiments.

[0037] Figure 9 This is a flowchart illustrating a process according to an embodiment. Detailed Implementation

[0038] Figure 1A system 100 for performing particle synthesis according to some embodiments is shown. System 100 includes a particle extraction unit 102 that extracts particles from a raw audio recording 111. Specifically, the particle extraction unit divides the raw audio recording into segments referred to as “particles”. The extracted particles are stored in a particle database (GDB) 104 (or simply “database”), which is accessed during rendering by a particle scheduling unit 106, which may also be referred to as a “particle selection unit” or “particle scheduler,” and is a component of a particle rendering function (GRF) 108. In some embodiments, one particle database exists for each procedural audio source, and each particle database is available for use in the GRF 108. Each particle stored in the particle database is associated with one or more vectors of one or more descriptor values, each vector corresponding to a specific descriptor.

[0039] When creating the particle database 104, the audio designer decides which aspects should be used as descriptors. In some cases, it can be a characteristic of the sound itself, such as pitch or loudness, but it can also be other aspects related to how the sound is produced, such as the speed of motion that produces a contact sound between two objects sliding against each other, or the opening angle of a door that produces a harsh sound when it is opened and closed. The descriptors should be chosen so that the renderer can dynamically regenerate the sound based on the target descriptor coordinates or trajectory.

[0040] As mentioned above, each particle stored in the particle database is associated with one or more descriptor values. Therefore, the particles in the original recording need to be annotated with descriptor values. When the descriptor is an audio feature, the descriptor value can be measured directly from the audio signal itself. In other cases, the descriptor value needs to be provided as additional metadata for the recording in some way. This can be done, for example, by recording data from some sensors during recording and providing that data in an accompanying file. In some cases, annotation can be done manually by creating a data log describing how the descriptor changes during recording, or it can be done manually for each extracted particle.

[0041] When granules are extracted from the raw record, descriptor values ​​are stored as metadata for each granule. Using descriptor values, each granule can reside in a multidimensional descriptor space, where the value of each descriptor describes its position along an axis within that space. If only one descriptor is used, the descriptor space is one-dimensional (1D), but if multiple descriptors are used, the dimensionality of the descriptor space increases.

[0042] Figure 2A An example of a two-dimensional (2D) descriptor space is shown, where each circle represents a particle. (e.g.) Figure 2AAs shown, each particle has a location (e.g., a point or region location) in the 2D descriptor space, which is called particle coordinates.

[0043] In one embodiment, the descriptor metadata of a granular sequence extracted from a record describes the trajectory within the descriptor space, which corresponds to how the descriptor evolved during the original record.

[0044] During rendering, particle scheduling (i.e., selecting which particles to render) is typically based on target descriptor coordinates in descriptor space. Target descriptor coordinates specify what descriptor values ​​the generated sound output should have, meaning that particles closest to those coordinates in descriptor space will be used most prominently. Target descriptor coordinates can come from many types of sources, such as a physics engine simulating virtual object interaction, live input parameters from manual controllers or other sensors, or predefined automation parameters.

[0045] One aspect of particle compositing rendering is that the repetition of the same particles often sounds very unrealistic and unnatural. If the particles are short, less than 50ms, repeating the same particles will produce a very metallic and static sound. If the particles are long, the repetition will sound like a repeating pattern, which often produces an illogical sound.

[0046] The selection of particles needs to avoid repeatedly selecting the same particles, but at the same time needs to select particles in the descriptor space that are close to the target descriptor coordinates. Therefore, in some embodiments of this disclosure, a weighted selection (e.g., weighted random selection) process is used, which generates a sequence of particles (i.e., an ordered set of particles) that closely follows the evolving target descriptor coordinates. An example of a particle sequence is: [particle-7, particle-6, particle-7, particle-9, particle-11, particle-10].

[0047] In some embodiments, each particle can be assigned a predefined weight (represented as p). i5 And a set of dynamic weights that can change over time. Predefined weights can be useful in situations where particles should not be used too frequently, but whose occasional use can add realistic variations to the generated sound as outliers. Another use case is to use predefined weights to control the frequency of particles representing, for example, birdsong, compared to particles representing background sounds of a forest.

[0048] The input from the audio source (e.g., a signal generated through user interaction) is mapped to target descriptor coordinates. Based on the target descriptor coordinates, one or more particles from a candidate particle set are selected for rendering using weighted selection (e.g., weighted random selection or selection of the particle with the highest weight). The selected particles can be rendered using standard particle compositing methods, where the sound designer can pre-specify metadata regarding overlap percentage and crossfade windows.

[0049] The input to the audio source in the control process can be changed in real time; therefore, the target descriptor coordinates can change over time as the input changes (even if the input does not change, the target descriptor coordinates can still change over time).

[0050] In one embodiment, the granular selection algorithm using weighted selection has the following steps.

[0051] Step 1: Obtain the target descriptor coordinates (e.g., map the input (such as user input or other input) to the target descriptor coordinates in the descriptor space).

[0052] Step 2: Adaptively determine the size of the neighborhood, for example, by calculating the ak value based on the target descriptor coordinates, or by calculating the radius value (r) based on the target descriptor coordinates. Alternatively, obtain pre-calculated k or radius values ​​from the metadata of the particle database.

[0053] Step 3: Select a set of candidate particles from the database using either the k-value or the radius value. For example, select the particles that are the k-nearest neighbors of the target descriptor coordinates from the database. This search can be performed using readily available, computationally efficient algorithms such as kD-tree search. As another example, include every particle whose coordinates are within a distance r from the target descriptor coordinates in the candidate particle set.

[0054] Step 4: Assign a final weight to each particle in the candidate particle set. The final weight assigned to a given particle can be determined based on the following:

[0055] i) The distance between the particle's position in the descriptor space (i.e., the particle's coordinates) and the target descriptor's coordinates in the descriptor space.

[0056] ii) The difference between the trend of the particle descriptor and the trajectory of the target descriptor.

[0057] iii) The time history of previously used particles,

[0058] iv) The time difference between particles in the original record and previously used particles (if they come from the same record), and / or

[0059] v) Predefined weights of particles in the database.

[0060] Step 5: Using the final weights assigned in Step 4, perform weighted particle selection from the candidate particle set. For example, perform a weighted random selection, or, as another example, select the particle with the highest or lowest final weight. In this way, particles are selected based on the target descriptor coordinates and further based on the final weights assigned to the particles in the candidate particle set.

[0061] If the target descriptor coordinates change, these steps are repeated. Otherwise, weighted particle selection continues (starting from step 4), where the weights are changed based on the time history of previous particles.

[0062] Step 2: Determine the value of k.

[0063] Traditionally, the value of k is a user-defined constant. However, keeping the value of k fixed all the time is undesirable, as doing so may result in selecting too few or too many granules, which in turn leads to underutilization of granules or selection of granules that are not similar to the target descriptor value, respectively.

[0064] Therefore, in one embodiment, this disclosure provides an adaptive k-selection based on the density of available particles in the region surrounding the target descriptor coordinates.

[0065] Figure 2A and Figure 2B The example illustrates why different k values ​​should be used for different target descriptor coordinates. Figure 2A In the middle, the target descriptor coordinates are close to a cluster of 3 particles.

[0066] However, in Figure 2B In these scenarios, the target descriptor coordinates are located near more particles. Using the same k value is not optimal in these situations. For Figure 2A In this case, k=3 is appropriate. If k>3, this will cause particles not in the cluster to be selected, resulting in discontinuous textures in the rendered sound. Figure 2B In this case, if k=3, too few particles will be selected. Here, a larger value will result in richer textures and less repetition, because a wider variety of particles can be selected.

[0067] The value of k should be adaptively selected based on the coordinates of the target descriptor in the descriptor space. In one embodiment, each particle in the database is assigned an optimal k value. Then, for the target descriptor coordinates, the value of k is set to be equal to the optimal k value assigned to the particle closest to the target descriptor coordinates.

[0068] The assignment of the optimal k-value for each particle in the database can be performed offline or during the construction of the particle database. The particles in the database are recorded. The distance to all other particles can be calculated, or a threshold can be set for the maximum number of neighbors to stop distance calculation.

[0069] Then sort the distances from lowest to highest. (Continuous) The difference between values ​​will exhibit abrupt changes at values ​​where the distance increases sharply. This is considered... The cutoff value, and Assigned to this particle. Add 1 to include the particle itself in the k value.

[0070] The standard used to determine the cutoff value will be to set an absolute threshold for the difference in sorted distances between consecutive neighbors or to increase the normalized percentage of consecutive sorted distances.

[0071] Alternatively, the concept of an adaptive radius can be used to select the candidate particle set. Current methods in the literature involve using a fixed circle centered on the target descriptor coordinates and selecting all particles within that fixed-radius circle to be included in the candidate particle set. Following the same reasoning as above, it can be beneficial to adaptively change the radius based on the position of the target descriptor coordinates. There, instead of choosing different k values, different radius values ​​will be used.

[0072] Figure 3A The calculation of the k-value for a particle 301, represented by a black circle, is shown. Figure 3A The image shows particle 301 and its six corresponding neighbors. Figure 3B The diagram shows the sorting distance of particle 301. Since there is a significant jump in the distance value for k=4, the optimal k value of 5 is assigned to particle 301.

[0073] Step 3: Weight Assignment

[0074] The factors that influence the final weights assigned to particles are described below. We represent the target descriptor as follows: And will be with particles The associated weights are represented as Descriptor indexes are used by It means that, among them, It refers to the number of descriptors or the dimension of the descriptor space. Dimension Particles at the location The descriptor value is determined by Given, and the target's descriptor value is... .

[0075] The following criteria influencing the probability of selecting a particle are expressed as a proportional relationship because the final values ​​of the weights are obtained after normalization, i.e., ensuring compatibility with all... The weights corresponding to each particle are added together to 1.

[0076] i) Distance from the target descriptor coordinates

[0077] Distance metrics are used to define the proximity between the coordinates of a target descriptor and the coordinates of other descriptors corresponding to the particle. An example of a distance metric is the weighted Euclidean distance, where the differences in coordinates in each dimension are weighted by the reciprocal of the standard deviation of the corresponding descriptor values.

[0078] .

[0079] The closer the particle's descriptor coordinates are to the target descriptor coordinates, the higher the associated weight. Let Particles Distance to the target descriptor coordinates. For example, particles. The probability of being selected is inversely proportional to the distance.

[0080] Therefore, it is possible to set... .

[0081] In some cases, different descriptors should not have the same degree of influence on particle selection. For example, if one descriptor is the pitch of a sound and another descriptor has a less strong influence on the perceived characteristics of the sound, then the distance in the dimension corresponding to the pitch can be assigned a higher weight than the distance in the dimension corresponding to the other descriptor. This can be achieved by adding additional variable weights to each dimension when calculating this distance. To achieve:

[0082] .

[0083] ii) Trend difference between the target descriptor trajectory and the original descriptor trajectory

[0084] When performing particle extraction, descriptor coordinates are used as the primary selection criterion. However, the trend of the original descriptor trajectory also provides important information about the particle. For example, if engine sound is modeled using a particle database with RPMs representing the engine, the trend of the descriptor corresponds to whether the engine is accelerating or decelerating at the time instance from which the particle is extracted in that record. A particle extracted from a portion of a record when the engine is accelerating will have a slightly lower pitch at the beginning than at the end, and will therefore be best suited for cases where the desired output is the sound of an accelerating engine.

[0085] In a more general case, when using a multidimensional descriptor space, the descriptor trend is a vector corresponding to the direction of the trajectory of how the descriptor changed during the original recording. Similarly, the trend of the target descriptor trajectory describes the direction in which the target descriptor coordinates are moving in the descriptor space.

[0086] This trend describes the direction and rate of change. Referring again to the engine example, if the particle database includes particles that correspond to the same RPM but have different accelerations, particles with accelerations similar to those of the target descriptor trajectory should be given priority.

[0087] Figure 4A The descriptor trace of the raw recording of the engine sound is shown, where the descriptor is the engine's RPM. The recording is divided into 15 particles. Figure 4BAn example of how particles are selected to match a target descriptor trajectory is shown, where the descriptor trend of the particles is not considered during particle selection. It can be seen that particles with increasing and decreasing RPMs are used in combination. The resulting descriptor trajectory exhibits irregular behavior, which may lead to a decrease in perceived quality, especially when the descriptor represents the pitch of a sound. Figure 4C In addition, particle selection also takes into account descriptor trends, so that only particles with decreasing RPM are used, which will produce a smoother sound.

[0088] Considering descriptor trends during particle selection is especially important for sound sources with different characteristics due to varying trends. For example, an engine may sound different when accelerating than when decelerating. Ensuring that descriptor trends are matched avoids the problem of using particles with different characteristics together.

[0089] The descriptor trend of a particle can be calculated as the difference between the descriptor value at the end of the particle and the descriptor value at the beginning of the particle, divided by the duration of the particle. In the case of a multidimensional descriptor space, the trend is a vector of the average rate of change of the descriptor coordinates over the particle's duration in the original record; for example, for three descriptors, the particle trend... It will be a three-dimensional vector:

[0090] .

[0091] When particles are selected during rendering, the trend of the target descriptor trajectory can be similarly calculated as the difference in descriptor coordinates since the last update divided by the time elapsed since the last update. .

[0092]

[0093] Then, the difference in trends can be calculated as follows:

[0094]

[0095] Then, the weight assigned to the particle can be calculated as follows: Functions of the norm, for example: p i2 , where m T It is a variable used to control how the probability decreases as the trend difference of the descriptor increases.

[0096] iii) Time history of previous particle use

[0097] The history of previously rendered particles is preserved within a certain time window to avoid duplication and reduce the probability of selecting a particle again when it has already been rendered. This reduction in probability is related to the current time instance. and granules Last rendered time instance The difference between them is proportional to a function.

[0098]

[0099] If no particles have been selected in the past or within a certain time window Then the value of the previous time instance will be set to zero, that is... .function It can be a linear function, a quadratic function, a logarithmic function, or a parameter. Any monotonically increasing function with non-negative output.

[0100] iv) Differences between time instances in the original records

[0101] For some sound sources, the evolution of a sound may not be entirely described by changes in descriptors. Sometimes, the evolution of a sound depends on what happened beforehand. For example, even when an old door is opened at the same speed, the creaking sound of the old door may be slightly different each time it is opened. In these cases, particles with similar descriptor values ​​and descriptor trends may sound very different, and combining them can lead to unnatural discontinuities not present in the original recording. One way to avoid these discontinuities is to assign higher probabilities to particles from the same part of the original recording as previously used particles. Particles from the same part of the original recording are expected to be closely related and similar to each other in characteristics, and are therefore good candidates when choosing the next particle.

[0102] To measure the proximity between one particle from a specific record and another particle from the same record, a time difference can be calculated corresponding to the difference between the time instances in which the two particles were extracted from that record. If the particles are close, this time difference is small. To calculate the time difference between two particles, metadata specifying which original record each particle was extracted from and in which time instance it was extracted can be used. This metadata thus provides a measure of the proximity between the two particles. This metadata can be concisely specified as two values: a record identifier assigned to the original record and a timestamp indicating the time instance in that record in which the particle can be found.

[0103] The weight of a particle can then be calculated based on the timestamp difference as follows: the smaller the difference, the higher the probability of selecting a particle for evaluation. For example, a particle... weight It can be calculated as:

[0104]

[0105] in, It refers to assigning to the extraction of particles from it The record identifier of the record, It is the record identifier assigned to the record from which previously rendered particles are extracted. and These are the corresponding timestamps of the two particles, and This is a design constant used to set the probability of using particles from another record. (Function) The time instance difference is used as input, and the probability of the particle is calculated. In one embodiment, the function... It decreases linearly as the time instance difference increases, and its slope is determined by the variable Specify:

[0106]

[0107] This has the following effect: as the difference between the various timestamps increases, the probability decreases from 1.0 to... But never lower than .

[0108] In one embodiment, record identifier It can also be set to refer to segments of a record; that is, a record can be divided into segments, each with its own index. This can be useful when a record contains segments that the renderer considers irrelevant.

[0109] Final weight

[0110] The final weight assigned to particle i (represented as) ) is achieved by accumulating the assignments given to particles at each stage. The weights are used to calculate, that is In some embodiments, not all five weights are required and can be skipped by setting the corresponding weight to 1.0 or by excluding it entirely from the calculation.

[0111] In one embodiment, by modifying the individual weights using a fractional index, different weights have varying degrees of influence on the final weight. For example:

[0112] ,

[0113] Among them, by using The score index, the weight of the first stage. It becomes less influential, and through the use of The score index, weight It has become less influential.

[0114] The weights of all particles in the candidate particle set are calculated according to the steps described above. Then, these weights are normalized (i.e., each weight is divided by the sum) so that the final weights become probability values.

[0115] , where k is the size of the candidate particle set.

[0116] Then, particles are selected based on their final weights (e.g., by sampling from a distribution). Note that the particle's temporal history and descriptor trend affect the final weight each time the particle is rendered, assuming the target descriptor coordinates remain constant. Therefore, as long as the target descriptor remains constant, if particle i is selected at time instance t, then at time instance t+h, the particle's final weight is influenced by its temporal history and descriptor trend. The final weight of particle i becomes...

[0117] ,in, .

[0118] In order to avoid Repeated particles We can Defined as:

[0119]

[0120] However, due to the normalization operation, other The probability of each particle also changes.

[0121] Figure 5 This is a flowchart illustrating a process 500 for rendering audio corresponding to an audio recording according to an embodiment, wherein the audio recording is divided into multiple particles. Process 500 may begin at step s502.

[0122] Step s502 includes obtaining the coordinates of a first target descriptor, wherein the coordinates of the first target descriptor identify a first position in an N-dimensional descriptor space, where N>0.

[0123] Step s504 includes defining a first candidate particle set based on the coordinates of a first target descriptor, wherein the first candidate particle set includes k1 particles from a plurality of particles, where k1>1.

[0124] Step s506 includes assigning a final weight to each particle in the first candidate particle set.

[0125] Step s508 includes selecting a particle from the first candidate particle set based on the assigned final weight, such that the probability of a given particle in the first candidate particle set being selected is a function of the final weight assigned to the given particle.

[0126] Step s510 includes rendering the selected particles.

[0127] Figure 6 This is a flowchart illustrating a process 600 for rendering audio corresponding to an audio recording according to an embodiment, wherein the audio recording is divided into multiple particles. Process 600 may begin at step s602.

[0128] Step s602 includes obtaining the coordinates of a first target descriptor, wherein the coordinates of the first target descriptor identify a first position in an N-dimensional descriptor space, where N>0.

[0129] Step s604 includes defining a first candidate particle set based on the coordinates of a first target descriptor, wherein the first candidate particle set includes k1 particles out of a plurality of particles, where k1>1, and the first candidate particle set includes a first particle and a second particle.

[0130] Step s606 includes assigning a final weight to each particle in the first candidate particle set, wherein assigning a final weight to each particle in the first candidate particle set includes assigning a first final weight to the first particle and assigning a second final weight to the second particle. The first final weight assigned to the first particle is: a function of the distance between the first target descriptor coordinates and the first particle, a function of the amount of time that has elapsed since the first particle was last rendered, a function of the trajectory associated with the first particle and the target trajectory, and / or a function of a measure of the proximity between the first particle and the most recently rendered particle.

[0131] Step s608 includes selecting particles from the first candidate particle set based on the assigned final weight.

[0132] Step s610 includes rendering the selected particles.

[0133] Particle grouping examples

[0134] An alternative approach to creating a granular database is to define granular groups, i.e., at least one group of granules. An example is a granular grouping of forest sound records, where one group includes granules containing birdsong, another group includes granules containing ambient forest sounds (such as the rustling of wind), and yet another group includes granules containing water sounds (e.g., the sound of a babbling brook). Weights can be assigned to each group.

[0135] For some sound sources, where descriptors representing aspects of the sound (e.g., engine RPM) are easily defined, selecting particles based on target locations in the descriptor space is often advantageous. For other sound sources with greater randomness, it is better to select particles based on particle groups instead. When selecting particles without a descriptor space, the external input can be an update of the weights assigned to the particle groups. However, it is also possible to combine these two aspects by using a descriptor space with target locations and simultaneously defining particle groups.

[0136] Granular rendering combining descriptor space and particle group

[0137] As mentioned above, the weight assigned to particle i (p i ) is a set of weights (such as the weight p mentioned above) i1 p i2 p i3 p i4 and p i5 The function of p. In one embodiment, when particle i is a member of the particle group, p i5 It is set to be equal to the weight assigned to the particle group to which particle i belongs (this weight is denoted as p). iG Therefore, in one embodiment, the final nonnormalized weight of the particles in the weighted random selection can be calculated as: pi = p i1 p i2 p i3 p i4 p iG .

[0138] In one embodiment, for each defined particle group, the particle group's metadata is stored in a particle database. The metadata includes a particle group identifier (GID) and weights assigned to the particle group. In one embodiment, the particle group metadata is stored as a list of GIDs with associated weights, where the weights associated with a GID are the weights assigned to the particle group identified by that GID. Table 1 below shows an example list of such examples:

[0139] Table 1

[0140]

[0141] In this example, four particle groups are defined, with GIDs ranging from 0 to 3, each with its initial weight. The weight of any group can be decreased by setting it to a smaller number. Setting the initial weight to a higher number will increase the probability of selecting particles belonging to that group during rendering. The initial weight can also be set to a number greater than 1.0. The list of predefined particle groups can be stored as global metadata in the particle database.

[0142] For each particle assigned to a particle group, the particle's metadata may include the GID of the group to which the particle was assigned. If a particle is not assigned to any group, this is equivalent to assigning the particle to a particle group with a weight set to 1.0.

[0143] This embodiment makes it easy to increase or decrease the probability of particle i being selected, relative to particles that are not assigned to any group, for example. For instance, when particle i is assigned to particle group J and a weight less than 1.0 is assigned to group J, the probability that the particle belongs to group J will be lower than the probability that the particle does not belong to any group. Similarly, if the weight of the group is set to be greater than 1.0, the probability that the particle belongs to group J will be higher than the probability that the particle does not belong to any group.

[0144] Granular rendering with only a database of particle groups

[0145] For a particle database that does not have a descriptor space concept but has defined particle groups, granular rendering may include: selecting particle groups from the database using weighted selection (e.g., using a weighted random selection algorithm based on group-based assignment weights or selecting the particle group with the highest weight), and then selecting particles from the selected particle groups. In some embodiments, a weighted selection process may be used to perform the selection of particles from the selected particle groups because, in some embodiments, each particle i in the selected particle group can be assigned a different weight, e.g., a weight equal to pi. For example, in some embodiments, pi = pi3pi4pi G More generally, for granular databases without a descriptor space concept, pi can be a function of weights (e.g., pi3 and pi4) that are independent of the descriptor space concept.

[0146] Using the same example as the previous section with 4 probability groups, probability group 1 will be sampled or selected more frequently compared to probability group 2. Once a probability group is selected, any particle can then be randomly selected from that group for rendering. In another embodiment, each particle group can also have a separate probability quality function (PMF), meaning that different particles within a group have different assignment weights, and in this embodiment, a weighted random selection algorithm can be used to select particles from the particle group.

[0147] Dynamic update of particle group weights

[0148] During rendering, the weights of particle groups can be dynamically updated. This means that the initial weights assigned to a particle group are updated, and all particles assigned to that group are affected by this updated weight. By changing the weights assigned to one or more particle groups, the rendering can be altered to reflect changes in user input or other changes in the virtual scene. Specifically, changes in user input or the virtual scene can cause GRF 108 to change the weights assigned to one or more particle groups. For example, based on the user's location in the virtual scene, the probability of hearing birdsong may be higher than the probability of hearing a car; therefore, GRF 108 can increase the weight assigned to the particle group containing birdsong while decreasing the weight assigned to the particle group containing car sounds. In another example of rendering engine sounds, GRF 108 can increase the weight of the particle group representing the sound of an engine running out of fuel in response to detecting that a virtual engine in the virtual scene is running out of fuel. Therefore, by controlling the probability of that group of particles, the particles in that group may become denser when the gas level in the tank becomes extremely low.

[0149] The weights of a grain group can be updated either by directly calling a GRF 108 function or by a higher-level application that monitors the state of the virtual scene and receives user input, specifying the grain group's GID and the new updated weights, for example: UpdateGrainGroupWeight(GID, new_weight).

[0150] Another way to update the probabilities of a particle group is to include the update in a bitstream update packet, which is decoded by the decoder and then fed to the renderer. The proposed method then provides a bitrate-efficient way to update the probabilities of many particles because a single update only requires the group index (GID) and the new probability, and this update may change the probabilities of thousands of particles. In the bitstream update packet, only the group index to be updated and the new probability need to be added.

[0151] Conditional weights

[0152] Each particle can be assigned one or more condition weights relative to one or more particle groups. For example, if a particle belongs to two particle groups, it can be assigned two condition weights: a first condition weight associated with the first particle group and a second condition weight associated with the second particle group. Table 2 below shows the metadata table for particle i, which includes records for each particle group to which particle i belongs. Each record includes a first field storing the GID of the particle group to which particle i belongs and a second field storing the condition weight of particle i.

[0153] Table 2: Metadata Records of Particle i

[0154]

[0155] In some embodiments, a particle may belong to only one group, in which case the table will have only a single record.

[0156] When defining a particle group, the creator can specify that if a particle group is selected (or all candidate particles belong to the same particle group), GRF 108 should treat each particle included in the particle group as having the same conditional weight. In this case, no additional particle group metadata needs to be added to the particle group's metadata record. This is referred to as "uniform" weight assignment to the particles in the particle group. Similarly, the creator can specify that if a particle group is selected by GRF 108 (or all candidate particles belong to the same particle group), GRF 108 assigns a conditional weight to each particle in the particle group such that one or more particles in the group can have a unique weight assigned to it. In this case, metadata (e.g., a flag) should be additionally added to the particle group's metadata record to indicate that each particle in the group should be assigned a group weight. This is referred to as "non-uniform assignment." Therefore, in some embodiments, if a particle belongs to a group, the final weight assigned to that particle depends on the selected particle group.

[0157] Non-uniform assignment is beneficial when particles belonging to a particle group can have different masses. Higher-mass particles in a particle group can be assigned a higher weight per particle. Similarly, another criterion could be particle length, where longer particles are assigned a higher weight per particle compared to shorter particles.

[0158] In GRF 108, where a set of candidate particles is first selected and each candidate particle belongs to the same group, conditional weights can be used. Candidate particles can be selected from the database based on the target location and using a KD-tree search, as described above, where some particles closest to the target location in the descriptor space are selected for further evaluation. As an example, assuming all candidate particles belong to group J and particle i is one of the candidate particles, the final weight assigned to particle i can be calculated as: pi = p i1 p i2 p i3 p i4 p i5 p iCj , where p iCj This is the conditional weight of particle i relative to group J. As a specific example, when the group is a group with GID=2, then as shown in Table 2 above, p... iCj It equals 1.3. In some embodiments, p i5 =p iG (That is, the weights assigned to particle groups with GID=2). In this embodiment, the conditional weights are only related to the weight values ​​of the particle groups (p). iG( ) together are effective, that is, in the absence of In the case of, or when the candidate particles include particles from different probability groups, p iC j cannot be used as a multiplication factor for a single particle. This is because p iCj It is the conditional probability value.

[0159] In another embodiment, the final weight assigned to particle i can be calculated as: pi = p i1 p i2 p i3 p i4 p iCj In other words, in this embodiment, p i5 Set to equal p iCj Instead of being equal to p iG .

[0160] If particles in the particle database are not specified using descriptor spatial locations, the step of selecting a set of candidate particles can be performed alternatively by first selecting a particle group. Particles belonging to the selected particle group are then candidate particles, and conditional per-particle weights are applied to these particles. As an example, assuming a particle group with GID=1 is selected, the final weight assigned to particle i (which is a member of the particle group with GID=1) can be calculated as: pi = p i1 p i2 p i3 p i4 p iC1 Among them, as shown in Table 2 above, p iC1 =0.7.

[0161] As mentioned above, the selection of particle groups can be accomplished using a weighted random selection algorithm, where the probability of random selection is based on the weights of different groups. For example, assuming three particle groups are assigned weights w1, w2, and w3 respectively, the probability of the first group being selected is w1 / (w1+w2+w3), the probability of the second group being selected is w2 / (w1+w2+w3), and the probability of the third group being selected is w3 / (w1+w2+w3).

[0162] Figure 9 This is a flowchart illustrating a process 900 for rendering audio corresponding to an audio recording according to an embodiment, wherein the audio recording is divided into multiple particles. Process 900 may begin at step s902. Step s902 includes selecting a particle from the multiple particles. Step s904 includes rendering the audio using the selected particle.

[0163] In one embodiment, step s902 includes: a) selecting a particle group from a set of particle groups including a first particle group and a second particle group, wherein each particle group included in the set of particle groups is assigned a weight, and selecting a particle group from the set of particle groups is based on the assigned weight; and b) selecting a particle from the selected particle group.

[0164] In another embodiment, step s902 includes: a) assigning a final weight to each particle in a candidate particle set including a first particle and a second particle, wherein assigning a final weight to each particle in the candidate particle set includes assigning a first final weight to the first particle and assigning a second final weight to the second particle; and b) selecting a particle from the candidate particle set based on the assigned final weight, wherein the first final weight assigned to the first particle is a function of the weight associated with the particle group to which the first particle belongs.

[0165] Example use cases

[0166] Figure 7A An XR system 700 according to one embodiment is shown, in which embodiments disclosed herein can be applied. For example... Figure 7A As shown, the XR system 700 includes: an XR head-mounted device 720 (e.g., XR goggles, XR glasses, XR head-mounted display (HMD), etc.) configured to be worn by a user and operable to display an XR scene to the user, such as a VR scene in which the user is virtually immersed; speakers 734 and 735 for generating sound for the user; and an input device 750 for receiving input from the user. In this example, the input device 750 takes the form of a joystick.

[0167] like Figure 7B As shown, the XR head-mounted device 720 may include an orientation sensing unit 721, a position sensing unit 722, and an XR rendering device (XRRD) 724. In this embodiment, the XRRD 724 includes an audio renderer (AR) 799, which includes a GDB 104 and a GRF 108.

[0168] The orientation sensing unit 721 is configured to detect changes in the user's orientation and provide information about the detected changes to the XR rendering device 724. In some embodiments, given an orientation change detected by the orientation sensing unit 721, the XR rendering device 724 determines an absolute orientation (relative to a coordinate system). In some embodiments, the orientation sensing unit 721 may include one or more accelerometers and / or one or more gyroscopes.

[0169] In addition to receiving input from sensing units 721 and 722, the XR rendering device 724 can also receive input from input device 750 and can also obtain XR scene configuration information (e.g., particle metadata). Based on these inputs and the XR scene configuration, the XR rendering device 724 renders the XR scene for the user in real time. That is, the XR rendering device generates XR content in real time, including, for example, video data provided to display driver 726 to display images included in the XR scene on display screen 727, and audio data provided to speaker driver 728 to play audio using speakers 734 and 735. GRF 108 can use particles selected from GDB 104 to generate audio data or a portion thereof, as described above. Although the XR rendering device 724 is shown in this embodiment as being within the XR headset 720, in other embodiments, the XR rendering device 724 or one or more of its components (e.g., GDB 104 and GRF 108) is located remotely from the XR headset 720. In this case, the XR headset 720 and the XR rendering device 724 have communication means (transmitter, receiver) for enabling the XR rendering device 724 to send XR content to the XR headset 720 (e.g., the XR rendering device or its components may be implemented in the "cloud").

[0170] Figure 8 This is a block diagram of an XR rendering device 724 for performing the methods disclosed herein, according to some embodiments. Figure 8 As shown, the XR rendering device 724 may include: a processing circuitry (PC) 802, which includes one or more processors (P) 855 (e.g., one or more general-purpose microprocessors and / or one or more other processors, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc.), which may be housed in a single enclosure or a single data center, or may be geographically distributed (i.e., the XR rendering device 724 may be a distributed computing device including two or more computers or a monolithic computing device consisting of a single computer); at least one network interface 848 (e.g., a physical interface or an over-the-air interface). The XR rendering device 724 includes a transmitter (Tx) 845 and a receiver (Rx) 847 for enabling the XR rendering device 724 to send and receive data from other nodes connected to and from a network 110 (e.g., an Internet Protocol (IP) network), a network interface 848 (physically or wirelessly) connected to the network 110 (e.g., the network interface 848 may be coupled to one or more antennas for enabling the XR rendering device 724 to wirelessly send / receive data); and a storage unit (also known as a "data storage system") 808, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. Figure 8As shown, GDB 104 can be stored in storage unit 808. In embodiments where PC 802 includes a programmable processor, a computer-readable storage medium (CRSM) 842 may be provided. CRSM 842 may store a computer program (CP) 843 including computer-readable instructions (CRI) 844. CRSM 842 may be a non-transitory computer-readable medium, such as a magnetic medium (e.g., a hard disk), an optical medium, a storage device (e.g., random access memory, flash memory), etc. In some embodiments, the CRI 844 of the computer program 843 is configured such that, when executed by PC 802, the CRI causes XR rendering device 724 to perform the steps described herein (e.g., the steps described herein with reference to the flowchart). In other embodiments, XR rendering device 724 may be configured to perform the steps described herein without requiring code. That is, for example, PC 802 may consist only of one or more ASICs. Therefore, the features of the embodiments described herein can be implemented in hardware and / or software.

[0171] Overview of various embodiments

[0172] A1. A method for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of particles, the method comprising: selecting particles from the plurality of particles; and rendering the audio using the selected particles, wherein selecting particles from the plurality of particles comprises: i) a) selecting a particle group from a set of particle groups including a first particle group and a second particle group, wherein each particle group included in the set of particle groups is assigned a weight, and selecting a particle group from the set of particle groups based on the assigned weight, and b) selecting a particle from the selected particle group; or ii) a) assigning a final weight to each particle in a set of candidate particles including a first particle and a second particle, wherein assigning a final weight to each particle in the set of candidate particles comprises assigning a first final weight to the first particle and assigning a second final weight to the second particle, and b) selecting a particle from the set of candidate particles based on the assigned final weight, wherein the first final weight assigned to the first particle is a function of a weight associated with the particle group to which the first particle belongs.

[0173] A2. The method according to embodiment A1, wherein selecting particles from a plurality of particles includes: a) selecting a particle group from a set of particle groups including a first particle group and a second particle group, wherein each particle group included in the set of particle groups is assigned a weight, and selecting a particle group from the set of particle groups is based on the assigned weight; and b) selecting particles from the selected particle group.

[0174] A3. The method according to embodiment A2, wherein selecting a particle group from the particle group set includes selecting a particle group from the particle group set using a weighted random selection algorithm and assigned weights.

[0175] A4. The method according to embodiment A2 or A3, wherein selecting particles from the selected particle group includes: assigning a final weight to each particle in the selected particle group; and selecting particles from the selected particle group based on the assigned final weight.

[0176] A5. The method according to embodiment A4, wherein assigning a final weight to each particle in the selected particle group comprises: assigning a first final weight to a first particle included in the selected particle group, and the first final weight is a function of conditional weights associated with the first particle and the selected particle group.

[0177] A6. The method according to embodiment A5, wherein the method further comprises: obtaining metadata of the first particle before assigning a final weight to the first particle, and the metadata of the first particle includes a record including a particle group identifier (GID) and conditional weights assigned to the selected particle group.

[0178] A7. The method according to embodiment A5 or A6, wherein the first final weight is also a function of the following: the distance between the first target location and the location of the first particle, the amount of time that has elapsed since the last rendering of the first particle, the trajectory and target trajectory associated with the first particle, and / or a measure of the proximity between the first particle and the previously rendered particle.

[0179] A8. The method according to any one of embodiments A2 to A7, wherein the method further includes obtaining first metadata of the first particle group and second metadata of the second particle group, the first metadata including a first particle group identifier (GID) assigned to the first particle group and a first weight assigned to the first particle group, and the second metadata including a second GID assigned to the second particle group and a second weight assigned to the second particle group.

[0180] A9. The method according to embodiment A1, wherein selecting a particle from a plurality of particles includes: assigning a final weight to each particle in the candidate particle set; and selecting a particle from the candidate particle set based on the assigned final weight, wherein the first final weight assigned to the first particle is a function of the weight associated with the particle group to which the first particle belongs.

[0181] A10. The method according to embodiment A9, wherein selecting a particle from a plurality of particles further comprises: selecting a particle from a plurality of particles further comprises: obtaining information identifying a first target location in an N-dimensional descriptor space before assigning a final weight to each particle in the candidate particle set, wherein N>0, and the first candidate particle set is defined based on the first target location.

[0182] A11. The method according to embodiment A9 or A10, wherein selecting a particle from a plurality of particles further includes: determining the particle group to which the first particle belongs before assigning a first final weight to the first particle; and obtaining metadata of the particle group to which the first particle belongs, wherein the metadata of the particle group to which the first particle belongs includes the weight assigned to the particle group, and the weight associated with the particle group to which the first particle belongs is the weight assigned to the particle group to which the first particle belongs.

[0183] A12. The method according to embodiment A9 or A10, wherein selecting a particle from a plurality of particles further includes: obtaining metadata of a first particle before assigning a first final weight to the first particle; and obtaining a weight associated with the particle group to which the first particle belongs from the metadata of the first particle.

[0184] A13. The method according to embodiment A12, wherein the method of selecting particles from a plurality of particles further includes: selecting particles from a plurality of particles further includes: determining that each particle included in the first candidate particle set is a member of the particle group to which the first particle belongs before assigning a first final weight to the first particle, and performing a step of obtaining a weight associated with the particle group to which the first particle belongs as a result of determining that each particle included in the first candidate particle set is a member of the particle group to which the first particle belongs.

[0185] A14. The method according to any one of embodiments A9 to A13, wherein the first final weight is also a function of the following: the distance between the first target location and the location of the first particle, the amount of time that has elapsed since the last rendering of the first particle, the trajectory and target trajectory associated with the first particle, and / or a measure of the proximity between the first particle and the previously rendered particle.

[0186] B1. A computer program including instructions that, when executed by the processing circuitry of a rendering device, cause the rendering device to perform the method according to any one of embodiments A1 to A14.

[0187] B2. A carrier comprising the computer program according to embodiment C1, wherein the carrier is one of an electrical signal, an optical signal, a radio signal, and a computer-readable storage medium.

[0188] C1. A rendering apparatus for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of particles, the rendering apparatus comprising: a memory; and processing circuitry coupled to the memory, wherein the rendering apparatus is configured to perform a method comprising: selecting particles from a plurality of particles; and rendering audio using the selected particles, wherein selecting particles from a plurality of particles comprises: i) a) selecting a particle group from a set of particle groups including a first particle group and a second particle group, wherein each particle group included in the set of particle groups is assigned a weight, and selecting a particle group from the set of particle groups based on the assigned weight, and b) selecting a particle from the selected particle group; or ii) a) assigning a final weight to each particle in a set of candidate particles including a first particle and a second particle, wherein assigning a final weight to each particle in the set of candidate particles comprises assigning a first final weight to a first particle and assigning a second final weight to a second particle, and b) selecting a particle from the set of candidate particles based on the assigned final weight, wherein the first final weight assigned to the first particle is a function of a weight associated with the particle group to which the first particle belongs.

[0189] C2. The rendering device according to embodiment C1, wherein the rendering device is further configured to perform the method according to any one of embodiments A2 to A14.

[0190] Although various embodiments have been described herein, it should be understood that they are presented by way of example only and not by way of limitation. Therefore, the breadth and scope of this disclosure should not be limited to any of the exemplary embodiments described above. Furthermore, any combination of the foregoing elements with all possible variations is included in this disclosure unless otherwise indicated or otherwise explicitly conflicted in the context.

[0191] Furthermore, although the process described above and shown in the accompanying drawings is presented as a series of steps, it is for illustrative purposes only. Therefore, it is conceivable that some steps may be added, some steps may be omitted, the order of steps may be rearranged, and some steps may be performed in parallel. Additionally, in this document, "a" means "at least one" or "one or more".

[0192] References

[0193] [1] US20180068487A1: Systems and methods for simulating sounds of avirtual object using procedural audio (Disney Enterprises Inc.).

[0194] [2] Farnell, Andy, "An introduction to procedural audio and itsapplication in computer games,” Audio mostly conference Vol.23.2007.

[0195] [3] D.Gabor, “Theory of communication.Part 1:The analysis ofinformation,” Journal of the Institution of Electrical Engineers-Part III:Radio and Communication Engineering, vol.93, No.26, pp.429–441, 1946.

[0196] [4] D.Schwarz, "Corpus-Based Concatenative Synthesis," in IEEE SignalProcessing Magazine, vol.24, No.2, pp.92-104, March 2007, doi:10.1109 / MSP.2007.323274.

[0197] [5] D.Schwarz "Distance mapping for corpus-based concatenativesynthesis."Sound and Music Computing (SMC) 2011.

[0198] [6] Aaron Einbond and Diemo Schwarz."Spatializing timbre with corpus-based concatenative synthesis."International Computer Music Conference Proceedings.Vol.2010.International Computer Music Association, 2010.

[0199] [7] D.Schwarz."A system for data-driven concatenative soundsynthesis."Digital Audio Effects (DAFx).2000.

[0200] [8] D.Schwarz et al."Real-time corpus-based concatenative synthesiswith catart."9th International Conference on Digital Audio Effects (DAFx).2006.

[0201] [9] Diemo Schwarz and Norbert Schnell."Descriptor-based sound texturesampling."Sound and music computing (SMC) 2010.

[0202]

[10] Stefan Kersten and Hendrik Purwins.“Sound texture synthesis withHidden Markov Tree models in the wavelet domain”.In Proceedings of theInternational Conference on Sound and Music Computing (SMC), Barcelona,Spain, July 2010.

[0203]

[11] Diemo Schwarz and Sean O'Leary."Smooth granular sound texturesynthesis by control of timbral similarity."Sound and Music Computing (SMC)2015.

[0204]

[12] Zechen Zhang, Nikunj Raghuvanshi, John Snyder, and SteveMarschner.2019.Acoustic texture rendering for extended sources in complexscenes.ACM Trans.Graph.38, 6, Article 222 (December 2019), 9 pages.https: / / doi.org / 10.1145 / 3355089.3356566.

[0205]

[13] Barrass, Stephen, and Matt Adcock."Interactive granularsynthesis of haptic contact sounds."Audio Engineering Society conference:22ndinternational conference:virtual, synthetic, and entertainment audio.AudioEngineering Society, 2002.

[0206]

[14] US20190094975A1:Haptic Effect Conversion System Using GranularSynthesis:Immersion Corp。

Claims

1. A method (900) for rendering audio corresponding to an audio recording (111), wherein, The audio recording is divided into multiple particles, and the method includes: Select (s902) particles from the plurality of particles; and The audio is rendered using the selected particles (s904), where, Selecting particles from the plurality of particles includes: i) a) Selecting a particle group from a set of particle groups including a first particle group and a second particle group, wherein each particle group included in the set of particle groups is assigned a weight, and selecting a particle group from the set of particle groups is based on the assigned weight; and b) Selecting particles from the selected particle group, or ii) a) Assigning a final weight to each particle in the candidate particle set, which includes a first particle and a second particle, wherein assigning a final weight to each particle in the candidate particle set includes assigning a first final weight to the first particle and assigning a second final weight to the second particle; and b) Selecting a particle from the candidate particle set based on the assigned final weight, wherein the first final weight assigned to the first particle is a function of the weight associated with the particle group to which the first particle belongs.

2. The method according to claim 1, wherein, Selecting particles from the plurality of particles includes: A particle group is selected from a set of particle groups including a first particle group and a second particle group, wherein each particle group included in the set of particle groups is assigned a weight, and the selection of a particle group from the set of particle groups is based on the assigned weight; and Select particles from the selected particle group.

3. The method according to claim 2, wherein, Selecting a particle group from the set of particle groups includes: selecting a particle group from the set of particle groups using a weighted random selection algorithm and assigned weights.

4. The method according to claim 2 or 3, wherein, Selecting particles from the selected particle group includes: Assign a final weight to each particle in the selected particle group; and Based on the assigned final weight, select particles from the selected particle group.

5. The method according to claim 4, wherein, Assigning a final weight to each particle in the selected particle group includes: assigning a first final weight to the first particle included in the selected particle group, and The first final weight is a function of the conditional weights associated with both the first particle and the selected particle group.

6. The method according to claim 5, wherein, The method further includes: Before assigning the first final weight to the first particle, obtain the metadata of the first particle, and The metadata of the first particle includes a record that includes a particle group identifier (GID) assigned to the selected particle group and the condition weight.

7. The method according to claim 5 or 6, wherein, The first final weight is also a function of the following: The distance between the first target location and the location of the first particle The amount of time that has elapsed since the last rendering of the first particle. The trajectory associated with the first particle and the target trajectory, and / or A measure of proximity between the first particle and previously rendered particles.

8. The method according to any one of claims 2 to 7, wherein, The method further includes: obtaining first metadata of the first particle group and second metadata of the second particle group. The first metadata includes a first particle group identifier (GID) assigned to the first particle group and a first weight assigned to the first particle group. The second metadata includes the second GID assigned to the second particle group and the second weight assigned to the second particle group.

9. The method according to claim 1, wherein, Selecting particles from the plurality of particles includes: Assign a final weight to each particle in the candidate particle set; and A particle is selected from the set of candidate particles based on the assigned final weight, wherein the first final weight assigned to the first particle is a function of the weight associated with the particle group to which the first particle belongs.

10. The method according to claim 9, wherein, Selecting a particle from the plurality of particles further includes: obtaining information identifying a first target location in an N-dimensional descriptor space, where N > 0, before assigning a final weight to each particle in the candidate particle set. The first set of candidate particles is defined based on the first target location.

11. The method according to claim 9 or 10, wherein, Selecting a particle from the plurality of particles further includes: before assigning the first final weight to the first particle. Determine the particle group to which the first particle belongs; and Obtain the metadata of the particle group to which the first particle belongs, wherein, The metadata of the particle group to which the first particle belongs includes the weight assigned to the particle group, and The weight associated with the particle group to which the first particle belongs refers to the weight assigned to the particle group to which the first particle belongs.

12. The method according to claim 9 or 10, wherein, Selecting a particle from the plurality of particles further includes: before assigning the first final weight to the first particle. Obtain the metadata of the first particle; and Obtain the weight associated with the particle group to which the first particle belongs from the metadata of the first particle.

13. The method according to claim 12, wherein, Selecting a particle from the plurality of particles further includes: determining, before assigning the first final weight to the first particle, that each particle included in the first candidate particle set is a member of the particle group to which the first particle belongs, and As a result of determining that each particle included in the first candidate particle set is a member of the particle group to which the first particle belongs, the step of obtaining the weight associated with the particle group to which the first particle belongs is performed.

14. The method according to any one of claims 9 to 13, wherein, The first final weight is also a function of the following: The distance between the first target location and the location of the first particle The amount of time that has elapsed since the last rendering of the first particle. The trajectory associated with the first particle and the target trajectory, and / or A measure of proximity between the first particle and previously rendered particles.

15. A computer program (843) comprising instructions (844) that, when executed by a processing circuit (502) of a rendering device (724), causes the rendering device to perform the method according to any one of claims 1 to 14.

16. A carrier comprising a computer program according to claim 15, wherein, The carrier is one of electrical signals, optical signals, radio signals, and computer-readable storage media (842).

17. A rendering device (724) for rendering audio corresponding to an audio recording (111), wherein, The audio recording is divided into multiple particles, and the rendering device includes: Memory (842); and Processing circuitry (802), coupled to the memory, wherein the rendering device is configured to execute a method comprising: Select (s902) particles from the plurality of particles; and The audio is rendered using the selected particles (s904), where, Selecting particles from the plurality of particles includes: i) a) Selecting a particle group from a set of particle groups including a first particle group and a second particle group, wherein each particle group included in the set of particle groups is assigned a weight, and selecting a particle group from the set of particle groups is based on the assigned weight; and b) Selecting particles from the selected particle group, or ii) a) Assigning a final weight to each particle in the candidate particle set, which includes a first particle and a second particle, wherein assigning a final weight to each particle in the candidate particle set includes assigning a first final weight to the first particle and assigning a second final weight to the second particle; and b) Selecting a particle from the candidate particle set based on the assigned final weight, wherein the first final weight assigned to the first particle is a function of the weight associated with the particle group to which the first particle belongs.

18. The rendering apparatus of claim 17, wherein, The rendering device is also configured to perform the method according to any one of claims 2 to 14.

Images