Particle interpolation for particle synthesis

Abstract

A method for rendering audio corresponding to an audio recording, wherein the audio recording is divided into multiple particles. The method includes obtaining information identifying the target position in an N-dimensional descriptor space, where N > 0. The method further includes: selecting a set of particles from a particle database based on the target position, wherein the selected set of particles includes first particles and second particles. The method further includes determining a first weight w1 for the first particles. The method further includes determining a second weight w2 for the second particles. The method further includes using the first weight, the second weight, the first particle, and the second particle to generate interpolated particles. The method further includes rendering the interpolated particles.

Description

Technical Field

[0001] The disclosed embodiments relate to particle synthesis. Background Technology

[0002] Audio rendering is the process of presenting audio, such as audio within extended reality (XR) scenes (e.g., virtual reality (VR), augmented reality (AR), or mixed reality (MR) scenes), to give the listener the impression that the sound is coming from a physical source within the scene at a certain location. This can be done via headphone speakers or other loudspeakers. If the presentation is done via headphone speakers, the processing used is called binaural rendering and utilizes spatial cues of human spatial hearing that allow determination of the direction from which the sound is coming. These cues involve interaural time delay (ITD), interaural intensity difference (ILD), and / or spectral difference.

[0003] Procedural audio refers to the creation of sound in real time as a response to real-time input. As an example, consider the sound of a car engine in a virtual space, where the sound changes based on the car's speed or acceleration. This mechanism is commonly used in video games for a better user experience. It is believed that for XR use cases (e.g., AR or VR), there are many that would benefit from dynamically generated sounds, allowing these sounds to react in real time to changes in the scene. For example, the sound generated when a user touches a surface or operates an engine. In reference [1], the sounds of virtual objects (e.g., swords, axes, or staffs) are simulated based on their position and orientation. There are fundamental and overtones. Both are modulated to change pitch, timbre, and amplitude to convey the speed of movement of the virtual object. Real-time input can come from the user via sensors (such as manual controllers or headphones), or it can be control data generated in real time by a software process (such as physics simulation) or predefined automated data. Regardless of how the input data is generated, the audio renderer needs to process the incoming data and generate sound in real time in response to that data.

[0004] There are many different approaches to procedural audio (see, for example, reference [2]), including synthetic sound synthesis using audio processing modules, machine learning methods trained on real recordings, and splicing synthesis methods that utilize original recordings and rearrange segments of those recordings to generate variants. Because these splicing synthesis methods use real recordings directly, the resulting audio sounds are very natural, thus enhancing the user experience.

[0005] Particle synthesis is a type of splicing synthesis in which sound recordings are divided into small segments called “particles” (see, for example, reference [3]). By carefully selecting the segments (particles) at rendering time, reasonably dynamic sounds can be generated.

[0006] The particle synthesis process comprises two main steps: (1) particle extraction and (2) particle synthesis. Particle extraction refers to the extraction of relevant particles from a long, raw recording. The extraction method depends on the type of sound source and the desired features to be extracted. Particle synthesis refers to the technique of selecting an appropriate particle order; this order can also be based on real-time user input.

[0007] Many sound design tools support particle extraction and synthesis, such as, for example, Audiokinectic Wwise's Soundseed Particles, Logic Pro's Alchemy, or FMOD's AudioMotors. These tools allow sound designers to extract particles manually or semi-automatically, and perform other simple manipulations. Designers can select the particle length, amplitude envelope, or shape of each particle, as well as other controls. In the case of AudioMotors, an automated particle extraction tool specifically designed for engine sounds is provided.

[0008] Particle extraction can be performed manually by sound designers or in a data-driven manner by identifying relevant features of audio for segmentation purposes. Relevant features include, for example, the pitch period of the tone, the spectral energy of a given frequency, the Mel-frequency cepstral coefficients (MFCC), and the local maxima of the amplitude envelope.

[0009] There are also many methods for particle synthesis. A common method is to randomly select particles and perform overlap and addition (OLA or overlap-addition) operations. This method is not suitable for all types of sound sources and does not capture the temporal correlation between adjacent particles.

[0010] Corpus-based concatenation synthesis (CBCS) methods are based on selecting particles from a corpus of sound segments sampled from a database of heterogeneous sound sources. They organize the corpus using descriptors associated with the sound segments and perform a search within the descriptor space to select the next particle. Note that the concept of a descriptor is not limited to features of the audio signal (see, for example, reference [4]). When a direct mapping between the desired effect and features in the audio signal is not possible, the user can annotate the particles with perceptual descriptors.

[0011] The descriptor space is multidimensional, where the number of dimensions is equal to the number of descriptors. Searching for suitable particles is performed in a computationally efficient manner by utilizing the weighted Euclidean distance between the target descriptor location (e.g., a point or region) in the descriptor space (hereinafter referred to as "target descriptor coordinates" or "target location") and the particle location in the descriptor space. Reference [5] proposes a warp function for the distance metric to better select the particle set and also avoid duplicate particles from previous renderings.

[0012] For efficient searching in the descriptor space, a kD-tree search is used. The k nearest neighbors of the target descriptor coordinates or particles within a radius r from the target descriptor coordinates are selected. In reference [6], the corpus is organized into regions so that particles from different regions are not selected when using a k nearest neighbor search.

[0013] The software CATERPILLAR (see reference [7]) performs splicing synthesis in an offline setting, where a sequence of target descriptors is given. The program uses the Viterbi algorithm to identify the particle sequence to match the target descriptor. The cost function is a combination of the distance to the target descriptor coordinates and the splicing cost based on the similarity of consecutive particles.

[0014] On the other hand, CataRT is a real-time system, so it randomly selects subsequent particles from the set of the k nearest neighbors or from the radius centered on the target descriptor coordinates (see, for example, reference [8]).

[0015] To achieve smoother transitions in particle-synthesized sounds, reference [9] uses feature descriptors such as pitch, loudness, spectral centroid, fundamental frequency, periodicity, and autocorrelation coefficient at hysteresis 1. Feature descriptors are computed for each particle, and a Gaussian mixture model (GMM) is used to capture the correlation between these feature descriptors, based on which particles are sampled for synthesis.

[0016] There are also several results, such as reference

[10] , which model or assign the transition probabilities between adjacent particles to establish Markov models for generating or synthesizing sound textures. The model parameters are estimated using records, but unlike particle synthesis, sound synthesis does not directly use records.

[0017] Reference

[11] discusses particle synthesis, in which, for the continuity of timbre, the next particle is selected based on the feature descriptor of the current particle. A kD-tree search is performed to select the candidate particle with the closest Euclidean distance to the current particle in the feature descriptor space.

[0018] When the audio corpus is sparse, reference [9] proposes a corpus expansion method based on feature descriptors. These techniques belong to the Feature Modulation Synthesis (FMS) category (see, for example, reference

[15] ), which identifies appropriate transformations to be applied to the audio based on target feature descriptors. Specific methods include pitch transformation, gain adjustment, and the use of filters. These methods may involve several steps and must be performed offline, i.e., before particle synthesis or rendering.

[0019] Particle synthesis typically uses overlay addition to produce a smooth output with a continuous stream of particles, without artifacts from abrupt changes or discontinuities between particles. Overlay addition is performed so that the perceived loudness is kept within the overlapping area of ​​consecutive particles. Otherwise, the rendering will sound uneven, fragmented, or simply fluctuating in loudness.

[0020] If consecutive particles are highly correlated and nearly in phase with each other, the mixture of two particles will be linearly additive; that is, if two particles with the same loudness are given a gain of 0.5, the combined mixture will have the same loudness as the two individual particles. In this case, the two particles are constructively additive, and the gains of the two particles should be set such that the total gain sums to 1.0. In this case, the mixture follows the linear mixing rule, where the mixture is a linear combination of the particles being mixed.

[0021] However, if the two particles (the first and second particles) are largely uncorrelated, the mixture of the first and second particles will be nonlinearly additive. This is because many samples of the first particle will have a different sign than the samples of the second particle they combine with, resulting in a lower amplitude rather than a higher amplitude when added. In this case, the two particles do not always additively add to each other. To keep the perceived loudness of the mixture the same as that of the two particles, the mixture should follow a different mixing rule, where the gain of each particle being mixed is set such that the signal power is maintained: P MIX = P1 = P2. The power of the signal is proportional to the square of the amplitude, which means the gain should be set according to the following formula: Where A1 and A2 are the amplitudes of the signals of the two particles to be mixed, and g1 and g2 are the gains used when performing the mixing. If the mixing has the same power as signals 1 and 2, it follows A... MIX Since A1 and A2 are the same, the gain should satisfy: For example, when g1 and g2 are equal, they should be set to... .

[0022] To achieve a smooth, overlapping additive result, the window function used needs to be selected based on the characteristics of the sound being rendered. For particle-dependent sounds, an amplitude-preserving window should be used, while for particle-independent sounds, a power-preserving window should be used.

[0023] Blending rules have an effect not only during overlap-addition processing, but also whenever other forms of particle blending are performed. For example, when the renderer performs interpolation between particles by weighted blending of two or more particles.

[0024] Using the correlation coefficient between the two particles, the work described in reference

[16] designed a custom or analytical window function for power preservation, since it may be difficult to use the threshold correlation coefficient value to determine the correlation level. Summary of the Invention

[0025] Several challenges exist. For example, obtaining a sufficiently dense particle database (i.e., containing sufficiently uniformly distributed particles) to provide smooth transitions between particles for all possible trajectories in the descriptor space is sometimes infeasible. Sometimes this involves a recording process, and it is not easy to capture sound representing all locations in the descriptor space, such as, for example, when recording the sound of a forest when the wind is not blowing at the correct speed.

[0026] Furthermore, even if a dense particle database were feasible, the reason for not using one is due to memory consumption. A particle database with a three-dimensional (3D) descriptor space would require storing a large number of particles to achieve dense particle coverage. Using a sparse database where the descriptor space is not uniformly filled with particles can solve the memory consumption problem; however, using a sparse database generally means that the rendering will not produce smooth-changing outputs because there will be large jumps between particles in the descriptor space, and this often results in unnatural, abrupt changes in the rendered sound.

[0027] Crossfade is widely used in loop-based audio rendering, where two or more audio loops are played in parallel, and amplitude crossfade is applied to achieve a smooth transition from one loop to another. However, achieving smooth interpolation is a more complex process when using particle synthesis with multidimensional descriptor spaces of particles with different lengths (representing different pitches).

[0028] Corpus expansion techniques described in the literature involve several steps, such as identifying the appropriate feature descriptors to be manipulated and applying the correct transformations to audio particles to synthesize new particles. Depending on the type of sound source, each corpus requires different steps, and may also require manual tuning of several parameters. The aim of these methods is to expand the corpus or database and then use it for rendering. The steps for corpus expansion typically cannot be performed during rendering time because they are computationally expensive.

[0029] Therefore, in one aspect, a method is provided for rendering audio corresponding to an audio recording, wherein the audio recording is divided into multiple particles. The method includes obtaining information identifying a target position in an N-dimensional descriptor space, where N > 0. The method further includes selecting a set of particles from a particle database based on the target position, wherein the selected set of particles includes a first particle and a second particle. The method further includes determining a first weight w1 for the first particle. The method further includes determining a second weight w2 for the second particle. The method further includes using the first weight, the second weight, the first particle, and the second particle to generate interpolated particles. The method further includes rendering the interpolated particles.

[0030] In another aspect, an apparatus is provided configured to perform a method for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of particles. The method includes obtaining information identifying a target position in an N-dimensional descriptor space, where N > 0. The method further includes selecting a set of particles from a particle database based on the target position, wherein the selected set of particles includes first particles and second particles. The method further includes determining a first weight w1 for the first particles. The method further includes determining a second weight w2 for the second particles. The method further includes using the first weight, the second weight, the first particle, and the second particle to generate interpolated particles. The method further includes rendering the interpolated particles. The apparatus may include a memory and processing circuitry coupled to the memory.

[0031] In another aspect, a computer program including instructions is provided that, when executed by processing circuitry of a device, causes the device to perform any of the methods disclosed herein. In one embodiment, a carrier containing a computer program is provided, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer-readable storage medium.

[0032] The advantages of the embodiments disclosed herein are that they are universal for databases of any size and type. Furthermore, the particles selected for interpolation are dynamic, as they are chosen based on their distance from the target descriptor location and are not repeated within a short period. The embodiments do not require many particles for interpolation, can work with just two particles, and can be scaled up to select multiple particles for interpolation if needed for a specific type of sound source. The embodiments are lightweight and can be executed during rendering time, thereby reducing the storage size of the particle database in memory. Attached Figure Description

[0033] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0034] Figure 1 A system according to an embodiment is shown.

[0035] Figure 2A An example two-dimensional descriptor space is shown.

[0036] Figure 2B An example two-dimensional descriptor space is shown.

[0037] Figure 3A An example two-dimensional descriptor space is shown.

[0038] Figure 3B The process for determining the optimal k value for a given particle, according to an embodiment, is illustrated.

[0039] Figure 4A An example of the descriptor trajectory of the original record is shown.

[0040] Figure 4B Examples of the original record's descriptor trajectory and the generated descriptor trajectory are shown.

[0041] Figure 4C Examples of the original record's descriptor trajectory and the generated descriptor trajectory are shown.

[0042] Figure 5 This is a flowchart illustrating a process according to an embodiment.

[0043] Figure 6 This is a flowchart illustrating a process according to an embodiment.

[0044] Figure 7A and Figure 7B A system according to some embodiments is shown.

[0045] Figure 8 This is a block diagram of an apparatus according to some embodiments.

[0046] Figure 9 The 2D descriptor space is shown to be non-uniformly covered by particles in GDB 104.

[0047] Figure 10 This is a flowchart illustrating a process according to an embodiment.

[0048] Figure 11 A particle database with several particle clusters is shown.

[0049] Figure 12 This is a flowchart illustrating a process according to an embodiment.

[0050] Figure 13 This is a flowchart illustrating a process according to an embodiment.

[0051] Figure 14A The rendering of the particle set over time is shown.

[0052] Figure 14B An example of rapid particle switching is shown.

[0053] Figure 15 The parameters used to determine whether a fast particle switching should be triggered are shown.

[0054] Figure 16A An example of fast particle switching without phase compensation is shown.

[0055] Figure 16B An example of fast particle switching with phase compensation is shown.

[0056] Figure 17 This is a flowchart illustrating a process according to an embodiment. Detailed Implementation

[0057] Particle selection

[0058] Figure 1 A system 100 for performing particle synthesis according to some embodiments is shown. System 100 includes a particle extraction unit 102 that extracts particles from a raw audio recording 111. That is, the particle extraction unit divides the raw audio recording into small segments, referred to as “particles.” The extracted particles are stored in a particle database (GDB) 104 (or simply “database”), which is accessed at rendering time by a particle scheduling unit 106, which may also be referred to as a “particle selection unit” or “particle scheduler,” and is a component of a particle rendering function (GRF) 108. In some embodiments, there is one particle database for each procedural audio source, and each particle database is available for the GRF 108. Each particle stored in the particle database is associated with one or more vectors of one or more descriptor values, each vector corresponding to a specific descriptor.

[0059] When creating the particle database 104, the audio designer decides what aspects should be used as descriptors. In some cases, this can be a characteristic of the sound itself, such as pitch or loudness, but it can also be other aspects related to how the sound is generated, such as the speed of motion that produces a contact sound between two objects sliding against each other, or the opening angle of a door that produces a harsh sound when it is opened and closed. Descriptors should be chosen so that the sound can be dynamically regenerated by the renderer given the target descriptor coordinates or trajectory.

[0060] As described above, each particle stored in the particle database is associated with one or more descriptor values. Therefore, the particles in the original recording need to be annotated with descriptor values. In cases where the descriptor is an audio feature, the descriptor value can be measured directly from the audio signal itself. In other cases, the descriptor value needs to be provided in some way as additional metadata of the recording. This can be achieved, for example, by recording data from some sensors during recording and providing that data in a companion file. In some cases, annotation can be done manually by creating a data log describing how the descriptor changes during recording, or it can be done manually for each extracted particle.

[0061] When extracting particles from one or more original records, a descriptor value is stored as metadata for each particle. Using the descriptor values, each particle can be positioned in a multidimensional descriptor space, where the value of each descriptor describes its position along an axis within that space. If only one descriptor is used, the descriptor space is one-dimensional (1D); however, if multiple descriptors are used, the dimensionality of the descriptor space increases.

[0062] Figure 2A An example of a two-dimensional (2D) descriptor space is shown, where each circle represents a particle. (e.g.) Figure 2A As shown, each particle has a location (e.g., a point or region location) in the 2D descriptor space, which is called the particle coordinates.

[0063] In one embodiment, the descriptor metadata for a particle sequence extracted from a record describes a trajectory in the descriptor space, corresponding to how the descriptor evolved during the original record.

[0064] During rendering, particle scheduling—the selection of particles to be rendered—is typically based on target descriptor coordinates in descriptor space. Target descriptor coordinates specify what descriptor values ​​the generated sound output should have, meaning that particles in descriptor space closest to those coordinates will be used most prominently. Target descriptor coordinates can come from many types of sources, such as physics engines simulating virtual body interactions, real-time input parameters from manual controllers or other sensors, and predefined automated parameters.

[0065] One aspect of particle synthesis rendering is that the repetition of the same particles often sounds very unrealistic and artificial. If the particles are short, less than 50ms, repeating the same particles will produce a very metallic and static sound. If the particles are long, the repetition will sound like a repetitive pattern, which often results in an unreasonable sound.

[0066] The selection of particles needs to avoid repeating the same particles, but at the same time select particles in the descriptor space that are close to the target descriptor coordinates. Therefore, in some embodiments, this disclosure uses a weighted selection (e.g., weighted random selection) process that generates a sequence of particles (i.e., an ordered set of particles) that closely follows the evolving target descriptor coordinates. An example of a particle sequence is: [particle-7, particle-6, particle-7, particle-9, particle-11, particle-10].

[0067] In some embodiments, each particle can be assigned a predefined weight (labeled p). i5 And a set of dynamic weights that can change over time. Predefined weights can be useful in situations where particles should not be used too frequently, but if used occasionally, they can add outliers to the generated sound, indicating real variation. Another use case is using predefined weights to control the frequency of particles representing, for example, birdsong, relative to particles representing background forest sounds. For example, in an embodiment where a set of one or more particle groups is defined and a first set of weights (wg1) is assigned to a first particle group within that set, and particle i is a member of the first particle group, then p i5 It can be set to wg1.

[0068] The input to the procedural audio source (e.g., a signal generated by user interaction) is mapped to target descriptor coordinates. Based on the target descriptor coordinates, one or more particles from a candidate particle set are selected for rendering using a weighted selection method (e.g., weighted random selection or selection where the particle with the highest weight is selected). The selected particles can be rendered using standard particle compositing methods, which include, for example, performing an overlap-addition operation using two selected particles, such as crossfading the selected particles with another particle, where metadata regarding the overlap percentage and crossfade windows can be pre-specified by the sound designer. Therefore, rendering particles encompasses not only rendering samples of all particles, but also a first set of rendered particle samples, followed by a set of blended samples generated by mixing a second set of particle samples with samples from another particle, and a second set of rendered particle samples.

[0069] The input to the programmed audio source can be changed in real time; therefore, the target descriptor coordinates can change over time as the input changes (even if the input does not change, the target descriptor coordinates can still change over time).

[0070] In one embodiment, the particle selection algorithm using weighted selection has the following steps.

[0071] Step 1: Obtain the target descriptor coordinates (e.g., map the input (such as user input or other input) to the target descriptor coordinates in the descriptor space).

[0072] Step 2: Adaptively determine the size of the neighborhood, for example, by calculating the k value based on the target descriptor coordinates, or by calculating the radius value (r) based on the target descriptor coordinates. Alternatively, obtain a pre-calculated k value or radius from the metadata of the particle database.

[0073] Step 3: Use the k-value or radius value to select a candidate particle set from the database. For example, select the k nearest neighbors of the target descriptor coordinates from the database. This search can be performed using readily available computationally efficient algorithms, such as kD-tree search. As another example, the candidate particle set includes each particle with descriptor coordinates within a distance r from the target descriptor coordinates.

[0074] Step 4: Assign a final weight to each particle in the candidate particle set. The final weight assigned to a given particle can be based on:

[0075] i) The distance between the position of the particle in the descriptor space (i.e., the particle coordinates) and the coordinates of the target descriptor in the descriptor space.

[0076] ii) Differences in particle descriptor trends compared to the target descriptor trajectory.

[0077] iii) The time history of previously used particles,

[0078] iv) The time difference between the particles in the original record and the particles used previously, if they come from the same record, and / or

[0079] v) Predefined weights of particles in the database.

[0080] Step 5: Perform a weighted selection of particles from the candidate particle set using the final weights assigned in Step 4. For example, perform a weighted random selection, or, as another example, select the particle with the highest or lowest final weight. Thus, particles are selected based on the target descriptor coordinates and further based on the final weights assigned to the particles in the candidate particle set.

[0081] If the target descriptor coordinates change, repeat the steps. Otherwise, continue the weighted selection of particles (starting from step 4) by utilizing the changes made to the weights based on the previous particle time history.

[0082] Step 2 - Determining the value of k.

[0083] Traditionally, the k value is a user-defined constant. However, it is not desirable to always keep the k value fixed, as doing so can lead to selecting too few or too many particles, which in turn can result in underutilization of particles or selection of particles that do not match the target descriptor value, respectively.

[0084] Therefore, in one embodiment, this disclosure provides an adaptive selection of k based on the density of particles available in the region surrounding the target descriptor coordinates.

[0085] exist Figure 2A and Figure 2B The example illustrates why different k values ​​should be used for different target descriptor coordinates. Figure 2A In the middle, the target descriptor coordinates are close to a cluster with 3 particles.

[0086] However, in Figure 2B In these scenarios, the target descriptor coordinates are closer to more particles. Using the same k value is not optimal in these situations. For Figure 2A In this case, k=3 is appropriate. If k > 3, this will result in the selection of particles not in the cluster, and thus cause discontinuities in the texture of the rendered sound. Figure 2B In this case, if k = 3, too few particles are selected. Larger values ​​will result in richer textures with less repetition, as a wider variety of particles can be chosen.

[0087] The k-value should be adaptively selected based on the target descriptor coordinates in the descriptor space. In one embodiment, each particle in the database is assigned an optimal k-value. Then, for the target descriptor coordinates, the k-value is set to be equal to the optimal k-value assigned to the particle closest to the target descriptor coordinates.

[0088] The assignment of the optimal k for each particle in the database can be performed offline or during the construction of the particle database. All distances from particle i to other particles in the database are recorded, or a threshold of the maximum number of neighbors for calculating the stopping distance can be used.

[0089] The distances are then sorted from lowest to highest. The difference between distances for consecutive k values ​​will have a sudden jump at values ​​where the distance increases sharply. This is considered the cutoff value for k, and k+1 is assigned to the particle. Adding 1 means including the particle itself in the k value.

[0090] The criterion for determining the cutoff value would be to set an absolute threshold for the difference in sorted distances between consecutive neighbors, or to increase the normalized percentage of consecutive sorted distances.

[0091] Alternatively, the concept of an adaptive radius can be used to select the candidate particle set. Existing methods in the literature involve using a fixed radius centered on the target descriptor coordinates and selecting all particles within a circle of that fixed radius to be included in the candidate particle set. Following the same reasoning as above, it may be beneficial to adaptively change the radius based on the position of the target descriptor coordinates. For this, instead of choosing different k values, different radius values ​​would be used.

[0092] Figure 3A The calculation of k for a specific particle 301, represented by the black circle, is shown. Figure 3A The image shows particle 301 and its six corresponding neighbors. Figure 3B In the diagram, sorted distances are shown for particle 301. Since jumps in distance values ​​are observable for k=4, the optimal k value of 5 is assigned to particle 301.

[0093] Step 3 - Weight Allocation

[0094] The following describes each factor that influences the final weights assigned to particles. The target descriptor is tagged as... and will be with particles The associated weights are labeled as The descriptor index is represented as ,in, This refers to the number of descriptors or the dimension of the descriptor space. In terms of dimension... Particles at the location The descriptor value is determined by Given, and the target's descriptor value is... .

[0095] The following criteria affecting the probability of selecting particles are expressed as proportional relationships because the final values ​​of the weights are obtained after normalization, that is, ensuring that the weights corresponding to all k particles add up to 1.

[0096] i) Distance from the target descriptor coordinates

[0097] Distance metrics are used to define the proximity between the coordinates of a target descriptor and the coordinates of other descriptors corresponding to the particle. An example of a distance metric is the weighted Euclidean distance, where the difference between coordinates in each dimension is weighted by the inverse of the standard deviation of the corresponding descriptor values.

[0098] .

[0099] The closer the particle's descriptor coordinates are to the target descriptor coordinates, the higher the associated weight. Let From particles Distance to the target descriptor coordinates. For example, particle. The probability of being selected can be inversely proportional to the distance.

[0100] Therefore, it is possible to set... .

[0101] In some cases, different descriptors should not have the same amount of influence on particle selection. For example, if one descriptor is the pitch of a sound and another descriptor has a less strong influence on the perceived characteristics of sound, then the distance in the dimension corresponding to the pitch can be given a higher weight than the distance in the dimension corresponding to the other descriptor. This can be achieved by adding additional variable weights to each dimension when calculating the distance. To achieve:

[0102] .

[0103] ii) Differences in trends between the target descriptor trajectory and the original descriptor trajectory

[0104] When performing particle extraction, descriptor coordinates are used as the primary selection criterion. However, the trend of the original descriptor trajectory also provides important information about the particles. For example, if engine sounds are modeled using a particle database, where one descriptor represents the engine's RPM, then the trend of that descriptor corresponds to the engine's acceleration or deceleration at a given time point in the record from which particles are extracted. Particles extracted from a portion of the record when the engine is accelerating will have a slightly lower pitch than at the beginning and end, and will therefore be best suited when the desired output is the sound of an accelerating engine.

[0105] In a more general case, when using a multidimensional descriptor space, the descriptor trend is a vector corresponding to the direction of the trajectory describing how the descriptor changes at the time of the original record. Similarly, the trend of the target descriptor trajectory describes the direction in which the target descriptor coordinates move in the descriptor space.

[0106] The trend describes both direction and rate of change. Referring again to the engine example, if the particle database includes particles corresponding to the same RPM but with different accelerations, then particles with accelerations similar to those of the target descriptor trajectory should be preferred.

[0107] Figure 4A The descriptor trajectory of the original recording of the engine sound is shown, where the descriptor is the engine's RPM. The recording is divided into 15 particles. Figure 4BAn example of how particles can be selected to match a target descriptor trajectory is shown, where the descriptor trend of the particles is not considered during particle selection. It can be seen that particles with increasing and decreasing RPMs are used in combination. The resulting descriptor trajectories exhibit irregular behavior, which may lead to a decrease in perceived quality, especially if the descriptor represents the pitch of a sound. Figure 4C In addition, particle selection also takes into account descriptor trends so that only particles with reduced RPM are used, which will result in a smoother sound.

[0108] Considering descriptor trends during particle selection is particularly important for sound sources, where characteristics differ for different trends. For example, an engine might sound different when accelerating compared to when decelerating. Ensuring matching with descriptor trends avoids the problems of using particles with different characteristics together.

[0109] The descriptor trend of a particle can be calculated as the difference between the descriptor value at the end of the particle's lifespan and its value at the beginning of the particle's lifespan. In the case of a multidimensional descriptor space, the trend is a vector of the average rate of change of the descriptor coordinates during the period describing the particle in the original record; for example, with three descriptors, the particle's trend t... G It will be a three-dimensional vector:

[0110] .

[0111] When particles are selected at render time, the trend of the target descriptor trajectory can be similarly calculated as the difference between the descriptor coordinates since the last update, divided by the time T elapsed since the last update. U .

[0112] .

[0113] Then, the difference in trends can be calculated as

[0114] .

[0115] Then, the weights assigned to the particles can be calculated as t. D The norm, for example: , where m T It is a variable that controls how the probability decreases as the difference in descriptor trend increases.

[0116] iii) The time history of previously used particles

[0117] The history of previously rendered particles is preserved within a certain time window to prevent them from being repeated and to reduce the probability of selecting a particle if it has already been rendered. This reduction in probability is related to the current time point. and particles The last point in time that was rendered The difference between them is a function proportional to the difference between them.

[0118] .

[0119] If particles If a time point has not been selected in the past or within a certain time window, the value of the last time point is set to zero. The function f can be linear, quadratic, logarithmic, or have an independent variable with non-negative output. Any monotonically increasing function.

[0120] iv) Difference in time points in the original records

[0121] For some sound sources, the way sound evolves may not be entirely described by changes in descriptors. Sometimes, sound evolves in a way that depends on what has happened previously. For example, the sharp sound of an old door might have a slightly different sharp sound each time it is opened, even if it is opened at the same speed, etc. In these cases, particles with similar descriptor values ​​and descriptor trends can sound very different, and combining them can lead to unnatural discontinuities not present in the original record. A way to avoid these discontinuities is to assign a higher probability to particles from the same part of the original record that are the same as previously used particles. Particles from the same part of the original record are expected to be closely related in properties and similar to each other, and therefore are good candidates when selecting the next particle.

[0122] To measure how close one particle from a particular record is to another particle from the same record, a time difference can be calculated, corresponding to the difference in time points between the records from which the two particles were extracted. If the particles are close, the time difference is small. To calculate the time difference between two particles, metadata can be used, which tells each particle which original record it was extracted from and at what time. Therefore, this metadata provides a measure of the closeness between the two particles. This metadata can be specified in a compact way as two values: a record identifier assigned to the original record and a timestamp identifying the time point in that record where the particle can be found.

[0123] Then, the weights for particles can be calculated based on different timestamps in this way: smaller differences give a higher probability of selecting the particle being evaluated. For example, the weight p for particle i... i It can be calculated as:

[0124] ,

[0125] Among them, R iR0 is the record identifier assigned to the record from which particle i is extracted, and R0 is the record identifier assigned to the record from which previously rendered particles are extracted. i t0 and t0 are the corresponding timestamps of the two particles, and b is a design constant that sets the probability of using a particle from the other record. The function f() takes the difference between the time points as input and calculates the probability of the particle. In one embodiment, the function f decreases linearly as the difference between the time points increases, where the slope is specified by the variable a:

[0126] .

[0127] This has the following effect: as the difference between the corresponding timestamps increases, the probability decreases from 1.0 to b, but never below b.

[0128] In one embodiment, the record identifier R i It can also be set to refer to segments of a record; that is, a record can be divided into segments, where each segment has its own index. This can be useful when a record contains segments that will not be considered relevant by the renderer.

[0129] Final weight

[0130] The final weight assigned to particle i (denoted as pi) is calculated by accumulating the weights assigned to particle i from each stage, i.e., In some embodiments, not all five weights are required, and they can be skipped by setting the corresponding weight to 1.0 or by excluding it from the calculation altogether.

[0131] In one embodiment, by modifying the individual weights using a fractional index, different weights can be given different degrees of influence on the final weight, such as, for example...

[0132] ,

[0133] Among them, by using the fractional exponent ½, the weight p from the first stage is made... i1 The impact is relatively small, and the weight p is made easier by using a fractional exponent of 1 / 4. i4 The impact is smaller.

[0134] The weights were calculated for all particles in the candidate particle set according to the steps described above. They are then normalized, that is, each of them is divided by the sum, so that the final weights become probability values.

[0135] , where k is the size of the candidate particle set.

[0136] Then, particles are selected based on their final weights (e.g., by sampling from a distribution). Note that while the target descriptor coordinates remain constant, the particle's temporal history and descriptor trend affect its final weight at each time point the particle is rendered. Therefore, as long as the target descriptor remains constant, if particle i is selected at time t, then at time t+h, the particle's final weight is influenced by its temporal history and descriptor trend. The final weight of particle i becomes...

[0137] , among which, middle, .

[0138] In order to avoid Repeating particle i can Defined as

[0139] .

[0140] However, due to normalization operations, other The probability of each particle also changes.

[0141] Figure 5 This is a flowchart illustrating a process 500 for rendering audio corresponding to an audio recording according to an embodiment, wherein the audio recording is divided into multiple particles. Process 500 may begin at step s502.

[0142] Step s502 includes: obtaining the coordinates of a first target descriptor, wherein the coordinates of the first target descriptor identify a first position in an N-dimensional descriptor space, where N > 0.

[0143] Step s504 includes: defining a first candidate particle set based on the coordinates of the first target descriptor, wherein the first candidate particle set includes k1 particles from a plurality of particles, where k1 > 1.

[0144] Step s506 includes: assigning a final weight to each particle in the first candidate particle set.

[0145] Step s508 includes: randomly selecting a particle from the first candidate particle set based on the assigned final weight, such that the probability of a given particle in the first candidate particle set being selected is a function of the final weight assigned to that given particle.

[0146] Step s510 includes: rendering the selected particles.

[0147] In some embodiments, each of the plurality of particles is associated with particle coordinates (e.g., a set of one or more descriptor values) that identify the position of a particle in an N-dimensional descriptor space, and the first candidate particle set is defined based on the first target descriptor coordinates and the particle coordinates.

[0148] In some embodiments, defining a first candidate particle set based on the coordinates of a first target descriptor and the coordinates of a particle includes: determining a nearest neighbor particle set consisting of k1 particles from a plurality of particles, wherein none of the plurality of particles not included in the nearest neighbor particle set is closer to the first target descriptor coordinates than any of the particles included in the nearest neighbor particle set, and the candidate set of candidate particles consists of particles included in the nearest neighbor particle set.

[0149] In some embodiments, the method further includes determining k1 based on the number of particles among a plurality of particles having particle coordinates within a threshold distance of the first target descriptor coordinates.

[0150] In some embodiments, each particle included in a plurality of particles is assigned an optimal k value, and the method includes: setting k1 equal to the optimal k value assigned to the particle among the plurality of particles that has the particle coordinates closest to the target descriptor coordinates.

[0151] In some embodiments, defining a first candidate particle set based on the coordinates of a first target descriptor and the coordinates of a particle includes: determining a first radius value r1; and including each particle in the first candidate particle set having particle coordinates within a distance r1 from the coordinates of the first target descriptor.

[0152] In some embodiments, the method further includes: determining r1 based on the number of particles among a plurality of particles having particle coordinates within a threshold distance of the first target descriptor coordinates.

[0153] In some embodiments, assigning a final weight to each particle in the first candidate particle set includes: assigning a first weight to a first particle included in the candidate particle set; determining a first final weight based on the first weight; and assigning the first final weight to the first particle.

[0154] In some embodiments, the first weight is a function of the distance between the coordinates of the first target descriptor and the first particle, a function of the amount of time that has elapsed since the first particle was last rendered, a function of the trajectory associated with the first particle and the target trajectory, or a function of a measure of proximity between the first particle and the most recently rendered particle (e.g., a time difference indicating the difference between the timestamp used for the first particle and the timestamp used for the most recently rendered particle, assuming that the first particle and the most recently rendered particle were extracted from the same record or the same segment).

[0155] In some embodiments, the method further includes: after randomly selecting a particle from a first candidate particle set based on the assigned final weight, assigning a new final weight to at least one particle in the first candidate particle set, or removing a rendered particle from the first candidate particle set; after assigning a new final weight to the rendered particle or removing a rendered particle from the first candidate particle set, randomly selecting another particle from the first candidate particle set based on the currently assigned final weight; and rendering the selected other particle.

[0156] In some embodiments, the method further includes: obtaining second target descriptor coordinates after randomly selecting particles from a first candidate particle set; defining a second candidate particle set based on the second target descriptor coordinates, wherein the second candidate particle set includes k2 particles from a plurality of particles, where k2 > 1; assigning a final weight to each particle in the second candidate particle set; randomly selecting particles from the second candidate particle set based on the assigned final weights, such that the probability of a given particle in the second candidate particle set being selected is a function of the final weight assigned to the given particle; and rendering the particles randomly selected from the second candidate particle set.

[0157] Figure 6 This is a flowchart illustrating a process 600 for rendering audio corresponding to an audio recording according to an embodiment, wherein the audio recording is divided into multiple particles. Process 600 may begin at step s602.

[0158] Step s602 includes obtaining the coordinates of a first target descriptor, wherein the coordinates of the first target descriptor identify a first position in an N-dimensional descriptor space, where N > 0.

[0159] Step s604 includes: defining a first candidate particle set based on the coordinates of the first target descriptor, wherein the first candidate particle set includes k1 particles out of a plurality of particles, where k1 > 1, and the first candidate particle set includes a first particle and a second particle.

[0160] Step s606 includes: assigning a final weight to each particle in the first candidate particle set, wherein assigning a final weight to each particle in the first candidate particle set includes: assigning a first final weight to the first particle and assigning a second final weight to the second particle. The first final weight assigned to the first particle is: a function of the distance between the first target descriptor coordinates and the first particle, a function of the amount of time that has elapsed since the first particle was last rendered, a function of the trajectory associated with the first particle and the target trajectory, and / or a function of a measure of the proximity between the first particle and the most recently rendered particle.

[0161] Step s608 includes: selecting particles from the first candidate particle set based on the assigned final weights.

[0162] Step s610 includes: rendering the selected particles.

[0163] In some embodiments, each of the plurality of particles is associated with particle coordinates (e.g., a set of one or more descriptor values) that identify the position of a particle in an N-dimensional descriptor space, and the first candidate particle set is defined based on the first target descriptor coordinates and the particle coordinates.

[0164] In some embodiments, defining a first candidate particle set based on the coordinates of a first target descriptor and the coordinates of a particle includes: determining a nearest neighbor particle set consisting of k1 particles from a plurality of particles, wherein none of the plurality of particles not included in the nearest neighbor particle set is closer to the first target descriptor coordinates than any of the particles included in the nearest neighbor particle set, and the candidate set of candidate particles consists of particles included in the nearest neighbor particle set.

[0165] In some embodiments, the method further includes determining k1 based on the number of particles among a plurality of particles having particle coordinates within a threshold distance of the first target descriptor coordinates.

[0166] In some embodiments, each particle included in a plurality of particles is assigned an optimal k value, and the method includes: setting k1 equal to the optimal k value assigned to the particle among the plurality of particles that has the particle coordinates closest to the target descriptor coordinates.

[0167] In some embodiments, defining a first candidate particle set based on the coordinates of a first target descriptor and the coordinates of a particle includes: determining a first radius value r1; and including each particle in the first candidate particle set having particle coordinates within a distance r1 from the coordinates of the first target descriptor.

[0168] In some embodiments, the method further includes: determining r1 based on the number of particles among a plurality of particles having particle coordinates within a threshold distance of the first target descriptor coordinates.

[0169] In some embodiments, the method further includes: after selecting a particle from a first candidate particle set based on an assigned final weight, assigning a new final weight to at least one particle in the first candidate particle set, or removing a rendered particle from the first candidate particle set; after assigning a new final weight to the rendered particle or removing a rendered particle from the first candidate particle set, selecting another particle from the first candidate particle set based on the currently assigned final weight; and rendering the selected other particle.

[0170] In some embodiments, the method further includes: after selecting particles from a first candidate particle set, obtaining second target descriptor coordinates; defining a second candidate particle set based on the second target descriptor coordinates, wherein the second candidate particle set includes k2 particles from a plurality of particles, where k2 > 1; assigning a final weight to each particle in the second candidate particle set; selecting particles from the second candidate particle set based on the assigned final weights; and rendering particles randomly selected from the second candidate particle set.

[0171] Particle interpolation

[0172] As described above, at rendering time, particle scheduling (i.e., selecting particles to render) is based on the target descriptor coordinates in the descriptor space. When GDB 104 has many particles uniformly filling the entire descriptor space, particle selection can be performed without frequently repeating the same particles and without using particles far from the target location. However, in some cases, GBD 104 may have too few particles or not enough particles uniformly filling the descriptor space. In these cases, particle selection becomes more limited. For some target locations in the descriptor space, there may be no nearby particles, or very few nearby particles, making them need to be repeated frequently.

[0173] If the particle scheduler cannot find a particle sufficiently close to the target location, or if all particles close to the target location have recently been used, a new particle can be created; this new particle is called an "interpolation particle." The method for creating an interpolation particle involves selecting multiple particles (referred to as a "set of particles") and calculating the interpolation particle based on the selected set of particles. It is important to select particles such that the set of particles orbits the target location. In one embodiment, the particles in the set are close to the target location, but not all particles are on the same side of the target.

[0174] Figure 9 A 2D descriptor space is shown, where particles in GDB 104 do not uniformly cover the descriptor space. (See diagram.) Figure 9 As shown, three particle clusters A, B, and C surround a target location indicated by X. If particle 901 in cluster B is selected as the first particle in a set of particles to be used for interpolation, the next particle selected is complementary to particle 901 relative to the target location.

[0175] In one embodiment, particle i is complementary to particle j relative to the target position if and only if, for each particle descriptor coordinate dimension in the particle database, particles i and j are on opposite sides of the target position. Using this definition, particle i is complementary to particle j relative to the target position if Gix is ​​less than Tx and Giy is greater than Ty, where Gix is ​​the x-coordinate of particle i, Giy is the y-coordinate of particle i, Tx is the x-coordinate of the target position, and Ty is the y-coordinate of the target position.

[0176] More generally, assuming that particle j is closer to the target position (Tx, Ty) than particle i, then relative to the 2D descriptor space, particle i and particle j are complementary under the following conditions:

[0177] If Gjx == Tx and Gjy < Ty, then if Giy > Ty, then particle i is complementary to particle j, or

[0178] If Gjx == Tx and Gjy > Ty, then if Giy < Ty, particle i and particle j are complementary, or

[0179] If Gjy == Ty and Gjx < Tx, then if Gix > Tx, particle i is complementary to particle j, or

[0180] If Gjy == Ty and Gjx > Tx, then if Gix < Tx, then particle i is complementary to particle j, or

[0181] If Gjx < Tx and Gjy < Ty, then if Gix ≥ Tx and Giy ≥ Ty, then particle i is complementary to particle j, or

[0182] If Gjx < Tx and Gjy > Ty, then if Gix ≥ Tx and Giy ≤ Ty, then particle i is complementary to particle j, or

[0183] If Gjx > Tx and Gjy < Ty, then if Gix ≤ Tx and Giy ≥ Ty, then particle i is complementary to particle j, or

[0184] If Gjx > Tx and Gjy > Ty, then if Gix ≤ Tx and Giy ≤ Ty, then particle i is complementary to particle j.

[0185] Particle 902 in cluster A is an example of a particle complementary to particle 901 relative to its target position. This is in Figure 9The following fact indicates that particle 902 is on the opposite side of the target position in both dimensions compared to particle 901. Adding particle 902 to this group of particles is ideal because it is not only complementary to particle 901 but also close to the target position. Particles from cluster C are all on the same side as particle 901 in the y-dimensional direction, and particles in cluster B are all on the same side as particle 901 in the x-dimensional direction; therefore, no particle in cluster B or C is complementary to particle 901 relative to the target position. However, any particle from cluster A satisfies the condition for being complementary to particle 901.

[0186] In one embodiment, the following steps are performed:

[0187] Step 1: Obtain the target location in the database descriptor space, and define the candidate particle set based on the target location using methods such as those described above.

[0188] Step 2: Use weighted selection (such as, for example, weighted random selection) to select the first particle from the candidate particle set. If the selected particle is within a threshold distance of the target location, render the particle without any interpolation; otherwise, continue to steps 3-7.

[0189] Step 3 (optional): Remove the first particle from the candidate particle set.

[0190] Step 4: For each particle in the candidate particle set, assign a final weight to the candidate particle, where the final weight is a function of whether the particle is complementary to the first particle relative to the target position. All other things being equal, candidate particles complementary to the first particle will have a higher final weight than candidate particles not complementary to the first particle. If the first particle is not removed from the candidate particle set, it is assigned a very low final weight (e.g., a weight of 0.00001).

[0191] In one embodiment, if there are K particles in the candidate particle set, then for i=1 to K, the final nonnormalized weight assigned to particle i in the candidate particle set can be calculated as: p i = p i1 p i2 p i3 p i4 p i5 p iCOMP , where p iCOMP It depends on whether particle i is complementary to the first particle. As an example, p COMP It can be set as follows: if particle i is complementary, then p iCOMP = X, otherwise, p iCOMPX is set to Y, where X > Y. Preferably, X is at least an order of magnitude larger than Y. For example, in one embodiment, X = 1 and Y = 0.0001. In some scenarios, where there may not be many particles to choose from in a sparse particle database (where there may not be many particles near the target location), it may be beneficial not to completely exclude non-complementary particles. Therefore, for this reason, Y is generally not set to equal 0.

[0192] Step 5: Select at least a second particle from the candidate particle set based on the assigned final weights.

[0193] Step 6: Set the length of the new interpolated particles. For example, if the sound in the database has pitch, the length is calculated as a weighted average of the lengths of the selected particles, where the weights depend on the distance from each particle to the target location. If the sound in the database has no pitch, the length is set based on the shortest particle among the selected particles.

[0194] Step 7: Use the selected particles to generate interpolated particles. For example, the interpolated particles can be a weighted mixture of the selected particles, where the weights depend on the distance from each selected particle to the target location. If the sounds in the database have pitch, each selected particle can be resampled to fit the length of the interpolated particles. If the sounds in the database do not have pitch, only the sub-parts of the selected particles corresponding to the length of the interpolated particles are used.

[0195] In the simplest case, the two selected particles closest to the target descriptor coordinates. and It was used to generate interpolation particles.

[0196] It is generally good to use as few particles as possible for interpolation, as mixing many particles together can result in scattered sound that is dissimilar to the original sound. Therefore, in one embodiment, only the first and second selected particles are used to derive the interpolation particles.

[0197] However, if more than two particles are specifically selected from a multidimensional particle database for interpolation, the condition for complementarity in a certain dimension can be relaxed for the selection of the third particle. Another criterion could be ensuring paired complementarity for subsequently selected particles. For example, in a 3D database, particles A and B satisfy the complementarity criterion in all dimensions, and particles B and C are complementary in all dimensions; however, particles A and C could be complementary only in the first two dimensions, and not in the third dimension.

[0198] Determine the length of the interpolation particles

[0199] In step 6, the length of the interpolated particles is set, referred to as the "target length." For weighted mixing of particles to occur, the particles need to have a matching length. The target particle length can be determined in different ways depending on the characteristics of the sound described by the particle database.

[0200] For some particle databases, the length of each particle corresponds to one or a specific number of tone cycles. In this case, the target length can be calculated as a weighted average of the lengths of the selected particles. For cases where only two particles are used to generate the interpolated particles, the target length (L...) Target ) can be calculated as: L Target = w1(L1) + w2(L2), where L1 and L2 are the lengths of the two selected particles, and w1 and w2 are weights calculated as inversely proportional to the distance of each selected particle from the target position: w1 = d2 / (d1+d2) and w2 = d1 / (d1+d2).

[0201] Distances d1 and d2 are Euclidean distances in the descriptor space. In cases where only one dimension of the particle database affects the pitch of the sound and thus the length of the particles, weights can be calculated using a distance metric, where only distances in that dimension are considered.

[0202] For other particle databases where particle lengths do not indicate pitch, the target length is less critical. A straightforward approach is to select a target length that matches the length of the shortest particle among those previously selected.

[0203] Create temporary particles for weighted mixing

[0204] Before the weighted mixing can be calculated to form the interpolated particles, a temporary version of the selected particles with the selected target particle length is created.

[0205] When a particle database represents a tone sound, the selected particles can be resampled using a resampling method that allows arbitrary resampling rates, such as linear resampling, sinc interpolation, or Lanczos resampling, or similar well-known techniques.

[0206] If the particle database does not represent pitch sounds, a sub-part of the selected particles that matches the length of the target particle can be used. This can be done by using the first sub-part of each selected particle, or by using a sub-part within each particle with a randomly selected starting point, where the randomly selected starting point is constrained such that the remaining samples of the particles correspond at least to the length of the target particle.

[0207] Interpolation particles are generated using weighted mixing.

[0208] In one embodiment, the interpolating particle is a weighted mixture of selected particles (assuming the selected particles have the same length), or a weighted mixture of a selected particle and a temporary particle derived from another selected particle (assuming the length of the selected particle is equal to the target length), or a weighted mixture of temporary particles (assuming no particle has a length equal to the target length). Therefore, the interpolating particle is equal to: w1(g1) + w2(g2), or w1(tg1) + w2(g2), or w1(g1) + w2(tg2), or w1(tg1) + w2(tg2), where tg1 is a first temporary particle derived from the first particle, and tg2 is a second temporary particle derived from the second particle. In one embodiment, the weights can be calculated in a similar manner to that used when selecting the target particle length, as being inversely proportional to the distance from the selected particle to the target position.

[0209] For example, if two particles g1 and g2 are selected and the distances of the selected particles from the target position are d1 and d2, respectively, then in one embodiment, the weights used in performing the weighted mixing are calculated as: w1 = (d2 / (d1+d2)) and w2 = (d1 / (d1+d2)). When consecutive particles are expected to be almost uncorrelated, the weights can be calculated according to a constant power mixing rule instead of a linear mixing rule. In this case, the weights can be calculated as w1 = sqrt(d2 / (d1+d2)) and w2 = sqrt(d1 / (d1+d2)). Other ways of creating weights (also called gains) are described below in the section on determining the optimal mixing window coefficients.

[0210] Cache of interpolated particles

[0211] After interpolated particles have been generated, they can be stored for later use. This reduces the complexity of calculating new interpolated particles later. In this case, like other particles in the database, the interpolated particles should be assigned indexes and / or descriptor coordinates so that the renderer can avoid reproducing the same interpolated particles. While caching interpolated particles reduces renderer complexity, caching increases memory consumption; therefore, limiting the number of cached particles may be beneficial. If desired, cached interpolated particles can also be written to a granularity database, making the proposed interpolation technique another method for corpus expansion.

[0212] Figure 10This is a flowchart illustrating a process 1000 for rendering audio corresponding to an audio recording according to an embodiment. The audio recording is divided into multiple particles. Process 1000 may begin at step s1002. Step s1002 includes: obtaining target descriptor coordinates, wherein the target descriptor coordinates identify the target position in an N-dimensional descriptor space, where N > 0. Step s1004 includes: selecting a set of particles from a particle database based on the target descriptor coordinates, wherein the selected set of particles includes a first particle and a second particle. Step s1006 includes: determining a first weight w1 for the first particle. Step s1008 includes: determining a second weight w2 for the second particle. Step s1010 includes: generating interpolated particles using the first weight, the second weight, the first particle, and the second particle. Step s1012 includes: rendering the interpolated particles.

[0213] In some embodiments, the second particle has a position in N-dimensional space, and the first particle is located in N-dimensional space at a complementary position to the position of the second particle in N-dimensional space.

[0214] In some embodiments, the interpolating particle is equal to: w1(g1) + w2(g2), or w1(tg1) + w2(g2), or w1(g1) + w2(tg2), or w1(tg1) + w2(tg2), where g1 is the first particle, g2 is the second particle, tg1 is the first temporary particle derived from the first particle, and tg2 is the second temporary particle derived from the second particle.

[0215] In some embodiments, the first particle has a length, and the second particle has a length, and generating the interpolated particle includes: determining the length of a target particle using a first length value specifying the length of the first particle and a second length value specifying the length of the second particle, wherein the length of the first particle is not equal to the length of the target particle; deriving a first temporary particle from the first particle having a length equal to the length of the target particle; and generating the interpolated particle using the first temporary particle, the second particle, a first weight, and a second weight.

[0216] In some embodiments, the first particle has a length, and the second particle has a length, and generating the interpolated particle includes: determining the length of a target particle using a first length value specifying the length of the first particle and a second length value specifying the length of the second particle, wherein the length of the first particle is not equal to the length of the target particle, and the length of the second particle is not equal to the length of the target particle; deriving a first temporary particle from the first particle having a length equal to the length of the target particle; deriving a second temporary particle from the second particle having a length equal to the length of the target particle; and generating the interpolated particle using the first temporary particle, the second temporary particle, a first weight, and a second weight.

[0217] In some embodiments, deriving a first temporary particle from a first particle includes: resampling the first particle to generate a first temporary particle, or selecting a sub-part of the first particle, wherein the first temporary particle is the selected sub-part.

[0218] In some embodiments, determining the target particle length using a first length value and a second length value includes: determining the target particle length using a first length value, a second length value, a first weight, and a second weight.

[0219] In some embodiments, the target particle length is equal to w1(L1) + w2(L2), where L1 is a first length value and L2 is a second length value.

[0220] In some embodiments, a first particle has a position in N-dimensional space, and a second particle has a position in N-dimensional space, a first weight is based on the distance from the position of the first particle in N-dimensional space to the target position in N-dimensional space (e.g., inversely proportional), and a second weight is based on the distance from the position of the second particle in N-dimensional space to the target position in N-dimensional space (e.g., inversely proportional).

[0221] In some embodiments, the first weight is equal to d2 / (d1 + d2), the second weight is equal to d1 / (d1 + d2), d1 is the distance from the position of the first particle in N-dimensional space to the target position in N-dimensional space, and d2 is the distance from the position of the second particle in N-dimensional space to the target position in N-dimensional space.

[0222] In some embodiments, selecting a set of particles from a particle database includes: defining a first candidate particle set based on a target position; selecting a first particle from the first candidate particle set; after selecting the first particle from the first candidate particle set, removing the first particle from the first particle set to form a second particle set; assigning a final weight to each particle included in the second particle set; and selecting a particle from the second candidate particle set based on the assigned final weight, wherein the particle selected from the second particle set is a second particle.

[0223] In some embodiments, selecting a set of particles from a particle database includes: defining a candidate particle set based on a target position; selecting a first particle from the candidate particle set; assigning a final weight to each particle included in the candidate particle set; and, after assigning the final weight to each particle included in the candidate particle set, selecting a particle from the candidate particle set based on the assigned final weight, wherein the particle selected from the second particle set is a second particle.

[0224] In some embodiments, assigning a final weight to the second particle includes: determining whether the second particle is complementary to the first particle relative to the target position; and assigning a complementary weight p to the second particle.COMP Where, if the second particle is complementary to the first particle relative to the target position, then p COMP The value of p is X; otherwise, p COMP The value of X is Y, where X is greater than Y; and the final weights for the second particle are calculated using the complementary weights assigned to the second particle. In some embodiments, X is at least ten times Y.

[0225] Determining the optimal mixing window coefficient.

[0226] When performing particle synthesis, the overlap-add technique can be used to provide a smooth transition from one particle to the next in a particle sequence. As previously described, it can be beneficial to adapt the choice of the overlap-add window to the characteristics of the signals in the particles to be mixed. If the signals are highly correlated, a linear mixing window should be used. However, if the signals are uncorrelated, a power-preserving mixing window should be used. In many cases, the optimal mixing window is somewhere between a linear window and a power-preserving window, because only a portion of the signal may be correlated.

[0227] To generate an optimized blended window (labeled W) O The mixing window coefficient (denoted as "m") (also known as the mixing window weight) can be used to provide continuous control over the mixing window from linear to power preservation. In one embodiment, the mixing window coefficient (m) is a value between 0.0 and 1.0, and W O = mW p + (1-m)W L Among them, W L It is a linearly overlapping window, and W P It is a power preservation overlapping window. Therefore, in this embodiment, the optimized hybrid window is a weighted average of the power preservation overlapping window and the linear overlapping window.

[0228] When m=1, only the power-preserving overlapping window will be used. When m=0, only the linear overlapping window will be used. For values ​​between 0 and 1, a blend of both windows will be used. Therefore, by properly tuning the blending window coefficients, an optimal blending window can be found.

[0229] In one embodiment, W is used for a pair of particles (the pair of particles to be mixed) undergoing overlapping addition processing. L W is a vector of scalar values ​​of X, i.e., W L = [w l [0], w l [1], ..., w l [X-1], where w l[x] = x / (X-1), X = min(L1, L2) × p, where L1 is the length of the first particle in the pair, L2 is the length of the second particle in the pair, and p is the overlap percentage. In one embodiment, W P It is also a vector of X scalar values, that is, W P = [w p [0],w p [1], ..., w p [X-1], where w p [x] = (w l [x]) 1 / 2 .

[0230] If the mixing window coefficient is m for both the first and second particles in the pair, then the weights g1 and g2 for the two particles at sample index n are g1[n] = m(w l [n]) 1 / 2 + (1-m)w l [n] and g2[n] = m(1-w l [n]) 1 / 2 + (1-m)(1-w l [n]).

[0231] X overlapping samples s[n] (for n=0 to X-1) will be a linear combination of X samples from two particles and weights g1 and g2, i.e., s[n] = g1[n]grain1[Sp1+n] + g2[n]grain2[Sp2+n] for n=0 to X-1, where grain1[] is the sample set including the first particle, grain2[] is the sample set including the second particle, and Sp1 and Sp2 are the starting positions for the first and second particles, respectively.

[0232] When using a particle database where particles have positions in the descriptor space, clusters of highly correlated particles and other uncorrelated particles may often exist. This can occur, for example, when the particle database describes a sound that is strong in one part of the descriptor space but more textured and noisy in another. By specifying the blending window coefficients for the different parts of the descriptor space, the most effective overlap window for the particles in that part can be selected.

[0233] Figure 11 A particle database with several particle clusters is shown. The mixing window used can be tuned for each particle cluster by specifying the mixing window coefficients at several positions in the descriptor space.

[0234] In one embodiment, a set of blending window coefficients is specified, where each coefficient is associated with a position in the descriptor space; that is, each coefficient is associated with a set of coordinates specifying the position in the descriptor space. The coefficients can be stored as metadata in a particle database. This allows optimized blending window coefficients to be pre-computed by performing correlation checks or other methods between particles to find an optimized blending window. Alternatively, the set of coefficients can be created manually, and tuned by the sound designer who created the particle database.

[0235] Using a list of blending window coefficients along with their descriptor spatial locations is a compact, efficient, and scalable way to specify blending windows for a database. In some cases, specifying only one blending window may be sufficient, while in others, specifying a large number of blending windows in different locations can be beneficial.

[0236] Below is an example of a data structure (table) for storing the mixing window coefficients used in a 3D particle database:

[0237]

[0238] Here, each entry (a row in the table) has a field that stores the coordinates of the location in the 3D descriptor space and an associated field that stores the blending window coefficients specified for that location. Other data structures (such as, for example, lists) can be used to store the coefficients and their corresponding coordinates.

[0239] At rendering time, the optimal (or final) blending window coefficients can be found for each particle based on its position in the descriptor space. In the simplest case, the coefficients closest to the particle in the descriptor space are selected from the list. Another embodiment can use a weighted sum of coefficients, where the weight of each coefficient is based on its distance from the particle; that is, the optimal blending window coefficients are: ,in, It is a function of the distance between the i-th coefficient in the list and the particle.

[0240] To efficiently retrieve the closest coefficients for a specific particle, the list of coefficients can be sorted, for example, by a KD-tree structure.

[0241] When mixing two particles (such as, for example, overlapping addition), coefficients for each of the two particles can be determined as described above, and then the optimal mixing window coefficient is based on these two determined coefficients. In one embodiment, the optimal mixing window coefficient is the larger of the two determined mixing window coefficients, i.e., m = max(m1, m2), where m1 is the coefficient determined for the first particle and m2 is the coefficient determined for the second particle. The logic behind this operation is that a higher value of the mixing window coefficient implies that the particles to be mixed may be less relevant.

[0242] For example, if one particle has a mixing window coefficient of 0.32 and another particle has a mixing window of 0.93, then m = 0.93 would be used. Even if the particle with a mixing window coefficient of 0.32 comes from a cluster of correlated particles, it will likely be less correlated with particles from clusters with low correlation. Therefore, it can be assumed that if particles have different mixing window values, these particles are almost uncorrelated. Thus, using the maximum mixing window coefficient is advantageous.

[0243] In one embodiment, the final mixing window coefficients are not based on the positions of the particles to be mixed, but rather on the current target position in the descriptor space. In this case, for example, a certain number of mixing window coefficients closest to the current target position are used as the basis for determining the optimized mixing window coefficients. This can be done by selecting the closest one or by performing some form of weighted mixing on a set of mixing window coefficients specified at points close to the target position.

[0244] Whenever a mixture of two or more particles is to be performed, the optimal mixing window coefficient can be used. In the case of overlapping addition, the mixing occurs within the overlapping region.

[0245] When mixing particles together, for example to form interpolated particles, the optimal mixing window coefficient (m) can also be used, as described above. As an example, the samples s of interpolated particles... int [] can be:

[0246] s int [n] = g1×grain1[n] + g2×grain2[n], for n=0 to L-1, where,

[0247] L is the length of the particle (in this embodiment, the particles have equal lengths).

[0248] grain1[] is a sample set including the first particle.

[0249] grain2[] is a sample set that includes the second particle.

[0250] g1 = m (w1) 1 / 2 + (1-m)w1; and

[0251] g2 = m (w2) 1 / 2 + (1-m)w2.

[0252] In one embodiment, w1 = d2 / (d1+d2) and w2 = d1 / (d1+d2), where d1 is the Euclidean distance between the first particle and the target position, and d2 is the Euclidean distance between the second particle and the target position. However, where only one dimension of the particle database affects the pitch of the sound and therefore the length of the particles, the weights w1 and w2 can be calculated using a distance metric, where only the distance in that dimension is considered.

[0253] Figure 12 This is a flowchart illustrating a process 1200 for rendering audio corresponding to an audio recording according to an embodiment. The audio recording is divided into multiple particles. Process 1200 may begin at step s1202.

[0254] Step s1202 includes: selecting a first particle from a plurality of particles, wherein the first particle is associated with a first position in an N-dimensional (ND) descriptor space.

[0255] Step s1204 includes: selecting a first set of one or more blending window coefficients based on a first position in the ND descriptor space.

[0256] Step s1206 includes: selecting a second particle from a plurality of particles, wherein the second particle is associated with a second position in an N-dimensional descriptor space.

[0257] Step s1208 includes: selecting a second set of one or more blending window coefficients based on a second position in the ND descriptor space.

[0258] Step s1210 includes: using a first set of mixed window coefficients and a second set of mixed window coefficients to obtain the final mixed window coefficients m.

[0259] Step s1212 generates a mixed sample S by mixing at least a portion of the first particle with at least a portion of the second particle using the final mixing window coefficient.

[0260] In some embodiments, the first particle includes a first sample set, the second particle includes a second sample set, and generating a mixed sample using the final mixing window coefficients includes generating a first mixed sample s[0] by calculating s = g1 × grain1_sample + g2 xgrain2_sample, where g1 is a function of the final mixing window coefficients, grain1_sample is a sample from the first sample set, g2 is a function of the final mixing window coefficients, and grain2_sample is a sample from the second sample set.

[0261] In some embodiments, g1 is another function of a first weight w1 associated with a first blending window and a second weight w2 associated with a second blending window, and g2 is another function of a third weight w3 associated with the first blending window and a fourth weight w4 associated with the second blending window.

[0262] In some embodiments, g1 = m × w1 + (1-m) × w2, and g2 = m × w3 + (1-m) × w4.

[0263] In some embodiments, w1 = (w2) 1 / 2 And w3 = (w4) 1 / 2 .

[0264] In some embodiments, w4 = 1 - w2.

[0265] In some embodiments, w2 = n / (L-1), n ​​≥ 0 and n ≤ (L-1), L = min(L1,L2) × p, L1 is the length of the first particle, L2 is the length of the second particle, and p is a predetermined overlap percentage.

[0266] In some embodiments, w2 = d1 / (d1+d2), w4 = d2 / (d1+d2), where d1 is the distance from the first position in the ND descriptor space to the target position in the descriptor space, and d2 is the distance from the second position in the ND descriptor space to the target position in the descriptor space.

[0267] In some embodiments, the first set of mixed window coefficients consists of a first mixed window coefficient m1, and obtaining the final mixed window coefficients includes: setting m equal to max(m1,m2), where m2 is a mixed window coefficient included in the second set of mixed window coefficients, or based on a mixed window coefficient included in the second set of mixed window coefficients.

[0268] In some embodiments, obtaining the final blended window coefficients includes: assigning weights to each blended window coefficient included in a first set of blended window coefficients; using the weights assigned to each blended window coefficient included in the first set of blended window coefficients, obtaining a weighted average of the blended window coefficients included in the first set of blended window coefficients; and setting m equal to max(m1, m2), where m1 is the weighted average of the blended window coefficients included in the first set of blended window coefficients, and m2 is a blended window coefficient included in a second set of blended window coefficients or based on the blended window coefficients included in the second set of blended window coefficients.

[0269] Figure 13 This is a flowchart illustrating a process 1300 for rendering audio corresponding to an audio recording according to an embodiment. The audio recording is divided into multiple particles. Process 1300 may begin at step s1302.

[0270] Step s1302 includes: selecting a first particle from a plurality of particles.

[0271] Step s1304 includes: selecting a second particle from a plurality of particles.

[0272] Step s1306 includes: i) selecting a set of two or more blending window coefficients based on the target location in the N-dimensional (ND) descriptor space, or ii) selecting a single blending window coefficient based on the target location in the ND descriptor space.

[0273] Step s1308 includes: generating a mixed sample S by mixing at least a portion of the first particle with at least a portion of the second particle using i) a single mixing window coefficient or ii) a derived mixing window coefficient derived using a set of two or more mixing window coefficients.

[0274] In some embodiments, the first particle includes a first sample set, the second particle includes a second sample set, and generating a mixed sample using the final mixing window coefficient includes: generating a first mixed sample s[0] by calculating s = g1 × grain1_sample + g2 xgrain2_sample, where g1 is a function of the final mixing window coefficient, grain1_sample is a sample from the first sample set, g2 is a function of the final mixing window coefficient, and grain2_sample is a sample from the second sample set.

[0275] In some embodiments, g1 is another function of a first weight w1 associated with a first blending window and a second weight w2 associated with a second blending window, and g2 is another function of a third weight w3 associated with the first blending window and a fourth weight w4 associated with the second blending window.

[0276] In some embodiments, g1 = m × w1 + (1-m) × w2, and g2 = m × w3 + (1-m) × w4, where m is a single blending window coefficient or a derived blending window coefficient.

[0277] In some embodiments, w1 = (w2) 1 / 2 And w3 = (w4) 1 / 2 .

[0278] In some embodiments, w4 = 1 - w2.

[0279] In some embodiments, w2 = n / (L-1), n ​​≥ 0 and n ≤ (L-1), L = min(L1,L2) × p, L1 is the length of the first particle, L2 is the length of the second particle, and p is a predetermined overlap percentage.

[0280] In some embodiments, w2 = d1 / (d1+d2), w4 = d2 / (d1+d2), where d1 is the distance from the first position in the ND descriptor space to the target position in the descriptor space, and d2 is the distance from the second position in the ND descriptor space to the target position in the descriptor space.

[0281] Fast particle switching

[0282] The above describes a method for scheduling particles from a particle database, where each particle has a designated position in a descriptor space. The target position in the descriptor space controls which particle should be selected next. As the target position is updated in real time, particles closer to the target position are selected. This allows control over how the sound evolves dynamically. Figure 14A A simplified example of this feature is shown in the figure.

[0283] Figure 14A This demonstrates how consecutive particles can be selected after the target position has been changed. Figure 14A In the case shown, the particle database is only one-dimensional, namely descriptor 1. On the x-axis, the timeline shows when particles are selected and how long each particle is used to generate audio output, i.e., how long each particle is rendered. Figure 14A The dashed lines in the diagram illustrate how the target's position changes over time. Figure 14A In this process, each complete particle is rendered before transitioning to the next particle.

[0284] However, typically, samples from the currently selected particle are played until the playback position reaches a predetermined sample position, which is usually near the end of the particle's sample position. When the playback position reaches the predetermined sample position, the next particle is selected, and a crossfade in / out (e.g., overlap / addition) is initiated between the two particles.

[0285] In some cases, the particle database needs to contain a considerable number of particles to preserve the natural sound of the original sound source. In such cases, particle selection may not be frequent enough to allow for fast and smooth transitions. For example, if the target position is moving rapidly through the descriptor space, the output of particle synthesis may have large jumps due to the fact that the target position has already moved a considerable distance in the descriptor space within the time it takes to play a single particle. In this situation, particle switching is too slow to keep up with the dynamic changes in the target position.

[0286] At the same time, using a forced higher particle switching rate may produce an unnatural output. If the particle database is designed with long particles, the sound designer intends the particles to be long, and using a forced particle switching rate will not allow the full particles to be played as intended.

[0287] Therefore, this disclosure proposes frequently evaluating the amount the target position has moved since the last particle was selected. The criterion is then evaluated to determine whether a fast particle switch should be initiated. In this way, longer particles will play out completely as long as the target position doesn't move too much. If the target position does move quickly, a fast particle switch can be triggered, and it can have fast dynamic behavior. Figure 14B This feature is shown in the image.

[0288] like Figure 14B As shown, a new particle can be selected, and the user can switch to that new particle before the end of the currently used particle has been reached. This allows for smoother sound transitions that more closely follow the target position without large jumps. Figure 14B To keep the illustration simple, overlapping areas are not shown.

[0289] Therefore, in one embodiment, a method is provided, comprising: obtaining an updated target position in a descriptor space, using the target position to determine a distance, for example, determining the distance the target position has moved since a previous time point, and determining, based on the determined distance, whether a fast particle switching condition is met.

[0290] In one embodiment, as a result of determining that the fast particle switching conditions are met, the renderer immediately transitions to the new particle. For example, the new particle is selected based on an updated target position, and overlap addition is initiated to transition to that new particle. As another example, the new particle is selected based on an updated target position, the renderer stops playing the current particle, and the renderer immediately begins playing the new particle.

[0291] Standards for triggering fast particle switching

[0292] In one embodiment, determining whether a fast particle switching condition is met includes determining the amount by which the target position has moved since the last particle was selected. For example, in one embodiment, the determination is made of the distance the target position has moved since the last particle was selected, and if the determined distance is greater than a threshold, a fast switching is triggered, i.e., the fast particle switching condition is met.

[0293] In another embodiment, determining whether a fast particle switching condition is met includes determining how well the currently used particle(s) match the updated location. For example, in one embodiment, if a single particle is currently being rendered, the updated target location is compared to the location of that single particle. If the distance between the target location and the particle is greater than a threshold distance, a fast particle switching is triggered. Alternatively, a fast switching is triggered if the distance has increased beyond the threshold since the current particle was selected.

[0294] As another example, when more than one particle is currently being rendered—for instance, if the renderer uses crossfade-in / fade-out with two or more particles—the updated target position can be compared to the positions of all currently used particles. For example, the distance from the updated target position to the nearest particle among those used can be calculated, and if that distance is greater than a threshold, a fast particle switch is triggered. Alternatively, a fast switch can be triggered if that distance has increased beyond a threshold since the current particle was selected.

[0295] Figure 15 This shows a scene where the two particles (particle 1501 and particle 1502) are used to generate audio output; that is, the two particles are rendered in a blended manner.

[0296] The distance (d2) between the updated target position 1512 and the line 1520 between the two used particles is calculated, and in one embodiment, a fast particle switching is triggered if the distance (d2) is greater than a predetermined value (also known as a threshold).

[0297] In another embodiment, to avoid continuously triggering fast particle switching in a sparse database where no particles are found near the target location, the change in distance since the current particle was selected can be compared to a threshold; that is, the value (d2 - d1) is compared to a threshold, where d1 is the distance to target location 1511 used when particles 1501 and 1502 were selected. In other words, how much the distance between the target location and the currently used particle has increased since the currently used particle was selected. If the increase in distance exceeds the threshold, fast particle switching can be triggered. If the increase in distance is less than the threshold, the currently selected particle may still be effective for the updated target location.

[0298] In another embodiment, d2 is compared to d1, and a fast particle switch is triggered if the distance has increased beyond a threshold (T); that is, if d2 > d1 + T, a fast particle switch is triggered. The value of T can be configured for each particle database. It can be written to the particle database, for example, as metadata. In one embodiment, the threshold is specified using a list of points in a descriptor space, where each point has its own specific threshold. This threshold can be calculated as a function of the inter-particle distances in the database.

[0299] If the database is organized by clusters, a list of thresholds is more suitable. For each cluster, its centroid can be selected as the point where the threshold is specified. The value of the threshold can be a function of the average distance between the particles and the centroid. The threshold can also be defined based on the distance between clusters.

[0300] Fast particle switching with phase compensation for tonal sounds

[0301] In particle database descriptions of sounds with prominent tones, where the length of each particle is proportional to the tone period, particle switching can lead to phase cancellation issues if a particle is not fully played before transitioning to the next particle using overlap addition. If the currently playing particle is positioned at 50% of its length and overlap addition with the next particle has been initiated, the two particles may be 180 degrees out of phase during the overlap addition period. Figure 16A As shown, this can lead to severe cancellation, where the two signals more or less cancel each other out.

[0302] like Figure 16A As shown, particle n is the currently rendered particle. At some point t, a rapid particle switch is triggered to this particle. In this case, the next particle selected to switch to (particle n+1) is out of phase with particle n in the overlapping region. This will cause severe waveform distortion during the overlap and will make a smooth transition impossible.

[0303] By tracking the current playback position and calculating the corresponding starting position in the next particle, phase can be maintained during particle switching and phase cancellation can be avoided, such as... Figure 16B As shown. More specifically, Figure 16B Phase compensation is shown to ensure that particle n+1 is in phase with particle n during overlap. Phase compensation is achieved by skipping the first portion of particle n+1, which corresponds to the amount of particle n used before the fast particle switch is triggered.

[0304] For example, in one embodiment, in response to the detection that the fast particle condition is met, the current playback position sp of the first particle is stored. For example, if, for example, when the condition is determined to be met, sample i of the currently rendered particle has been played, but sample i+1 has not yet been played, then sp is set to equal i+1. Then, the mixed sample set s[] is obtained as follows: s[n] = g1[n]grain_n[sp+n] + g2[n]grain_n+1[sp+n], for n = 0 to X-1, where grain_n[] is the sample set including particle n, grain_n+1[] is the sample set including particle n+1, and X is the length of the overlapping region.

[0305] Figure 17 This is a flowchart illustrating process 1700 according to an embodiment. Process 1700 may begin at step s1702. Step s1702 includes: selecting a first particle from a plurality of particles. Step s1704 includes: rendering at least a first portion of the first particle. Step s1706 includes: while the first particle is rendered, obtaining information indicating an updated target position in the ND descriptor space, and determining, based on the updated target position, whether a fast particle switching condition is met. Step s1708 includes: as a result of determining that the fast particle switching condition is met, transitioning from the first particle to a second particle.

[0306] In some embodiments, transitioning from a first particle to a second particle includes: stopping the rendering of the first particle; and rendering at least a portion of the second particle.

[0307] In some embodiments, the conversion from a first particle to a second particle includes: generating a blended sample by mixing at least a portion of the first particle with at least a first portion of the second particle; and rendering the blended sample.

[0308] In some embodiments, the method further includes rendering at least a second portion of the second particle after rendering the blended sample.

[0309] In some embodiments, the method further includes: obtaining information indicating a first target location in the ND descriptor space before selecting a first particle, wherein the first particle is selected from a plurality of particles based on the first target location; and determining the distance between the updated target location and the first target location, wherein determining whether a fast particle switching condition is met includes: comparing the distance with a predetermined value.

[0310] In some embodiments, the method further includes: determining the distance between the position of a first particle in the ND descriptor space and the updated target position, and determining whether the fast particle switching condition is met includes: comparing the distance with a predetermined value.

[0311] In some embodiments, rendering at least a portion of the first particle includes: generating a mixed sample using samples from the first particle and samples from the third particle, the first particle having a position in the ND descriptor space, the third particle having a position in the ND descriptor space, and determining whether the fast particle switching condition is satisfied is further based on the positions of the first particle and the third particle in the ND descriptor space.

[0312] In some embodiments, determining whether the fast particle switching condition is met includes: determining a first distance between the updated target position and the position of the first particle; determining a second distance between the updated target position and the position of the third particle; comparing the first distance and the second distance; determining, based on the comparison, that the first distance is less than the second distance; comparing the first distance with a predefined value; and determining, based on the comparison, whether the fast particle switching condition is met.

[0313] In some embodiments, determining whether the fast particle switching condition is met includes: determining a first distance between the updated target position and a straight line passing through the positions of the first particle and the third particle; comparing the first distance with a predefined value; and based on the comparison, determining whether the fast particle switching condition is met.

[0314] In some embodiments, the method further includes: obtaining information indicating a first target position in the ND descriptor space before selecting a first particle, wherein the first particle is selected from a plurality of particles based on the first target position, and determining whether the fast particle switching condition is met includes: determining a first distance between the first target position and a straight line passing through the position of the first particle and the position of the third particle; determining a second distance between the updated target position and the straight line; determining the difference between the first distance and the second distance; comparing the difference with a predefined value; and determining whether the fast particle switching condition is met based on the comparison.

[0315] Example use cases

[0316] Figure 7AAn XR system 700 according to one embodiment is shown, in which the embodiments disclosed herein can be applied. Figure 7A As shown, the XR system 700 includes: an XR headset 720 (e.g., XR goggles, XR glasses, XR head-mounted display (HMD), etc.) configured to be worn by a user and operable to display an XR scene to the user, such as, for example, a VR scene in which the user is virtually immersed; speakers 734 and 735 for generating sound for the user; and an input device 750 for receiving input from the user. In this example, the input device 750 takes the form of a joystick.

[0317] like Figure 7B As shown, the XR headset 720 may include an orientation sensing unit 721, a position sensing unit 722, and an XR rendering device (XRRD) 724. In this embodiment, the XRRD 724 includes an audio renderer (AR) 799, which includes a GDB 104 and a GRF 108.

[0318] Orientation sensing unit 721 is configured to detect changes in the user's orientation and provide information about the detected changes to XR rendering device 724. In some embodiments, XR rendering device 724 determines an absolute orientation (relative to a coordinate system) given a change in orientation detected by orientation sensing unit 721. In some embodiments, orientation sensing unit 721 may include one or more accelerometers and / or one or more gyroscopes.

[0319] In addition to receiving input from sensing units 721 and 722, the XR rendering device 724 can also receive input from input device 750 and can also obtain XR scene configuration information (e.g., particle metadata). Based on these inputs and the XR scene configuration, the XR rendering device 724 renders the XR scene for the user in real time. That is, the XR rendering device generates XR content in real time, including, for example, video data provided to display driver 726 so that display driver 726 will display images included in the XR scene on display screen 727, and audio data provided to speaker driver 728 so that speaker driver 728 will play audio for speakers 734 and 735 used. The audio data, or portions thereof, may be generated by GRF 108 using particles selected from GDB 104, as described above. Although in this embodiment the XR rendering device 724 is shown inside the XR headset 720, in other embodiments the XR rendering device 724 or one or more of its components (such as, for example, GDB 104 and GRF 108) are located remotely from the XR headset 720. In this case, the XR headset 720 and the XR rendering device 724 have communication means (transmitter, receiver) for enabling the XR rendering device 724 to send XR content to the XR headset 720 (e.g., the XR rendering device or its components may be implemented in the "cloud").

[0320] Figure 8 This is a block diagram of an XR rendering device 724 for performing the methods disclosed herein, according to some embodiments. Figure 8 As shown, the XR rendering device 724 may include: a processing circuitry (PC) 802, which includes one or more processors (P) 855 (such as, for example, one or more general-purpose microprocessors and / or one or more other processors, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc.), which may coexist in a single housing or a single data center or may be geographically distributed (e.g., the XR rendering device 724 may be a distributed computing device including two or more computers, or a monolithic computing device consisting of a single computer); at least one network interface 848 (e.g., a physical interface or spatial interface). The XR rendering device 724 includes a transmitter (Tx) 845 and a receiver (Rx) 847, enabling it to send and receive data to and from other nodes connected to (physically or wirelessly) a network 110 (e.g., an Internet Protocol (IP) network) connected to the network interface 848 (e.g., the network interface 848 may be coupled to an antenna arrangement including one or more antennas to enable the XR rendering device 724 to wirelessly send / receive data); and a storage unit (also referred to as a “data storage system”) 808, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. Figure 8 As shown, GDB 104 can be stored in storage unit 808. In embodiments where PC 802 includes a programmable processor, a computer-readable storage medium (CRSM) 842 may be provided. CRSM 842 may store a computer program (CP) 843 including computer-readable instructions (CRI) 844. CRSM 842 may be a non-transitory computer-readable medium, such as magnetic media (e.g., hard disk), optical media, memory devices (e.g., random access memory, flash memory), etc. In some embodiments, the CRI 844 of computer program 843 is configured such that when executed by PC 802, the CRI causes XR rendering device 724 to perform the steps described herein (e.g., the steps described herein with reference to the flowcharts). In other embodiments, XR rendering device 724 may be configured to perform the steps described herein without requiring code. That is, for example, PC 802 may consist only of one or more ASICs. Therefore, the features of the embodiments described herein can be implemented in hardware and / or software.

[0321] Overview of various embodiments

[0322] A1. A method for rendering audio corresponding to an audio recording, wherein the audio recording is divided into multiple particles, the method comprising: obtaining information identifying a target position in an N-dimensional descriptor space, wherein N > 0; selecting a set of particles from a particle database based on the target position, wherein the selected set of particles includes a first particle and a second particle; determining a first weight w1 for the first particle; determining a second weight w2 for the second particle; using the first weight, the second weight, the first particle, and the second particle to generate interpolated particles; and rendering the interpolated particles.

[0323] A2. The method according to embodiment A1, wherein the second particle has a position in the N-dimensional space, and the first particle is located in the N-dimensional space at a complementary position to the position of the second particle in the N-dimensional space.

[0324] A3. According to the method described in A1 or A2, wherein the interpolating particle is equal to: w1(g1) + w2(g2), or w1(tg1) + w2(g2), or w1(g1) + w2(tg2), or w1(tg1) + w2(tg2), where g1 is the first particle, g2 is the second particle, tg1 is a first temporary particle derived from the first particle, and tg2 is a second temporary particle derived from the second particle.

[0325] A4. The method according to embodiment A1 or A2, wherein the first particle has a length and the second particle has a length, and generating the interpolated particle comprises: determining a target particle length using a first length value specifying the length of the first particle and a second length value specifying the length of the second particle, wherein the length of the first particle is not equal to the target particle length; deriving a first temporary particle from the first particle having a length equal to the target particle length; and generating the interpolated particle using the first temporary particle, the second particle, the first weight, and the second weight.

[0326] A5. The method according to embodiment A1 or A2, wherein the first particle has a length and the second particle has a length, and generating the interpolated particle comprises: determining a target particle length using a first length value specifying the length of the first particle and a second length value specifying the length of the second particle, wherein the length of the first particle is not equal to the target particle length, and the length of the second particle is not equal to the target particle length; deriving a first temporary particle from the first particle having a length equal to the target particle length; deriving a second temporary particle from the second particle having a length equal to the target particle length; and generating the interpolated particle using the first temporary particle, the second temporary particle, the first weight, and the second weight.

[0327] A6. The method according to embodiment A4 or A5, wherein deriving the first temporary particle from the first particle includes: resampling the first particle to generate the first temporary particle, or selecting a sub-part of the first particle, wherein the first temporary particle is the selected sub-part.

[0328] A7. The method according to embodiments A4, A5 or A6, wherein determining the target particle length using the first length value and the second length value includes: determining the target particle length using the first length value, the second length value, the first weight and the second weight.

[0329] A8. The method according to embodiment A7, wherein the length of the target particle is equal to w1(L1) + w2(L2), where L1 is the first length value and L2 is the second length value.

[0330] A9. The method according to any one of embodiments A1 to A8, wherein the first particle has a position in the N-dimensional space, and the second particle has a position in the N-dimensional space, the first weight is based on the distance from the position of the first particle in the N-dimensional space to the target position in the N-dimensional space (e.g., inversely proportional), and the second weight is based on the distance from the position of the second particle in the N-dimensional space to the target position in the N-dimensional space (e.g., inversely proportional).

[0331] A10. The method according to any one of embodiments A1 to A9, wherein the first weight is equal to: d2 / (d1 + d2), the second weight is equal to: d1 / (d1 + d2), d1 is the distance from the position of the first particle in the N-dimensional space to the target position in the N-dimensional space, and d2 is the distance from the position of the second particle in the N-dimensional space to the target position in the N-dimensional space.

[0332] A11. The method according to any one of embodiments A1 to A10, wherein selecting a set of particles from the particle database comprises: defining a first candidate particle set based on the target position; selecting a first particle from the first candidate particle set; after selecting the first particle from the first candidate particle set, removing the first particle from the first particle set to form a second candidate particle set; assigning a final weight to each particle included in the second candidate particle set; and selecting a particle from the second candidate particle set based on the assigned final weight, wherein the particle selected from the second particle set is the second particle.

[0333] A12. The method according to any one of embodiments A1 to A10, wherein selecting a set of particles from the particle database comprises: defining a candidate particle set based on the target position; selecting a first particle from the candidate particle set; assigning a final weight to each particle included in the candidate particle set; and, after assigning the final weight to each particle included in the candidate particle set, selecting a particle from the candidate particle set based on the assigned final weight, wherein the particle selected from the candidate particle set is the second particle.

[0334] A13. The method according to embodiment A11 or A12, wherein assigning a final weight to the second particle includes: determining whether the second particle is complementary to the first particle relative to the target position; assigning a complementary weight p to the second particle. COMP Wherein, if the second particle is complementary to the first particle relative to the target position, then p COMPThe value of p is X; otherwise, p COMP The value of X is Y, where X is greater than Y; and the final weights for the second particle are calculated using the complementary weights assigned to the second particle.

[0335] A14. The method according to embodiment A13, wherein X is at least ten times Y.

[0336] B1. A computer program comprising instructions that, when executed by processing circuitry of a rendering device, cause the rendering device to perform the method according to any one of embodiments A1 to A14.

[0337] B2. A carrier comprising a computer program according to embodiment B1, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer-readable storage medium.

[0338] C1. A rendering apparatus for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of particles, the rendering apparatus comprising: a memory; and processing circuitry coupled to the memory, wherein the rendering apparatus is configured to perform a method comprising: obtaining information identifying a target position in an N-dimensional descriptor space, wherein N > 0; selecting a set of particles from a particle database based on the target position, wherein the selected set of particles includes a first particle and a second particle; determining a first weight w1 for the first particle; determining a second weight w2 for the second particle; generating interpolated particles using the first weight, the second weight, the first particle, and the second particle; and rendering the interpolated particles.

[0339] C2. The rendering apparatus according to embodiment C1, wherein the rendering apparatus is further configured to perform the method according to any one of embodiments A2 to A14.

[0340] Although various embodiments have been described herein, it should be understood that they are presented by way of example only and not limitation. Therefore, the breadth and scope of this disclosure should not be limited by any of the exemplary embodiments described above. Moreover, unless otherwise indicated herein or clearly contradicted by the context, any combination of the elements described above in all possible variations is covered by this disclosure.

[0341] Furthermore, although the process described above and illustrated in the accompanying drawings is shown as a series of steps, this is merely for illustrative purposes. Therefore, it should be expected that some steps may be added, some steps may be omitted, the order of the steps may be rearranged, and some steps may be performed in parallel. Further, as used herein, "a" means "at least one" or "one or more".

[0342] References

[0343] [1] US20180068487A1: Systems and methods for simulating sounds of a virtual object using procedural audio (The Disney Enterprises).

[0344] [2] Farnell, Andy, “An introduction to procedural audio and its application in computer games”, Audio Major Conference, Vol. 23, 2007.

[0345] [3] D. Gabor, “Theory of communication. Part 1: The analysis of information”, Journal of the Institute of Electrical Engineers - Part 3: Radio and Communications Engineering, Vol. 93, No. 26, pp. 429–441, 1946.

[0346] [4] D. Schwarz, “Corpus-Based Concatenative Synthesis”, IEEE Journal of Signal Processing, Vol. 24, No. 2, pp. 92-104, March 2007, doi: 10.1109 / MSP.2007.323274.

[0347] [5] D. Schwarz “Distance mapping for corpus-based concatenative synthesis”, Sound and Music Computing (SMC) 2011.

[0348] [6] Aaron Einbond and Diemo Schwarz. “Spatializing timbre with corpus-based concatenative synthesis”, Proceedings of the International Conference on Computer Music, Vol. 2010, International Association for Computer Music, 2010.

[0349] [7] D. Schwarz, “A system for data-driven concatenative sound synthesis”, Digital Sound Effects (DAFx), 2000.

[0350] [8] D. Schwarz et al., “Real-time corpus-based concatenative synthesis with catart”, 9th International Conference on Digital Audio (DAFx), 2006.

[0351] [9] Diemo Schwarz and Norbert Schnell, “Descriptor-based sound texturesampling”, Sound and Music Computing (SMC) 2010.

[0352]

[10] Stefan Kersten and Hendrik Purwins, “Sound texture synthesis with Hidden Markov Tree models in the wavelet domain”, Proceedings of the International Conference on Sound and Music Computing (SMC), Barcelona, ​​Spain, July 2010.

[0353]

[11] Diemo Schwarz and Sean O'Leary, “Smooth granular sound texture synthesis by control of timbral similarity”, Sound and Music Computation (SMC) 2015.

[0354]

[12] Zechen Zhang, Nikunj Raghuvanshi, John Snyder and Steve Marschner, 2019, Acoustic texture rendering for extended sources in complex scenes, ACM Trans. Graph. 38, 6, Item 222 (December 2019), 9 pages. https: / / doi.org / 10.1145 / 3355089.3356566.

[0355]

[13] Barrass, Stephen and Matt Adcock, “Interactive granular synthesis of haptic contact sounds”, Audio Engineering Society Conference: 22nd International Conference: Virtual, Synthetic and Entertainment Audio, Audio Engineering Society, 2002.

[0356]

[14] US20190094975A1: Haptic Effect Conversion System Using GranularSynthesis, Immersive Technology.

[0357]

[15] T. Park, J. Biguenet, Z. Li, C. Richardson and T. Scharr, “Featuremodulation synthesis (FMS)”, Proc. ICMC (Copenhagen, Denmark), 2007.

[0358]

[16] Fink, Marco, Martin Holters and Udo Zölzer, “Signal-matched power-complementary cross-fading and dry-wet mixing”, Proceedings of the 19th International Conference on Digital Audio (DAFx-16), 2016.

Claims

1. A method (1000) for rendering audio corresponding to an audio recording (111), wherein, The audio recording is divided into multiple particles, and the method includes: Obtain (s1002) information to identify the target location in the N-dimensional descriptor space, where N > 0; Based on the target position, a set of particles is selected from the particle database (s1004), wherein the selected set of particles includes a first particle and a second particle; Determine (s1006) the first weight w1 used for the first particle; Determine (s1008) the second weight w2 used for the second particle; The first weight, the second weight, the first particle, and the second particle are used to generate (s1010) interpolation particles; and Render the interpolated particles (s1012).

2. The method according to claim 1, wherein, The second particle has a position in the N-dimensional space, and The first particle is located in a complementary position in the N-dimensional space relative to the position of the second particle in the N-dimensional space.

3. The method of claim 1 or 2, wherein, The interpolating particle is equal to: w1(g1) + w2(g2), or w1(tg1) + w2(g2), or w1(g1) + w2(tg2), or w1(tg1) + w2(tg2), where, g1 is the first particle. g2 is the second particle. tg1 is the first temporary particle derived from the first particle, and tg2 is the second temporary particle derived from the second particle.

4. The method according to claim 1 or 2, wherein, The first particle has a length, and the second particle has a length, and Generating the interpolation particles includes: The length of the target particle is determined using a first length value specifying the length of the first particle and a second length value specifying the length of the second particle, wherein the length of the first particle is not equal to the length of the target particle; Derive a first temporary particle with a length equal to the length of the target particle from the first particle; and The interpolation particle is generated using the first temporary particle, the second particle, the first weight, and the second weight.

5. The method according to claim 1 or 2, wherein, The first particle has a length, and the second particle has a length, and Generating the interpolation particles includes: The target particle length is determined using a first length value specifying the length of the first particle and a second length value specifying the length of the second particle, wherein the length of the first particle is not equal to the target particle length, and the length of the second particle is not equal to the target particle length; Derive a first temporary particle with a length equal to that of the target particle from the first particle; Derive a second temporary particle with a length equal to that of the target particle from the second particle; and The interpolation particles are generated using the first temporary particle, the second temporary particle, the first weight, and the second weight.

6. The method of claim 4 or 5, wherein, Deriving the first temporary particle from the first particle includes: The first particle is resampled to generate the first temporary particle, or Select a sub-part of the first particle, where the first temporary particle is the selected sub-part.

7. The method according to claim 4, 5 or 6, wherein, Determining the target particle length using the first length value and the second length value includes: using the first length value, the second length value, the first weight, and the second weight to determine the target particle length.

8. The method according to claim 7, wherein, The length of the target particle is equal to w1(L1) + w2(L2), where, L1 is the first length value, and L2 is the second length value.

9. The method according to any one of claims 1 to 8, wherein, The first particle has a position in the N-dimensional space, and The second particle has a position in the N-dimensional space. The first weight is based on the distance from the position of the first particle in the N-dimensional space to the target position in the N-dimensional space, and The second weight is based on the distance from the position of the second particle in the N-dimensional space to the target position in the N-dimensional space.

10. The method according to any one of claims 1 to 9, wherein, The first weight is equal to: d2 / (d1 + d2), The second weight is equal to: d1 / (d1 + d2), d1 is the distance from the position of the first particle in the N-dimensional space to the target position in the N-dimensional space, and d2 is the distance from the position of the second particle in the N-dimensional space to the target position in the N-dimensional space.

11. The method of any one of claims 1 to 10, wherein, Selecting a set of particles from the particle database includes: Based on the target location, a first candidate particle set is defined; Select the first particle from the first candidate particle set; After selecting the first particle from the first candidate particle set, the first particle is removed from the first particle set to form a second candidate particle set. For each particle included in the second candidate particle set, assign a final weight to that particle; Based on the assigned final weights, particles are selected from the second candidate particle set, wherein the particles selected from the second particle set are the second particles.

12. The method of any one of claims 1 to 10, wherein, Selecting a set of particles from the particle database includes: Based on the target location, a candidate particle set is defined; Select the first particle from the candidate particle set; For each particle included in the candidate particle set, a final weight is assigned to that particle; After assigning the final weight to each particle included in the candidate particle set, a particle is selected from the candidate particle set based on the assigned final weight, wherein the particle selected from the candidate particle set is the second particle.

13. The method of claim 11 or 12, wherein, Assigning the final weight to the second particle includes: Determine whether the second particle is complementary to the first particle relative to the target position; Assign complementary weights p to the second particle COMP Wherein, if the second particle is complementary to the first particle relative to the target position, then p COMP The value of p is X; otherwise, p COMP The value of is Y, where X is greater than Y; and The final weights for the second particle are calculated using the complementary weights assigned to the second particle.

14. The method of claim 13, wherein, X is at least ten times that of Y.

15. A computer program (843) comprising instructions (844) that, when executed by a processing circuitry (802) of a rendering device (724), causes the rendering device to perform the method according to any one of claims 1 to 14.

16. A carrier containing the computer program of claim 15, wherein the carrier is one of an electronic signal, an optical signal, a radio frequency signal, or a computer readable storage medium. The carrier is one of electronic signals, optical signals, radio signals, and computer-readable storage media (842).

17. A rendering device (724) for rendering audio corresponding to an audio recording (111), wherein The audio recording is divided into multiple particles, and the rendering device includes: Memory (842); and Processing circuitry (802), coupled to the memory, wherein the rendering device is configured to perform a method (100) comprising: Obtain (s1002) information to identify the target location in the N-dimensional descriptor space, where N > 0; Based on the target position, a set of particles is selected from the particle database (s1004), wherein the selected set of particles includes a first particle and a second particle; Determine (s1006) the first weight w1 used for the first particle; Determine (s1008) the second weight w2 used for the second particle; The first weight, the second weight, the first particle, and the second particle are used to generate (s1010) interpolation particles; and Render the interpolated particles (s1012).

18. The rendering device of claim 17, wherein, The rendering device is also configured to perform the method according to any one of claims 2 to 14.

Images