Method and apparatus for particle scheduling for particle synthesis

Abstract

Various embodiments of this disclosure disclose methods and apparatus for particle scheduling in particle synthesis. A method for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of particles. In one embodiment, the method includes: obtaining first target descriptor coordinates, wherein the first target descriptor coordinates identify a first position in an N-dimensional descriptor space, where N > 0. The method further includes: defining a first candidate particle set based on the first target descriptor coordinates, wherein the first candidate particle set includes k1 particles from the plurality of particles, where k1 > 1. The method further includes: assigning a final weight to each particle in the first candidate particle set. The method further includes: randomly selecting particles from the first candidate particle set based on the assigned final weights, such that the probability of a given particle in the first candidate particle set being selected is a function of the final weight assigned to the given particle. The method further includes: rendering the selected particles.

Description

[0001] This application is a divisional application of patent application No. 202480019036.6, filed on March 13, 2024, entitled "Particle Scheduling for Particle Synthesis". Technical Field

[0002] Examples involving particle synthesis are disclosed. Background Technology

[0003] Audio rendering is the process of presenting audio, such as audio within an extended reality (XR) scene (e.g., virtual reality (VR), augmented reality (AR), or mixed reality (MR) scene), to give the listener the impression that the sound comes from a physical source at a specific location within the scene. This can be done via headphone speakers or other loudspeakers. If the presentation is done via headphone speakers, the processing used is called binaural rendering and utilizes spatial cues of human spatial hearing, making it possible to determine the direction from which the sound is coming. These cues involve interaural time delay (ITD), interaural level difference (ILD), and / or spectral difference.

[0004] Process audio refers to the generation of sound in real time as a response to on-site input. As an example, consider the sound of a car engine in a virtual space, where the sound changes based on the car's speed or acceleration. This mechanism is commonly used in video games to achieve a better user experience. It is believed that for XR (e.g., AR or VR) use cases, there are many sounds that would benefit from being dynamically generated so that they can react in real time to changes in the scene. For example, the sound generated when a user touches a surface or operates an engine. In reference [1], the sound of a virtual object is simulated based on the position and orientation of the virtual object (e.g., a sword, axe, or club). There are fundamental and overtones. Both are modulated to change the pitch, timbre, and amplitude to convey the speed of movement of the virtual object. On-site input can come from the user via sensors such as hand-held controllers or headphones, or it can be control data generated in real time by some software process such as physical simulation or predefined automated data. Regardless of how the input data is generated, the audio renderer needs to process the incoming data and generate sound in real time in response to that data.

[0005] There are many different approaches for procedural audio (see, for example, reference [2]), including sound synthesis using audio processing modules, machine learning methods trained on real recordings, and splicing synthesis methods that utilize original recordings and rearrange segments of those recordings to generate variations. Because these splicing synthesis methods use real recordings directly, the resulting audio sounds are very natural, thus enhancing the user experience.

[0006] Particle synthesis is a type of splicing synthesis in which sound recordings are divided into small segments called “particles”. (See, for example, reference [3]). By carefully selecting the segments (particles) during rendering, realistic, dynamically changing sounds can be generated.

[0007] The particle synthesis process comprises two main steps: (1) particle extraction and (2) particle synthesis. Particle extraction refers to the extraction of relevant particles from a long, original recording. The extraction method depends on the type of sound source and the desired features to be extracted. Particle synthesis refers to the technique of selecting an appropriate particle order; this ordering can also be based on real-time user input.

[0008] Many sound design tools support particle extraction and synthesis, such as Soundseed Particles for Audiokinectic Wwise, Alchemy for Logic Pro, and AudioMotors for FMOD. These tools allow sound designers to extract particles manually or semi-automatically, along with other simple manipulations. Among other controls, designers can select the particle length, amplitude envelope, or shape of each particle. In the case of AudioMotors, an automated particle extraction tool specifically designed for engine sounds is provided.

[0009] Particle extraction can be done manually by the sound designer or in a data-driven manner by identifying relevant features of the audio for segmentation purposes. Relevant features include, for example, pitch period in the case of tonal sounds, spectral energy at a given frequency, Mel-frequency cepstral coefficients (MFCC), and local maxima of the amplitude envelope.

[0010] Many other methods exist for particle synthesis. One common method is to randomly select particles and perform an overlap-addition (OLA) operation. This method is not applicable to all types of sound sources and cannot capture the temporal correlation between adjacent particles.

[0011] Corpus-based concatenation synthesis (CBCS) methods select particles based on a corpus of sound segments sampled from a database of heterogeneous sound sources. They organize the corpus using descriptors associated with the sound segments and perform a search within the descriptor space to select the next particle. Note that the concept of descriptors is not limited to features of the audio signal (see, for example, reference [4]). When a direct mapping between the desired effect and features in the audio signal is not possible, the user can annotate the particles with perceptual descriptors.

[0012] The descriptor space is multidimensional, where the dimension is equal to the number of descriptors. The search for appropriate particles is carried out in a computationally efficient manner by utilizing the weighted Euclidean distance between the target descriptor location (e.g., a point or region) in the descriptor space (hereinafter referred to as the “target descriptor coordinates”) and the particle location in the descriptor space. Reference [5] proposes a warp function for the distance measurement to better select the particle set and also avoid repeating previously rendered particles.

[0013] To perform an efficient search in the descriptor space, a kD-tree search is used. The k nearest neighbors of the target descriptor coordinates or particles within a radius 'r' of the target descriptor coordinates are selected. In reference [6], the corpus is organized into regions so that particles from different regions are not selected when using a k nearest neighbor search.

[0014] The software CATERPILLAR (see reference [7]) performs splicing synthesis in an offline setting of the target descriptor sequence. The program uses the Viterbi algorithm to identify the particle sequence to match the target descriptor. The cost function is a combination of the distance from the target descriptor coordinates and the splicing cost based on the similarity of consecutive particles.

[0015] On the other hand, CataRT is a real-time system, so it randomly selects subsequent particles from a set of particles that are either the k nearest neighbors or the radii centered on the target descriptor coordinates (see, for example, reference [8]).

[0016] To achieve smooth transitions in the synthesized sound of particles, reference [9] uses feature descriptors such as pitch, loudness, spectral centroid, fundamental frequency, period, and autocorrelation coefficients of lag 1. Feature descriptors are computed for each particle, and a Gaussian mixture model (GMM) is used to capture the correlation between these feature descriptors, from which particles are sampled for synthesis. Reference

[10] discusses particle synthesis, where the next particle is selected based on the feature descriptor of the current particle for timbre continuity. A kD-tree search is performed to select the candidate particle that is closest to the current particle in the feature descriptor space in terms of Euclidean distance.

[0017] Granular synthesis is used in different approaches but within the context of rendering the acoustic texture of extended sources in a virtual scene (see reference

[11] ). The described method numerically solves the wave / sound propagation equations for the entire virtual scene with source pulses distributed across the sound sources (e.g., a river) to compute the event loudness density (ELD). The ELD is obtained by deconvolving the source pulses with the sound pressure at location x over multiple time steps to compute the aggregated impulse response at a given location. The extracted time-varying impulse response is then used to accumulate ELD statistics, which are histograms of peak loudness in the impulse response. The desired particle density (e.g., particles per second) at a given listener location is obtained by scaling the total number of events at that location. The particle amplitude is then extracted from the ELD as a distribution. This is done for all sound sources in each scene for a given listener location.

[0018] Particle synthesis is also discussed in relation to haptics and gestures. Changes in a user's gesture or touch are mapped to changes in audio features, resulting in interactive audio rendering. For example, in reference

[12] , the amplitude of the sound is manipulated based on user input. The haptic input force and the energy of the audio signal are quadratically related. In reference

[13] , audio data is used as input for haptic effects. Summary of the Invention

[0019] Several challenges exist. For example, in existing systems, the method for selecting the k nearest neighbors of the target descriptor coordinates in the descriptor space is fixed and not adaptive, which can lead to spurious output when the neighborhood is not compact and the particle database includes clusters of different sizes. While in some systems (e.g., CataRT) the user can adaptively select the neighborhood size, it remains 3D, beyond which visualization is impossible and therefore cannot be adjusted by the user. Furthermore, particles in the neighborhood of the target descriptor coordinates are uniformly and randomly selected without distinguishing between the most likely and least likely particles to be played. Therefore, conventional systems cannot produce natural sound output, especially when the user input does not linger long enough at the target descriptor coordinates. In such cases, it is desirable to render the most representative particles closest to the target descriptor coordinates, rather than randomly selected ones. Furthermore, while some systems (e.g., CATERPILLAR) use the Viterbi algorithm to find the next particle that minimizes the cost function describing the discontinuities introduced by the splicing of the next particle, this optimization serves to minimize discontinuities rather than avoid unnatural vocalizations caused by repeating the same particles or particle patterns.

[0020] Therefore, in one aspect, a method for rendering audio corresponding to an audio recording is provided, wherein the audio recording is divided into a plurality of particles. In one embodiment, the method includes: obtaining first target descriptor coordinates, wherein the first target descriptor coordinates identify a first position in an N-dimensional descriptor space, where N > 0. The method further includes: defining a first candidate particle set based on the first target descriptor coordinates, wherein the first candidate particle set includes k1 particles from the plurality of particles, where k1 > 1. The method further includes: assigning a final weight to each particle in the first candidate particle set. The method further includes: randomly selecting particles from the first candidate particle set based on the assigned final weights, such that the probability of a given particle in the first candidate particle set being selected is a function of the final weight assigned to the given particle. The method further includes: rendering the selected particles.

[0021] In another embodiment, the method includes: obtaining first target descriptor coordinates, wherein the first target descriptor coordinates identify a first position in an N-dimensional descriptor space, where N > 0; and defining a first candidate particle set based on the first target descriptor coordinates, wherein the first candidate particle set includes k1 particles from the plurality of particles, where k1 > 1, and the first candidate particle set includes a first particle and a second particle. The method further includes: assigning a final weight to each particle in the first candidate particle set, wherein assigning a final weight to each particle in the first candidate particle set includes: assigning a first final weight to the first particle and assigning a second final weight to the second particle. The method further includes: selecting particles from the first candidate particle set based on the assigned final weights, and rendering the selected particles, wherein the first final weights assigned to the first particles are: a function of the distance between the first target descriptor coordinates and the first particle, a function of the amount of time that has elapsed since the first particle was last rendered, a function of the trajectory and target trajectory associated with the first particle, and / or a function of a measure of proximity between the first particle and the most recently rendered particle (e.g., a time difference, which indicates the difference between the timestamp used for the first particle and the timestamp used for the most recently rendered particle, assuming that the two particles were extracted from the same record or the same segment).

[0022] In another aspect, a computer program including instructions is provided that, when executed by processing circuitry of an apparatus, cause the apparatus to perform any of the methods disclosed herein. In one embodiment, a carrier containing the computer program is provided, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer-readable storage medium. In another aspect, an apparatus configured to perform the methods disclosed herein is provided. The apparatus may include a memory and processing circuitry coupled to the memory.

[0023] The advantages of the embodiments disclosed herein lie in their granular synthesis, which uses real-time target descriptor coordinates as input to generate natural particle progressions. By using weighted selection (i.e., particle selection processing using weights assigned to particles), precise control and a combination of natural variations in sound are achieved. The solution's low complexity allows for the rendering of numerous particle sound sources even on devices with limited computing power. The low real-time complexity also enables the solution to produce reactive and responsive sounds without latency. Attached Figure Description

[0024] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0025] Figure 1 A system according to an embodiment is shown.

[0026] Figure 2A An example two-dimensional descriptor space is shown.

[0027] Figure 2B An example two-dimensional descriptor space is shown.

[0028] Figure 3A An example two-dimensional descriptor space is shown.

[0029] Figure 3B The process for determining the optimal k value for a given particle, according to an embodiment, is illustrated.

[0030] Figure 4A An example audio signal is shown.

[0031] Figure 4B An example scheduling of particles is shown.

[0032] Figure 4C An example scheduling of particles according to an embodiment is shown.

[0033] Figure 5 This is a flowchart illustrating a process according to an embodiment.

[0034] Figure 6 This is a flowchart illustrating a process according to an embodiment.

[0035] Figure 7A and Figure 7B A system according to some embodiments is shown.

[0036] Figure 8 This is a block diagram of an apparatus according to some embodiments. Detailed Implementation

[0037] Figure 1 A system 100 for implementing particle synthesis according to some embodiments is shown. System 100 includes a particle extraction unit 102 that extracts particles from a raw audio recording 111. That is, the particle extraction unit divides the raw audio recording into small segments called “particles”. The extracted particles are stored in a particle database 104, which is accessed during rendering by a particle scheduling unit 106 (also called a particle selection unit), a component of a particle rendering unit 108. In some embodiments, one particle database exists for each procedural audio source, and each particle database is available to the rendering unit 108. Each particle stored in the particle database is associated with one or more vectors of one or more descriptor values, each vector corresponding to a specific descriptor.

[0038] When creating the particle database 104, the audio designer decides which aspects should be used as descriptors. In some cases, this might be characteristics of the sound itself, such as pitch or loudness, but it could also be other aspects related to how the sound is generated, such as the speed of movement that generates contact sound between two objects sliding relative to each other, or the opening angle of a door that generates a harsh sound when it is opened and closed. The descriptors should be chosen so that, given the target descriptor coordinates or trajectory, the sound can be dynamically regenerated by the renderer.

[0039] As described above, each particle stored in the particle database is associated with one or more descriptor values. Therefore, the particles in the original recording need to be annotated with descriptor values. Where the descriptor is an audio feature, the descriptor value can potentially be measured directly from the audio signal itself. In other cases, the descriptor value needs to be provided as additional metadata for the recording in some way. This can be done, for example, by recording data from some sensors during recording and providing that data in an accompanying file. In some cases, annotation can be done manually by creating a data log explaining how the descriptor changes during recording, or it can be done manually for each extracted particle.

[0040] When particles are extracted from the original record, descriptor values ​​are stored as metadata for each particle. Using descriptor values, each particle can be located in a multidimensional descriptor space, where the value of each descriptor describes its position along an axis within that space. If only one descriptor is used, the descriptor space is one-dimensional (1D), but the dimensionality of the descriptor space increases if more descriptors are used. Figure 2A The example shows a two-dimensional (2D) descriptor space, where each circle represents a particle. (See diagram for example.) Figure 2A As shown, each particle has a location (e.g., a point or region) in the 2D descriptor space, which is called particle coordinates.

[0041] In one embodiment, the descriptor metadata of a granular sequence extracted from a record describes a trajectory within the descriptor space that corresponds to how the descriptor evolved during the original record.

[0042] During rendering, particle scheduling (i.e., the selection of particles to render) is based on target descriptor coordinates in descriptor space. Target descriptor coordinates specify what descriptor values ​​the generated sound output should have, meaning that particles closest to those coordinates in descriptor space will be used most frequently. Target descriptor coordinates can come from many types of sources, such as physics engines simulating virtual body interactions, field input parameters from hand controllers or other sensors, and predefined automation parameters.

[0043] A key aspect of particle compositing rendering is that the repetition of the same particles often sounds very unrealistic and artificial. If the particles are short (less than 50 milliseconds), repeating the same particles will result in a very metallic and static sound. If the particles are long, the repetition sounds like a repetitive pattern, which usually makes the sound unrealistic.

[0044] The scheduling of particles needs to avoid duplication of the same particles, but at the same time select particles that are close to the target descriptor coordinates in the descriptor space. Therefore, this disclosure uses a weighted selection (e.g., weighted random selection) process that generates a sequence of particles (i.e., an ordered set of particles) that closely follows the evolving target descriptor coordinates. An example of a particle sequence is: [particle-7, particle-6, particle-7, particle-9, particle-11, particle-10].

[0045] Each particle can be assigned a predefined weight and a set of dynamic weights that can change over time. Predefined weights can be useful when particles should not be used too frequently, but whose occasional use can add outliers to the generated sound, indicating genuine variation. Another use case is using predefined weights to control the frequency of particles representing, for example, birdsong, compared to particles representing background forest sounds.

[0046] The input from the audio source (e.g., a signal generated by user interaction) is mapped to target descriptor coordinates. Based on the target descriptor coordinates, one or more particles are selected from the candidate particle set for rendering using weighted selection (e.g., weighted random selection or selection where the particle with the highest weight is chosen). The selected particles are rendered using standard particle synthesis methods, where metadata regarding overlap percentage and crossfade windows is pre-specified by the sound designer.

[0047] The input to the audio source in the control process can be changed in real time; therefore, the target descriptor coordinates can change over time as the input changes (even if the input does not change, the target descriptor coordinates can still change over time).

[0048] In one embodiment, the granular scheduling algorithm using weighted selection has the following steps.

[0049] Step 1: Obtain the target descriptor coordinates (e.g., map inputs such as user input or other inputs to target descriptor coordinates in the descriptor space).

[0050] Step 2: Adaptively determine the size of the neighborhood, for example, by calculating the k-value or the radius (r) based on the target descriptor coordinates. Optionally, obtain a pre-calculated k-value or radius from the metadata of the particle database.

[0051] Step 3: Use the k value or radius value to select a candidate particle set from the database. For example, select the particles that are the k nearest neighbors of the target descriptor coordinates from the database. This search can be implemented using readily available computationally efficient algorithms such as kD-tree search. As another example, include each particle in the candidate particle set that has particle coordinates within r of the target descriptor coordinates.

[0052] Step 4: Assign a final weight (also called a probability value) to each particle in the candidate particle set. The final weight assigned to a given particle can be based on:

[0053] i) The distance between the particle location (i.e., particle coordinates) in the descriptor space and the target descriptor coordinates in the descriptor space.

[0054] ii) The difference between the trend of the particle descriptor and the trajectory of the target descriptor.

[0055] iii) The time history of previously used particles,

[0056] iv) The time difference between particles in the original record and previously used particles (if they come from the same record), and / or

[0057] v) Predefined probability weights for particles in the database.

[0058] Step 5: Using the final weights assigned in Step 4, perform a weighted selection of particles from the candidate particle set. For example, perform a weighted random selection, or, as another example, select the particle with the highest or lowest final weight. In this way, the particle is selected based on the target descriptor coordinates and also on the final weights assigned to the particles in the candidate particle set.

[0059] If the target descriptor coordinates change, these steps are repeated. Otherwise, the weighted selection of particles continues (starting from step 4), where the weights are changed based on the time history of previous particles.

[0060] Step 2 – Determining the value of k.

[0061] Typically, the value of k is a user-defined constant. However, it is not desirable to always keep the value of k fixed, as doing so may result in selecting too few or too many granules, which in turn may lead to underutilization of the granules or scheduling of granules that are not similar to the target descriptor value.

[0062] Therefore, in one embodiment, this disclosure provides an adaptive selection of k based on the density of particles available in the region surrounding the target descriptor coordinates.

[0063] An example of why different k values ​​should be used for different target descriptor coordinates is provided below. Figure 2A and Figure 2B As shown in [the image]. Figure 2A In the middle, the target descriptor coordinates are close to a cluster of 3 particles, while... Figure 2B In this case, the target descriptor coordinates are closer to more particles. Using the same k value is not optimal in both situations. For Figure 2A In this case, k = 3 is appropriate. If k > 3, this will result in selecting particles that are not in the cluster, and thus cause discontinuities in the texture of the rendered sound. Figure 2B In this case, if k = 3, too few particles are selected. Here, since a variety of particles can be selected, a larger value will result in a richer texture with less repetition.

[0064] The value of k should be adaptively chosen based on the target descriptor coordinates in the descriptor space. In one embodiment, each particle in the database is assigned an optimal k value. Then, for target descriptor coordinates, the value of k is set to be equal to the optimal k value assigned to the particle closest to that target descriptor coordinate.

[0065] The assignment of the optimal k for each particle in the database can be performed offline or during the construction of the particle database. Alternatively, all distances from particle i to other particles in the database can be recorded, or a threshold for the maximum number of neighbors for calculating the stopping distance can exist.

[0066] The distances are then sorted from lowest to highest. The difference between consecutive values ​​of k will have a sudden jump at values ​​where the distance increases sharply. This is considered the cutoff value for k, and k+1 is assigned to that particle. Adding 1 is to include the particle itself in the value of k.

[0067] The criterion for determining the cutoff value would be to set an absolute threshold for the difference in sorted distances between consecutive neighbors, or to increase the normalized percentage of consecutive sorted distances.

[0068] Alternatively, the concept of an adaptive radius can be used to select the candidate particle set. Current methods in the literature involve using a fixed radius with the target descriptor coordinates as the center, and selecting all particles within a circle of that fixed radius to be included in the candidate particle set. Following the same reasoning as above, it may be beneficial to adaptively change the radius based on the position of the target descriptor coordinates. In this case, instead of choosing different k values, different radius values ​​are used.

[0069] Figure 3A The calculation of k for a specific particle 301, represented by a black circle, is shown. Figure 3A The image shows particle 301 and its six corresponding neighbors. Figure 3B The diagram shows the sorting distance for particle 301. Because a jump in distance values ​​can be observed for k = 4, the optimal k value assigned to particle 301 is 5.

[0070] Step 3 - Weight Allocation

[0071] The following describes each factor that influences the final weights assigned to particles. The target descriptor is represented as... The weights (also called probabilities) associated with particle i are represented as follows: The descriptor index is determined by... It means that, among them This refers to the number of descriptors or the dimension of the descriptor space. The descriptor value of particle i at dimension j is determined by... Given, and the target's descriptor value is... .

[0072] The following criteria affecting the probability of selecting a particle are expressed as a proportional relationship because the final value of the weights is obtained after normalization (i.e., ensuring that the sum of the weights corresponding to all k particles is 1).

[0073] i) Distance from the target descriptor coordinates

[0074] Distance metrics are used to define the proximity between the coordinates of a target descriptor and the coordinates of other descriptors corresponding to a particle. An example of a distance metric is the weighted Euclidean distance, where the difference between coordinates in each dimension is weighted by the inverse of the standard deviation of the corresponding descriptor values.

[0075]

[0076] The closer the particle's descriptor coordinates are to the target descriptor coordinates, the higher the associated weight. Let Let be the distance from particle i to the target descriptor coordinates. For example, the probability of particle i being selected can be inversely proportional to the distance. Therefore, it is possible to set .

[0077] In some cases, different descriptors should not have the same influence on granularity selection. For example, if one descriptor is the pitch of a sound, and another descriptor has a less strong influence on the perceptual features of the sound, then the distance in the dimension corresponding to the pitch can be given a higher weight than the distance in the dimension corresponding to the other descriptor. This can be achieved by adding additional variable weights to each dimension when calculating the distance. To achieve:

[0078]

[0079] ii) Trend difference between target and original descriptor trajectories

[0080] When performing particle extraction, descriptor coordinates are used as the primary selection criterion. However, the trend of the original descriptor trajectory also provides important information about the particles. For example, if engine sound is modeled using a granular database of descriptors representing engine RPM (revolutions per minute), the trend of the descriptors corresponds to the engine's acceleration or deceleration at the moment in the record from which particles are extracted. Particles extracted from a portion of the record when the engine is accelerating will have a slightly lower pitch at the beginning than at the end, and therefore will be best suited when the desired output is the sound of an accelerating engine.

[0081] In a more general case, when using a multidimensional descriptor space, the descriptor trend is a vector corresponding to the direction of the trajectory, which describes how the descriptor changed during the original recording. Similarly, the trend of the target descriptor trajectory describes the direction in which the target descriptor coordinates are moving within the descriptor space.

[0082] The trend describes both the direction and rate of change. Referring again to the engine example, if the granularity database includes particles corresponding to the same RPM but with different accelerations, then particles corresponding to accelerations similar to those of the target descriptor trajectory should be preferred.

[0083] Figure 4A The descriptor trace of the raw recording of the engine sound is shown, where the descriptor is the engine's RPM. The recording is divided into 15 particles. Figure 4B An example of how particles can be selected to match a target descriptor trajectory is shown, where the descriptor trend of the particles is not considered during particle selection. It can be seen that particles with both increasing and decreasing RPMs are used in combination. The resulting descriptor trajectory exhibits irregular behavior, which can lead to a deterioration in perceived quality, especially if the descriptor represents the pitch of a sound. Figure 4C In addition, particle selection also takes into account descriptor trends so that only particles with reduced RPM are used, which will result in a smoother sound.

[0084] Considering descriptor trends during particle scheduling is particularly important for sound sources, where the characteristics differ for different trends. For example, an engine may sound different when accelerating compared to when decelerating. Ensuring that descriptor trends match avoids the problem of particles with different characteristics being used together.

[0085] The descriptor trend of a particle can be calculated as the difference between the descriptor values ​​at the end of the particle and those at the beginning, divided by the duration of the particle. In the case of a multidimensional descriptor space, the trend is a vector describing the average rate of change of the descriptor coordinates during the particle's duration in the original record; for example, for three descriptors, the particle trend t... G It will be a three-dimensional vector:

[0086]

[0087] When rendering time-scheduled particles, the trend of the target descriptor trajectory can be similarly calculated as the difference between the descriptor coordinates since the last update, divided by the time T elapsed since the last update. U .

[0088]

[0089] Then, the difference in trends can be calculated as:

[0090]

[0091] Then, based on t D The norm is used to calculate the weights assigned to the particles, for example: , where mT It is a variable that controls how the probability decreases as the difference in the descriptor trend increases.

[0092] iii) Time history of previously used particles

[0093] The history of previously rendered particles is maintained within a specific time window to prevent them from being repeated, and the probability of selecting a particle is reduced if it has already been rendered. This reduction in probability is related to the current time *t* and the time *i* was last rendered. The difference between them is a function proportional to the difference between them.

[0094]

[0095] If no particle i was selected in the past or within a specific time window, the value of the previous time step is set to zero. .function It can be an independent variable with a non-negative output. Linear, quadratic, logarithmic, or any monotonically increasing function.

[0096] iv) Time difference in the original record

[0097] For some sound sources, the way sound evolves may not be entirely described by changes in descriptors. Sometimes sound evolves in a way that depends on earlier events. For example, even when an old door is opened at the same speed, the harsh sound of the old door may be slightly different each time it is opened, and so on. In these cases, particles with similar descriptor values ​​and descriptor trends may sound very different, and combining them can lead to unnatural discontinuities that are not present in the original recording. One way to avoid these discontinuities is to assign higher probabilities to particles from the same part of the original recording that are similar to previously used particles. Particles from the same part of the original recording are expected to be closely related in features and similar to each other, and therefore are good candidates when selecting the next particle.

[0098] To measure how close one particle from a specific record is to another particle from the same record, a time difference can be calculated, corresponding to the difference in the times from which the two particles were extracted from the records. If the particles are close, the time difference is small. To calculate the time difference between two particles, metadata can be used, which tells each particle which original record it was extracted from and at what time. Therefore, this metadata provides a measurement of the proximity between the two particles. This metadata can be specified in a compact way as two values: a record identifier assigned to the original record and a timestamp indicating the time in which the particle can be found in that record.

[0099] Then, weights for particles can be calculated based on the difference timestamps, such that smaller differences give a higher probability of selecting the particles being evaluated. For example, the weight p for particle i... i It can be calculated as:

[0100]

[0101] Among them, R i R0 is the record identifier assigned to the record from which particle i is extracted, and R0 is the record identifier assigned to the record from which previously rendered particles were extracted. i t0 and t0 are the corresponding timestamps of the two particles, and b is a design constant that sets the probability of using a particle from the other record. The function f() takes the time difference as input and calculates the probability of the particle. In one embodiment, the function f decreases linearly as the time difference increases, where the slope is specified by the variable a:

[0102]

[0103] This has the effect that as the difference between the corresponding timestamps increases, the probability decreases from 1.0 to b, but never falls below b.

[0104] In one embodiment, the record identifier R i It can also be set to reference multiple segments of a record; that is, a record can be divided into multiple segments, each with its own index. This can be useful when a record contains multiple segments that are not considered related by the renderer.

[0105] Final weight

[0106] The final weight assigned to particle i (i.e., p) i ) is calculated by accumulating the weights assigned to particle i from each level, that is, ,in, These are the pre-determined weights mentioned above. In some embodiments, not all five weights are required, and the corresponding weights can then be skipped by setting the corresponding weight to 1.0 or by completely excluding the corresponding weight from the calculation.

[0107] In one embodiment, by modifying individual weights using a fractional index, different weights are assigned different levels of influence on the final weight, for example...

[0108]

[0109] Among them, by using the fractional index Make the weight p from the first level i1 The effect is smaller, and the weight p is made smaller by using a fractional exponent of 1 / 4.i4 The impact is even smaller.

[0110] Calculate the weights for all particles in the candidate particle set according to the steps described above. Then they must be normalized, that is, each of them is divided by the sum.

[0111] , where k is the size of the candidate particle set.

[0112] Then, particles are selected based on their final weights (e.g., by sampling from a distribution). Note that, with the target descriptor coordinates unchanged, the particle's temporal history and descriptor trend influence the final weight at each time the particle is rendered. Therefore, as long as the target descriptor remains constant, if particle i is selected at time t, then at time t+h, the particle's final weight is influenced by its temporal history and descriptor trend. The final weight of particle i becomes... , among which, middle, .

[0113] In order to avoid For repeating particle i, it can be Defined as:

[0114]

[0115] However, due to the normalization operation, other The probability of each particle also changes.

[0116] Figure 5 This is a flowchart illustrating a process 500 for rendering audio corresponding to an audio recording according to an embodiment, wherein the audio recording is divided into multiple particles. Process 500 may begin at step s502.

[0117] Step s502 includes: obtaining the coordinates of a first target descriptor, wherein the coordinates of the first target descriptor identify a first position in an N-dimensional descriptor space, where N > 0.

[0118] Step s504 includes: defining a first candidate particle set based on the coordinates of the first target descriptor, wherein the first candidate particle set includes k1 particles from a plurality of particles, where k1 > 1.

[0119] Step s506 includes: assigning a final weight to each particle in the first candidate particle set.

[0120] Step s508 includes: randomly selecting particles from the first candidate particle set based on the assigned final weights, such that the probability of a given particle in the first candidate particle set being selected is a function of the final weights assigned to that given particle.

[0121] Step s510 includes: rendering the selected particles.

[0122] Figure 6 This is a flowchart illustrating a process 600 for rendering audio corresponding to an audio recording according to an embodiment, wherein the audio recording is divided into multiple particles. Process 600 may begin at step s602.

[0123] Step s602 includes: obtaining the coordinates of a first target descriptor, wherein the coordinates of the first target descriptor identify a first position in an N-dimensional descriptor space, where N > 0.

[0124] Step s604 includes: defining a first candidate particle set based on the coordinates of the first target descriptor, wherein the first candidate particle set includes k1 particles out of a plurality of particles, wherein k1 > 1, and the first candidate particle set includes a first particle and a second particle.

[0125] Step s606 includes: assigning a final weight to each particle in the first candidate particle set, wherein assigning a final weight to each particle in the first candidate particle set includes: assigning a first final weight to the first particle and assigning a second final weight to the second particle. The first final weight assigned to the first particle is: a function of the distance between the first target descriptor coordinates and the first particle, a function of the amount of time that has elapsed since the first particle was last rendered, a function of the trajectory associated with the first particle and the target trajectory, and / or a function of a measurement of the proximity between the first particle and the most recently rendered particle.

[0126] Step s608 includes: selecting particles from the first candidate particle set based on the assigned final weights.

[0127] Step s610 includes: rendering the selected particles.

[0128] Example use cases

[0129] Figure 7A An XR system 700 according to one embodiment is shown, in which embodiments disclosed herein may be applied. Figure 7A As shown, the XR system 700 includes: an XR headset 720 (e.g., XR goggles, XR glasses, XR head-mounted display (HMD), etc.) configured to be worn by a user and operable to display an XR scene to the user (e.g., an XR scene in which the user is virtually immersed); speakers 734 and 735 for generating sound for the user; and an input device 750 for receiving input from the user (in this example, the input device 750 takes the form of a joystick).

[0130] Figure 7BThe XR headset 720 shown includes an orientation sensing unit 721, a position sensing unit 722, and an XR rendering device 724. In this embodiment, the XR rendering device 724 includes an audio renderer, which includes a particle database 104 and a particle rendering unit 108.

[0131] The orientation sensing unit 721 is configured to detect changes in the user's orientation and provide information about the detected changes to the XR rendering device 724. In some embodiments, the XR rendering device 724 determines an absolute orientation (relative to some coordinate system) given a detected change in orientation detected by the orientation sensing unit 721. In some embodiments, the orientation sensing unit 721 may include one or more accelerometers and / or one or more gyroscopes.

[0132] In addition to receiving input from sensing units 721 and 722, the XR rendering device 724 can also receive input from input device 750 and obtain XR scene configuration information (e.g., particle metadata). Based on these inputs and the XR scene configuration, the XR rendering device 724 renders the XR scene for the user in real time. That is, the XR rendering device generates XR content in real time, which includes, for example, video data provided to display driver 726 and audio data provided to speaker driver 728, wherein the video data causes display driver 726 to display images included in the XR scene on display screen 727, and the audio data causes speaker driver 728 to play audio (audio data or portions thereof may be generated by particle rendering unit 108) for speakers 734 and 735 in use. Although the XR rendering device 724 is shown as being located within the XR headset 720 in this embodiment, in other embodiments, the XR rendering device 724 (or one or more components thereof, such as the particle database 104 and the particle rendering unit 108) is located remotely from the XR headset 720. In this case, the XR headset 720 and the XR rendering device 724 have communication components (transmitter, receiver) to enable the XR rendering device 724 to send XR content to the XR headset 720 (e.g., the XR rendering device or its components may be implemented in the "cloud").

[0133] Figure 8 This is a block diagram of an XR rendering device 724 for implementing the methods disclosed herein, according to some embodiments. Figure 8As shown, the XR rendering device 724 may include: a processing circuitry (PC) 802, which may include one or more processors (P) 855 (e.g., one or more general-purpose microprocessors and / or one or more other processors, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc.), which may coexist in a single enclosure or a single data center, or may be geographically distributed (i.e., the XR rendering device 724 may be a distributed computing device); at least one network interface 848 (e.g., a physical interface or an air interface), which includes a transmitter (Tx) 845 and a receiver (R) x) 847 is used to enable the XR rendering device 724 to send data to and receive data from other nodes connected to network 110 (e.g., an Internet Protocol (IP) network), and network interface 848 is (physically or wirelessly) connected to network 110 (e.g., network interface 848 may be coupled to an antenna arrangement including one or more antennas to enable the XR rendering device 724 to wirelessly send / receive data); and storage unit (also referred to as a "data storage system") 808, which may include one or more non-volatile storage devices and / or one or more volatile storage devices (e.g., ... Figure 8 As shown, the particle database (DB) can be stored in storage unit 808. In embodiments where PC 802 includes a programmable processor, a computer-readable storage medium (CRSM) 842 may be provided. CRSM 842 may store a computer program (CP) 843 including computer-readable instructions (CRI) 844. CRSM 842 may be a non-transitory computer-readable medium, such as magnetic media (e.g., hard disk), optical media, storage devices (e.g., random access memory, flash memory), etc. In some embodiments, the CRI 844 of computer program 843 is configured such that, when executed by PC 802, the CRI causes XR rendering device 724 to perform the steps described herein (e.g., the steps described herein with reference to the flowchart). In other embodiments, XR rendering device 724 may be configured to perform the steps described herein without requiring code. That is, for example, PC 802 may include only one or more ASICs. Therefore, the features of the embodiments described herein may be implemented in hardware and / or software.

[0134] Overview of various embodiments

[0135] A1. A method for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of particles, the method comprising: obtaining first target descriptor coordinates, wherein the first target descriptor coordinates identify a first position in an N-dimensional descriptor space, wherein N > 0; defining a first candidate particle set based on the first target descriptor coordinates, wherein the first candidate particle set includes k1 particles from the plurality of particles, wherein k1 > 1; assigning a final weight to each particle in the first candidate particle set; randomly selecting a particle from the first candidate particle set based on the assigned final weight, such that the probability of a given particle in the first candidate particle set being selected is a function of the final weight assigned to the given particle; and rendering the selected particle.

[0136] A2. The method according to embodiment A1, wherein each of the plurality of particles is associated with particle coordinates (e.g., a set of one or more descriptor values), the particle coordinates identifying the position of the particle in the N-dimensional descriptor space, and the first candidate particle set is defined based on the first target descriptor coordinates and the coordinates of the particles.

[0137] A3. The method according to embodiment A2, wherein defining the first candidate particle set based on the first target descriptor coordinates and the particle coordinates includes: determining a nearest neighbor particle set including k1 particles from the plurality of particles, wherein any particle not included in the nearest neighbor particle set is not closer to the first target descriptor coordinates than any particle included in the nearest neighbor particle set, and the candidate set of candidate particles includes the particles included in the nearest neighbor particle set.

[0138] A4. The method according to embodiment A3 further includes: determining k1 based on the number of particles among the plurality of particles having particle coordinates within a threshold distance of the coordinates of the first target descriptor.

[0139] A5. The method according to embodiment A3, wherein each particle included in the plurality of particles is assigned an optimal k value, and the method includes: setting k1 to be equal to the optimal k value assigned to the particle among the plurality of particles having particle coordinates closest to the coordinates of the first target descriptor.

[0140] A6. The method according to embodiment A2, wherein defining the first candidate particle set based on the coordinates of the first target descriptor and the coordinates of the particle includes: determining a first radius value r1; and including in the first candidate particle set each of the plurality of particles having particle coordinates that are within r1 of the coordinates of the first target descriptor.

[0141] A7. The method according to embodiment A6 further includes: determining r1 based on the number of particles among the plurality of particles having particle coordinates within a threshold distance of the coordinates of the first target descriptor.

[0142] A8. The method according to any one of embodiments A1-A7, wherein assigning a final weight to each particle in the first candidate particle set comprises: assigning a first weight to a first particle included in the candidate particle set; determining a first final weight based on the first weight; and assigning the first final weight to the first particle.

[0143] A9. The method according to embodiment A8, wherein the first weight is: a function of the distance between the first target descriptor coordinates and the first particle, a function of the amount of time that has elapsed since the first particle was last rendered, a function of the trajectory associated with the first particle and the target trajectory, or a function of a measurement of the proximity between the first particle and the most recently rendered particle (e.g., a time difference, which indicates the difference between the timestamp used for the first particle and the timestamp used for the most recently rendered particle, assuming that the two particles were extracted from the same record or the same segment).

[0144] A10. The method according to any one of embodiments A1 to A9 further includes: after randomly selecting a particle from the first candidate particle set based on the assigned final weight, assigning a new final weight to the rendered particle or removing the rendered particle from the first candidate particle set; after assigning a new final weight to the rendered particle or removing the rendered particle from the first candidate particle set, randomly selecting another particle from the first candidate particle set based on the currently assigned final weight; and rendering the selected other particle.

[0145] A11. The method according to any one of embodiments A1 to A9 further includes: obtaining second target descriptor coordinates after randomly selecting particles from the first candidate particle set; defining a second candidate particle set based on the second target descriptor coordinates, wherein the second candidate particle set includes k2 particles from the plurality of particles, where k2 > 1; assigning a final weight to each particle in the second candidate particle set; randomly selecting particles from the second candidate particle set based on the assigned final weights, such that the probability of a given particle in the second candidate particle set being selected is a function of the final weight assigned to the given particle; and rendering the particles randomly selected from the second candidate particle set.

[0146] B1. A method for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of particles, the method comprising: obtaining first target descriptor coordinates, wherein the first target descriptor coordinates identify a first position in an N-dimensional descriptor space, wherein N > 0; defining a first candidate particle set based on the first target descriptor coordinates, wherein the first candidate particle set includes k1 particles from the plurality of particles, wherein k1 > 1, and the first candidate particle set includes a first particle and a second particle; assigning a final weight to each particle in the first candidate particle set, wherein assigning a final weight to each particle in the first candidate particle set includes: assigning a first final weight to the first particle and assigning a second final weight to the second particle; selecting a particle from the first candidate particle set based on the assigned final weight; and rendering the selected particle, wherein the first final weight assigned to the first particle is: a function of the distance between the first target descriptor coordinates and the first particle, a function of the amount of time elapsed since the first particle was last rendered, a function of the trajectory and target trajectory associated with the first particle, and / or a function of a measurement of the proximity between the first particle and the most recently rendered particle.

[0147] B2. The method according to embodiment B1, wherein each of the plurality of particles is associated with particle coordinates (e.g., a set of one or more descriptor values), the particle coordinates identifying the position of the particle in the N-dimensional descriptor space, and the first candidate particle set is defined based on the first target descriptor coordinates and the coordinates of the particles.

[0148] B3. The method according to embodiment B2, wherein defining the first candidate particle set based on the first target descriptor coordinates and the particle coordinates includes: determining a nearest neighbor particle set including k1 particles from the plurality of particles, wherein any particle not included in the nearest neighbor particle set is not closer to the first target descriptor coordinates than any particle included in the nearest neighbor particle set, and the candidate set of candidate particles includes the particles included in the nearest neighbor particle set.

[0149] B4. The method according to embodiment B3 further includes: determining k1 based on the number of particles among the plurality of particles having particle coordinates within a threshold distance of the coordinates of the first target descriptor.

[0150] B5. The method according to embodiment B3, wherein each particle included in the plurality of particles is assigned an optimal k value, and the method includes: setting k1 to be equal to the optimal k value assigned to the particle among the plurality of particles having particle coordinates closest to the coordinates of the first target descriptor.

[0151] B6. The method according to embodiment B2, wherein defining the first candidate particle set based on the coordinates of the first target descriptor and the coordinates of the particle comprises: determining a first radius value r1; and including in the first candidate particle set each of the plurality of particles having particle coordinates that are within r1 of the coordinates of the first target descriptor.

[0152] B7. The method according to embodiment B6 further includes: determining r1 based on the number of particles among the plurality of particles having particle coordinates within a threshold distance of the coordinates of the first target descriptor.

[0153] B8. The method according to any one of embodiments B1 to B7 further includes: after selecting a particle from the first candidate particle set based on the assigned final weight, assigning a new final weight to the rendered particle or removing the rendered particle from the first candidate particle set; after assigning a new final weight to the rendered particle or removing the rendered particle from the first candidate particle set, selecting another particle from the first candidate particle set based on the currently assigned final weight; and rendering the selected other particle.

[0154] B9. The method according to any one of embodiments B1 to B8 further includes: after selecting particles from the first candidate particle set, obtaining second target descriptor coordinates; defining a second candidate particle set based on the second target descriptor coordinates, wherein the second candidate particle set includes k2 particles from the plurality of particles, wherein k2 > 1; assigning a final weight to each particle in the second candidate particle set; selecting particles from the second candidate particle set based on the assigned final weights; and rendering particles randomly selected from the second candidate particle set.

[0155] C1. A computer program comprising instructions that, when executed by processing circuitry of an XR rendering apparatus, cause the XR rendering apparatus to perform the method according to any one of claims A1-A11 or B1-B10.

[0156] C2. A carrier comprising a computer program according to embodiment C1, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer-readable storage medium.

[0157] D1. An XR rendering apparatus for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of particles, the XR rendering apparatus comprising: a memory; and processing circuitry coupled to the memory, wherein the apparatus is configured to implement a method comprising: obtaining first target descriptor coordinates, wherein the first target descriptor coordinates identify a first position in an N-dimensional descriptor space, wherein N > 0; defining a first candidate particle set based on the first target descriptor coordinates, wherein the first candidate particle set comprises k1 particles from the plurality of particles, wherein k1 > 1; assigning a final weight to each particle in the first candidate particle set; randomly selecting a particle from the first candidate particle set based on the assigned final weight, such that the probability of a given particle in the first candidate particle set being selected is a function of the final weight assigned to the given particle; and rendering the selected particle.

[0158] D2. The XR rendering apparatus according to embodiment D1, wherein the XR rendering apparatus is further configured to implement the method according to any one of claims A2 to A11.

[0159] E1. An XR rendering apparatus for rendering audio corresponding to an audio recording, wherein the audio recording is divided into a plurality of particles, the XR rendering apparatus comprising: a memory; and processing circuitry coupled to the memory, wherein the apparatus is configured to implement a method comprising: obtaining first target descriptor coordinates, wherein the first target descriptor coordinates identify a first position in an N-dimensional descriptor space, wherein N > 0; defining a first candidate particle set based on the first target descriptor coordinates, wherein the first candidate particle set includes k1 particles from the plurality of particles, wherein k1 > 1. The first candidate particle set includes a first particle and a second particle; assigning a final weight to each particle in the first candidate particle set, wherein assigning a final weight to each particle in the first candidate particle set includes: assigning a first final weight to the first particle and assigning a second final weight to the second particle; selecting a particle from the first candidate particle set based on the assigned final weight; and rendering the selected particle, wherein the first final weight assigned to the first particle is: a function of the distance between the first target descriptor coordinates and the first particle, a function of the amount of time elapsed since the first particle was last rendered, a function of the trajectory and target trajectory associated with the first particle, and / or a function of a measurement of the proximity between the first particle and the most recently rendered particle.

[0160] E2. The XR rendering apparatus according to embodiment E1, wherein the XR rendering apparatus is further configured to implement the method according to any one of claims B2 to B10.

[0161] While various embodiments have been described herein, it should be understood that they are rendered in an illustrative and not limiting manner only. Therefore, the breadth and scope of this disclosure should not be limited to any of the exemplary embodiments described above. Furthermore, unless otherwise stated herein or clearly contradicted by the context, this disclosure covers any combination of the foregoing elements in all possible variations.

[0162] Furthermore, although the processes described above and shown in the accompanying drawings are presented as a sequence of steps, this is for illustrative purposes only. Therefore, it is conceivable that steps may be added, omitted, rearranged, or performed in parallel. Additionally, as used herein, "a" means "at least one" or "one or more".

[0163] References

[0164] [1] US20180068487A1: Systems and methods for simulating the sound of virtual objects using process audio (Disney Enterprises, Inc.).

[0165] [2] Farnell, Andy, “An introduction to procedural audio and its application in computer games”, Audio Mostly Conference, Vol. 23, 2007.

[0166] [3] D. Gabor, “Theory of communication. Part 1: The analysis of information”, Journal of the Institute of Electrical Engineers - Part III: Radio and Communications Engineering, Vol. 93, No. 26, pp. 429–441, 1946.

[0167] [4] D. Schwarz, “Corpus-Based Concatenative Synthesis”, IEEE Journal of Signal Processing, Vol. 24, No. 2, pp. 92-104, March 2007, doi: 10.1109 / MSP.2007.323274.

[0168] [5] D. Schwarz “Distance mapping for corpus-based concatenative synthesis”, Sound and Music Computation (SMC) 2011.

[0169] [6] Aaron Einbond and Diemo Schwarz, “Spatializing timbre with corpus-based concatenative synthesis”, Proceedings of the International Conference on Computer Music, Vol. 2010, International Association for Computer Music, 2010.

[0170] [7] D. Schwarz, “A system for data-driven concatenative sound synthesis”, Digital Audio Effects (DAFx), 2000.

[0171] [8] D. Schwarz et al. “Real-timecorpus-based concatenative synthesis with catart”, 9th International Conference on Digital Audio Effects (DAFx), 2006.

[0172] [9] Diemo Schwarz and Norbert Schnell, “Descriptor-based sound texture sampling”, Sound and Music Computing (SMC), 2010.

[0173]

[10] Diemo Schwarz and Sean O'Leary, "Smooth granular sound texture synthesis by control of timbralsimilarity", Sound and Music Computation (SMC), 2015.

[0174]

[11] Zechen Zhang, Nikunj Raghuvanshi, John Snyder and Steve Marschner, 2019, Acoustic texture rendering for extended sources in complex scenes, ACM Transactions, Graph. 38, 6, Article 222 (December 2019), 9 pages. https: / / doi.org / 10.1145 / 3355089.3356566.

[0175]

[12] Barrass, Stephen, and Matt Adcock, “Interactive granular synthesis of haptic contact sounds,” Audio Engineering Society Conference: 22nd International Conference: Virtual, Synthetic and Entertainment Audio. Audio Engineering Society, 2002.

[0176]

[13] US20190094975A1: Haptic Effect Conversion System Using Granular Synthesis: Immersion Corp.

Claims

1. A method (600) for rendering audio corresponding to an audio recording, wherein, The audio recording is divided into multiple particles, and the method is implemented by an audio renderer, the method including: Obtain (s602) the coordinates of the first target descriptor, wherein the coordinates of the first target descriptor identify the first position in the N-dimensional descriptor space, where N > 0; Based on the coordinates of the first target descriptor, a first candidate particle set is defined (s604), wherein the first candidate particle set includes k1 particles from the plurality of particles, where k1 > 1, and the first candidate particle set includes a first particle and a second particle. Assign a final weight to each particle in the first candidate particle set (s606), wherein assigning a final weight to each particle in the first candidate particle set includes: assigning a first final weight to the first particle and assigning a second final weight to the second particle; Based on the assigned final weights, select particle (s608) from the first candidate particle set; and The selected particles for rendering (s610), where, The first final weight assigned to the first particle is: A function of the amount of time that has elapsed since the first particle was last rendered. A function of the trajectory associated with the first particle and the target trajectory, and / or A function that measures the proximity between the first particle and the most recently rendered particle.

2. The method according to claim 1, wherein, Each of the plurality of particles is associated with particle coordinates, which identify the position of the particle in the N-dimensional descriptor space. The first candidate particle set is defined based on the coordinates of the first target descriptor and the coordinates of the particles.

3. The method according to claim 2, wherein, Defining the first candidate particle set based on the coordinates of the first target descriptor and the coordinates of the particles includes: determining the nearest neighbor particle set of k1 particles from the plurality of particles, wherein, Any particle not included in the nearest neighbor particle set is not closer to the first target descriptor coordinates than any particle included in the nearest neighbor particle set. The candidate set of candidate particles includes particles included in the nearest neighbor particle set.

4. The method according to claim 3, further comprising: k1 is determined based on the number of particles among the plurality of particles that have particle coordinates within a threshold distance of the coordinates of the first target descriptor.

5. The method according to claim 3, wherein, Each particle included in the plurality of particles is assigned an optimal k value, and The method includes setting k1 to an optimal k value equal to the particle among the plurality of particles that has the particle coordinates closest to the coordinates of the first target descriptor.

6. The method according to claim 2, wherein, The first candidate particle set is defined based on the coordinates of the first target descriptor and the coordinates of the particle, including: Determine the first radius value r1; and The first candidate particle set includes each particle among the plurality of particles whose coordinates are within r1 of the coordinates of the first target descriptor.

7. The method according to claim 6, further comprising: r1 is determined based on the number of particles among the plurality of particles that have particle coordinates within a threshold distance of the coordinates of the first target descriptor.

8. The method according to any one of claims 1 to 7, further comprising: After selecting particles from the first candidate particle set based on the assigned final weights, assign new final weights to the rendered particles or remove the rendered particles from the first candidate particle set. After assigning a new final weight to the rendered particle or removing the rendered particle from the first candidate particle set, another particle is selected from the first candidate particle set based on the currently assigned final weight. as well as Render the selected particle.

9. The method according to any one of claims 1 to 8, further comprising: After selecting particles from the first candidate particle set, the coordinates of the second target descriptor are obtained; Based on the coordinates of the second target descriptor, a second candidate particle set is defined, wherein the second candidate particle set includes k2 particles from the plurality of particles, where k2 > 1; Assign a final weight to each particle in the second candidate particle set; Based on the assigned final weights, particles are selected from the second candidate particle set; and Render particles randomly selected from the second candidate particle set.

10. An extended reality XR rendering device (724) for rendering audio corresponding to an audio recording, wherein, The audio recording is divided into multiple particles, and the XR rendering device includes: Memory (842); and Processing circuitry (802), coupled to the memory, wherein the apparatus is configured to implement method (600), the method comprising: Obtain (s602) the coordinates of the first target descriptor, wherein the coordinates of the first target descriptor identify the first position in the N-dimensional descriptor space, where N > 0; Based on the coordinates of the first target descriptor, a first candidate particle set is defined (s604), wherein the first candidate particle set includes k1 particles from the plurality of particles, where k1 > 1, and the first candidate particle set includes a first particle and a second particle. Assign a final weight to each particle in the first candidate particle set (s606), wherein assigning a final weight to each particle in the first candidate particle set includes: assigning a first final weight to the first particle and assigning a second final weight to the second particle; Based on the assigned final weights, select particle (s608) from the first candidate particle set; and The selected particles for rendering (s610), where, The first final weight assigned to the first particle is: A function of the amount of time that has elapsed since the first particle was last rendered. A function of the trajectory associated with the first particle and the target trajectory, and / or A function that measures the proximity between the first particle and the most recently rendered particle.

11. The XR rendering device according to claim 10, wherein, The XR rendering device is also configured to implement the method according to any one of claims 1 to 9.

Images