A method, apparatus, and program for simulating the acoustics of objects generated by the movement of a virtual character.

Abstract

Multiple motion parameters are determined from the simulated object mesh. These motion parameters include information on the virtual character's motion velocity, the object's deformation rate, and the object's deformation region size. Based on the first audio parameter control and the multiple motion parameters, a friction sound is retrieved from a friction audio database containing multiple sample friction sounds. The first audio parameter control is configured to control the characteristics of the friction sound. Based on the second audio parameter control and the multiple motion parameters, a crumpling sound is retrieved from a crumpling audio database containing multiple sample crumpling sounds. The second audio parameter control is configured to control the characteristics of the crumpling sound.

Description

Technical Field

[0001] The present disclosure generally relates to synthesizing sound for simulated physical objects within a virtual environment.

Background Art

[0002] The background description provided herein is for the purpose of generally presenting the background of the disclosure. Within the scope described in this background section, the research of the presently named inventors and aspects of this description that may not have been recognized as prior art at the time of filing are not, explicitly or implicitly, recognized as prior art to the present disclosure.

[0003] Over the past few decades, the creation of realistic visual effects in computer animation and games has made remarkable progress. However, in these computer-generated realities, it is also important to incorporate sound for a more immersive experience. Sound plays an important role in enhancing human perception in the virtual world. For example, environmental sounds are often used to set the mood and pace of a scene, and digital audio effects are used to provide feedback on a player's actions in a video game.

Summary of the Invention

[0004] Aspects of the present disclosure include methods, apparatus, and non-transitory computer-readable storage media for sound synthesis. In some examples, the apparatus for sound synthesis includes a processing circuit.

[0005] A method is provided for simulating the acoustics of an object generated by the movement of a virtual character, in accordance with aspects of this disclosure. In the method, a plurality of motion parameters related to the movement are determined from a simulated object mesh. The plurality of motion parameters represent the motion velocity information of the virtual character, the deformation rate of the object, and the deformation region size of the object. Based on a first audio parameter control and the plurality of motion parameters, friction sounds are obtained from a friction audio database, which includes a plurality of sample friction sounds related to the movement of the virtual character. The first audio parameter control is configured to control the characteristics of the friction sounds (e.g., pitch or volume). Based on a second audio parameter control and the plurality of motion parameters, crumpling sounds are obtained from a crumpling audio database, which includes a plurality of sample crumpling sounds related to the deformation of the object, and the second audio parameter control is configured to control the characteristics of the crumpling sounds (e.g., pitch or volume).

[0006] In the example, the motion velocity information of the virtual character includes multiple motion velocities (e.g., average sliding velocity), the deformation rate of the object indicates the total buckling energy, and the deformation region size of the object indicates the buckling region size. In the example, the first audio parameter control is a first RTPC such as a velocity real-time parameter control (RTPC) (e.g., friction velocity RTPC). The second audio parameter control is a second RTPC. The second RTPC may include a clamp ring size RTPC and a clamp ring strength RTPC.

[0007] In some embodiments, the object includes one of a piece of cloth, a rope, and the hair of a virtual character.

[0008] In the example, vertex information is extracted from multiple vertices of the simulated object mesh in order to determine multiple motion parameters. The vertex information includes the vertex positions and vertex normals of the multiple vertices.

[0009] In the example, to extract vertex information, the vertex positions and normals of multiple vertices are extracted from the CPU skin mesh renderer, based on the fact that the simulated object mesh was generated by a CPU-based simulator. Each vertex position is converted from local coordinates to world coordinates. It is further determined whether each vertex normal is normalized according to a pre-set standard.

[0010] In the example, to extract vertex information, vertex positions and vertex normals are extracted from level of detail (LOD) renderer data, based on the fact that the simulated object mesh was generated by a GPU-based simulator.

[0011] In this embodiment, motion velocity information includes multiple motion velocities (multiple average sliding velocities). To determine multiple motion parameters, the bone positions of multiple bones of a virtual character associated with the motion are determined. One or more nearest vertices among multiple vertices associated with each bone position are determined. Each of the multiple motion velocities is determined based on each bone position and one or more nearest vertices corresponding to each bone position.

[0012] In this embodiment, the mean curvature around each of the multiple vertices is calculated to determine multiple motion parameters. Based on the mean curvature of the multiple vertices, the deformation rate associated with the multiple vertices is determined.

[0013] In the example, to determine multiple motion parameters, the deformation region size (e.g., buckling region size) is determined based on the total number of vertices used to calculate the mean curvature.

[0014] In the example, multiple sampled friction sounds related to the movement of a virtual character are recorded to form a friction audio database. The multiple sampled friction sounds are grouped into multiple subcontainers based on the characteristics of the multiple sampled friction sounds. The multiple subcontainers are stored in a looping random container within the friction audio database.

[0015] In the example, multiple sampled clamping sounds related to the movement of a virtual character are recorded to form a clamping audio database. The multiple sampled clamping sounds are grouped into multiple subcontainers based on the intensity of the multiple sampled clamping sounds. The multiple subcontainers are stored in a discrete blend container according to a predetermined order in the clamping audio database.

[0016] In the example, the second RTPC further includes a clamping strength control configured to select one of several subcontainers in the clamping audio database for playback based on the intensity level of the deformation rate, and a clamping size control configured to control the volume of multiple sample clamping sounds.

[0017] In this embodiment, a correlation is determined between a first audio parameter control (e.g., velocity RTPC) and motion velocity information (multiple motion velocities) of a virtual character. Friction sounds are further extracted from a friction audio database based on the first audio parameter control.

[0018] In the embodiment, a first correlation between the deformation rate and the clamping strength control is determined. A second correlation between the deformation area size and the clamping size control is also determined. The clamping sound is extracted from a clamping audio database based on the clamping strength control and the clamping size control.

[0019] Apparatus is provided in accordance with another aspect of this disclosure. Apparatus comprises a processing circuit. The processing circuit may be configured to perform one or a combination of methods for simulating the acoustics of objects generated by the movement of a virtual character.

[0020] Aspects of this disclosure also provide a non-temporary computer-readable medium that, when executed by at least one processor, stores instructions causing at least one processor to perform one or a combination of methods for simulating the acoustics of objects generated by the movement of a virtual character.

[0021] Further features, properties, and various advantages of the disclosed subject matter will become clearer from the detailed description below and the accompanying drawings. [Brief explanation of the drawing]

[0022] [Figure 1] This is a schematic diagram of an exemplary procedural fabric sound synthesizer according to several embodiments. [Figure 2] This is a schematic diagram of an exemplary motion driver according to several embodiments. [Figure 3] This is a diagram illustrating an exemplary acoustic synthesizer according to several embodiments. [Figure 4] A flowchart illustrating an exemplary process of acoustic simulation according to some embodiments of this disclosure is shown. [Figure 5] This is a schematic diagram of a computer system according to an embodiment. [Modes for carrying out the invention]

[0023] In the present disclosure, BP is an abbreviation for blueprint visual scripting. CSS is an abbreviation for concatenative sound synthesis. FPS is an abbreviation for frames per second. LERP is an abbreviation for linear interpolation. LOD is an abbreviation for level of detail. PCG is an abbreviation for procedural content generation. RC is an abbreviation for random container. RTPC is an abbreviation for real-time parameter control. RTPC is configured to control specific properties of various objects in real time based on real-time parameter changes that occur within a game. The objects can be Wwise objects such as acoustics, containers, control busses, effects, etc. For example, in a racing game, the volume and pitch of a car's engine sound can be controlled based on the speed of the car and the revolutions per minute (RPM) of the engine. UE5 is an abbreviation for Unreal Engine 5. Wwise is an abbreviation for Wave works interactive sound engine.

[0024] In the first relevant example, a data-driven system is provided to generate sounds for physics-based cloth animation using waveform concatenation speech synthesis (CSS). The synthesis process may involve two steps. In the first step, the movement of the cloth animation is analyzed, and two parametric acoustic models (friction and crumpling) are used to generate an initial low-quality target sound. In the second step, a CSS-based synthesis process is used, which can select a sequence of microsound units from a pre-recorded cloth acoustic database. The sequence of microsound units can then be concatenated to match the target sound from the first step. However, the mapping process between the first and second steps relies on an experienced acoustic designer who can manually provide multiple correspondences between the target sound and the database sound. While cloth animation can render high-fidelity sound, the solution in the first related example has long simulation times (e.g., 0.5 to 4 hours), lengthy manual mapping (e.g., approximately 5 to 15 minutes with 2 to 5 iterations), and a limited number of available animations. Therefore, it is difficult to directly apply to real-time applications such as games.

[0025] In a second related example, a simplified data-driven system is provided that uses different trade-off sets to generate clothing sounds that enable real-time synthesis for video games. Similar to the approach described in the first related example, the solution of the second related example can also drive the synthesis of cloth acoustics from actual recordings using a friction and crimping sound model and a manual feature warping technique. Instead of first generating a low-quality target sound, a number of simplified acoustic models can be used. Although a number of simplified acoustic models may output low-precision parameters, the simplified acoustic models can directly drive the waveform continuous synthesis of the recorded database. The manual feature association process according to the second related example can use a user-defined warp function with a small number of parameters to be specified. Furthermore, the manual feature association process does not require an acoustic designer, or alternatively, the acoustic designer is only required once (within a few minutes) and can be reused for the same type of cloth. The computational time associated with the second related example can be significantly reduced (e.g., 0.5 - 6.54 ms). However, there are some trade-offs in the final acoustic quality, particularly notable in the transitions between different acoustic units. Furthermore, the synthesizer of the second related example may be built on a particle-based cloth simulator that only operates on the CPU and may not be able to synthesize the sounds of complex interactions such as the clothing of a character. Furthermore, the friction contact model in the second related example may only calculate the average velocity of all contact particles, and the crimping energy model in the second related example may only consider a single collective buckling event. Therefore, the solution of the second related example may be less accurate for very complex interactions. Finally, the second related example does not include an interface for the parameter adjustment process. Therefore, the user may not be able to easily customize the sound based on their needs to find the "best" (or most appropriate) result.

[0026] This disclosure provides a data-driven system for synthesizing sound for simulated physical objects. The data-driven system can synthesize sound in real time in several respects. Simulated physical objects include physically based object simulations within a virtual environment. For example, physically based objects may include clothing worn by a character in a virtual environment such as a video game. For a given object simulator, such as a CPU / GPU-based clothing simulator, the system can use a clothing motion driver to automatically analyze the geometry of the fabric (or garment) pieces and extract a number of acoustically generated motion parameters to drive synthesis. These acoustically generated motion parameters may be extracted, for example, on a frame-by-frame basis. The acoustically generated motion parameters can then be mapped to corresponding acoustic clips from a pre-recorded database using a cloth synthesizer, thereby allowing the final cloth sound to be rendered in real time. Compared to the solutions in the first or second related example, the system of this disclosure can achieve faster computation speeds while limiting compromises to acoustic quality. The solution of this disclosure can also help improve the efficiency of acoustic designers by reducing the labor-intensive manual work used in related methods such as Foley-based methods.

[0027] In one aspect, the method and / or system of the Disclosure can be applied to any type of simulated object that may generate friction and / or crumpling events, such as in-game virtual environments that use cloth simulations. Aspects of the Disclosure can be applied to certain types of simulated objects, such as non-rigid objects. The method and / or system of the Disclosure can also be extended to work in games and other virtual environments that use other objects that generate sound based on acoustic generation properties similar to cloth, such as rope simulations and hair simulations that require audio output corresponding to rope simulations and hair simulations. For example, the audio system of the Disclosure can be implemented in an Unreal Engine 5 (UE5) environment in the form of a blueprint function (BP) library, and users can easily create game objects within any character class.

[0028] An exemplary objective of this disclosure is to design and implement a data-driven audio system, such as a real-time data-driven system, for object simulations (e.g., cloth, rope, hair). A real-time data-driven audio system can have the ability to automatically and efficiently synthesize audio based on control parameters from the simulation. A main framework of the audio system (100) can be shown in Figure 1. As shown in Figure 1, the audio system (or system) (100) may include a cloth simulator (102) configured to generate a simulated cloth mesh (or garment mesh). In an example, the cloth simulator (102) may be a CPU-based simulator. In an example, the cloth simulator (102) may be a GPU-based simulator. In another example, the cloth simulator (102) may be a software-based simulator. The system (100) may include a motion driver (104). The motion driver (104) may be configured to perform a runtime (or real-time) geometric analysis on the simulated cloth mesh. For example, the motion driver (104) may extract vertex information (e.g., vertex positions and vertex normals) of the simulated cloth mesh. The motion driver (104) can further output several parametric acoustic models (or motion-related parameters) to drive sound synthesis. The system (100) may include an acoustic synthesizer (106) that can automatically detect sound generation events based on reported parameters (e.g., motion-related parameters from the motion driver (104)) and control the playback of cloth audio assets (108) from the audio engine (112). Still referring to Figure 1, the system (100) may also include audio assets (108) that can store sample sounds such as friction sounds and crumpling sounds. The audio engine (112) can be any suitable audio engine such as Wwise, FMOD, SoLoud, etc.The audio engine (112) can compress sample sounds within the audio asset (108), control sound synthesis in the sound synthesizer (106), and control the player interface (110) to play back the synthesized sounds generated by the sound synthesizer (106).

[0029] In embodiments, friction sounds may include sounds produced by surface friction of an object (e.g., cloth). Surface friction may be triggered by the movement (e.g., dancing, jumping, or running) of a virtual character (e.g., a person or animal) in a virtual environment. Crumpling sounds may include sounds produced by surface deformation of an object. Surface deformation of an object may be triggered by the movement of a virtual character in a virtual environment. Different materials or styles of objects may produce different friction sounds or different crumpling sounds. For example, the pitch or volume of a friction sound may be affected by the material (e.g., cotton or nylon) or style (e.g., shirt, T-shirt, or dress) of a piece of clothing.

[0030] It should be noted that the cloth simulator (102), motion driver (104), acoustic synthesizer (106), audio asset (108), player interface (110), and audio engine (112) may be implemented by one or more software modules, hardware modules, or a combination thereof. Software modules (e.g., computer programs) may be developed using a computer programming language, and software modules may be executed by one or more processors to perform the functions of the software module. Hardware modules may be implemented using processing circuits and / or memory to perform functions.

[0031] Compared to the first and second related examples, system (100) in Figure 1 can compute motion (e.g., dancing, running, jumping) using a more efficient processing pipeline. This significantly improves computation speed and reduces memory and CPU power consumption. System (100) can also be more user-friendly for acoustic designers by providing a more user-friendly interface with meaningful parameters to control acoustic properties. Furthermore, system (100) can be made more modular and adaptable to various physics engines (e.g., UE5 Chaos) and audio engines (e.g., Wwise) by separating the audio system from both the cloth simulator (e.g., (102)) and the audio engine (e.g., (112)).

[0032] Figure 2 shows an exemplary motion driver (200) of the present disclosure. In this example, the motion driver (200) may be analogous to the motion driver (104) of system (100). As shown in Figure 2, the motion driver (200) may include three main components: a mesh analyzer (202), a friction driver (204), and a clamping ring driver (206). In some embodiments, the motion driver (200) may further include a character bone analyzer (208). The character bone analyzer (208) may be configured to identify the bone positions of a character.

[0033] Referring still to Figure 2, the mesh analyzer (202) can extract vertex information relevant to motion analysis. In a second relevant example, such vertex information is typically stored in a predefined function that can be called from the simulator. For example, all vertex positions may be accessed using a particle position function from a VICODynamics cloth simulation plugin that runs on the CPU only. However, not all simulators can provide such a function (e.g., UE5 Chaos). Therefore, to increase the versatility of the system (100), the mesh analyzer (202) may be designed to work with both CPU-based and GPU-based cloth simulations. In the case of a CPU-based simulator, the vertex information is typically stored in a CPU skin mesh renderer that the analyzer can access directly. The vertex information can include both vertex positions and vertex normals. In some embodiments, an additional step may be applied to convert all positions to their corresponding world transformations. Local coordinates may represent coordinate values ​​around the character, and world coordinates may represent coordinate values ​​in the entire game environment. The additional step may also ensure that all normals are properly normalized. In the example, normals can be guaranteed to be normalized to a pre-set value (or pre-set standard) according to the simulator's guarantee function. In the case of a GPU-based simulator, vertex information can be accessed from the LOD renderer data. The Mesh Analyzer (202) can scan all clothing data from any thread, then extract the simulated cloth data (or clothing data), and output the corresponding data position, normals, and transformations (e.g., transformation from local coordinates to world coordinates).

[0034] Furthermore, two drive models can be introduced to analyze the associated acoustic generation behavior from cloth (or clothing). As shown in Figure 2, a friction driver (204) and a clamping driver (206) can be applied to analyze the associated acoustic generation behavior. In the relevant real-time system, the friction driver can use position-based collision analysis to calculate the average velocity of all contacting vertices. The size of the contact area is also used as another characteristic. However, this method may only work with cloth (or clothing) of simple shapes such as squares or rectangles with few vertices (e.g., less than 1000). When dressing a character in complex cloth (or clothing) such as a dress, the relevant system may not be able to prove sufficient to approximate all pitch / volume changes with a single set of parameters. Furthermore, if raycasting is used to detect such collisions, the computation may increase when the total number of vertices exceeds 1000.

[0035] In this disclosure, the friction driver (204) can address the aforementioned problems of the associated system by, for example, utilizing the bone position of the character (e.g., a character in a virtual environment) based on the character's posture, and by detecting sliding contact events of an object where sound should be synthesized only around a part of the character, such as around a specific bone (e.g., a bone associated with movement). For example, an acoustic designer can identify which part or bone of the character is most relevant to sound synthesis. The most relevant part may be identified based on which part of the character is in the most contact with the cloth (or clothing).

[0036] In the example, bone positions may be supplied to a friction driver (204) via a character bone analyzer (208). The bone positions, in relation to the character's movement in each frame, may then be used to find the nearest vertex on the cloth based on a user-defined delta distance. Next, the average contact velocity between a bone (or bone region) and the mesh vertices touching the bone region may be calculated using a backward difference method (e.g., a backward difference function). In some embodiments, the average position of all vertices touching the bone region may be used to calculate the average contact velocity.

[0037] For the sake of brevity, the contact area size parameter is not extracted in some examples, as the actual perceptual difference between large and small sliding events may be small. Therefore, the audio database construction process in subsequent steps (e.g., in an acoustic synthesizer) can be simplified.

[0038] The clamping driver (206) can measure changes in the cloth mesh (or garment mesh). Changes can be measured in each frame. Furthermore, the measured changes may include changes in curvature within the cloth mesh. The total buckling energy can be calculated by estimating (or calculating) the mean curvature around each vertex in the cloth mesh using surface triangulation, and then summing the energy changes for each vertex where the sign of the mean curvature changes between two frames. Thus, each vertex can have its own buckling energy if the mean curvature of each vertex changes sign between two frames. The sum of the buckling energies of the vertices is equal to the total buckling energy shown in Figure 2. The total buckling energy can indicate the rate (or degree) of deformation of the cloth. The size of the buckling region can also be used as a feature parameter to drive the synthesis of acoustics. The buckling region size can indicate the size of the deformation region of the cloth.

[0039] To improve the performance of system(100), one or more optimization methods may be added. A first optimization method may impose constraints to prevent the calculated mean curvature from going outside the boundaries (or constraints) at the edges of the cloth mesh, leading to an abnormal total energy. A second optimization method may add a vertex resolution scaling factor to control the total number of mesh vertices that need to be calculated in each frame. Depending on the hardware configuration, the two optimization methods can effectively improve computation speed, as the clamping driver(206) may be the most expensive component in the entire system(100).

[0040] As shown in Figure 2, when synthesizing sounds generated by cloth, the mesh analyzer (202) of the motion driver (200) can receive a simulated cloth mesh and then extract the vertex positions and vertex normals of the cloth mesh. The mesh analyzer (202) may be configured to extract the vertex positions and vertex normals of other object meshes in other examples. The vertex positions can be supplied to the friction driver (204). The friction driver (204) may further receive character pose information (e.g., bone positions) from the character bone analyzer (208). Based on the vertex positions and bone positions, the friction driver (204) can generate motion velocity information, such as the average sliding velocity (or motion velocity) per bone associated with the character's movement. The crumpling driver (206) can then generate crumpling information, such as the total buckling energy and buckling region size, based on the received vertex positions and vertex normals.

[0041] Figure 3 shows an exemplary acoustic synthesizer (300). In this example, acoustic synthesizers (300) and (106) may have the same configuration. As shown in Figure 3, acoustic synthesizers (300) may include four main components: an audio database asset, an audio parameter control (e.g., real-time parameter control (RTPC)), friction and crumpling feature mapping, and an audio player (302). The audio database asset may further include a friction audio database (304) and a crumpling audio database (306). The friction audio database (304) may store sampled friction sounds that can be associated with character movement. The crumpling audio database (306) may store sampled crumpling sounds that can be associated with character movement or deformation of an object (e.g., cloth, rope, or hair).

[0042] An RTPC may be configured to control the playback of sounds extracted from an audio database. For example, an RTPC can control the pitch and / or volume of sounds extracted from an audio database. An RTPC may include multiple velocity (or friction velocity) RTPC(312), size (or crampring size) RTPC(314), and strength (or crampring strength) RTPC(316). A velocity RTPC(312) may be configured to control the playback characteristics (e.g., pitch or volume) of a friction sound based on the character's motion velocity information. A size RTPC(314) may be configured to control the playback characteristics (e.g., volume) of a crampring sound based on the buckling region size. A strength RTPC(316) may be configured to control the playback characteristics (e.g., pitch) of a crampring sound based on buckling energy, such as the strength level of the buckling energy. Friction and crampring feature correspondences may include multiple friction feature warping(308) and crampring feature warping(310). The friction feature warping (308) can map the average sliding velocity to the velocity RTPC (312) based on the warping function or lerp function. The clamping feature warping (310) can map the size RTPC (314) to the buckling region size and the strength RTPC (316) to the total buckling energy.

[0043] The lerp function can perform linear interpolation. The lerp function can typically be used to find a point at a certain distance along a line between two endpoints. lerp allows the user to linearly interpolate between two values. Given minimum and maximum values ​​(a and b) and an interpolated value (t), it can be specified that the interpolated value (t) should return a point on a scale between a and b. Mathematically, the lerp function can be defined, for example, lerp(a,b,t) = a + (ba) × t.

[0044] Still referring to Figure 3, the audio databases (e.g., friction audio database (304) and clamping audio database (306)) may include pre-recorded cloth friction sound clips and clamping sound clips. In the first and second relevant examples, the database construction process is complex and may require the construction of numerous devices, thus necessitating further effort. In the example, a mechanical roller may be used to record friction sounds. According to this example, a first cloth sample is wrapped around the roller, and a second cloth sample is pressed against the first cloth sample. The roller is rotated to record the friction sound. In the example, two custom-made metal handles may be used to record clamping sounds. According to this example, a square piece of cloth is attached to the two metal handles, and a friction-free clamping sound is produced by manually deforming the cloth using an up-and-down shearing motion. Furthermore, each type of friction sound needs to be recorded at varying speeds for three different contact area sizes, and each type of clamping sound needs to be recorded at varying intensities for three different cloth (or garment) sizes. The post-processing step allows for numerous complex data splitting and labeling operations. All processed acoustic units are stored in a large matrix.

[0045] To simplify the process of building the audio database, the system (100) may use an improved recording method. One or more audio databases may be configured to store specific types of sounds. For example, an experienced audio designer may conduct research, and based on the results of that research, specific types of sounds suitable for the improved recording method may be used in the system (100).

[0046] In the example of the friction audio database (304), a large number of sliding motions (or sampled friction sounds) may be recorded first. Acoustic clips associated with sliding motions may be grouped together. The grouped acoustic clips may be placed in several random subcontainers (e.g., RC_A, RC_B, etc.). Another looping random container may be used to enclose all of these subcontainers, and each container may crossfade with another container for a predetermined duration.

[0047] In the example of a crumpling audio database (306), a number of fabric creases (or sampled crumpling sounds) may be recorded first. The fabric creases may then be grouped into random subcontainers. The fabric creases may be grouped based on different intensities (e.g., RC_Low, RC_Med, and RC_High). A discrete blend container may be used to store these subcontainers in a specific order (from low intensity to high intensity). An audio parameter controller (e.g., RTPC) may be added to directly control the playback of the recorded sounds from these databases. Audio parameter controls may be used to enable control of specific characteristics of objects in a virtual environment based on parameter changes occurring within the virtual environment, including changes in characters within the virtual environment. In some examples, specific characteristics of objects may be controlled in real time based on real-time parameter changes. Further post-processing is not required, but may be performed in some aspects. Although RTPC is used as an example of audio parameter control in this disclosure, other audio parameter controls may be used.

[0048] Next, several RTPCs (e.g., (312), (314), and (316)) can be used to control the playback of recorded sounds in an audio database (e.g., (304) and (306)) in order to create correspondence information (or correlation) with a parametric acoustic model from the motion driver (e.g., average sliding velocity, total buckling energy, and buckling region size). For the friction audio database (304), for example, a friction velocity RTPC (312) ranging from 0 to 100 may be created to control the pitch and volume of the friction sound for each bone. In the example, each velocity RTPC can control the friction sound associated with each bone. For the clamping audio database (306), a clamping strength RTPC (316) may be created to select whether to play its clamping random container (RC) based on the input strength level of the total buckling energy, and a clamping size RTPC (314) may be used to control the volume of the output clamping sound based on the buckling region size.

[0049] Table 1 shows exemplary values ​​and functions of the RTPC of this disclosure. For example, as shown in Table 1, the friction velocity RTPC can have values ​​from 0 to 100. The friction velocity RTPC can change the friction volume within the range of -200 dB to 0 dB and the friction pitch within the range of -300 cents to 300 cents. [Table 1]

[0050] A key function of the acoustic synthesizer (300) is the feature mapping process, which maps parameters from the motion driver (e.g., average sliding velocity, total buckling energy, and buckling area size) to corresponding RTPCs or other audio parameter controls to control audio playback. In the case of frictional motion, the lerp friction function (or warping friction function) included in the friction feature warping (308) may be applied to map the average sliding velocity to the velocity RTPC (312). The lerp friction function (or warping friction function) can specify minimum and maximum sliding velocities, or select different strengths based on the area of ​​the cloth mesh. For example, the lerp function can warp each input average sliding velocity around a particular bone into the space of the friction velocity RTPC (312). The user can then specify minimum and maximum sliding velocities to eliminate undesirable motion. The velocity interpolation time can also be set to fine-tune the period over which the velocity RTPC (312) changes toward a new value. The speed RTPC (312) can, accordingly, control the playback of friction sounds extracted from the friction audio database (304).

[0051] In the case of wrinkling operation, the lerp clamping function (or warping clamping function) may be employed in the clamping feature warping (310). This can map the total fabric buckling energy to strength RTPC(316) and the fabric buckling area size to size RTPC(314). To eliminate idle operation, minimum and maximum values ​​for both inputs (e.g., total buckling energy and buckling area size) can also be set by the user. Thus, the lerp clamping warping function may be configured to map the total buckling energy to strength RTPC(316) and the buckling area size to size RTPC(314). Strength RTPC(316) and size RTPC(314) can, accordingly, control the playback of clamping sounds extracted from the clamping audio database (306) based on the total buckling energy and buckling area size. These feature warping techniques (e.g., friction feature warping (308) and crumpling feature warping (310)) can effectively provide real-time feedback to the audio player (302), resulting in higher quality and more accurate cloth audio that appropriately responds to different movements on the cloth (or clothing). Depending on the software engine (or game engine) used, control based on feature warping techniques may be implemented using different player interfaces. For example, in UE5, the BP library may be incorporated to play and control the character's cloth audio.

[0052] This disclosure provides a procedural audio system that can automatically synthesize cloth acoustics based on a pre-recorded audio database and user-defined parameters. The technical solution of this disclosure does not require expensive physical calculations and can improve related real-time data-driven synthesis with more efficient analysis and better output quality. Compared to the first and second related examples, the system (100) of this disclosure can offer the following advantages: (1) Improved Usability: In the first relevant example, an offline pre-calculation process and some manual intervention may be required. In the second relevant example, a more complex database construction process may be required, there may be no interface for parameter tuning, and it may depend on a custom CPU-based cloth simulator. In contrast, the system of the present disclosure requires less manual work, has a simpler database construction process, and has a more intuitive interface for parameter tuning. Furthermore, the system of the present disclosure can generate more complex cloth acoustics (e.g., character clothing) and can work seamlessly with both CPU-based and GPU-based cloth solvers. (2) Improved FPS: The first related example is CPU-intensive and may not be suitable for real-time cloth simulation. The second related example and the system of this disclosure can run in real time. To further compare frames per second (FPS), both the system of the second related example and the system of this disclosure were implemented using the same dress style (3908 vertices) on a dance character with the same audio database. With an AMD Ryzen 3970X 32-Core 3.70GHz CPU and an NVIDIA GeForce RTX 3080 GPU, the average FPS of the system of the second related example was 25.78, while the average FPS of the system of this disclosure was 50.83. (3) More accurate motion: The system of this disclosure can detect frictional motion based on bone position and compute more efficient clamping events that isolate edge cases. In the second relevant example, events are estimated based only on the average parameters of the entire cloth (or garment). The second relevant example is valid for simple square cloth (or garment), but its accuracy may decrease when the character is wearing more complex cloth (or garment). (4) Better sound quality: The synthesis process in the second related example may be based on CSS, in which case transitions between sound units are usually ignored or not properly implemented. The system of this disclosure provides sound designers with greater flexibility and control over editing, tuning, and adjusting source recordings directly within the audio engine by controlling the actual audio synthesis and playback simply by manipulating the RTPC or other audio parameter controls. As a result, the sound quality of the fabric is significantly improved.

[0053] It should be noted that this disclosure is not limited to the embodiments provided in Figures 1-3. This disclosure may include the following embodiments: (1) A procedural acoustic synthesizer for other types of objects (including, but not limited to, ropes, hair, and other types of soft bodies) that use a similar motion and data-driven approach in virtual environments, video games, and other simulations. (2) A motion driver that includes different components and output parameters, such as an aerodynamic driver that may be used to drive the synthesis of wind noise generated by an object. (3) An acoustic synthesizer using different warping functions or feature mapping techniques. The acoustic synthesizer is customizable and can be easily replaced with a different solution, but the key concept is to dynamically map motion parameters to control parameters in an audio database. (4) Although audio databases use different construction processes and control parameters, they can arrive at a similar audio asset structure.

[0054] Figure 4 shows a flowchart illustrating the acoustic simulation process (400) according to an embodiment of the present disclosure. The process begins at (S401) and proceeds to (S410).

[0055] In (S410), several motion parameters associated with the movement are determined from the simulated object mesh. These motion parameters represent the virtual character's motion velocity information, the object's deformation rate, and the object's deformation region size.

[0056] In (S420), a friction sound is retrieved from the friction audio database based on the first audio parameter control and multiple motion parameters. The friction audio database contains multiple sample friction sounds associated with the movement of the virtual character. The first audio parameter control is configured to control the characteristics of the friction sound (e.g., pitch or volume).

[0057] In (S430), a clamping sound is retrieved from the clamping audio database based on the second audio parameter control and multiple motion parameters. The clamping audio database contains multiple sample clamping sounds associated with the deformation of the object, and the second audio parameter control is configured to control the characteristics of the clamping sound (e.g., pitch or volume).

[0058] In the example, the motion velocity information of the virtual character includes multiple motion velocities (e.g., average sliding velocity), the deformation rate of the object indicates the total buckling energy, and the deformation region size of the object indicates the buckling region size. In the example, the first audio parameter control is a first RTPC such as a velocity real-time parameter control (RTPC) (e.g., friction velocity RTPC). The second audio parameter control is a second RTPC. The second RTPC may include a clamp ring size RTPC and a clamp ring strength RTPC.

[0059] In some embodiments, the object includes one of a piece of cloth, a rope, and the hair of a virtual character.

[0060] In the example, vertex information is extracted from multiple vertices of the simulated object mesh in order to determine multiple motion parameters. The vertex information includes the vertex positions and vertex normals of the multiple vertices.

[0061] In the example, to extract vertex information, the vertex positions and normals of multiple vertices are extracted from the CPU skin mesh renderer, based on the fact that the simulated object mesh was generated by a CPU-based simulator. Each vertex position is converted from local coordinates to world coordinates. It is further determined whether each vertex normal is normalized according to a pre-set standard.

[0062] In the example, to extract vertex information, vertex positions and vertex normals are extracted from level of detail (LOD) renderer data, based on the fact that the simulated object mesh was generated by a GPU-based simulator.

[0063] In this embodiment, motion velocity information includes multiple motion velocities (multiple average sliding velocities). To determine multiple motion parameters, the bone positions of multiple bones of a virtual character associated with the motion are determined. One or more nearest vertices among multiple vertices associated with each bone position are determined. Each of the multiple motion velocities is determined based on each bone position and one or more nearest vertices corresponding to each bone position.

[0064] In this embodiment, the mean curvature around each of the multiple vertices is calculated to determine multiple motion parameters. Based on the mean curvature of the multiple vertices, the deformation rate associated with the multiple vertices is determined.

[0065] In the example, to determine multiple motion parameters, the deformation region size (e.g., buckling region size) is determined based on the total number of vertices used to calculate the mean curvature.

[0066] In the example, multiple sampled friction sounds associated with the movement of a virtual character are recorded to form a friction audio database. The multiple sampled friction sounds are grouped into multiple subcontainers based on the characteristics of the multiple sampled friction sounds. The multiple subcontainers are stored in a looping random container within the friction audio database.

[0067] In the example, multiple sampled clamping sounds associated with the movement of a virtual character are recorded to form a clamping audio database. The multiple sampled clamping sounds are grouped into multiple subcontainers based on the intensity of the multiple sampled clamping sounds. The multiple subcontainers are stored in a discrete blend container according to a predetermined order in the clamping audio database.

[0068] In the example, the second RTPC further includes a clamping strength control configured to select one of several subcontainers in the clamping audio database for playback based on the intensity level of the deformation rate, and a clamping size control configured to control the volume of multiple sample clamping sounds.

[0069] In this embodiment, a correlation is determined between a first audio parameter control (e.g., a velocity parameter control) and motion velocity information (multiple motion velocities) of a virtual character. Friction sounds are further extracted from a friction audio database based on the first audio parameter control.

[0070] In the embodiment, a first correlation between the deformation rate and the clamping strength control is determined. A second correlation between the deformation area size and the clamping size control is also determined. The clamping sound is extracted from a clamping audio database based on the clamping strength control and the clamping size control.

[0071] Next, the process proceeds to (S400) and terminates.

[0072] Process (400) may be appropriately adapted. The steps of Process (400) may be changed and / or omitted. Additional steps may be added. Any appropriate order of implementation may be used.

[0073] The aforementioned technology may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 5 shows a computer system (500) suitable for implementing a particular embodiment of the subject matter disclosed.

[0074] Computer software can be coded in any suitable machine code or computer language that follows mechanisms such as assembly, compilation, and linking, to generate code that includes instructions that can be executed directly or through interpretation, microcode execution, etc., by one or more central processing units (CPUs), graphics processing units (GPUs), etc.

[0075] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, game consoles, and devices for the Internet of Things.

[0076] The components shown in Figure 5 with respect to the computer system (500) are illustrative and are not intended to imply any limitation on the scope of use or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should not be construed as having any dependencies or requirements on any one or combination of components described in the exemplary embodiments of the computer system (500).

[0077] The computer system (500) may include certain human interface input devices. Such human interface input devices may respond to input from one or more users, for example, through haptic input (e.g., keyboard, swipe, dataglobe motion), voice input (e.g., voice, clapping), visual input (e.g., gestures), or olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as sound (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images taken from a still camera), or video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0078] The input human interface device may include one or more of the following: keyboard (501), mouse (502), trackpad (503), touchscreen (510), data glove (not shown), joystick (505), microphone (506), scanner (507), and camera (508) (only one of each is shown).

[0079] The computer system (500) may also include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., touchscreens (510), data gloves (not shown), or tactile feedback via joysticks (505), however, there may also be tactile feedback devices that do not function as input devices), audio output devices (e.g., speakers (509), headphones (not shown)), visual output devices (e.g., CRT screens, LCD screens, plasma screens, OLED screens, with or without touchscreen input functionality and with or without tactile feedback functionality, some of which may be capable of outputting two-dimensional visual output or output in more than three dimensions by means such as stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0080] The computer system (500) may also include human-accessible storage devices and their associated media, such as CD / DVD ROM / RW (520) including CD / DVD or similar media (521), thumb drives (522), removable hard disks or solid-state drives (523), legacy magnetic media, such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD-based devices, such as security dongles (not shown), and the like.

[0081] Those skilled in the art will understand that the term “computer-readable medium” as used in relation to the subject matter currently disclosed does not include transmission media, carrier waves, or other transient signals.

[0082] The computer system (500) may also include an interface (554) to one or more communication networks (555). The networks may be, for example, wireless, wireline, or optical. The networks may further be local, wide-area, metropolitan, vehicle and industrial, real-time, latency-tolerant, etc. Examples of networks include local area networks such as Ethernet®, cellular networks including wireless LAN, GSM, 3G, 4G, 5G, LTE, etc., TV wireline or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial TV, and vehicle and factory networks including CANBus. Certain networks generally require an external network interface adapter attached to a specific general-purpose digital port or peripheral bus (549) (e.g., a USB port on the computer system (500)). Others are generally integrated into the core of the computer system (500) by attachment to a system bus as described below (e.g., an Ethernet network to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (500) can communicate with other entities. Such communication can be unidirectional and receivable only (e.g., broadcast TV) or unidirectional and receivable only (e.g., CANBus to a specific CANBus device), or it can be bidirectional to other computer systems using, for example, a local or wide-area digital network. Specific protocols or protocol stacks are available for use with each of the networks and network interfaces described above.

[0083] The above-mentioned human interface devices, human-accessible memory devices, and network interfaces may be attached to the core (540) of the computer system (500).

[0084] The core (540) may include one or more central processing units (CPUs) (541), graphics processing units (GPUs) (542), dedicated programmable processing units in the form of field-programmable gate areas (FPGAs) (543), hardware accelerators (544) for specific tasks, graphics adapters (550), etc. These devices may be connected via a system bus (548) along with read-only memory (ROM) (545), random access memory (RAM) (546), internal mass storage devices such as internal user-inaccessible hard drives, SSDs, etc. (547). In some computer systems, the system bus (548) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripherals may be attached directly to the core's system bus (548) or via a peripheral bus (549). For example, a screen (510) may be connected to a graphics adapter (550). Architectures for peripheral buses include PCI and USB.

[0085] The CPU (541), GPU (542), FPGA (543), and accelerator (544) are capable of executing certain instructions that can be combined to constitute the computer code described above. This computer code can be stored in ROM (545) or RAM (546). Temporary data can also be stored in RAM (546), while persistent data can be stored, for example, in a built-in mass storage device (547). High-speed storage and retrieval to any of the memory devices can be made possible by using cache memory. Cache memory may be closely associated with one or more CPUs (541), GPUs (542), mass storage devices (547), ROMs (545), RAM (546), etc.

[0086] Computer-readable media may contain computer code for performing various computer implementation operations. The media and computer code may be specifically designed and configured for the purposes of this disclosure, or they may be of a type that is well known and available to those skilled in the art in computer software technology.

[0087] As an example, and not as an limitation, a computer system having an architecture (500), specifically a core (540), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) that runs software embodied in one or more tangible computer-readable media. Such computer-readable media may be media related to user-accessible mass storage introduced above, in addition to specific storage devices of the core (540) that have a non-transient nature, such as core-integrated mass storage (547) or ROM (545). Software that implements various aspects of this disclosure is stored in such devices and is executable by the core (540). The computer-readable media may include one or more memory devices or chips, depending on the specific needs. The software can cause the core (540), and specifically the processor (including a CPU, GPU, FPGA, etc.) within it, to execute certain processes or specific parts of certain processes described herein, including defining data structures stored in RAM (546) and modifying such data structures according to processes defined by the software. Additionally, or alternatively, a computer system may provide functionality as a result of logic (e.g., accelerators (544)) hardwired or otherwise embodied in the circuitry, which can operate in place of or with the software to perform specific processes or specific parts of specific processes described herein. References to software may, as necessary, include logic, and vice versa. References to computer-readable media may, as necessary, include circuitry storing software for execution (e.g., integrated circuits (ICs)), circuitry embodying logic for execution, or both. This disclosure also encompasses any suitable combination of hardware and software.

[0088] Any use of “at least one of…” or “one of…” within this disclosure is intended to include any one or a combination of the elements described. For example, references to “at least one of A, B, or C,” “at least one of A, B, and C,” “at least one of A, B, and / or C,” and “at least one of A through C” are intended to include A only, B only, C only, or any combination thereof. References to “one of A or B” or “at least one of A and B” are intended to include A or B or (A and B). The use of “one of…” does not preclude any combination of the elements described, where applicable, such as when the elements are not mutually exclusive.

[0089] While this disclosure has described some aspects, there are various substitutes, replacements, and equivalents within the scope of this disclosure. Therefore, as will be apparent, a person skilled in the art can conceive of numerous systems and methods that embody the principles of this disclosure and thus fall within its spirit and scope, even if not expressly illustrated or described herein.

[0090] [References] This application claims priority based on U.S. Patent Application No. 18 / 447883, filed on 10 October 2023, with the title of the invention "SOUND SYNTHESIZER FOR VIRTUAL ENVIRONMENTS," the prior U.S. Patent Application is incorporated herein by reference in its entirety.

Claims

1. A method by which a processor simulates the acoustics of objects generated by the movement of a virtual character, The step of determining a plurality of motion parameters related to the motion from a simulated object mesh, wherein the plurality of motion parameters represent the motion velocity information of the virtual character, the deformation rate of the object, and the deformation region size of the object, A step of obtaining friction sounds from a friction audio database based on a first audio parameter control and a plurality of motion parameters, wherein the friction audio database includes a plurality of sample friction sounds related to the movement of the virtual character, and the first audio parameter control is configured to control the characteristics of the friction sounds. A step of acquiring a clamping sound from a clamping audio database based on a second audio parameter control and the plurality of motion parameters, wherein the clamping audio database includes a plurality of sample clamping sounds related to the deformation of the object, and the second audio parameter control is configured to control the characteristics of the clamping sound. A method of having.

2. The object includes one of the following: a piece of cloth, a rope, and the hair of the virtual character. The method according to claim 1.

3. The step of determining the aforementioned multiple motion parameters is: The process further includes the step of extracting vertex information for multiple vertices of the simulated object mesh, The vertex information includes the vertex positions of the plurality of vertices and the vertex normals of the plurality of vertices. The method according to claim 1.

4. The step of extracting the vertex information is as follows: Based on the fact that the simulated object mesh was generated by a CPU-based simulator, Extract the vertex positions and vertex normals of the plurality of vertices from the CPU skin mesh renderer. Each of the aforementioned vertex positions is converted from local coordinates to world coordinates, The process further includes determining whether each of the vertex normals is normalized according to a pre-set standard. The method according to claim 3.

5. The step of extracting the vertex information is as follows: Based on the fact that the simulated object mesh was generated by a GPU-based simulator, The process further includes the step of extracting the vertex positions and vertex normals from level-of-detail (LOD) renderer data. The method according to claim 3.

6. The aforementioned motion speed information includes multiple motion speeds, The step of determining the aforementioned multiple motion parameters is: The steps include determining the bone positions of multiple bones of the virtual character related to the aforementioned movement, The steps include determining one or more of the multiple vertices associated with each of the bone positions, A step of determining each of the plurality of motion velocities based on the position of each bone and the one or more nearest vertices corresponding to each bone position. This also includes, The method according to claim 3.

7. The step of determining the aforementioned multiple motion parameters is: The steps include: calculating the mean curvature around each of the aforementioned multiple vertices; A step of determining the deformation rate associated with the plurality of vertices based on the mean curvature of the plurality of vertices. This also includes, The method according to claim 3.

8. The step of determining the aforementioned multiple motion parameters is: The step further includes determining the size of the deformation region based on the total number of vertices used to calculate the mean curvature, The method according to claim 7.

9. The steps include recording the plurality of sample friction sounds related to the movement of the virtual character, The steps include: grouping the multiple sample friction sounds into multiple subcontainers based on the characteristics of the multiple sample friction sounds; The steps include storing the plurality of subcontainers in a looping random container within the friction audio database. The method according to claim 1, further comprising:

10. The steps include recording the plurality of sample clamping sounds related to the movement of the virtual character, The steps include: grouping the multiple sample-clamping sounds into multiple subcontainers based on the intensity of the multiple sample-clamping sounds; The steps include storing the plurality of subcontainers in a discrete blend container in an order pre-set in the clamping audio database, and The method according to claim 1, further comprising:

11. The aforementioned second audio parameter control is, A clamping strength control is configured to select one of the multiple subcontainers in the clamping audio database for playback based on the strength level of the deformation rate, A clamping size control configured to control the volume of the multiple sample clamping sounds, This also includes, The method according to claim 10.

12. The step of obtaining the aforementioned friction sound is: The steps include determining the correlation between the first audio parameter control and the motion speed information of the virtual character, The steps include: extracting the friction sound from the friction audio database based on the first audio parameter control; This also includes, The method according to claim 11.

13. The step of obtaining the aforementioned clamping sound is: The steps include determining a first correlation between the deformation rate and the clamp ring strength control, and a second correlation between the deformation region size and the clamp ring size control, A step of extracting the clamping sound from the clamping audio database based on the clamping strength control and the clamping size control. This also includes, The method according to claim 11.

14. A device that simulates the sound of objects generated by the movement of a virtual character, The device has a processing circuit, which is configured to perform the method described in any one of claims 1 to 13. Device.

15. A program, when executed by a processor, is configured to cause the processor to perform the method described in any one of claims 1 to 13.

Images