Audio visualizing method, device, computer device and storage medium

Abstract

This application relates to an audio visualization method, apparatus, computer device, and storage medium. The method includes: in response to imported target audio, extracting audio intensity information of the target audio and generating an audio intensity curve; based on the audio intensity curve, driving predetermined geometric elements to generate a dynamic visual sequence; converting the dynamic visual sequence into an animation format resource; creating a particle system and loading the animation format resource; and playing the dynamic visual sequence through the particle system to achieve synchronized playback of the target audio visualization. This application enables audio visualization in application environments where audio-driven graphics generation is not feasible.

Description

Technical Field

[0001] This application relates to the field of game audio processing technology, and in particular to an audio visualization method, apparatus, computer device, and storage medium. Background Technology

[0002] In the fields of game development and interactive media, audio visualization is a technique that converts sound data into graphical effects in real time. It works by analyzing the amplitude data in an audio waveform and mapping that value to attributes of graphical elements (such as size, position, or color) to drive basic visual dynamics.

[0003] Currently, in the two mainstream game engines, Unity and Unreal Engine, achieving the above functionality mainly relies on two paths: one is to utilize the engine's native audio components and scripting systems (such as Unity's AudioSource combined with C# scripts, or Unreal's MetaSounds system); the other is to integrate third-party dedicated plugins (such as Unity's AudioVisualizer plugin or Unreal's Audio Analyzer plugin). These technical solutions all follow the basic process of sampling amplitude, preset mapping, and driving graphics.

[0004] However, in application environments lacking built-in audio-driven graphics generation capabilities, the relevant technologies cannot directly create visual effects that are synchronized with audio in real time. Summary of the Invention

[0005] Therefore, it is necessary to address the aforementioned technical problems by providing an audio visualization method, apparatus, computer device, and storage medium that enables audio visualization in application environments where audio-driven graphics generation is not successful.

[0006] Firstly, this application provides an audio visualization method, the method comprising:

[0007] In response to the imported target audio, the audio intensity information of the target audio is extracted, and an audio intensity curve is generated; Based on the audio intensity curve, a dynamic visual sequence is generated by driving predetermined geometric elements. Convert dynamic visual sequences into animation format resources; Create a particle system and load animation format resources. Play dynamic visual sequences through the particle system to achieve synchronized playback of target audio visualization.

[0008] In one embodiment, based on an audio intensity curve, predetermined geometric elements are driven to generate a dynamic visual sequence, including: Multiple vertices are generated based on preset path lines; Assign the values ​​corresponding to the audio intensity curves to the normal direction of the vertices to determine the initial velocity of particle emission in the dynamic visual sequence, and generate a swarm of particles that jump with the audio as the dynamic visual sequence.

[0009] In one embodiment, converting a dynamic visual sequence into an animation format resource includes: Convert dynamic visual sequences into vertex animation texture format resources; The format resources for vertex animation textures include at least: a position map that records the position information of the model's vertices, a lookup map that records the additional attributes of the model's vertices, and a configuration file containing animation metadata.

[0010] In one embodiment, based on an audio intensity curve, predetermined geometric elements are driven to generate a dynamic visual sequence, including: Create a baseline curve containing multiple vertices; Map the numerical values ​​of the audio intensity curve to the positional attribute of at least one endpoint in the reference curve; Configure physical simulation properties for the reference curve and solve them to make the reference curve generate waveform motion driven by the endpoints; The motion data of the baseline curve is mapped to a preset target geometry, which generates a dynamic visual sequence synchronized with the audio.

[0011] In one embodiment, the target geometry is the terrain outline of a preset game scene.

[0012] In one embodiment, mapping motion data of a reference curve to a preset target geometry includes: Establish a correspondence between the vertices of the baseline curve and the vertices of the terrain contour line; The motion data of the vertices of the baseline curve are assigned to the vertices of the corresponding terrain contour lines.

[0013] In one embodiment, configuring physical simulation properties for the baseline curve and solving it includes: Configure fabric simulation properties for the baseline curve; Set at least two endpoints of the baseline curve as fixed constraints; The amplitude and shape of the waveform motion can be controlled by adjusting the tensile and bending stiffness parameters of the fabric.

[0014] In one embodiment, converting a dynamic visual sequence into an animation format resource includes: Convert dynamic visual sequences into skeletal animation format resources.

[0015] In one embodiment, the audio intensity information of the target audio is extracted to generate an audio intensity curve, including: The original audio waveform of the target audio is smoothed to form a continuous audio envelope curve, which serves as the audio intensity curve.

[0016] In one embodiment, smoothing the original audio waveform of the target audio includes: Extract the amplitude channel of the audio waveform; Use packet nodes to extract the peaks of the amplitude channel to generate dynamic curves; The dynamic curve is sampled and smoothed to filter out high-frequency jitter.

[0017] In one embodiment, creating a particle system and loading animation format resources includes: Turn off the default compression option for texture files in animation format resources; Configure materials and shaders in the particle system that match the animation format resources, and set the animation frame length and frame rate corresponding to the target audio duration to load the animation format resources.

[0018] In one embodiment, the method further includes: When playing dynamic visual sequences through a particle system, auxiliary visual effects are added to enhance the visualization of the target audio visualization during synchronized playback; among them, auxiliary visual effects include at least one of the following: halo effect, dynamic lighting, and light effect text.

[0019] Secondly, this application also provides an audio visualization device, the device comprising: The audio intensity extraction module is used to extract the audio intensity information of the imported target audio and generate an audio intensity curve in response to the imported target audio. The visual sequence generation module is used to generate dynamic visual sequences based on audio intensity curves and driven by predetermined geometric elements. The sequence conversion module is used to convert dynamic visual sequences into animation format resources; The audio visualization playback module is used to create particle systems and load animation format resources. It plays dynamic visual sequences through particle systems to achieve synchronized playback of target audio visualizations.

[0020] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0021] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0022] The audio visualization method, apparatus, computer equipment, and storage medium provided in this application import target audio and extract and generate an audio intensity curve characterizing audio changes, providing a precise data-driven source for audio visualization. By programmatically driving pre-determined geometric elements using this audio intensity curve, a high-quality dynamic visual sequence highly matched to the target audio can be generated. By converting the generated dynamic visual sequence into standardized animation format resources, an intermediate technical bridge is built from the content creation stage to the playback stage. Finally, a particle system can be created and these pre-made animation format resources loaded, synchronized with audio playback. In this way, the generation process of audio visualization effects can be divided into two stages: animation data pre-generation and pre-made animation playback. By advancing the computationally intensive creation processes such as audio intensity curve extraction, geometric element driving, and dynamic visual sequence conversion, and delivering the results to the target playback environment in the form of recognizable and playable animation format resources, high-quality, highly synchronized audio visualization effects can be achieved even in application environments lacking built-in audio-driven graphics generation capabilities through standardized resource loading and playback mechanisms, effectively expanding the audiovisual performance capabilities of such application environments. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating an audio visualization method provided in an embodiment of this application; Figure 2 This is a schematic flowchart illustrating the steps for generating a dynamic visual sequence of particle swarms, as provided in an embodiment of this application. Figure 3 This application provides a schematic flowchart illustrating the steps for generating a dynamic visual sequence of waveform curves in an embodiment of the present application. Figure 4 A schematic map of a game scene provided for an embodiment of this application; Figure 5 A schematic diagram of the terrain outline of a game scene provided in an embodiment of this application; Figure 6 A schematic diagram of the curve height of a terrain outline provided in an embodiment of this application; Figure 7 A schematic diagram illustrating the extraction of audio intensity information of a target audio source, provided as an embodiment of this application; Figure 8 A schematic diagram of a musical fountain whose terrain contour line changes with the target audio disturbance, provided as an embodiment of this application; Figure 9 A schematic diagram of a bouncing coil texture for configuring fabric simulation properties provided in an embodiment of this application; Figure 10 A schematic diagram of a skeletal animation format resource provided in an embodiment of this application; Figure 11 This application provides an illustration of the effect of a dynamic visual sequence combining terrain outlines within a preset game scene. Figure 12 This is a schematic diagram of the structure of an audio visualization device provided in an embodiment of this application; Figure 13 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0026] In the fields of game development and interactive media, audio visualization is a technique that converts sound data into graphical effects in real time. It works by analyzing the amplitude data in an audio waveform and mapping that value to attributes of graphical elements (such as size, position, or color) to drive basic visual dynamics.

[0027] Currently, in the two mainstream game engines, Unity and Unreal Engine, achieving the above functionality mainly relies on two paths: one is to utilize the engine's native audio components and scripting systems (such as Unity's AudioSource combined with C# scripts, or Unreal's MetaSounds system); the other is to integrate third-party dedicated plugins (such as Unity's AudioVisualizer plugin or Unreal's Audio Analyzer plugin). These technical solutions all follow the basic process of sampling amplitude, preset mapping, and driving graphics.

[0028] However, in application environments lacking built-in audio-driven graphics generation capabilities, the relevant technologies cannot directly create visual effects that are synchronized with audio in real time.

[0029] In one exemplary embodiment, Figure 1 This is a flowchart illustrating an audio visualization method provided in an embodiment of this application, as shown below. Figure 1As shown, an audio visualization method is provided. This example illustrates the method's application to a terminal. It is understood that this method can also be applied to a server, or to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, and tablets. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. In this embodiment, the method includes the following steps S101 to S104. Wherein: S101. In response to the imported target audio, extract the audio intensity information of the target audio and generate an audio intensity curve.

[0030] In this context, target audio refers to a specific sound file to be visualized, such as game background music or sound effects. Audio intensity information refers to a physical quantity quantized from the sound signal that characterizes changes in its energy or volume, typically amplitude. An audio intensity curve is a dynamic waveform curve with time on the horizontal axis and audio intensity on the vertical axis, used to accurately describe the continuous trend of audio intensity changes over time.

[0031] For example, a terminal or computer device providing audio visualization can load a target audio file via a file import function. The terminal can invoke or configure an audio analysis module to process the imported target audio data, extracting amplitude (or loudness) data and connecting and processing these discrete intensity values ​​into a smooth, continuous curve. This curve can serve as the data source for subsequently driving visual elements.

[0032] As an example, target audio can be imported into visual sequence processing software. Visual sequence processing software refers to professional digital content creation tools with procedural modeling, animation simulation, and data generation capabilities, such as Houdini and Maya. For instance, in Houdini, a target audio file named "theme_song.wav" can be imported through its audio channel processing network. Then, the packet node is used to extract the peaks of the audio waveform amplitude channel, and the extracted peak connections are smoothed using the hysteresis filter node to filter out high-frequency jitter, ultimately generating a smooth audio envelope curve that varies over time, serving as the audio intensity curve.

[0033] S102. Based on the audio intensity curve, drive the predetermined geometric elements to generate a dynamic visual sequence.

[0034] In this context, "driving" refers to the process of controlling or influencing the properties or behaviors of geometric elements using the values ​​of the audio intensity curve as input parameters. Geometric elements refer to manipulable basic graphic units, such as points (vertices), lines (curves), and surfaces (mesh volumes). In practice, geometric elements can be created and driven in visual sequence processing software. A dynamic visual sequence refers to a sequence of time-varying states (such as position, shape, and color) generated by the aforementioned geometric elements based on audio driving, which can be represented as a pre-generated animation.

[0035] For example, the terminal can use the generated audio intensity curve to map the corresponding values ​​(which may be scaled, offset, etc.) to specific attributes of the created geometric elements. For instance, the curve value of the audio intensity curve can be assigned to the initial emission velocity of a group of particles, or the curve value can be assigned to the height of a control point on a curve. Through this mapping relationship, the movement or deformation of the geometric elements will be synchronized with the rhythm and intensity of the target audio, thereby generating an animation sequence that matches the content of the target audio, i.e., a dynamic visual sequence.

[0036] As an example, in the visual sequence processing software Houdini, the terminal can generate 512 vertices based on a preset line representing the stream path; the audio intensity curve generated by S101 in each frame can be assigned to the vector of the normal direction (which can be the Y-axis) of each of these 512 vertices to determine the initial velocity of particle emission; based on this driving force, a particle swarm is generated with these vertices as the emission source, and the spray height and intensity of the particle swarm change in real time with the value of the audio intensity curve, thereby generating a dynamic visual sequence of a "musical fountain" that jumps with the audio.

[0037] S103. Convert the dynamic visual sequence into an animation format resource.

[0038] The conversion step refers to the process of changing the generated animation data format into another standardized or target platform-compatible format through operations such as baking and exporting. In practice, the animation format resources can be resources that a pre-configured game engine can recognize and play. The pre-configured game engine refers to the video game development platform that will ultimately run and display the audio-visual effects; it can be a game engine that does not have direct audio-visualization capabilities for game content creation, such as the Messier engine. The animation format resources refer to standardized data files or resource sets that the target playback environment (such as the game engine) can directly recognize, import, and play through its animation or rendering system.

[0039] For example, a terminal providing audio visualization can, after creating a dynamic visual sequence, use an export function or third-party plugin to package and output the dynamic visual sequence and its related models, textures, and other data into a specific format file compatible with the target playback environment (such as a game engine), which is an animation format resource that can be recognized and played. For example, particle animation can be baked into a vertex animation texture (VAT) resource package, or curve animation can be baked into a skeletal animation FBX file with keyframes.

[0040] As an example, the terminal can convert the generated dynamic visual sequence, similar to a "musical fountain," through the vertex animation texture baking node in the visual sequence processing software Houdini, and output a vertex animation texture format resource. This format resource may include: an EXR format position map that records the position information of the vertices of the particle swarm model for each frame, a TGA format lookup map that records the additional attributes of the vertices (such as normals), and a JSON configuration file that describes the total number of animation frames as 600 and the frame rate as 30fps.

[0041] S104. Create a particle system and load animation format resources. Play the dynamic visual sequence through the particle system to achieve synchronized playback of the target audio visualization.

[0042] In this context, a particle system refers to a component or module used to simulate and render a large number (which can be a preset number) of small, independently moving images (particles) to create complex visual effects (such as smoke, flames, and water flow). In practice, particle systems can be created within a game engine. The loading step refers to importing the corresponding files of externally generated animation format resources into the target playback environment and associating them with specific game objects or components. The synchronized playback step ensures that when the target audio is played in the game, the animation progress presented by the particle system is synchronized with the audio playback progress in time, thus achieving a unified audio-visual effect.

[0043] For example, the terminal can create a particle system component or effect object in the editor of the target playback environment (such as a game engine). The terminal can then assign the generated animation format resources (such as VAT texture sets or skeletal animation files) to the corresponding material, model, or animation controller of the particle system. By configuring the particle system's playback parameters (such as start time and playback speed) and linking them with the playback commands of the in-game audio player, the particle system can accurately reproduce and play a pre-made dynamic visual sequence, synchronized with the target audio, when the game is running.

[0044] As an example, the terminal implementing audio visualization creates a new particle system in the pre-configured game engine Messiah. Then, the position map and lookup map files from the vertex animation texture format resources output by S103 are decompressed using their default compression options in the engine resource management and configured for a dedicated vertex animation material for this particle system. Next, in the material's parameter panel, the animation frame length is set to 600 and the frame rate to 30 to match the 20-second duration of the target audio. Finally, when the target audio is played in the game scene, the loaded animation format resources are played synchronously through this particle system, thus achieving synchronized audio visualization playback.

[0045] In this embodiment, by importing the target audio and extracting and generating an audio intensity curve characterizing audio changes, a precise data-driven source is provided for audio visualization. By programmatically driving predetermined geometric elements using this audio intensity curve, a high-quality dynamic visual sequence highly matched to the target audio can be generated. By converting the generated dynamic visual sequence into standardized animation format resources, an intermediate technical bridge is built between the content creation stage and the playback stage. Finally, a particle system can be created and these pre-made animation format resources loaded, synchronized with audio playback. Thus, the audio visualization effect generation process can be divided into two stages: animation data pre-generation and pre-made animation playback. By prioritizing the computationally intensive creation processes such as audio intensity curve extraction, geometric element driving, and dynamic visual sequence conversion, and delivering the results to the target playback environment in a recognizable and playable animation format resource format, high-quality, highly synchronized audio visualization effects can be achieved even in application environments lacking built-in audio-driven graphics generation capabilities through standardized resource loading and playback mechanisms, effectively expanding the audiovisual performance capabilities of such application environments. In an exemplary embodiment, Figure 2 This application provides a schematic flowchart illustrating the steps for generating a dynamic visual sequence of particle swarms, as shown in the embodiments of this application. Figure 2 As shown, in step S102, based on the audio intensity curve, predetermined geometric elements are driven to generate a dynamic visual sequence, including: S201. Generate multiple vertices based on preset path lines.

[0046] S202. Assign the value corresponding to the audio intensity curve to the normal direction of the vertex to determine the initial velocity of particle emission in the dynamic visual sequence, and generate a particle swarm that jumps with the audio as the dynamic visual sequence.

[0047] In this context, the preset path lines refer to one or more curves pre-drawn by the artist according to scene requirements, which can be used to define the spatial distribution of particle emission sources. Vertices refer to the sampling points that constitute the path lines. The normal direction refers to the direction perpendicular to the tangent of the path curve where the vertex is located in 3D space, and can be used to define the initial ejection direction of the particles. The initial velocity of particle emission refers to the velocity vector of the particle at the moment of its creation. A particle swarm refers to a visual aggregate composed of a large number of particles that follow the same or similar motion rules.

[0048] For example, the terminal can create a custom curve as a preset path line in visual sequence processing software (such as Houdini) based on scene terrain or design requirements, and subdivide the curve to obtain a series of vertices distributed along the curve. The terminal can associate the previously generated audio intensity curve with the time axis, and in each frame, use the value of the audio intensity curve at the current moment as a scalar multiplier, assigning it to the normal direction vector at each vertex. In this way, each vertex can obtain an initial velocity vector in each frame that is proportional to the audio intensity and directed in the normal direction. The terminal can use these vertices as emission sources, setting up particle emitters so that the emitted particles inherit this initial velocity. Thus, when the audio intensity is high, the initial velocity assigned to the vertex is large, and the particles are ejected high and powerfully; when the audio intensity is low, the initial velocity is small, and the particles are ejected low and slowly, thereby generating a group of particles that jump with the rhythm of the music, forming a dynamic visual sequence.

[0049] As an example, in a game map scene, a path can be drawn along the outline of a stream and subdivided into 512 vertices. The values ​​of the processed theme song's audio intensity curve are assigned to the Y-axis (normal) direction of each vertex. When the theme song reaches its climax (high intensity value), all vertices are given an increased initial upward velocity for the particles, simulating a particle fountain effect "gushing" upwards along the stream; when the music subsides, the fountain height decreases.

[0050] In this embodiment, by directly mapping abstract audio intensity data to the physical parameters (initial velocity) of particle emission, precise and programmatic driving of the motion state of the particle swarm can be achieved from the audio signal. This enables the generation of complex particle animations (such as musical fountains) with physical realism and high synchronization with music intensity, providing a reliable and expressive dynamic visual sequence source for achieving such high-quality audio visualization effects in a self-developed game engine that lacks this functionality.

[0051] In an exemplary embodiment, step S103, converting the dynamic visual sequence into an animation format resource, includes: Convert dynamic visual sequences into vertex animation texture format resources; The format resources for vertex animation textures include at least: a position map that records the position information of the model's vertices, a lookup map that records the additional attributes of the model's vertices, and a configuration file containing animation metadata.

[0052] Vertex Animation Texture (VAT) refers to a technique that bakes animation data (such as position, normals, rotations, etc.) of mesh vertices onto a series of texture maps. A position map is a special type of texture where the RGB color values ​​of the pixels encode the 3D spatial coordinates (X, Y, Z) of each vertex in the model mesh at a specific animation frame. A lookup map is another special type of texture used to encode other animation properties of vertices, such as normal direction, tangent, vertex color, or custom properties, which can be used to assist in calculating lighting or performing other shader effects during playback. A configuration file is a structured data file (such as JSON format) used to store metadata about the animation, including information such as the total number of animation frames, frame rate, and the range of vertex position data values ​​(minimum / maximum boundaries), allowing the game engine to correctly parse and play VAT resources.

[0053] For example, the terminal can generate a dynamic visual sequence of particle swarms that jumps with the audio in visual sequence processing software, and then use a VAT export tool or node (such as the Labs Vertex Animation Textures node). This tool can capture the spatial form of the particle swarm in each frame (which can be regarded as a dynamic model composed of vertices) and compress and encode the position information of all vertices in all frames into one or more position maps. At the same time, additional attributes such as vertex normals are encoded into lookup maps. The terminal can also generate a configuration file that records key information such as the animation duration and frame rate. Finally, the terminal can output a resource package containing a static model (representing the form of the first frame), position maps, lookup maps, and configuration file as a vertex animation texture format resource.

[0054] As an example, the terminal can use Houdini's VAT tool to bake a previously generated musical fountain particle animation (600 frames in total). The output can include: an FBX model representing the basic shape of the fountain, an EXR format position map (recording the motion trajectory of all vertices within 600 frames), a TGA format lookup map, and a JSON file describing "600 frames of animation, 30fps frame rate, position range is...".

[0055] In this embodiment, by converting dynamic visual sequences into Vertex Animation Texture (VAT) resources, complex, real-time particle physics simulation results can be solidified into standardized texture and model files. This format can be efficiently read by the shader systems of most game engines, enabling high-performance animation playback on the GPU. Thus, even in self-developed game engines lacking real-time audio-driven particle simulation capabilities, simply loading and playing these pre-computed VAT resources allows for the lossless reproduction of complex particle animation effects generated by visual sequence processing software and strictly synchronized with audio, achieving high-quality audio visualization even in engines lacking this functionality.

[0056] In one exemplary embodiment, Figure 3 This application provides a schematic flowchart illustrating the steps for generating a dynamic visual sequence of waveform curves, as shown in the embodiments below. Figure 3 As shown, in step S102, based on the audio intensity curve, predetermined geometric elements are driven to generate a dynamic visual sequence, including: S301. Create a baseline curve containing multiple vertices.

[0057] S302, Mapping the numerical values ​​of the audio intensity curve to the positional attribute of at least one endpoint in the reference curve.

[0058] S303. Configure physical simulation properties for the reference curve and perform calculations to make the reference curve generate waveform motion under the drive of the endpoints.

[0059] S304. Map the motion data of the reference curve to the preset target geometry, so that the target geometry generates a dynamic visual sequence synchronized with the audio.

[0060] In this context, a baseline curve refers to a basic linear geometry used to receive audio drivers and perform physical simulations, which can be formed by sequentially connecting a series of vertices. Endpoints refer to the starting and / or ending points of the baseline curve. Positional attributes refer to the coordinates (X, Y, Z) of the vertices in three-dimensional space. Physical simulation attributes refer to the set of parameters that assign the geometry the ability to simulate real physical behaviors (such as cloth or soft-body dynamics). Solution refers to the process by which visual sequence processing software calculates the morphological changes of the geometry over time based on physical attributes, constraints, and driving conditions. Waveform motion refers to a propagating motion pattern, similar to waves or the trembling of silk. Target geometry refers to the visual element with a specific artistic shape that is ultimately intended to be presented in the game. Motion data refers to the sequential information of vertex positions changing over time.

[0061] For example, a terminal implementing audio visualization can create a straight line composed of multiple vertices (e.g., 512) as a reference curve in visual sequence processing software. The terminal can assign the value of the audio intensity curve to the position of the starting point (vertex 0) of this reference curve on the Y-axis (height) in real time. Simultaneously, the terminal's Y-axis position can also be determined by multiplying the starting point's position by a coefficient to create a difference. The terminal can add a cloth simulation attribute to this reference curve, setting its starting and ending points as fixed constraints, while the middle section acts as freely deformable cloth. When the audio drives the starting point to move up and down, it's like shaking one end of a taut piece of silk; through the physical calculation of the cloth, the fluctuations propagate along the curve, forming a natural waveform motion. The terminal can completely map (or copy) the waveform motion data obtained from this reference curve onto another preset target geometry with a specific shape (e.g., a winding terrain outline), thereby causing that target geometry to also generate a waveform animation synchronized with the audio.

[0062] As an example, the terminal creates a straight line with 512 vertices in Houdini. The Y-coordinate of vertex 0 is assigned the value of the music wave curve, and the Y-coordinate of vertex 511 is set to 1.2 times that of vertex 0. Cloth simulation is added to the line, fixing the beginning and end points and setting appropriate tension and bending stiffness. The result is a wavy line that undulates with the music. The motion data of each vertex on this wavy line is then copied to a pre-drawn map boundary curve of a game scene, also with 512 vertices, to synchronize the map boundary line with the music wave effect.

[0063] In this embodiment, a layered design—using audio-driven endpoints, physical simulation to transmit fluctuations, and data mapping to target geometry—enables the generation of highly natural and expressive audio-driven continuous surface animations (such as flowing light ribbons). The rhythm of the audio is transformed into visual waveform motion, and physical simulation ensures the smoothness and realism of the motion. This makes it possible to achieve complex linear or curved audio visualization effects beyond simple jumping in the target playback environment (such as a game engine).

[0064] In one exemplary embodiment, Figure 4 A schematic map of a game scene provided for an embodiment of this application; Figure 5 This application provides a schematic diagram of the terrain outline of a game scene as an embodiment of the present application; for example... Figure 4 , Figure 5 As shown, the target geometry is the terrain outline of the preset game scene.

[0065] The preset game scene refers to a specific level or area environment in a game project. The terrain outline refers to the curve used to delineate the terrain boundaries, paths, or specific areas of the scene.

[0066] For example, the terminal can directly design the graphics that will ultimately be used to present the audio visualization as iconic geographical features in the game scene. For instance, the audio waveform can be mapped to the ridgeline of a fairy mountain, the shoreline of a river, or the boundary of a specific magical area, making these scene elements themselves "breathe" or "dance" with the music.

[0067] As an example, in a game scene, the target geometry was set as the outline of a mountain range surrounding a central lake. Using the method described above, the entire mountain range's outline gently undulates in sync with the rhythm of the background music, greatly enhancing the atmosphere and immersion of the scene.

[0068] In this embodiment, by deeply integrating audio visualization with specific game scene art elements (terrain outlines), the visualization effect becomes part of the scene narrative and emotional expression. This not only enables audio visualization in game engines without native functionality, but also creates a unique, high-quality audiovisual experience closely integrated with the game world, enhancing the game's artistic expression.

[0069] In some exemplary embodiments, Figure 6 A schematic diagram of the curve height of a terrain outline provided in an embodiment of this application, such as... Figure 6 As shown, by multiplying time by a custom initial velocity by a custom multiplier, plugging the result into the sine function formula, and then taking the absolute value in the positive direction, a gradual curve can be obtained. The height of the curve can be set to the Y-axis height of the normal at each point. Therefore, the terrain outline of the game scene can be preset as a carrier for audio visualization, allowing the height of the terrain outline to dynamically change according to the intensity of the target audio.

[0070] In an exemplary embodiment, step S101, extracting the audio intensity information of the target audio and generating an audio intensity curve, includes: The original audio waveform of the target audio is smoothed to form a continuous audio envelope curve, which serves as the audio intensity curve.

[0071] The original audio waveform refers to the amplitude signal directly read from the audio file, containing a large amount of high-frequency detail and instantaneous changes. Smoothing refers to using digital signal processing algorithms (such as low-pass filtering and envelope tracking) to filter out high-frequency noise and overly subtle fluctuations in the signal, retaining its main, trend-based intensity changes. The audio envelope curve is a smooth curve that outlines the overall change in the amplitude of the audio signal; it reflects the macroscopic fluctuations in sound intensity, rather than microscopic vibrations.

[0072] For example, the terminal can import the target audio into the CHOP (channel processing) network of the visual sequence processing software to obtain the raw amplitude channel data of the target audio. This raw amplitude channel data may fluctuate very violently and densely. The terminal can use an envelope generator node to track the peaks (peaks) of the raw waveform, thereby obtaining a dynamic curve reflecting the maximum volume by connecting these peak points. To further smooth the changes in the dynamic curve and make it more suitable for driving visual elements, the terminal may use a lag filter node or a similar smoothing node to filter out excessively fast or small jitters remaining in the curve, ultimately obtaining a smooth audio envelope curve that clearly reflects the intensity of musical segments and rhythms, i.e., an audio intensity curve.

[0073] As an example, import a piece of music with strong drum beats; the original amplitude waveform is riddled with spikes. After extracting the peaks using envelope nodes, a curve is obtained that follows the drum beats but still exhibits some subtle variations. Then, a hysteresis filter is used for smoothing, transforming each drum beat into a smooth slope. This smoothed curve can then be used as the audio intensity curve to drive the visual input.

[0074] In this embodiment, by smoothing the original audio waveform to generate an audio envelope curve, the most important rhythmic and dynamic variations for human perception and visual expression can be effectively extracted from the audio, while filtering out irrelevant high-frequency details that may cause visual jitter or affect aesthetics. This ensures that the generated audio intensity curve is stable, controllable, and musically expressive, providing a high-quality driving data foundation for generating smooth, natural, and emotionally resonant dynamic visual sequences in visual sequence processing software. This, in turn, guarantees better visual effects for the audio visualization presented in the game engine.

[0075] In one exemplary embodiment, smoothing the original audio waveform of the target audio includes: Extract the amplitude channel of the audio waveform; Use packet nodes to extract the peaks of the amplitude channel to generate dynamic curves; The dynamic curve is sampled and smoothed to filter out high-frequency jitter.

[0076] In this context, the amplitude channel refers to the data channel in the audio file that stores changes in the amplitude of sound. A packet node is a signal processing node used to track local maxima (peaks) or minima (troughs) of the input signal and output its envelope. Sampling smoothing refers to the process of reducing the frequency of curve sampling points or applying filtering algorithms to reduce the undulation details of the dynamic curve itself, making the dynamic curve smoother.

[0077] For example, extracting the amplitude channel of an audio waveform can be achieved in an audio channel processing node within visual sequence processing software. This audio channel processing node refers to a tool node in visual sequence processing software specifically designed for reading, analyzing, and processing audio signal data, such as the File In node in the CHOP Network. The terminal can import the target audio file using the "File In" node within the CHOP Network of the visual sequence processing software and select the amplitude channel of the target audio as the processing object. The terminal can connect an "Envelope" node to the amplitude channel. This node can analyze the amplitude within each small time window, find its peak points, and connect them to form a dynamic curve that follows the rise and fall of the music's intensity. Since this curve may still contain many small jitters caused by high-frequency components or instantaneous impacts in the music, the terminal can connect a "Lag" node or adjust the sampling rate of the "Resample" node to smooth the curve, filtering out these high-frequency jitters, ultimately obtaining a clean and smooth audio intensity curve.

[0078] As an example, when using Houdini as visual sequence processing software, the path in the Houdini CHOP network is as follows: File In node (reads the amplitude of music.wav), Envelope node (generates the peak envelope), and Lag node (sets the smoothing time constant to filter out fast jitter). The final output channel is the smoothed audio intensity curve.

[0079] In this embodiment, a standardized audio preprocessing workflow consisting of amplitude extraction, packet generation, and smoothing filtering can automatically and repeatably generate high-quality driving curves from any target audio. This can be completed entirely during the content creation stage without relying on the game engine's audio analysis capabilities, ensuring the stability and consistency of the data source. This preprocessing step can improve the high-quality audio performance achieved by game engines without built-in functionality.

[0080] In some exemplary embodiments, Figure 7 This is a schematic diagram illustrating the extraction of audio intensity information of a target audio file, as provided in an embodiment of this application. Figure 7 As shown, by creating a new audio channel processing node and importing the audio file corresponding to the target audio, after retaining only the waveform amplitude channel, an envelope node is used to extract the peak connection to form an initial dynamic curve. This curve is then smoothed by reducing segmentation, manual adjustment, and using a hysteresis filter node, ultimately generating a smooth audio intensity curve. The corresponding attributes can be created in the geometry editing network of visual sequence processing software to read this audio intensity curve data.

[0081] In a visual programming network, the normal vector of each terrain contour vertex can be multiplied by the real-time value of the audio intensity curve. The result is then assigned a velocity attribute to that vertex, serving as the initial velocity for particle emission. A particle emitter inheriting this initial velocity can then be created, and perturbation fields can be added to enrich the motion details. Playing this over time generates a particle fountain animation that moves in sync with the music, such as... Figure 8 As shown, Figure 8 This is a schematic diagram of a musical fountain whose terrain contour line changes with the target audio disturbance, provided as an embodiment of this application.

[0082] In an exemplary embodiment, step S304, mapping the motion data of the reference curve to a preset target geometry, includes: Establish a correspondence between the vertices of the baseline curve and the vertices of the terrain contour line; The motion data of the vertices of the baseline curve are assigned to the vertices of the corresponding terrain contour lines.

[0083] The correspondence refers to establishing a one-to-one matching connection between the vertex sets of two geometries (the baseline curve and the terrain contour). The motion data assignment refers to directly copying or calculating the position offset or absolute coordinate data of the source vertex (the vertex on the baseline curve) in each frame and assigning it to the target vertex (the corresponding vertex on the terrain contour).

[0084] For example, the terminal can ensure in the visual sequence processing software that the reference curve and the target terrain contour have exactly the same number of vertices (e.g., 512 each). A correspondence is established using the vertex index numbers (e.g., 0th, 1st, ..., 511th). After each frame is solved, the terminal can read the position information of each vertex on the reference curve and directly write this information into the position attributes of the vertices with the same index number on the terrain contour. In this way, the shape of the terrain contour can completely inherit the waveform motion of the reference curve.

[0085] As an example, the baseline curve has 512 points (index 0-511), and the terrain contour line also has 512 points (index 0-511). After the calculation is completed, the terminal assigns the Y-axis coordinate change value of the point with index 0 on the baseline curve to the Y-axis coordinate of the point with index 0 on the terrain contour line; the same operation is performed on the points with indices 1, 2, ..., 511, thereby achieving complete mapping of motion data.

[0086] In this embodiment, a highly efficient, accurate, and lossless motion data transfer method is achieved through direct data assignment corresponding to vertex indices. This method ensures that the carefully modulated, audio-driven physical waveform motion in visual sequence processing software can be completely and accurately reproduced onto arbitrarily complex target graphics. This allows artists to freely design the final visual appearance of the graphics without worrying about the loss of motion effects, thus achieving a perfect combination of artistic creativity and audio-driven processes in game engines lacking real-time generation capabilities.

[0087] In an exemplary embodiment, step S303, configuring physical simulation properties for the reference curve and performing a solution, includes: Configure fabric simulation properties for the baseline curve; Set at least two endpoints of the baseline curve as fixed constraints; The amplitude and shape of the waveform motion can be controlled by adjusting the tensile and bending stiffness parameters of the fabric.

[0088] Fabric simulation properties refer to a set of parameters that simulate the physical behavior of fabrics or flexible sheets, including mass, damping, tensile resistance, and bending resistance. Fixed constraints refer to completely locking the position of a vertex or set of vertices in the physical simulation, preventing them from moving under stress. Tensile stiffness refers to the fabric's ability to resist tensile deformation along its length; a higher value means it is less likely to be stretched. Bending stiffness refers to the fabric's ability to resist bending deformation; a higher value makes it stiffer and less prone to soft wrinkles.

[0089] For example, a terminal implementing audio visualization can add a cloth solver to a baseline curve in visual sequence processing software and assign cloth properties to the cloth solver. The terminal can add the first and last vertices of the baseline curve to a group of fixed points, so that the baseline curve maintains a fixed spatial position in the simulation (or is moved by audio but not freely due to internal forces). Artists or technicians can fine-tune the effect by adjusting the "Stretch Stiffness" and "Bend Stiffness" parameters in the cloth properties panel: decreasing the stiffness makes the curve appear very soft, with large fluctuations and slow transmission, like a gentle ribbon; increasing the stiffness makes the curve more elastic, with fast fluctuation transmission and a compact shape, like a vibrating string.

[0090] As an example, the terminal can set both the tensile and bending stiffness of the reference curve to high values, thus simulating a resilient and rhythmic light strip effect. When the audio drive endpoint moves up and down rapidly, the entire curve quickly transmits a clear, crisp waveform, visually powerful.

[0091] In this embodiment, by introducing physically based cloth simulation and making key parameters (stiffness) adjustable, a highly controllable and natural representation of audio-driven waveform motion is provided. This allows the generated dynamic visual sequences to not only be synchronized with the audio, but also possess physical realism and rich artistic adjustability. Thus, it ensures that even in self-developed game engines that cannot perform such real-time physics simulation, the final audio visualization effect can still possess professional-grade dynamic quality.

[0092] In some exemplary implementations, a straight line consisting of multiple vertices can be created as a reference curve. The first and last endpoints of the reference curve are selected as a constraint group. A new audio channel processing node reads the waveform data of the target audio and assigns a custom audio intensity attribute. The real-time value of the audio intensity attribute is multiplied by a magnification factor and then mapped to the Y-axis height position of the starting point of the reference curve. Simultaneously, the Y-axis height position of the ending point of the reference curve is set to the current height of the starting point multiplied by another magnification factor, linking the motion of the two ends. Cloth simulation attributes are added to the reference curve, and the endpoints in the constraint group are set as fixed constraint points. The waveform shape is controlled by adjusting the tensile and bending stiffness parameters of the cloth. Cloth calculation is performed and the calculation accuracy is improved to obtain a curve that oscillates with the audio. The terrain outline of the game scene with the same number of vertices, pre-drawn, is used to inherit all the motion data of the reference curve through point-to-point mapping. After copying multiple curves, they are connected and skinned into a continuous surface mesh using a skinning tool, such as... Figure 9 As shown, Figure 9 This is a schematic diagram of a bouncing coil texture for configuring fabric simulation attributes, provided in an embodiment of this application.

[0093] In an exemplary embodiment, step S103, converting the dynamic visual sequence into an animation format resource, includes: Convert dynamic visual sequences into skeletal animation format resources.

[0094] Among them, skeletal animation format resources refer to standard 3D model animation files that contain skeletal hierarchical structure, skinning weight information, and keyframe animation data, with common formats such as FBX.

[0095] For example, after the terminal generates a dynamic visual sequence of terrain contours in the visual sequence processing software, it can use a skeleton rigging and baking tool (such as the Dem Bones Skinning Converter node). This tool can automatically generate a skeleton chain for this animated curved mesh (dynamic visual sequence) and calculate the weight relationship (skinning) between the mesh vertices and bones. Next, the terminal can bake the vertex animation onto this skeleton, generating standard animation data with keyframes for each bone. Finally, an FBX file with skeletal animation can be output as a skeletal animation format resource.

[0096] As an example, for a mountain outline mesh in a game scene that fluctuates with the music, the terminal can use a plugin to automatically generate 80 bones and attach skins. The waveform animation (600 frames) of the mountain mesh can be baked onto the rotation and translation keyframes of these 80 bones and exported as an FBX file.

[0097] In this embodiment, complex, audio-driven mesh deformation animations generated programmatically can be converted into the game engine's most native and widely supported skeletal animation format, achieving perfect compatibility with the game engine's existing animation pipeline. Skeletal animation serves as the foundation for game engines to process character and object animations, boasting well-optimized playback performance and a mature toolchain. Therefore, this conversion strategy enables complex audio-driven scene animations to be played easily and with high performance, just like playing a character animation, even in game engines lacking dedicated audio visualization capabilities. This reduces integration complexity and ensures operational efficiency.

[0098] In some exemplary implementations, a bone rigging and baking plugin is used to automatically generate bones for animated surface meshes and bake vertex animations into standard keyframe skeletal animation format resources, such as... Figure 10 As shown, Figure 10 This is a schematic diagram of a skeletal animation format resource provided in an embodiment of this application. This skeletal animation format resource is imported into a target game engine, a particle system or model entity is created, and the animation is referenced. After assigning the final texture to the model and adjusting the material parameters, the special effects are placed in the game scene. Finally, auxiliary effects such as glow patches and light-effect text can be combined to form a complete audio-visual effect.

[0099] In an exemplary embodiment, step S104, creating a particle system and loading animation format resources, includes: Turn off the default compression option for texture files in animation format resources; Configure materials and shaders in the particle system that match the animation format resources, and set the animation frame length and frame rate corresponding to the target audio duration to load the animation format resources.

[0100] In this context, texture files refer to position maps and lookup maps used in Vertex Animation Textures (VAT). Default compression options refer to image compression algorithms (such as DXTC / DXTn, ASTC) automatically applied to textures in the game engine's resource import pipeline to improve memory and storage efficiency. Disabling compression means marking the texture import settings as "uncompressed" or using lossless formats such as "RGBA Float" to ensure that precise numerical data encoded in the texture is not corrupted by lossy compression during reading. Matching materials and shaders refer to the material and shader programs specifically written or configured in the game engine for parsing and playing specific animation format resources (such as VAT). Animation frame length refers to the total number of frames contained in the animation resource. Frame rate refers to the number of frames played per second (FPS). Target audio duration refers to the length of the visual audio clip that needs to be synchronized.

[0101] For example, after importing a VAT resource package (containing an EXR position map and a TGA lookup map) into a game engine (such as the Messiah engine), the terminal can locate these two maps in the game engine's resource manager or texture import settings and set their compression format to "None" (uncompressed) or "HDR / High Precision" type to prevent data distortion. The terminal can then create a new material within the game engine, assigning or writing a shader capable of reading the VAT map and calculating vertex positions based on the current time. This VAT material can then be selected for use in the particle system or model rendering component. In the particle system or material's parameter panel, the total number of animation frames and frame rate parameters can be correctly set based on the VAT configuration file (JSON) or known information; for example, a total of 600 frames and a frame rate of 30fps correspond to a 20-second audio clip.

[0102] As an example, in the Messiah engine, the compression options for the pos.exr and lookup.tga textures of the VAT for the musical fountain can be turned off. Create a material named "VAT_Particle", use a custom VAT shader, and connect the two textures to the shader's "Position Tex" and "Lookup Tex" input slots, respectively. In the particle emitter's rendering module, select model rendering and specify the VAT_Particle material, then set "TotalFrames: 600" and "Play Rate: 30" in the material instance parameters.

[0103] In this embodiment, by disabling texture compression to ensure data accuracy, configuring dedicated materials for data decoding, and accurately setting playback parameters based on audio duration, a complete method can be obtained to correctly reproduce and play pre-made animation format resources within the game engine. This series of operations ensures that complex animations created in visual sequence processing software and strictly synchronized with audio can be reproduced losslessly, smoothly, and synchronously in the target game engine. Therefore, the final visualization effect can be achieved even in game engines that lack real-time audio visualization generation capabilities.

[0104] In one exemplary embodiment, a particle system can be created and animation format resources can be loaded in a game engine that does not have built-in audio-driven graphics generation capabilities.

[0105] Game engines that lack built-in audio-driven graphics generation capabilities refer to game development frameworks that do not provide features like Unity's AudioSource visualization API or Unreal Engine's Niagara audio analysis module, or similar functionality that allows for real-time mapping of audio data and control of graphics parameters. These engines typically require developers to implement the complete logic from audio analysis to graphics control themselves.

[0106] For example, the typical scenario targeted by the method in this application is a game engine such as the self-developed Messiah engine. Such game engines may focus on core modules such as graphics rendering, physics, and animation, but do not provide advanced out-of-the-box features for real-time audio and graphics linkage. This application is designed to achieve high-quality audio visualization in such environments.

[0107] As an example, a game project uses a self-developed Messiah engine, which does not provide nodes or functions similar to those in Unreal Engine that directly bind audio spectrum to particle velocity. By applying the various embodiments of the audio visualization method provided in this application, complex mountain contour fluctuations and particle fountain effects that change synchronously with the theme music can be successfully implemented in the game.

[0108] In this embodiment, by explicitly specifying its applicability to game engines that lack built-in audio-driven graphics generation capabilities, the core technical problem to be solved and the uniqueness of its application scenario are clearly defined. This application aims to provide a method that fills the gap in advanced audio visualization content creation capabilities of such engines through external toolchains and data conversion. This allows development teams using such engines to efficiently create top-tier audio visualization content without waiting for updates to the engine's underlying functions or undertaking arduous low-level development.

[0109] In one exemplary embodiment, the method further includes: When playing dynamic visual sequences through a particle system, auxiliary visual effects are added to enhance the visualization of the target audio visualization during synchronized playback; among them, auxiliary visual effects include at least one of the following: halo effect, dynamic lighting, and light effect text.

[0110] Among these, auxiliary visual effects refer to visual elements added outside the core dynamic visual sequence to enrich and enhance the overall visual expressiveness of the core audio-visual effects. A halo is a two-dimensional planar graphic, typically semi-transparent, with color gradients or luminous edges, used to simulate light scattering or create atmosphere. Dynamic lighting refers to an artificial light source whose brightness, color, position, or illumination range can change over time. Illuminated text refers to graphic elements composed of luminous lines or particles used to display specific text content.

[0111] For example, the terminal can create and configure the auxiliary visual effects within the same scene space while playing a dynamic visual sequence through a particle system in the game engine. The terminal can create a planar mesh with a semi-transparent luminescent material as a halo, placing the halo around or in the background of the core visualization effect; it can create point light sources or spotlights as dynamic lights, and make the brightness or color of the dynamic lights change according to the audio intensity curve or the sine function of time; it can create a 3D text model with an luminescent shader as light effect text, displaying lyrics or titles related to the audio content. The activation, parameter changes, or animation playback of these auxiliary visual effects are synchronized with the playback progress of the core particle system or have a preset correlation, which can collectively constitute a more layered and visually impactful composite audio visualization effect.

[0112] As an example, when the terminal plays a mountain outline (dynamic visual sequence) that fluctuates with the music in the Messiah Engine, a huge, color-gradient circular halo can be added behind the mountains to simulate the atmosphere of moonlight; several point light sources can be added to the scene as dynamic lights, with their colors slowly cycling between cool and warm colors; at the same time, a scene name composed of luminous particles can be created above the scene as light effect text, and the overall brightness and darkness of the light effect text can be synchronized with the main rhythm of the audio and flicker slightly.

[0113] In this embodiment, by simultaneously adding auxiliary visual effects such as halo effects, dynamic lighting, and light-effect text on top of the dynamic visual sequence playback, the core effect can be supplemented and enhanced from multiple visual dimensions, including environmental atmosphere, lighting tone, and theme identification. This not only improves the visual richness, layering, and artistic appeal of the overall audio-visual presentation, but also directly conveys semantic information related to the audio through elements such as light-effect text. This allows for the creation of a more immersive and expressive comprehensive audio-visual experience within the game engine, further enhancing the final visualization level and artistic effect of the synchronized audio-visual playback.

[0114] In some exemplary implementations, this skeletal animation resource is imported into the target game engine, a particle system or model entity is created, and the animation is referenced. After assigning final textures to the model and adjusting material parameters, the effects are placed into the game scene. Finally, auxiliary effects such as glow patches and light-effect text can be combined to create a complete audio-visual effect, such as... Figure 11 As shown, Figure 11 This is a schematic diagram illustrating the effect of a dynamic visual sequence combined with terrain outlines within a preset game scene, as provided in an embodiment of this application.

[0115] In some exemplary embodiments, the visual sequence processing software is specifically Houdini software. The game engine is specifically NetEase's self-developed Messiah Engine.

[0116] In some exemplary implementations, more complex waveform control can be achieved through VEX code when generating audio intensity curves, for example, using the formula y = abs(sin(@ptnum). chf(“frequency”) + @Time chf(“speed”)) The `chf("amplitude")` function generates a custom bouncing waveform that can be used for hybrid driving, where `@ptnum` is the vertex number used to determine the phase of different vertices in the waveform.

[0117] In this context, `@ptnum` is a unique index number for each geometric vertex in visual sequence processing software (such as Houdini), typically an integer starting from 0 and incrementing. In the formula, it determines the initial phase position of different vertices in the waveform; vertices with different indices correspond to different points on the sine wave, thus achieving spatial fluctuations along the vertex sequence. `chf("frequency")` can be a user-defined floating-point channel parameter named "frequency". It controls the spatial frequency of the waveform, i.e., the density of fluctuations along the vertex index direction (the direction of `@ptnum`). A larger value results in more cycles within the same vertex index range, leading to denser fluctuations. `@Time` is a system variable representing the current time (usually in seconds or frames). Its introduction allows the waveform to change dynamically over time, generating animation effects. `chf("speed")` is a user-defined floating-point channel parameter named "speed". It controls the rate at which the waveform changes over time, i.e., the time frequency; a larger value results in faster phase changes within the same time interval, making the waveform appear to move faster visually. `chf("amplitude")` is a user-defined floating-point channel parameter named "amplitude"; it controls the amplitude of the waveform, i.e., the overall scaling factor of the final output value y. The larger the value, the greater the range of y-value fluctuation. `sin(·)` is the sine function, which generates a waveform that periodically changes within the interval [-1, 1] based on the input angle value; it is the core mathematical function constituting the basic periodic change. `abs(·)` is the absolute value function; it flips the negative part of the sine function output to a positive value, thus transforming the waveform that originally oscillated between positive and negative values ​​into a waveform with all positive values ​​and bouncing or pulsed characteristics, simulating a trajectory similar to a bouncing ball.

[0118] Alternatively, this formula can be derived by spatially phase-assigning the sine wave (@ptnum). frequency), time-driven dynamics (@Time) speed), amplitude scaling ( A custom bouncing waveform, which can be used for hybrid driving, is generated using amplitude and waveform rectification (abs()). By adjusting the frequency, speed, and amplitude parameters, the spatial distribution, motion speed, and intensity of change of the waveform can be flexibly controlled, thereby driving visual elements to produce complex and rhythmic dynamic effects.

[0119] In some exemplary embodiments, the number of segment vertices of the preset path line can be customized, such as 512, but is not limited to this and can be adjusted according to the effect requirements.

[0120] In some exemplary implementations, the number of bones in skeletal animation can be set as needed and according to the limitations of the game engine, for example, between 80 and 128, to achieve a balance between detail accuracy and performance.

[0121] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0122] The audio visualization device provided in the embodiments of this application is described below. The audio visualization device has the same inventive concept as the audio visualization method described above. The solution to the problem provided by the device is similar to the solution described in the method described above. Therefore, the specific limitations of one or more audio visualization device embodiments provided below can be referred to the limitations of the audio visualization method above. The audio visualization device described below and the audio visualization method described above can be referred to each other, and will not be repeated here.

[0123] In one exemplary embodiment, Figure 12 This is a schematic diagram of the structure of an audio visualization device provided in an embodiment of this application, as shown below. Figure 12 As shown, the audio visualization device 120 includes: an audio intensity extraction module 1210, a visual sequence generation module 1220, a sequence conversion module 1230, and an audio visualization playback module 1240, wherein: The audio intensity extraction module 1210 is used to extract the audio intensity information of the target audio in response to the import of the target audio and generate an audio intensity curve.

[0124] The visual sequence generation module 1220 is used to generate dynamic visual sequences based on audio intensity curves and driven by predetermined geometric elements.

[0125] The sequence conversion module 1230 is used to convert dynamic visual sequences into animation format resources.

[0126] The audio visualization playback module 1240 is used to create a particle system and load animation format resources. It plays dynamic visual sequences through the particle system to achieve synchronized playback of the target audio visualization.

[0127] In an exemplary embodiment, the visual sequence generation module 1220 is used to generate multiple vertices based on preset path lines; assign the values ​​corresponding to the audio intensity curve to the normal direction of the vertices to determine the initial velocity of particle emission in the dynamic visual sequence, and generate a particle swarm that jumps with the audio as the dynamic visual sequence.

[0128] In an exemplary embodiment, the sequence conversion module 1230 is used to convert a dynamic visual sequence into a vertex animation texture format resource; wherein, the vertex animation texture format resource includes at least: a position map that records the position information of the model vertex, a lookup map that records the additional attributes of the model vertex, and a configuration file containing animation metadata.

[0129] In an exemplary embodiment, the visual sequence generation module 1220 is used to create a reference curve containing multiple vertices; map the values ​​of the audio intensity curve to the position attributes of at least one endpoint of the reference curve; configure physical simulation attributes for the reference curve and perform calculations to generate waveform motion of the reference curve under the drive of the endpoints; and map the motion data of the reference curve to a preset target geometry to generate a dynamic visual sequence synchronized with the audio.

[0130] In one exemplary embodiment, the target geometry is the terrain outline of a preset game scene.

[0131] In an exemplary embodiment, the visual sequence generation module 1220 is used to establish a correspondence between the vertices of the reference curve and the vertices of the terrain contour line; and to assign the motion data of the vertices of the reference curve to the corresponding vertices of the terrain contour line.

[0132] In an exemplary embodiment, the visual sequence generation module 1220 is used to configure fabric simulation properties for a reference curve; set at least two endpoints of the reference curve as fixed constraints; and control the amplitude and shape of the waveform motion by adjusting the tensile stiffness and bending stiffness parameters of the fabric.

[0133] In one exemplary embodiment, the sequence conversion module 1230 is used to convert dynamic visual sequences into skeletal animation format resources.

[0134] In an exemplary embodiment, the audio intensity extraction module 1210 is used to smooth the original audio waveform of the target audio to form a continuous audio envelope curve as the audio intensity curve.

[0135] In an exemplary embodiment, the audio intensity extraction module 1210 is used to extract the amplitude channel of the audio waveform; extract the peak of the amplitude channel using a packet node to generate a dynamic curve; and perform sampling smoothing on the dynamic curve to filter out high-frequency jitter.

[0136] In an exemplary embodiment, the audio visualization playback module 1240 is used to disable the default compression option of texture files in the animation format resource; configure materials and shaders that match the animation format resource in the particle system, and set the animation frame length and frame rate corresponding to the target audio duration, so as to load the animation format resource.

[0137] In an exemplary embodiment, the audio visualization playback module 1240 is further configured to add auxiliary visual effects to enhance the visualization of the target audio visualization synchronized playback when playing a dynamic visual sequence through a particle system; wherein, the auxiliary visual effects include at least one of the following: halo, dynamic lighting, and light effect text.

[0138] Each module in the aforementioned audio visualization device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0139] In one exemplary embodiment, this application also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the audio visualization methods described above.

[0140] In one exemplary embodiment, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the audio visualization methods described above.

[0141] In one exemplary embodiment, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the audio visualization methods described in the above embodiments.

[0142] Indicatively, such as Figure 13 As shown, Figure 13 This is a schematic diagram of the internal structure of a computer device 1300 provided in an embodiment of this application. The computer device 1300 can be provided as a server. (Refer to...) Figure 13The computer device 1300 includes a processor 1302, which further includes one or more processors, and memory resources represented by memory 1301 for storing instructions executable by the processor 1302, such as a computer program. The computer program stored in memory 1301 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processor 1302 is configured to execute instructions to perform the audio visualization method of any of the above embodiments. The computer device 1300 may operate on an operating system stored in memory 1301, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0143] The computer device 1300 may also include a power supply component 1303 configured to perform power management of the computer device 1300, a wired or wireless network interface 1304 configured to connect the computer device 1300 to a network, and an input / output (I / O) interface 1305. Wireless operation may be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by a processor, the computer program implements an audio visualization method. The display unit 1307 of the computer device is used to form a visually visible image and may be a display screen, a projection device, or a virtual reality imaging device. The display screen may be an LCD screen or an e-ink display screen. The input device 1306 of the computer device may be a touch layer covering the display screen, buttons, a trackball, or a touchpad located on the computer device casing, or an external keyboard, touchpad, or mouse, etc.

[0144] Those skilled in the art will understand that Figure 13 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0145] It should be noted that if this application involves user information and / or user data, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0146] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0147] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0148] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An audio visualization method, characterized in that, The method includes: In response to the import of target audio, the audio intensity information of the target audio is extracted, and an audio intensity curve is generated; Based on the audio intensity curve, predetermined geometric elements are driven to generate a dynamic visual sequence; The dynamic visual sequence is converted into an animation format resource; A particle system is created and the animation format resource is loaded. The dynamic visual sequence is played through the particle system to achieve synchronized playback of the target audio visualization.

2. The method according to claim 1, characterized in that, The step of generating a dynamic visual sequence by driving predetermined geometric elements based on the audio intensity curve includes: Multiple vertices are generated based on preset path lines; Assign the value corresponding to the audio intensity curve to the normal direction of the vertex to determine the initial velocity of particle emission in the dynamic visual sequence, and generate a particle swarm that jumps with the audio as the dynamic visual sequence.

3. The method according to claim 2, characterized in that, The step of converting the dynamic visual sequence into an animation format resource includes: Convert the dynamic visual sequence into a vertex animation texture format resource; The vertex animation texture format resource includes at least: a position map that records the position information of the model vertices, a lookup map that records the additional attributes of the model vertices, and a configuration file containing animation metadata.

4. The method according to claim 1, characterized in that, The step of generating a dynamic visual sequence by driving predetermined geometric elements based on the audio intensity curve includes: Create a baseline curve containing multiple vertices; Map the numerical values ​​of the audio intensity curve to the positional attribute of at least one endpoint of the reference curve; Configure physical simulation properties for the reference curve and perform calculations to make the reference curve generate waveform motion under the drive of the endpoint; The motion data of the reference curve is mapped to a preset target geometry, so that the target geometry generates a dynamic visual sequence synchronized with the audio.

5. The method according to claim 4, characterized in that, The target geometry is the terrain outline of a preset game scene.

6. The method according to claim 5, characterized in that, The step of mapping the motion data of the reference curve to a preset target geometry includes: Establish a correspondence between the vertices of the reference curve and the vertices of the terrain contour line; The motion data of the vertices of the reference curve are assigned to the vertices of the corresponding terrain contour lines.

7. The method according to claim 4 or 5, characterized in that, The step of configuring physical simulation properties for the baseline curve and performing calculations includes: Configure fabric simulation properties for the baseline curve; Set at least two endpoints of the reference curve as fixed constraints; The amplitude and shape of the waveform motion are controlled by adjusting the tensile and bending stiffness parameters of the fabric.

8. The method according to claim 1 or 4, characterized in that, The step of converting the dynamic visual sequence into an animation format resource includes: The dynamic visual sequence is converted into a skeletal animation format resource.

9. The method according to claim 1, characterized in that, The step of extracting the audio intensity information of the target audio and generating an audio intensity curve includes: The original audio waveform of the target audio is smoothed to form a continuous audio envelope curve, which serves as the audio intensity curve.

10. The method according to claim 9, characterized in that, The smoothing process of the original audio waveform of the target audio includes: Extract the amplitude channel of the audio waveform; The peaks of the amplitude channel are extracted using the packet node to generate a dynamic curve; The dynamic curve is sampled and smoothed to filter out high-frequency jitter.

11. The method according to claim 1, characterized in that, The creation of the particle system and loading of the animation format resources include: Turn off the default compression option for texture files in the animation format resources; Configure materials and shaders that match the animation format resource in the particle system, and set the animation frame length and frame rate corresponding to the target audio duration to load the animation format resource.

12. The method according to claim 1, characterized in that, The method further includes: When the dynamic visual sequence is played through the particle system, auxiliary visual effects are added to enhance the visualization of the target audio-visual synchronized playback; wherein, the auxiliary visual effects include at least one of the following: halo, dynamic lighting, and light effect text.

13. An audio visualization device, characterized in that, The device includes: The audio intensity extraction module is used to extract the audio intensity information of the imported target audio and generate an audio intensity curve in response to the import of the target audio. A visual sequence generation module is used to drive predetermined geometric elements based on the audio intensity curve to generate a dynamic visual sequence. The sequence conversion module is used to convert the dynamic visual sequence into animation format resources; The audio visualization playback module is used to create a particle system and load the animation format resources, and play the dynamic visual sequence through the particle system to achieve synchronized playback of the target audio visualization.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

Images