A method and system for real-time intelligent sound effect matching based on multi-source motion feature fusion

Abstract

This application relates to a method and system for real-time intelligent sound effect matching based on multi-source motion feature fusion. The method includes: performing spatiotemporal alignment preprocessing on acquired motion capture data to obtain motion capture sequence data; extracting features from the motion capture sequence data using a multi-scale temporal convolutional network to obtain motion latent space vectors, which represent the intrinsic attributes of the motion; inputting the motion latent space vectors as conditional vectors into a generative sound effect model to generate a target sound effect spectrogram corresponding to the motion latent space vectors; performing waveform synthesis on the target sound effect spectrogram to obtain a digital audio signal stream; and using the digital audio signal stream to instruct the game engine to read and play matching audio packets from the audio queue based on the timestamp of the current rendering frame. This solution can enhance the immersive audiovisual experience of games.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and in particular relates to a method and system for real-time intelligent sound effect matching based on multi-source motion feature fusion. Background Technology

[0002] In the field of motion and sound effect matching in game development, existing technologies mostly use a combination of single motion feature extraction and manually preset sound effect mapping rules to achieve motion and sound effect matching, and lack an efficient collaborative processing mechanism for motion data and sound effect data.

[0003] On the one hand, feature mining of motion capture data focuses only on single dimensions such as limb movement trajectory and joint angles, failing to effectively integrate multi-source features such as the force, speed, rhythm, or emotional tendency of the movement. This results in insufficient completeness and accuracy of motion feature representation, making it difficult to support accurate sound effect generation. On the other hand, sound effect generation often relies on simple searches of fixed sample libraries, failing to achieve intelligent generation and stylized adaptation of sound effects based on the fused multi-source motion features. Furthermore, the synchronization strategies for motion data processing, sound effect generation, and game engine playback are rudimentary, exhibiting significant processing delays and easily leading to asynchrony between sound effects and motion presentation. Simultaneously, the motion feature processing and sound effect matching logic of existing solutions struggles to adapt to the personalized needs of different game scene styles and character settings. Moreover, under mainstream game hardware configurations, they are prone to excessive resource consumption and unstable frame rates. Overall, they fail to meet the core requirements of modern games for high precision, real-time performance, and high adaptability in motion-sound effect matching, hindering the improvement of game audiovisual immersion. Summary of the Invention

[0004] Therefore, it is necessary to provide a method and system for real-time intelligent sound effect matching based on multi-source motion feature fusion to address the above-mentioned technical problems, which can enhance the immersive audiovisual experience of games.

[0005] Firstly, this application provides a real-time intelligent sound effect matching method based on multi-source action feature fusion, including:

[0006] The acquired motion capture data is preprocessed with spatiotemporal alignment to obtain motion capture sequence data.

[0007] Feature extraction based on a multi-scale temporal convolutional network is performed on motion capture sequence data to obtain action latent space vectors, which are used to characterize the intrinsic attributes of actions.

[0008] The motion latent space vector is input as a conditional vector into the generative sound effect model to generate a target sound effect spectrogram corresponding to the motion latent space vector.

[0009] Waveform synthesis is performed on the target sound effect spectrogram to obtain a digital audio signal stream; the digital audio signal stream is used to instruct the game engine to read the matching audio packet from the audio queue according to the timestamp of the current rendering frame for playback.

[0010] In one embodiment, the acquired motion capture data undergoes spatiotemporal alignment preprocessing to obtain motion capture sequence data, including:

[0011] Obtain the hardware timestamps corresponding to the motion capture data, which includes human joint angle data, limb movement trajectory data, and facial muscle movement data;

[0012] Based on hardware timestamps, human joint angle data, limb movement trajectory data and facial muscle movement data are aligned on the time axis according to the time sequence to generate time-aligned multi-source data frames.

[0013] Based on the preset target sampling frequency, cubic spline interpolation is performed on the multi-source data frames to generate interpolated data frames with uniform time intervals.

[0014] By registering human joint angle data, limb movement trajectory data, and facial muscle movement data at the same time point, motion capture sequence data is obtained.

[0015] In one embodiment, feature extraction based on a multi-scale temporal convolutional network is performed on the motion capture sequence data to obtain an action latent space vector, including:

[0016] A sliding window process is applied to the motion capture sequence data, setting a fixed-length time window to extract motion data segments within the current time window from the motion capture sequence data;

[0017] The action data segments within the current time window are input into a multi-scale temporal convolutional network. The network then performs dilated convolution processing through multiple parallel temporal convolutional blocks to generate multiple temporal feature maps at different scales.

[0018] Multiple temporal feature maps of different scales are spliced ​​and fused along the feature channel dimension to obtain a fused multi-scale temporal feature map.

[0019] The fused multi-scale temporal feature map is input into a fully connected layer for dimensionality reduction mapping to obtain the action latent space vector.

[0020] In one embodiment, the generative sound effect model includes a generator network and a discriminator network, and the training method for the generative sound effect model includes:

[0021] Obtain the historical action latent space vector corresponding to the historical action data, and the real sound effect spectrogram corresponding to the historical action data;

[0022] The historical action latent space vector is used as a conditional vector and input into the generator network to generate a predicted sound effect spectrogram.

[0023] The predicted sound effect spectrogram and the actual sound effect spectrogram are respectively input into the discriminator network, and the discriminator network outputs their respective discrimination scores;

[0024] The adversarial loss function value is calculated based on the discrimination score, and the network parameters of the generator network and the discriminator network are updated by backpropagation based on the adversarial loss function value.

[0025] In one embodiment, waveform synthesis is performed on the target sound effect spectrogram to obtain a digital audio signal stream, including:

[0026] Obtain the scene style label of the current game scene, input the scene style label into the pre-trained style embedding layer for embedding encoding, and obtain the style vector;

[0027] The style vector is input into the first fully connected layer and the second fully connected layer respectively. The scaling parameter vector is obtained by linear transformation through the first fully connected layer, and the translation parameter vector is obtained by linear transformation through the second fully connected layer. The scaling parameter vector and the translation parameter vector are used as affine transformation parameters.

[0028] The target audio effect spectrogram and affine transformation parameters are input into a Wavenet-based vocoder. The waveform is reconstructed through the vocoder, and during the reconstruction process, the feature map of the intermediate layer of the vocoder is affinely transformed according to the affine transformation parameters to obtain a digital audio signal stream.

[0029] Secondly, this application also provides an intelligent real-time sound effect matching system based on multi-source motion feature fusion, used to implement the intelligent real-time sound effect matching method based on multi-source motion feature fusion as provided in the first aspect. The system includes:

[0030] The data preprocessing module is used to perform spatiotemporal alignment preprocessing on the acquired motion capture data to obtain motion capture sequence data;

[0031] The feature extraction module is used to extract features from motion capture sequence data based on a multi-scale temporal convolutional network to obtain action latent space vectors, which are used to characterize the intrinsic attributes of actions.

[0032] The sound effect generation module is used to input the motion latent space vector as a conditional vector into the generative sound effect model to generate a target sound effect spectrogram corresponding to the motion latent space vector.

[0033] The audio matching module is used to synthesize the waveform of the target sound effect spectrogram to obtain a digital audio signal stream; the digital audio signal stream is used to instruct the game engine to read the matching audio packet from the audio queue according to the timestamp of the current rendering frame for playback.

[0034] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a real-time intelligent sound effect matching method based on multi-source motion feature fusion as provided in the first aspect.

[0035] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a real-time intelligent sound effect matching method based on multi-source motion feature fusion as provided in the first aspect.

[0036] The aforementioned intelligent real-time sound effect matching method and system based on multi-source motion feature fusion preprocesses motion capture data to obtain standardized motion capture sequence data. It then uses a multi-scale temporal convolutional network to extract features from the motion capture sequence data to obtain a motion latent space vector that characterizes the intrinsic attributes of the motion. This motion latent space vector is used as a conditional vector input into a generative sound effect model to generate a corresponding target sound effect spectrogram. Finally, waveform synthesis processing is performed on the target sound effect spectrogram to obtain a digital audio signal stream that can be used by the game engine to read audio packets from the audio queue according to the current rendering frame timestamp for playback. This scheme can fully mine and efficiently fuse multi-source motion features, improve the completeness and accuracy of motion feature representation, avoid dependence on a fixed sound effect sample library, achieve intelligent sound effect generation highly adapted to motion features, enhance adaptability to different game scenes and character settings, and optimize system resource consumption during operation, thereby improving the audiovisual immersion of the game. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 A flowchart illustrating a real-time intelligent sound effect matching method based on multi-source motion feature fusion provided in an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram of the structure of an intelligent real-time sound effect matching system based on multi-source motion feature fusion, provided in an embodiment of the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0041] First, a brief introduction to the terms used in the embodiments of this application will be given.

[0042] The motion latent space vector is a low-dimensional dense vector representation obtained by nonlinearly mapping motion capture sequence data through a deep learning network. This vector is not a simple encoding of the action, but rather a high-order abstract expression of the action's intrinsic attributes (such as intensity, speed, rhythm changes, or movement style). By compressing high-dimensional temporal motion data into the latent space, the similarity, gradation, and style differences between actions can be measured through geometric distance in the vector space, laying the mathematical foundation for intelligent mapping from discrete actions to continuous sound effects.

[0043] Generative sound effect models are data-driven models built on deep generative networks. Their core function is to learn the complex nonlinear mapping between action latent space vectors and sound effect spectrograms. Unlike the "tag-sample" retrieval mechanism used in traditional sound effect matching, this model can dynamically generate entirely new, adapted sound effect content based on the input action features, rather than selecting from a fixed library. This generative mechanism allows sound effects to continuously and gradually change with subtle variations in action, breaking the limitations of discrete retrieval and providing technical possibilities for personalized and diversified expression of sound effects.

[0044] Affine transformation parameters are scaling and translation coefficients used to modulate the features of intermediate layers in a neural network during the stylization process of audio signals. These parameters are not fixed values ​​set manually, but rather dynamically generated vectorized representations based on the style tags of the current game scene (such as "ancient style" or "sci-fi"). By applying affine transformation parameters to the internal feature map of the vocoder, it is possible to finely control perceptual dimensions such as timbre and spatial sense while maintaining the core structure of the original sound effects. This achieves end-to-end stylized audio synthesis and avoids the resource redundancy of maintaining multiple sound effect libraries required in traditional methods.

[0045] Based on the above definitions, the implementation environment of the intelligent sound effect real-time matching method based on multi-source motion feature fusion provided in this application embodiment is described. Indicatively, the implementation environment includes: a terminal, sensors, and a processor. The terminal is used to run the game program, render the game screen, and play the matched sound effects, providing an interactive scenario for the user. Sensors include, but are not limited to, inertial measurement units (IMUs), optical motion capture sensors, skeletal joint angle sensors, accelerometers, angular velocity sensors, pressure sensors, and microphone arrays (used to collect acoustic reference signals accompanying movements), used to comprehensively capture the user's or character's limb movement trajectory, movement force, speed, rhythm, and related action-related data. The processor includes, but is not limited to, a central processing unit, a multi-core processor, or an artificial intelligence chip, used to perform core computational tasks such as spatiotemporal alignment preprocessing of motion data, multi-source feature extraction, generative sound effect model calculation, and waveform synthesis; these are not limited here.

[0046] Based on the above explanations of terms and implementation environments, the application scenarios of the embodiments of this application will be described. The intelligent real-time sound effect matching method based on multi-source action feature fusion provided in the embodiments of this application can be applied to scenarios including but not limited to the following:

[0047] In the development of action role-playing games (ARPGs), every attack, block, or spell cast by a player-controlled character contains rich interactive information through variations in the intensity, speed, and rhythm of the movements. Traditional methods typically match fixed sword-swing sound effects based solely on the type of attack action, resulting in a lack of differentiation in the sound effects of light thrusts and heavy slashes. By applying this technical solution, the system captures joint angle changes and limb movement trajectory data generated when the player controls the character, extracting the acceleration and impact characteristics contained within, and generating corresponding latent space vectors for the actions. A generative sound effect model dynamically synthesizes a spectrum of sword-swing sound effects containing different tones and decay durations based on these vectors, ensuring that each attack by the character produces a differentiated sound that perfectly matches the intensity of the action, greatly enhancing the impact and realism of combat scenes.

[0048] In the interactive scenarios of multiplayer competitive games, the complex interactive actions between different characters (such as throws and combos) and the collisions between characters and complex environments (such as metal floors and wooden platforms) place extremely high demands on the real-time matching of sound effects. Traditional solutions often suffer from audio-visual asynchrony due to temporal disorder when processing multi-source data, or are unable to adapt to material changes due to rigid sound effect library retrieval logic. This technical solution ensures precise synchronization of multi-character interactive actions on the timeline by performing spatiotemporal alignment preprocessing on facial muscle movement data and limb movement trajectory data from motion capture data. Subsequently, the motion latent space vector extracted by a multi-scale temporal convolutional network can simultaneously encode action type and environmental interaction information. Based on this, the generative sound effect model synthesizes footsteps and collision sound effects that are matched in real time with the action progress and ground material. Through a timestamp-driven playback mechanism, it is ensured that all sound effects remain synchronized with the action presentation in intense multiplayer battles.

[0049] In open-world adventure games, the same action (such as walking) performed by a character in different environments (such as dark caves, snowy mountaintops, or bustling towns) requires drastically different environmental sound effects and stylized auditory feedback. Traditional sample library-based solutions require pre-storing a large number of scene-specific sound effects, which not only consumes a huge amount of storage space but also results in abrupt and jarring transitions. This technical solution introduces a stylized processing mechanism in the waveform synthesis stage. When a character enters different scenes, the system obtains the scene style tag and generates a corresponding style vector. Through a feature linear modulation module, the feature map of the vocoder's intermediate layer is subjected to an affine transformation, thereby adjusting the reverberation, spatiality, and timbre in real time while preserving the core features of the original footsteps. This allows the same walking action of the character in different scenes to present a natural, smooth, and stylistically distinct auditory effect, enhancing the immersion and exploration enjoyment of the open world.

[0050] As an illustration, the intelligent sound effect real-time matching method based on multi-source motion feature fusion provided in this application embodiment can also be applied to other application scenarios. This is only an example and does not limit the specific application scenarios.

[0051] In one exemplary embodiment, such as Figure 1 As shown, a real-time intelligent sound effect matching method based on multi-source motion feature fusion is provided. This embodiment illustrates the application of this method to a terminal in the aforementioned implementation environment. It is understood that this method can also be applied to a server, and also to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps 101 to 104:

[0052] Step 101: Perform spatiotemporal alignment preprocessing on the acquired motion capture data to obtain motion capture sequence data.

[0053] Specifically, motion capture data can be acquired through various hardware devices. For example, optical motion capture systems (such as Vicon and OptiTrack) capture human joint angles and limb movement trajectories, and in conjunction with inertial measurement units (IMUs) or facial capture devices (such as Faceware), facial muscle movement data is acquired synchronously. This data contains raw information such as limb joint positions, movement trajectories, and movement time. Due to factors such as device synchronization deviations and environmental interference during the acquisition process, the raw data may have problems such as temporal misalignment and inconsistent spatial coordinates. This method first performs spatiotemporal alignment preprocessing on the acquired motion capture data. During the preprocessing, temporal alignment uses linear interpolation or dynamic time warping algorithms to unify the non-isochronous data output by different acquisition devices to the same time sampling frequency. Spatial alignment maps the joint coordinates from different acquisition perspectives to a unified world coordinate system through coordinate system transformation. Through the above processing, temporally continuous and spatially consistent motion capture sequence data is obtained. This data provides a standardized and reliable input basis for subsequent feature extraction, effectively avoiding the impact of defects in the raw data on the accuracy of subsequent processing.

[0054] Step 102: Perform feature extraction on the motion capture sequence data based on a multi-scale temporal convolutional network to obtain the action latent space vector, which is used to characterize the intrinsic attributes of the action.

[0055] Specifically, motion capture sequence data is essentially high-dimensional temporal data containing both temporal and spatial dimensions. To fully explore the multi-source intrinsic features of movements, such as intensity, speed, rhythm, and emotional tendency, this method employs a multi-scale temporal convolutional network to extract features from the motion capture sequence data. The multi-scale temporal convolutional network captures short-term motion detail features (such as instantaneous velocity and force changes in joint movements) and long-term temporal regularity features (such as rhythmic fluctuations and emotional expression trends) in the motion capture sequence by setting convolutional kernels of different sizes. Internally, the network fuses features at different scales through hierarchical convolutional operations, while introducing batch normalization and activation functions to enhance the non-linear expressive power of the features. Finally, a fully connected layer maps the fused high-dimensional features into a low-dimensional motion latent space vector. This vector can comprehensively and accurately represent the core intrinsic attributes of the movement, providing high-quality feature support for subsequent intelligent sound effect matching.

[0056] Step 103: Input the motion latent space vector as a conditional vector into the generative sound effect model to generate a target sound effect spectrogram corresponding to the motion latent space vector.

[0057] For example, the generative sound effect model can adopt a conditional generative adversarial network or conditional variational autoencoder architecture to dynamically generate sound effect data that conforms to specific attributes based on the input conditional information. This method inputs the action latent space vector as a conditional vector into the generative sound effect model. The model associates the action attributes represented by the action latent space vector with the sound effect generation process. First, the latent space vector is expanded in dimension and its features are mapped to match the input dimension of the model's generative network. Then, through the layer-by-layer operation of the generative network, a target sound effect spectrogram corresponding to the action attributes is generated from random noise. The target sound effect spectrogram contains key information such as the frequency distribution and amplitude changes of the sound effect, and its features are highly consistent with the multi-source features of the action. This breaks through the limitations of traditional fixed sample library retrieval and realizes intelligent generation and stylistic adaptation of sound effects. For example, if the action latent space vector encodes a fast boxing action, the generator will output a spectrogram with concentrated high-frequency energy and rapid decay; if it encodes a slow charging action, it will output a spectrogram with rich low-frequency energy and a flat envelope, so as to achieve a continuous nonlinear mapping from the action feature space to the sound effect feature space, thereby improving the sound effect matching efficiency.

[0058] Step 104: Perform waveform synthesis on the target sound effect spectrogram to obtain a digital audio signal stream; the digital audio signal stream is used to instruct the game engine to read the matching audio packet from the audio queue according to the timestamp of the current rendering frame for playback.

[0059] Specifically, this method performs waveform synthesis on the target sound effect spectrogram to obtain a digital audio signal stream that can be recognized and played by the game engine. For example, a WaveNet-based neural vocoder can be used as the waveform synthesis tool. This vocoder can recover the phase information of the audio from the input Mel spectrogram and reconstruct the intermediate audio signal stream in the time domain through sample-by-sample autoregressive generation or parallel generation. Furthermore, to ensure that the audio signal can be accurately synchronized with the game screen, this method can generate a hardware acquisition timestamp of the original motion capture data on which each audio packet is based during or after synthesis. These audio data packets with original timestamps are pushed to the game engine's audio queue, which stores the data packets in order of timestamp. When the game engine renders each frame, the engine obtains the precise rendering timestamp of the current frame and actively pulls the audio data packet that matches the rendering timestamp from the audio queue for playback. For example, when the rendering frame corresponds to time point t, the engine retrieves the audio packet with the timestamp closest to t from the queue for mixing and output. This timestamp-based retrieval mechanism replaces the traditional first-in-first-out push mechanism, fundamentally avoiding audio-visual misalignment caused by data processing latency, and ensuring that the sound effects heard and the visuals seen by players are matched in time during each attack, movement, or interaction.

[0060] The aforementioned intelligent real-time sound effect matching method based on multi-source motion feature fusion preprocesses motion capture data to obtain standardized motion capture sequence data. It then uses a multi-scale temporal convolutional network to extract features from the motion capture sequence data, obtaining a motion latent space vector that characterizes the intrinsic attributes of the motion. This motion latent space vector is used as a conditional vector input into a generative sound effect model to generate a corresponding target sound effect spectrogram. Finally, waveform synthesis processing is performed on the target sound effect spectrogram to obtain a digital audio signal stream that can be used by the game engine to read audio packets from the audio queue according to the current rendering frame timestamp for playback. This scheme can fully mine and efficiently fuse multi-source motion features, improve the completeness and accuracy of motion feature representation, avoid dependence on a fixed sound effect sample library, achieve intelligent sound effect generation highly adapted to motion features, enhance adaptability to different game scenes and character settings, optimize system resource consumption during operation, and improve the audiovisual immersion of the game.

[0061] In one embodiment, the acquired motion capture data undergoes spatiotemporal alignment preprocessing to obtain motion capture sequence data, including:

[0062] Obtain the hardware timestamps corresponding to the motion capture data, which includes human joint angle data, limb movement trajectory data, and facial muscle movement data.

[0063] Specifically, in actual game development or real-time interactive applications, motion capture data typically originates from multiple acquisition devices, such as optical motion capture systems for capturing large joint movements, inertial measurement units for recording minute joint accelerations, and facial capture cameras for capturing facial expression changes. These devices operate independently, each generating data streams at different sampling frequencies, and each data frame is precisely timed by hardware at the moment of generation—a hardware timestamp. This hardware timestamp forms the basis for subsequent multi-source data synchronization, ensuring that data from different sources can be processed under a unified time reference. The data acquired by this method specifically includes human joint angle data, such as the rotation angles of joints like the shoulder, elbow, and wrist; limb movement trajectory data, such as the position coordinate sequence of the wrist and ankle in three-dimensional space; and facial muscle movement data, such as the displacement or activation intensity of key points like the corners of the mouth, eyebrows, and eyes. This raw data with hardware timestamps forms the input basis for subsequent processing.

[0064] Based on hardware timestamps, human joint angle data, limb movement trajectory data, and facial muscle movement data are aligned along the time axis in chronological order to generate time-aligned multi-source data frames.

[0065] Specifically, because different data streams differ in their acquisition start time and sampling interval, directly fusing the original data can lead to time misalignment in the representation of the same action event across different data channels. To address this issue, this method uses hardware timestamps as the sole reference to reorder all data streams. For example, assuming the timestamps for optical data are t1, t2, and t3, and the timestamps for inertial data are τ1, τ2, τ3, and τ4, this method establishes a unified time axis and inserts all data points into their corresponding positions on that time axis according to their timestamp values. For any selected time reference point, if a data stream does not have a corresponding original data point at that moment, it is temporarily left blank. After this alignment operation, the originally scattered and asynchronous data streams are organized into a series of time-aligned multi-source data frames. Each frame is actually a data set containing multiple timestamp indices, providing a structured data foundation for subsequent interpolation processing.

[0066] Based on the preset target sampling frequency, cubic spline interpolation is performed on the multi-source data frames to generate interpolated data frames with uniform time intervals.

[0067] Specifically, after time axis alignment, the data distribution along the time axis remains non-uniform, which is disadvantageous for deep learning models that require a fixed input size. To address this, this method sets a uniform target sampling frequency, such as 120Hz. For channels with missing data at the target time point, the method employs cubic spline interpolation to complete the data. The principle of cubic spline interpolation is to construct a cubic polynomial between adjacent known data points. This polynomial not only passes through the known points but also ensures continuous first and second derivatives at the connection points, thus forming a smooth curve. For example, for a target interpolation time t between two known times t1 and t2 for joint angle data, the smooth interpolation result at time t can be calculated by solving a system of linear equations based on the angle values ​​and their derivatives at times t1 and t2. Compared to linear interpolation, cubic spline interpolation can better maintain the acceleration continuity of the original motion and avoid abrupt changes. The data frames obtained after interpolation have a strictly uniform interval in the time dimension, and the data in each frame are calculated based on the original measurement values ​​through physically reasonable mathematical methods, ensuring the smoothness and accuracy of the data.

[0068] By registering human joint angle data, limb movement trajectory data, and facial muscle movement data at the same time point, motion capture sequence data is obtained.

[0069] Specifically, after time alignment and interpolation, various types of data are synchronized in time, but they are still defined in different spatial coordinate systems: optically captured trajectory data is typically described in the world coordinate system, joint angle data provided by the inertial measurement unit (IMU) is defined in the sensor's own local coordinate system, while facial capture data depends on the camera's image coordinate system or the head's local coordinate system. To fuse these heterogeneous data into a unified motion representation, this method performs spatial registration, mapping all data to the same global reference coordinate system. For example, this method can first determine a global coordinate system, such as the world coordinate system of the optical motion capture system; for the human joint angle data measured by the IMU, using pre-calibrated sensor installation position and orientation information, a rotation matrix is ​​used to convert the angles in the local coordinate system into spatial poses in the global coordinate system; specifically, let the rotation matrix from the sensor's local coordinate system to the global coordinate system be... Then the joint orientation in the global coordinate system can be determined by... Multiplying by a local quaternion yields the result. For facial muscle motion data, based on the spatial relationship between the facial capture device and head tracking markers, the 2D image features are first converted into 3D points in the camera coordinate system using camera intrinsics. Then, based on the position and orientation of the head tracking markers in the global coordinate system, these 3D points are transformed into the global coordinate system, thus precisely binding facial expression changes to the overall head movement. After completing the coordinate transformation of all data sources, the spatially registered joint angles, limb trajectories, and facial muscle data at the same time point are combined into a complete multi-source data frame according to a predefined format. Arranging this data frame in chronological order yields spatially aligned motion capture sequence data. Each frame in this sequence data fully depicts the human body's motion state within a unified spatiotemporal framework, providing spatially consistent and temporally regular input for subsequent feature extraction based on graph convolutional networks or temporal convolutional networks.

[0070] In one embodiment, feature extraction based on a multi-scale temporal convolutional network is performed on the motion capture sequence data to obtain an action latent space vector, including:

[0071] A sliding window process is applied to the motion capture sequence data, setting a fixed-length time window to extract motion data segments within the current time window from the motion capture sequence data.

[0072] Specifically, since a single data frame can only describe an instantaneous posture and cannot reflect the dynamic evolution of the action, it is necessary to combine consecutive frames into segments to extract temporal features. This method employs a sliding window mechanism, setting a fixed-length time window, such as containing 64 or 128 frames. This length needs to cover the complete cycle of common actions; for example, a punching action corresponds to approximately 50 to 60 frames at a 120Hz sampling rate. The sliding window slides across the motion capture sequence data with a preset step size, starting from the beginning of the sequence, and each time capturing all data frames within the window's coverage area, forming a three-dimensional motion data segment. The dimension of this motion data segment can be represented as... ,in The time window length, The number of feature channels for each data frame (e.g., joint angles, trajectory coordinates, facial muscle activation values, etc.). This represents the number of skeletal nodes or facial keypoints. This operation segments continuous streaming data into independent samples that the model can process, while preserving the temporal continuity and contextual information of the actions, laying the foundation for subsequent temporal modeling.

[0073] The action data segments within the current time window are input into a multi-scale temporal convolutional network. The network then performs dilated convolution on multiple parallel temporal convolutional blocks to generate multiple temporal feature maps at different scales.

[0074] Specifically, the speed and rhythm of human or character movements are diverse. Fast movements, such as snapping fingers, may only last a few frames, while slow movements, such as the cloud hands in Tai Chi, may span dozens of frames. A single fixed-scale convolutional kernel can hardly capture such cross-temporal scale changes simultaneously. To address this, this method can employ a multi-scale temporal convolutional network architecture, with multiple parallel temporal convolutional blocks within the network, each configured with a different dilation coefficient. For the input action data fragment Each temporal convolutional block extracts features through dilated convolution. Dilated convolution expands the receptive field by inserting holes between the kernel elements. Its computation can be represented as: for the input one-dimensional temporal features... and convolution kernel Dilated convolution at time position The output at is ,in For expansion rate, The kernel size is [size]. For the convolution kernel at position The weight, Indicates in Interval before time The input values ​​at each time step. The expansion rate is adjusted. For example, setting it to 1, 2, and 4 can respectively implement convolution operations of sampling adjacent frames, sampling at one-frame intervals, and sampling at three-frame intervals. Among these, a small dilation coefficient (such as...) Convolutional blocks with a moderate dilation factor (e.g., ) can capture the details of instantaneous changes between adjacent frames. Convolutional blocks with large dilation coefficients (such as those used in the model) can capture changes in motion rhythm over a medium time span. Convolutional blocks can capture long-term motion trends. Each temporal convolutional block processes a segment of the input motion data. After dilated convolution, the corresponding temporal feature map is output. Its dimension can be represented as ,in The time length after convolution. This represents the number of feature channels output by the convolutional block. Through this multi-scale parallel dilated convolution design, the network can capture the evolution of actions at different time scales without significantly increasing the number of parameters.

[0075] Multiple temporal feature maps of different scales are spliced ​​and fused along the feature channel dimension to obtain a fused multi-scale temporal feature map.

[0076] Specifically, to integrate action feature information at different scales, this method combines the temporal feature maps output by multiple parallel temporal convolutional blocks. (in (where the number of parallel blocks is ), the blocks are concatenated and fused along the feature channel dimension. The concatenation operation can be represented as... ,in This represents the concatenation function along the channel dimension. After concatenation, the fused multi-scale temporal feature map is obtained. The dimension is In other words, the new number of feature channels is the sum of the number of output channels of each parallel block. This splicing and fusion method preserves the unique information of features at each scale, while constructing a more complete feature space and avoiding the one-sided representation caused by single-scale features. The fused multi-scale temporal feature map can simultaneously cover the detailed changes, rhythmic patterns, and overall trends of the action, laying the foundation for subsequent accurate representation of action attributes.

[0077] The fused multi-scale temporal feature map is input into a fully connected layer for dimensionality reduction mapping to obtain the action latent space vector.

[0078] Specifically, the fused multi-scale temporal feature map Since it is still high-dimensional feature data, using it directly would increase the computational complexity of subsequent processing. This method first transforms the three-dimensional feature data... Flattened into a one-dimensional feature vector ,in Then, the flattened feature vector is input into a fully connected layer composed of multiple neurons for dimensionality reduction mapping. The computation process of the fully connected layer can be represented as follows: ,in This is the weight matrix. Let be the dimension of the target latent space vector. For bias vectors, For activation functions (such as (Or linear activation); through this linear transformation and nonlinear activation, the method maps high-dimensional vectors into low-dimensional dense action latent space vectors. This allows the latent space vector of the action to efficiently preserve the core feature information of the action, while possessing the characteristics of low dimensionality and strong representational ability. The distance and direction in its vector space can reflect the similarity and difference of different actions in intrinsic attributes such as intensity, speed, and rhythm, and can be directly used as input conditions for subsequent sound effect generation models. This embodiment achieves in-depth mining and efficient integration of multi-scale temporal features of actions through innovative designs such as multi-scale dilated convolution, channel dimension splicing and fusion, and dimensionality reduction of fully connected layers, so as to improve the completeness and accuracy of action feature representation and provide high-quality input representation for subsequent intelligent mapping from action to sound effects.

[0079] In one embodiment, the generative sound effect model employs a generative adversarial network architecture, including a generator network and a discriminator network. The training method for this generative sound effect model may include the following steps:

[0080] Obtain the historical action latent space vector corresponding to the historical action data, as well as the real sound effect spectrogram corresponding to the historical action data.

[0081] Specifically, the historical motion data originates from various character motion capture data accumulated during game development, covering body movements and facial expressions in different scenes and styles. This method first performs spatiotemporal alignment preprocessing and multi-scale temporal convolutional network feature extraction on the historical motion data, consistent with the aforementioned embodiments, to obtain the corresponding historical motion latent space vector. The corresponding real sound effect spectrogram is standard sound effect frequency domain data that has been professionally produced and precisely matched with the historical motion data in the time dimension. It is obtained by performing a short-time Fourier transform on real audio files, which can accurately reflect the correspondence between motion and sound effects, providing reliable supervision signals for model training. In the actual training phase, a large-scale paired dataset needs to be constructed, where each pair of data contains a historical motion sample and its corresponding real sound effect sample. The historical motion data originates from a motion capture database that has been preprocessed with spatiotemporal alignment. Using the aforementioned multi-scale temporal convolutional network feature extraction method, each historical motion sample is converted into a historical motion latent space vector. This vector is a low-dimensional representation of the historical action's intrinsic attributes such as force, speed, and rhythm. Simultaneously, for the same historical action sample, it's also necessary to obtain its corresponding sound effect recordings in real-world scenarios, such as the sound of a sword swing, footsteps, or a punch breaking through the air, recorded synchronously in a professional recording studio. These raw audio data are then subjected to a short-time Fourier transform to convert them into a two-dimensional Mel-spectrum representation, i.e., the real sound effect spectrum. ,in Number of Mel frequency channels, For the number of time frames; the historical action latent space vector Compared with real sound effect spectrum diagram Pairings form the basic data units used for supervised training, and the quality of these paired data directly determines the accuracy of the action-sound effect mapping relationship that the model ultimately learns.

[0082] The historical action latent space vector is input as a conditional vector into the generator network, which then generates a spectrogram of the predicted sound effect.

[0083] For example, this method can input the historical action latent space vector as a conditional vector into the generator network. The generator network aims to learn a mapping function from the action latent space to the sound effect spectrogram. Its input includes two parts: one is the historical action latent space vector as a condition. Secondly, noise vectors randomly sampled from the standard normal distribution. The purpose of introducing noise vectors is to increase the diversity of generated results and avoid the model producing a deterministic, single output; and The vectors are concatenated to form a joint input vector. Generator networks typically employ an upsampling architecture based on transposed convolutions, consisting of multiple stacked deconvolutional layers. The joint input vector is first projected onto an appropriate dimension through a fully connected layer, then reshaped into a two-dimensional initial feature map. Subsequently, a series of upsampling operations are used to progressively expand the time and frequency dimensions, ultimately outputting a predicted audio spectrogram. Its size and the actual sound spectrum Completely identical; in the early stages of training, the generator produces... The initial results are often quite rough, but with each training iteration, the generator gradually learns to generate increasingly realistic sound effect spectra based on the input conditional vector.

[0084] The predicted sound effect spectrogram and the actual sound effect spectrogram are respectively input into the discriminator network, and the discriminator network outputs their respective discrimination scores.

[0085] Specifically, the discriminator network can employ a convolutional neural network architecture, acting similarly to a binary classifier to distinguish whether the input spectrogram is real (from training data) or fake (from the generator). The discriminator's input can be a single spectrogram or a combination of a spectrogram and a conditional vector to enhance the conditionality of the discrimination. For example, a PatchGAN-based discriminator can be used, which divides the input spectrogram into multiple overlapping patches, outputs a discrimination probability independently for each patch, and finally averages them to obtain the overall discrimination score. When the input is a real sound effect spectrogram... When the discriminator expects to output a high score close to 1, it is marked as "true"; when the input is the spectrogram of the predicted sound effect generated by the generator... At this time, the discriminator expects to output a low score close to 0, which is marked as "false". The discriminator outputs a score... and This reflects the current discriminator's ability to judge the authenticity of the input spectrogram, and also provides gradient signals for the optimization of the generator.

[0086] The adversarial loss function value is calculated based on the discrimination score, and the network parameters of the generator network and the discriminator network are updated by backpropagation based on the adversarial loss function value.

[0087] For example, this method can calculate the adversarial loss function value based on the discrimination score output by the discriminator network. The adversarial loss function adopts the binary cross-entropy loss function, which is constructed by measuring the distribution difference between the predicted sound effect spectrogram and the real sound effect spectrogram, as well as the classification error of the discriminator network. Then, with the goal of minimizing the adversarial loss function value, the network parameters of the generator network and the discriminator network are updated by backpropagation using the stochastic gradient descent algorithm. During the training process, the two networks are alternately and iteratively optimized, so that the predicted sound effect spectrogram generated by the generator network becomes closer and closer to the real sound effect spectrogram, while improving the discrimination ability of the discriminator network, until the loss function value converges to a stable range. Specifically, the training process is essentially a process of mutual game and alternating optimization between the generator and the discriminator. For the discriminator network, its goal is to minimize the discrimination error between real samples and fake samples. The loss function of the discriminator is usually defined as... ,in Expressing expectations, The output score of the discriminator on the true spectrogram, The generator network's output score is the discriminator's score for the generated spectrogram. Its objective is to deceive the discriminator as much as possible, making the generated spectrogram appear real to the discriminator. The generator's loss function is typically defined as... This loss value encourages the generator to produce spectrograms that enable the discriminator to output high scores. Furthermore, within the framework of conditional generative adversarial networks, feature matching loss can be introduced, which compares the L1 distance of features extracted by the discriminator's intermediate layers to real and generated samples to stabilize the training process. After calculating the loss function value, the gradient of the loss function with respect to each network parameter is calculated using the backpropagation algorithm, and optimization algorithms such as Adam are used to update the weight parameters of the generator and discriminator networks. In each iteration, the discriminator parameters are typically updated several times first, followed by the generator parameters, alternating until the model converges. When training is complete, the generator network has learned to generate realistic and diverse sound effect spectrograms based on the input action latent space vectors, while the discriminator network serves as an auxiliary training tool and is no longer used in the inference phase.

[0088] In summary, this embodiment constructs a conditional generative adversarial network (GAN) containing a generator and a discriminator, and employs an adversarial training mechanism. This enables the model to autonomously learn a complex nonlinear mapping from action features to sound effect spectra from a large amount of historical action-sound effect pairing data. This method not only generates sound effects that highly match the actions, but also ensures the diversity of the generated results by introducing noise vectors. This overcomes the limitation of traditional sample library-based methods that can only output fixed sound effects. The trained generator network can be directly used for real-time inference, generating corresponding sound effect spectra based on the real-time extracted action latent space vectors, providing high-quality input for subsequent waveform synthesis and synchronized playback.

[0089] In one embodiment, this technical solution achieves the generation of a digital audio signal stream that combines action adaptability and scene consistency through a waveform synthesis strategy that integrates scene style features and affine transformations, ensuring that the sound effects are deeply matched with the game scene style. To this end, the method of waveform synthesis of the target sound effect spectrogram to obtain the digital audio signal stream may include the following steps:

[0090] Obtain the scene style label of the current game scene, input the scene style label into the pre-trained style embedding layer for embedding encoding, and obtain the style vector.

[0091] Specifically, scene style tags are predefined classification identifiers based on the current game scene type, covering different scene categories such as combat, exploration, and dialogue. These tags can be output in real-time by the game engine during scene switching or retrieved through scene configuration files. In actual game environments, different game scenes have drastically different auditory aesthetic requirements. For example, science fiction scenes often require cold, metallic sound effects, ancient Chinese style scenes require a warm, ethereal auditory experience, while horror scenes require deep, oppressive ambient sound effects. To adaptively adjust the generated sound effects according to the scene style during waveform synthesis, this method introduces a style control mechanism. For example, this method can read the current scene's identifier information from the game engine's scene management module, map this identifier information to predefined scene style tags, such as discrete tags like "science fiction," "ancient Chinese style," "horror," or "realism," and input these scene style tags into the style embedding layer. The style embedding layer is essentially a learnable lookup table, its structure being a shape... Embedding matrix ,in The total number of predefined style categories, This represents the dimension of the style vector (e.g., 128 dimensions). For the input style label... (For example, the index value for "science fiction" is 2), the style embedding layer uses table lookup operations. Obtain the corresponding style vector The style vector is a dense representation of style features in a continuous vector space, where the direction and distance reflect the similarities and differences between different styles. The style embedding layer itself is jointly optimized with the vocoder network during the end-to-end training of the entire sound effects generation system, thereby learning style representations that can effectively guide timbre adjustment.

[0092] The style vector is input into the first fully connected layer and the second fully connected layer respectively. The scaling parameter vector is obtained by linear transformation through the first fully connected layer, and the translation parameter vector is obtained by linear transformation through the second fully connected layer. The scaling parameter vector and the translation parameter vector are used as affine transformation parameters.

[0093] Specifically, to inject style information into the waveform generation process, this method employs feature linear modulation technology, modulating the feature map inside the vocoder through affine transformation, and then using the style vector obtained in the previous step. The inputs are fed into two independent fully connected layers. The first fully connected layer is used to generate the scaling parameter vector, and its calculation process can be represented as follows: ,in This is the weight matrix of the first fully connected layer. For bias vectors, The number of channels in the feature map to be modulated. This refers to the scaling parameter vector; the second fully connected layer is used to generate the translation parameter vector, and its calculation process can be represented as follows: ,in This is the weight matrix of the second fully connected layer. For bias vectors, That is, the translation parameter vector, and Combined, they form the affine transformation parameter pair. The dimensions of both vectors are equal to the number of channels in the feature map to be modulated in the vocoder. Maintaining consistency allows for independent scaling and translation operations on each feature channel, enabling fine-tuning of the audio signal's timbre characteristics.

[0094] The target audio effect spectrogram and affine transformation parameters are input into a Wavenet-based vocoder. The waveform is reconstructed through the vocoder, and during the reconstruction process, the feature map of the intermediate layer of the vocoder is affinely transformed according to the affine transformation parameters to obtain a digital audio signal stream.

[0095] Specifically, a WaveNet-based vocoder is a deep autoregressive network or a parallel streaming network that can analyze the input Mel spectrogram. A high-fidelity time-domain waveform is reconstructed from the vocoder, which consists of multiple stacked residual blocks. Each residual block contains a dilated convolutional layer, a gated activation unit, and skip connections. To inject style information into the waveform generation process, this method introduces a feature linear modulation mechanism into each residual block or a specific intermediate layer of the vocoder. For example, for the feature map of a certain intermediate layer in the vocoder... ,in For the number of channels, Given the time step length, this method utilizes the aforementioned affine transformation parameters. Modulation is applied to it, and the modulation process can be represented as follows: ,in This indicates element-wise multiplication. Broadcast to the time dimension and Multiplication is performed at each time step. Similarly, the broadcast performs an addition operation. This affine transformation operation is equivalent to independently scaling and offsetting each channel of the feature map, thereby adjusting the intensity and bias of the acoustic features represented by that channel (such as the energy of a specific frequency band, the sharpness of the harmonic structure, etc.), achieving fine-grained control over the timbre style of the generated audio. The modulated feature map... It continues to propagate forward, participating in subsequent waveform reconstruction processes. Ultimately, the vocoder outputs a one-dimensional digital audio signal stream. ,in This represents the total number of audio sampling points. Since the affine transformation parameters are dynamically generated based on scene style labels, the final output audio signal stream retains the core sound features matching the action while adjusting its timbre to conform to the current scene style requirements. This embodiment converts discrete scene labels into continuous style vectors through a style embedding layer, then generates affine transformation parameters through two fully connected layers, and injects these parameters into the intermediate feature map of the WaveNet vocoder using feature linear modulation technology. This achieves dynamic and fine-grained adjustment of sound effect style during waveform synthesis. This method eliminates the need to maintain multiple independent sound effect libraries and post-mixing, enabling real-time generation of style-adaptive sound effects based on different game scenes. This improves the flexibility and scene adaptability of sound effect generation while maintaining high fidelity and real-time performance in waveform synthesis.

[0096] In summary, the intelligent real-time sound effect matching method based on multi-source motion feature fusion provided in this application acquires motion capture data containing human joint angles, limb movement trajectories, and facial muscle movement data. After hardware timestamp alignment, cubic spline interpolation, and multi-source data registration to complete spatiotemporal alignment preprocessing, standardized motion capture sequence data is obtained. Then, motion data fragments are extracted using a sliding window and input into a multi-scale temporal convolutional network. Parallel dilated convolution extracts temporal features at different scales, channel-dimensional splicing and fusion, and dimensionality reduction mapping using fully connected layers to obtain motion latent space vectors representing the core attributes of the motion. Subsequently, a generative sound effect model containing a generator network and a discriminator network is used. The network parameters are optimized through adversarial training between historical motion latent space vectors and real sound effect spectrograms. The latent space vector is used as a conditional input to the generator network to obtain the target sound effect spectrogram. Combined with the affine transformation parameters obtained by embedding encoding and fully connected layer transformation of the current game scene style label, waveform reconstruction is performed through a WaveNet-based vocoder, and affine transformation is applied to the intermediate layer feature map to generate a digital audio signal stream. This stream is then read and played by the game engine from the audio queue according to the rendering frame timestamp. This achieves deep fusion and accurate representation of multi-source motion features, freeing it from dependence on a fixed sound effect sample library. It enables intelligent generation of sound effects and scene stylization adaptation. At the same time, time alignment, network optimization, and synchronous playback mechanisms ensure real-time synchronization of motion and sound effects, reduce system resource consumption, improve adaptability to different game scenes and character settings, and effectively enhance the audiovisual immersion of the game.

[0097] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0098] Based on the same inventive concept, this application also provides an intelligent real-time sound effect matching system 10 based on multi-source motion feature fusion for implementing the aforementioned intelligent real-time sound effect matching method based on multi-source motion feature fusion. The solution provided by this system is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the intelligent real-time sound effect matching system 10 based on multi-source motion feature fusion provided below can be found in the limitations of the intelligent real-time sound effect matching method based on multi-source motion feature fusion described above, and will not be repeated here.

[0099] In one exemplary embodiment, such as Figure 2 As shown, a real-time intelligent sound effect matching system 10 based on multi-source motion feature fusion is provided to implement the aforementioned real-time intelligent sound effect matching method based on multi-source motion feature fusion. The system includes:

[0100] Data preprocessing module 11 is used to perform spatiotemporal alignment preprocessing on the acquired motion capture data to obtain motion capture sequence data;

[0101] Feature extraction module 12 is used to extract features from motion capture sequence data based on multi-scale temporal convolutional network to obtain action latent space vector, which is used to characterize the intrinsic attributes of the action.

[0102] The sound effect generation module 13 is used to input the motion latent space vector as a conditional vector into the generative sound effect model to generate a target sound effect spectrogram corresponding to the motion latent space vector.

[0103] The audio matching module 14 is used to synthesize the waveform of the target sound effect spectrogram to obtain a digital audio signal stream; the digital audio signal stream is used to instruct the game engine to read the matching audio packet from the audio queue according to the timestamp of the current rendering frame for playback.

[0104] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the intelligent sound effect real-time matching method based on multi-source motion feature fusion as described above.

[0105] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0106] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0107] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A real-time intelligent sound effect matching method based on multi-source motion feature fusion, characterized in that, The method includes: The acquired motion capture data is preprocessed with spatiotemporal alignment to obtain motion capture sequence data. Feature extraction based on a multi-scale temporal convolutional network is performed on the motion capture sequence data to obtain action latent space vectors, which are used to characterize the intrinsic attributes of actions. The motion latent space vector is input as a conditional vector into the generative sound effect model to generate a target sound effect spectrogram corresponding to the motion latent space vector; The target sound effect spectrogram is subjected to waveform synthesis to obtain a digital audio signal stream; the digital audio signal stream is used to instruct the game engine to read the matching audio packet from the audio queue according to the timestamp of the current rendering frame and play it.

2. The method according to claim 1, characterized in that, The process of performing spatiotemporal alignment preprocessing on the acquired motion capture data to obtain motion capture sequence data includes: Obtain the hardware timestamp corresponding to the motion capture data, wherein the motion capture data includes human joint angle data, limb movement trajectory data and facial muscle movement data; Based on the hardware timestamp, the human joint angle data, the limb movement trajectory data and the facial muscle movement data are aligned on the time axis in chronological order to generate time-aligned multi-source data frames. According to the preset target sampling frequency, the multi-source data frame is subjected to cubic spline interpolation to generate an interpolated data frame with a uniform time interval. The motion capture sequence data is obtained by registering the human joint angle data, the limb movement trajectory data, and the facial muscle movement data at the same time point.

3. The method according to claim 1, characterized in that, The step of extracting features from the motion capture sequence data using a multi-scale temporal convolutional network to obtain the action latent space vector includes: The motion capture sequence data is processed by sliding windowing, a fixed-length time window is set, and motion data segments within the current time window are extracted from the motion capture sequence data; The action data segment within the current time window is input into a multi-scale temporal convolutional network. The network performs dilated convolution processing through multiple parallel temporal convolutional blocks to generate multiple temporal feature maps of different scales. The multiple time-series feature maps of different scales are spliced ​​and fused along the feature channel dimension to obtain a fused multi-scale time-series feature map. The fused multi-scale temporal feature map is input into a fully connected layer for dimensionality reduction mapping to obtain the action latent space vector.

4. The method according to claim 1, characterized in that, The generative sound effect model includes a generator network and a discriminator network, and the training method of the generative sound effect model includes: Obtain the historical action latent space vector corresponding to the historical action data, and the real sound effect spectrogram corresponding to the historical action data; The historical action latent space vector is input as a conditional vector into the generator network, and the generator network generates a predicted sound effect spectrogram. The predicted sound effect spectrogram and the actual sound effect spectrogram are respectively input into the discriminator network, and the discriminator network outputs their respective discrimination scores. The adversarial loss function value is calculated based on the discrimination score, and the network parameters of the generator network and the discriminator network are updated by backpropagation based on the adversarial loss function value.

5. The method according to claim 1, characterized in that, Waveform synthesis is performed on the target sound effect spectrogram to obtain a digital audio signal stream, including: Obtain the scene style label of the current game scene, and input the scene style label into the pre-trained style embedding layer for embedding encoding to obtain the style vector; The style vector is input into the first fully connected layer and the second fully connected layer respectively. The scaling parameter vector is obtained by linear transformation through the first fully connected layer, and the translation parameter vector is obtained by linear transformation through the second fully connected layer. The scaling parameter vector and the translation parameter vector are used as affine transformation parameters. The target audio effect spectrogram and the affine transformation parameters are input into a Wavenet-based vocoder. Waveform reconstruction is performed through the vocoder, and during the reconstruction process, the feature map of the intermediate layer of the vocoder is subjected to affine transformation according to the affine transformation parameters to obtain the digital audio signal stream.

6. A real-time intelligent sound effect matching system based on multi-source motion feature fusion, used to implement the method according to any one of claims 1 to 5, characterized in that, The system includes: The data preprocessing module is used to perform spatiotemporal alignment preprocessing on the acquired motion capture data to obtain motion capture sequence data; The feature extraction module is used to extract features from the motion capture sequence data based on a multi-scale temporal convolutional network to obtain an action latent space vector, which is used to characterize the intrinsic attributes of the action. The sound effect generation module is used to input the motion latent space vector as a conditional vector into the generative sound effect model to generate a target sound effect spectrogram corresponding to the motion latent space vector. An audio matching module is used to synthesize waveforms from the target sound effect spectrogram to obtain a digital audio signal stream; the digital audio signal stream is used to instruct the game engine to read the matching audio packet from the audio queue according to the timestamp of the current rendering frame for playback.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

Images