A method and system for enhancing footsteps in a game scene

Abstract

This invention relates to a method and system for enhancing footsteps in a game scene. The method includes pre-constructing a mixed audio sample containing footsteps and background sound and generating a corresponding enhanced target audio signal; determining training equalizer parameters for frequency response correction based on the two; forming training data corresponding to the frequency domain characteristics of the mixed audio sample; and training a neural network model using the training data; embedding the trained neural network model and parameters into an embedded audio processing chip; acquiring real-time audio signals through the neural network model; obtaining amplitude spectrum features through time-frequency transformation of a convolutional layer; analyzing their temporal correlation through a recurrent layer; and outputting equalizer parameters through a fully connected layer; and performing equalization processing on the real-time audio signal according to the equalizer parameters to enhance game footsteps and outputting the signal. This achieves selective enhancement of weak footstep signals in a complex game sound field while maintaining spatial positioning accuracy and a natural listening experience.

Description

Technical Field

[0001] This invention relates to the field of audio signal processing technology, and in particular to a method and system for enhancing footsteps in a game scene. Background Technology

[0002] In games involving spatial auditory judgment, auditory information is one of the most important sources of information for players to obtain battlefield situational awareness. The footsteps of enemy characters carry crucial tactical information, including the enemy's location, distance, movement speed, and the material of the surface they are touching. However, in actual game scenarios, footsteps have significant weak signal characteristics, including concentrated energy in a narrow mid-to-high frequency band in the frequency domain; transient signals in the time domain with extremely short durations; and in complex game sound fields, they are easily masked by high-energy events such as gunshots and explosions, as well as ambient background noise.

[0003] Existing footstep enhancement solutions mainly include: traditional frequency band enhancement methods, which statically boost the gain of a specific frequency range, but this method cannot distinguish footsteps from other interference sounds within the same frequency band and will simultaneously amplify interference signals such as gunshots; PC-based plug-ins, which process the system audio stream in real time, but require continuous high-performance CPU resources and cannot be deployed on embedded platforms; game engine event detection, which relies on open interfaces from game developers, and varies greatly between different games, lacking universality; and deep learning sound source recognition, but existing models have large parameter scales and high computational complexity, making it difficult to achieve low-latency real-time operation on low-power embedded chips. Summary of the Invention

[0004] To address the issues of existing footstep enhancement methods lacking specificity and being difficult to run in real-time on embedded platforms, the main objective of this invention is to provide a method and system for enhancing footsteps in game scenes. This involves offline construction of a multi-dimensional footstep dataset and training a lightweight neural network. The neural network includes convolutional layers for extracting frequency domain features, recurrent layers for analyzing temporal correlations, and fully connected layers for outputting equalizer parameters. This allows for real-time prediction of parametric equalizer coefficients on embedded audio devices, achieving selective enhancement of weak footstep signals in complex game sound fields while maintaining spatial positioning accuracy and a natural listening experience.

[0005] To achieve the above objectives, the present invention provides a method for enhancing footsteps in a game scene, comprising the following steps: A mixed audio sample containing footsteps and background sounds is pre-constructed and a corresponding enhanced target audio signal is generated. Based on the mixed audio sample and the enhanced target audio signal, training equalizer parameters for frequency response correction are determined. The frequency domain features of the mixed audio sample are mapped to the equalizer parameters to form training data. A neural network model is trained using the training data. The neural network model includes a convolutional layer for extracting frequency domain features, a recurrent layer for analyzing temporal correlation, and a fully connected layer for outputting equalizer parameters. The trained neural network model and parameters are stored in the storage medium of the audio processing device, which is an embedded audio processing chip. The neural network model acquires real-time audio signals of the game scene, performs time-frequency transformation on the real-time audio signals through the convolutional layer to obtain amplitude spectrum features, analyzes the time-series correlation of the amplitude spectrum features through the recurrent layer, and outputs equalizer parameters through the fully connected layer. The real-time audio signal is processed according to the equalizer parameters to enhance the game footsteps, and the equalized audio signal is output to the audio playback device.

[0006] Furthermore, the step of pre-constructing a mixed audio sample containing footsteps and background sounds includes: By controlling variables, footstep and background sound audio were collected in multiple game scenes, covering various ground materials, movement methods, collection distances, and environmental noise conditions. The collected audio data was divided into a footsteps sub-dataset and a background noise sub-dataset; Footstep audio is randomly selected from the footstep audio subset, background audio is randomly selected from the background sound subset, and a signal-to-noise ratio parameter is randomly generated. The amplitude of the background sound audio is adjusted proportionally according to the signal-to-noise ratio parameter and then superimposed with the footstep sound audio to obtain a mixed audio sample.

[0007] Furthermore, the step of generating the corresponding enhanced target audio signal includes: Separate the footsteps component and the background sound component from the mixed audio samples; The footstep component is amplified by a fixed factor, which is preset according to the enhancement requirements to increase the energy proportion of the footstep in the mixed audio sample; The amplified footsteps component is then superimposed on the background sound component to obtain the enhanced target audio signal.

[0008] Further, the step of determining the training equalizer parameters for frequency response correction based on the mixed audio samples and the enhanced target audio signal includes: The time-frequency transformations of the mixed audio sample and the enhanced target audio signal are performed respectively to obtain the amplitude spectra of the mixed audio sample and the enhanced target audio signal; The amplitude spectra of the mixed audio samples and the enhanced target audio signal are normalized based on a preset reference frequency. Calculate the difference between the two normalized amplitude spectra at each frequency point to obtain the error curve; The error curve is subjected to mean compensation processing within a preset frequency band; Invert the compensated error curve to obtain the reverse compensation curve; The reverse compensation curve is subjected to bidirectional slope limiting, high-frequency gain limiting, and global gain clipping in sequence to obtain the physically constrained compensation curve. The compensation curve after the physical constraints are fitted to a finite number of training equalizer parameters.

[0009] Furthermore, the processing steps for bidirectional slope constraint include: Define the maximum permissible slope of attenuation as the frequency sampling point changes; The reverse compensation curve is scanned point by point from low frequency to high frequency, and the rise amplitude between adjacent frequency sampling points is limited to not exceed the maximum allowable slope to obtain the first limiting curve. The reverse compensation curve is scanned point by point from high frequency to low frequency, and the rise amplitude between adjacent frequency sampling points is limited to not exceed the maximum allowable slope to obtain the second limiting curve. The minimum value of the first limiting curve and the second limiting curve are combined to obtain the compensation curve after slope limitation.

[0010] Further, the step of fitting the physically constrained compensation curve to a finite number of training equalizer parameters includes: Set up a multi-band peak equalizer, each of which includes a center frequency, gain parameter, and quality factor parameter. The center frequency is preset according to the frequency band distribution to cover the frequency band where footstep energy is concentrated. Construct the overall frequency response of the multi-segment peak equalizer combination; A target function is established with the goal of minimizing the error between the overall frequency response and the compensation curve obtained after sequential bidirectional slope limiting, high-frequency gain attenuation, and maximum gain limiting. Within the preset range of gain parameters and quality factor parameters, determine the gain parameters and quality factor parameters that minimize the objective function.

[0011] Furthermore, the step of acquiring real-time audio signals of the game scene through the neural network model includes: Receive real-time audio streams output by the game device via a digital audio interface; The real-time audio stream is cached and divided into audio frames of fixed length, and a multi-frame buffer is constructed to store multiple consecutive audio frames. The audio frames are used to construct multi-frame temporal information input to the convolutional layer.

[0012] Furthermore, the step of obtaining amplitude spectrum features by performing time-frequency transformation on the real-time audio signal through the convolutional layer includes: The audio frame is used as the input to the convolutional layer; The convolutional layer performs time-frequency transformation on the audio frame, converting the time-domain audio signal into a frequency-domain representation; The energy distribution of the audio frame at each frequency point is extracted by taking the modulus of the time-frequency transformation result to obtain the amplitude spectrum features; The amplitude spectrum features of multiple consecutive frames are concatenated along the time dimension to construct a temporal feature matrix, which is then used as the input to the recurrent layer of the neural network model.

[0013] Further, the step of performing equalization processing on the real-time audio signal according to the equalizer parameters includes: The gain parameters and quality factor parameters output by the fully connected layer are combined with the preset center frequency and sampling rate to convert them into bi-second-order filter coefficients. The real-time audio signal is filtered according to the coefficients of the dual second-order filter. The filtered audio signal is output to the audio playback device.

[0014] The present invention also provides a footstep sound enhancement system for game scenes, comprising: The model training unit is used to pre-construct mixed audio samples containing footsteps and background sounds and generate corresponding enhanced target audio signals. Based on the mixed audio samples and the enhanced target audio signals, training equalizer parameters for frequency response correction are determined. The frequency domain features of the mixed audio samples are mapped to the equalizer parameters to form training data. The training data is used to train a neural network model. The neural network model includes a convolutional layer for extracting frequency domain features, a recurrent layer for analyzing temporal correlation, and a fully connected layer for outputting equalizer parameters. A solidified storage unit is used to solidify the trained neural network model and parameters in the storage medium of an audio processing device, wherein the audio processing device is an embedded audio processing chip. The audio acquisition unit uses Yahoo to acquire real-time audio signals of the game scene through the neural network model, performs time-frequency transformation on the real-time audio signals through the convolutional layer to obtain amplitude spectrum features, analyzes the time series correlation of the amplitude spectrum features through the recurrent layer, and outputs equalizer parameters through the fully connected layer. The equalization processing unit performs equalization processing on the real-time audio signal according to the equalizer parameters to enhance the game footsteps, and outputs the equalized audio signal to the audio playback device.

[0015] The method and system for enhancing footsteps in game scenes provided by this invention have the following beneficial effects: This invention constructs a multi-dimensional realistic footsteps dataset and combines a random signal-to-noise ratio mixing strategy with a clearly defined footsteps enhancement target signal to simulate complex game sound field environments and improve model robustness. It introduces physical constraints such as error compensation, bidirectional slope constraints, and high-frequency gain constraints in the equalization parameter solution to generate smooth and stable compensation curves, avoiding overcompensation and instability issues. Furthermore, it employs a lightweight neural network to predict a small number of PEQ parameters instead of the complete spectrum, reducing computational complexity and enabling low-latency, low-power real-time operation on embedded platforms. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a method for enhancing footsteps in a game scene according to an embodiment of the present invention; Figure 2 This is a structural block diagram of a footstep sound enhancement system in a game scene according to an embodiment of the present invention.

[0017] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] Reference Figure 1 This is a flowchart illustrating a method for enhancing footsteps in a game scene according to the present invention, comprising the following steps: S1, a mixed audio sample containing footsteps and background sounds is pre-constructed and a corresponding enhanced target audio signal is generated. Based on the mixed audio sample and the enhanced target audio signal, training equalizer parameters for frequency response correction are determined. The frequency domain features of the mixed audio sample are matched with the equalizer parameters to form training data. The training data is used to train a neural network model. The neural network model includes a convolutional layer for extracting frequency domain features, a recurrent layer for analyzing temporal correlation, and a fully connected layer for outputting equalizer parameters. S2, The trained neural network model and parameters are stored in the storage medium of the audio processing device, wherein the audio processing device is an embedded audio processing chip; S3, real-time audio signals of the game scene are obtained through the neural network model, the amplitude spectrum features are obtained by performing time-frequency transformation on the real-time audio signals through the convolutional layer, the time series correlation of the amplitude spectrum features is analyzed through the recurrent layer, and the equalizer parameters are output through the fully connected layer. S4, perform equalization processing on the real-time audio signal according to the equalizer parameters to enhance the game footsteps, and output the equalized audio signal to the audio playback device.

[0020] In one embodiment, a method for enhancing footsteps in a game scene is provided, employing a two-stage architecture of "offline training modeling and online real-time inference".

[0021] The offline training phase aims to construct a highly robust dataset specifically for game footsteps, and to train and optimize the neural network model based on this dataset. Addressing the issues of insufficient training data coverage and low scene realism in existing technologies, this invention obtains representative footstep samples by constructing a multi-dimensional footstep acquisition matrix. Footstep data is collected from multiple mainstream FPS / TPS games, recording multiple acoustic conditions, including: ground material (grass, metal, wood, cement, carpet, etc.), movement mode (walking, running, jumping, landing, etc.), acquisition distance (1m, 3m, 5m, 10m, 20m, etc.), and environmental noise conditions (indoor sound field, outdoor sound field, with gunfire interference, without gunfire interference, etc.). By controlling variables, footstep sample data covering different game environments, different action types, and different propagation conditions are obtained. By randomly generating different signal-to-noise ratio parameters, various complex scenarios such as long-distance gunfire interference, indoor reverberation, wind interference, and footsteps being masked by environmental noise can be simulated. The collected audio data was divided into two subsets: footsteps subset (which stores audio samples with pure footstep features) and background sound subset (which stores ambient sound samples that do not contain footsteps, including gunshots, ambient noise, voice acting, etc.).

[0022] To address the issue of ambiguous training targets in traditional methods, this invention constructs the input and target signals required for model training through random signal-to-noise ratio (SNR) mixing and enhanced target signal generation. A footstep audio track is randomly selected from the footstep audio dataset, and a background audio track is randomly selected from the background sound dataset. A random SNR parameter is generated, representing the ratio of footstep energy to background sound energy, with a value ranging from low to high SNR (e.g., -10dB to 20dB), used to simulate different game scenarios from strong noise interference to relatively quiet environments. An amplitude adjustment coefficient for the background sound audio is calculated based on the SNR parameter, ensuring that the energy ratio between the adjusted background sound audio and the footstep audio meets the requirements of the SNR parameter. The amplitude of the background sound audio is proportionally adjusted according to the SNR parameter and then superimposed with the footstep audio to obtain a mixed footstep signal. By using random SNR, various complex scenarios such as long-range gunfire interference, indoor echo, and wind noise interference can be simulated. The footstep component in the aforementioned mixed signal is amplified by a fixed factor. The amplified footsteps are then superimposed on the original background audio to obtain the enhanced target audio signal. The fixed factor (e.g., 3 times) is chosen to ensure the footsteps remain identifiable while avoiding excessive amplification that could cause audio distortion. This fixed factor is applied to the time-domain waveform of the footstep component, ensuring that the amplified footstep component dominates in energy relative to the background audio component. This step ensures that the footsteps in the target signal have a higher energy proportion than the background audio, serving as a supervised target for neural network training, guiding the network to learn the mapping relationship from the mixed signal to the desired enhancement effect.

[0023] To address the problems of overcompensation and unstable frequency response curves inherent in traditional automatic equalization methods, this invention introduces a multiple physical constraint mechanism to solve for equalizer parameters that are engineering-feasible. Time-frequency transformations, such as short-time Fourier transforms, are performed on the mixed signal and the target signal to obtain their amplitude spectra, resulting in their frequency response curves. This invention normalizes the amplitude spectra of the mixed audio sample and the enhanced target audio signal using a preset reference frequency. This preset reference frequency is used to perform an overall shift of the frequency response curves, eliminating the overall gain offset between the mixed audio sample and the enhanced target audio signal. In this embodiment, the preset reference frequency is 1kHz, one of the frequencies most sensitive to human hearing and a commonly used reference point in audio processing. The gain values ​​of the mixed audio sample and the enhanced target audio signal at 1kHz are calculated separately, and their respective frequency response curves are shifted overall, making the gain of both zero at 1kHz. This normalization process eliminates the overall gain difference between the mixed audio sample and the enhanced target audio signal, allowing subsequent error analysis to focus only on the relative shape of the frequency response. This invention does not limit the use of 1kHz; other frequency points (such as 500Hz, 2kHz, etc.) can also be used as reference frequencies. The difference between the frequency response values ​​of the mixed signal and the target signal at discrete frequency sampling points is calculated to obtain the original error curve, eliminating the influence of overall gain offset. To avoid individual narrowband dips or peaks causing deviations in error judgment, the error curve is statistically averaged within a preset frequency band (e.g., 100Hz-10kHz) to obtain the average bias of the frequency band. This preset frequency band covers the main energy frequency band of footsteps and common game audio frequency bands. The average bias of the frequency band is subtracted from the entire error curve, making the mean error within the preset frequency band approach zero, thereby reducing the influence of overall gain deviation on local error judgment, especially reducing false dips or peaks at key reference frequencies such as 1kHz caused by overall offset. Subtracting the average bias of the frequency band from the entire error curve yields a compensated error curve, eliminating the interference of narrowband peaks and valleys on error judgment. By compensating for the error curve, the average error within the preset frequency band is brought close to zero, reducing the impact of the overall gain deviation on the local error judgment, especially reducing the false dips or peaks caused by the overall offset at key reference frequencies such as 1kHz.

[0024] Calculate the inverse error curve, i.e., the compensation curve to be applied. Perform peak-valley detection on the inverse error curve to identify significant structural features and determine the frequency regions that require focused processing. Only peaks and valleys with a prominence exceeding a preset threshold (e.g., 1 dB) are considered valid features to avoid over-processing of minor fluctuations. Set bidirectional slope limits to ensure the compensation curve meets physical feasibility requirements and avoids excessively steep rises or falls. Without slope limits, the generated compensation curve may exhibit physically unfeasible spikes or discontinuous structures, leading to impractical filters or instability. Define the maximum permissible slope that decays with frequency: in , representing the maximum allowable slope value in the low-frequency region; ; For discrete frequency sampling points; exponential term The attenuation factor has a value at frequency. Increase and decrease, make Approximately at low frequencies (Allowing a larger slope), gradually approaching 0 at high frequencies (with a gentler slope). The bidirectional slope setting allows for a larger slope in the low-frequency region (for rapid compensation in areas where footstep energy is concentrated), while the slope in the high-frequency region is gentler (avoiding over-enhancement leading to distortion), thus preventing unattainable spikes or discontinuous structures. The bidirectional limits of LTR (from low to high frequencies) and RTL (from high to low frequencies) work together to ensure the smoothness of the compensation curve.

[0025] LTR slope control (constraint from low frequency to high frequency) scans point by point from low frequency to high frequency, applying constraints, with initial conditions as follows: That is, the first frequency point remains unchanged. For i=2,3,...,N, the calculation is performed recursively: Where N is the total number of frequency sampling points. This indicates the frequency of the curve after LTR limiting. The value at that point, the min function represents taking the smaller of the two values. This represents the maximum gain that can be achieved by starting from the previous frequency sampling point and increasing at the maximum allowable slope. This is the gain value after limiting the previous frequency sampling point; Indicates from arrive The maximum allowable gain increment, where The frequency interval between adjacent frequency sampling points. This represents the maximum permissible slope for this frequency band; adding the two values ​​yields the maximum gain achievable starting from the previous frequency point and rising at the maximum permissible slope. The smaller of the current point's original gain value and this maximum gain value is used to ensure that the increase in the compensation curve from low to high frequencies does not exceed the permissible range. The smaller of the current point's original gain value and this maximum gain value is used to ensure that the increase in the compensation curve from low to high frequencies does not exceed the permissible range.

[0026] RTL slope control (constraint from high frequency to low frequency) involves scanning point-by-point from high frequency to low frequency to apply constraints, with initial conditions as follows: That is, the last frequency point remains unchanged. For i = N-1, N-2, ..., 1, the calculation is performed recursively: in This indicates the curve value after RTL limiting at a certain frequency. The value at that location, This represents the maximum gain value achievable by rising at the maximum permissible slope from the next frequency sampling point. The smaller of the original gain value at the current point and the maximum gain value is taken to ensure that the rise of the compensation curve from high frequency to low frequency does not exceed the permissible range. This is the gain value after limiting the sampling point at the next frequency; Indicates from arrive The maximum allowable gain increment, where The frequency interval between adjacent frequency sampling points. This represents the maximum permissible slope for that frequency band; adding the two values ​​yields the maximum gain achievable starting from the next frequency point with the maximum permissible slope. The smaller of the original gain value at the current point and this maximum gain value is taken to ensure that the rise of the compensation curve from high to low frequencies does not exceed the permissible range. The RTL slope limits the rise of the compensation curve from high to low frequencies, suppressing abrupt changes in the reverse direction.

[0027] Merge the curves after two-way constraints by taking the minimum value: in, This indicates the compensation curve after bidirectional slope limiting at a certain frequency. The minimum value is chosen because the LTR limit ensures that the rise in frequency from low to high does not exceed the allowable value, and the RTL limit ensures that the rise in frequency from high to low does not exceed the allowable value. Taking the smaller of the two ensures that the compensation curve satisfies the slope constraint in both directions, thus obtaining the most stringent smooth curve. Through bidirectional slope constraint, the obtained compensation curve has a smooth transition overall, without physically unrealizable peaks or discontinuous structures.

[0028] Because high-frequency frequencies are highly sensitive to sound and prone to distortion or over-amplification in practical audio systems, it is necessary to limit the equalization compensation amplitude in the high-frequency range. A high-frequency attenuation weighting function is constructed, which exhibits a smooth transition on a logarithmic frequency coordinate system. The weighting value in the low-frequency range is close to 1, allowing full-amplitude compensation; the weighting value in the high-frequency range gradually decays to a preset attenuation coefficient, limiting the compensation amplitude in the high-frequency range. This high-frequency attenuation weighting function is implemented using a sigmoid function, and its transition band start and end frequencies are preset based on the frequency response characteristics of the audio system. To compensate for the attenuation weight as frequency increases, the high-frequency gain is limited, resulting in a compensation curve after high-frequency limiting. This avoids excessive amplification of high-frequency noise, maintains the natural listening experience, and suppresses high-frequency distortion. in, The attenuation weight increases with frequency, and the value ranges between (0,1]. The weight is close to 1 in the low-frequency region and gradually decreases in the high-frequency region. This indicates the compensation curve after high-frequency limiting at a certain frequency. The value at that location.

[0029] Let the maximum allowable gain be... The final compensation curve is obtained after cropping: in, This indicates the maximum gain value allowed by the system, used to avoid excessive gain adjustment that could lead to audio distortion or device overload; This indicates the final compensation curve after global gain clipping at the frequency. The value at that location.

[0030] The continuous frequency response compensation curve is fitted to a finite number of training equalizer parameters. A filter bank containing M peak equalizers is constructed, where the parameters of the m-th peak equalizer include... , , The center frequency is preset according to an octave band distribution, for example: The selection of the above center frequency is based on the commonly used frequency band division in audio processing, covering the main frequency bands within the range that the human ear can hear, especially the mid-to-high frequency region where footsteps are concentrated.

[0031] The overall frequency response of the M-band peak equalizer combination is the product of the responses of each band: in This indicates that the m-th peak equalizer is at the frequency... The amplitude response at that point, This represents the overall frequency response after combining the M-band peak equalizers. Converting this to decibels yields the PEQ fitting curve: Among them, | 20log 10 The decibel conversion formula converts linear amplitude to decibel values ​​in the logarithmic domain, making the frequency response more linear in auditory perception. This indicates that the PEQ fitting curve is at a frequency The gain value at the specified location is expressed in decibels (dB).

[0032] Constructing the error function Where N is the total number of frequency sampling points, Indicates frequency The deviation between the PEQ fitted curve and the target compensation curve is calculated; the squaring operation makes both positive and negative deviations count as positive values ​​and amplifies the effect of larger deviations; the summation is used to obtain the total squared error E at all frequency points, which is used to measure the closeness between the PEQ fitted curve and the target compensation curve, and the optimization objective is to minimize E.

[0033] Summing over i=1 to N. Applying gain constraints. and quality factor constraints The optimal PEQ parameter set is solved using constrained optimization methods (such as constrained nonlinear least squares methods). Using the above method, a large number of training samples of "input audio features and PEQ control parameters" are obtained in the offline stage.

[0034] To address the challenge of existing deep learning solutions running on embedded platforms, this invention employs a lightweight neural network to predict a limited number of PEQ parameters, rather than directly predicting the entire spectrum or waveform, significantly reducing computational complexity. In traditional audio enhancement methods, neural networks typically need to directly predict the spectral mask or the enhanced time-domain waveform of the entire audio signal. For example, when using STFT to represent audio in the frequency domain, each frame of the spectrum usually contains hundreds of frequency sampling points, resulting in an output dimension of hundreds of dimensions. In contrast, this invention predicts a limited number of parametric equalizer control parameters via a neural network. For instance, for 10 PEQ segments, only 10 gain parameters and 10 Q-value parameters need to be predicted, totaling 20 parameters. This reduces the output dimension from hundreds to tens of dimensions, significantly decreasing the computational burden.

[0035] The neural network takes a multi-frame amplitude spectrum matrix of a mixed signal as input and includes convolutional layers, recurrent layers, and fully connected layers connected in sequence. The convolutional layers perform time-frequency transformation on the input audio and extract its modulus, outputting amplitude spectrum features. These features are then used to extract local frequency features, capturing subtle differences in the frequency domain between footsteps and sounds like gunshots or explosions. The recurrent layers analyze the temporal correlation of the amplitude spectrum features, combining amplitude spectrum features from multiple consecutive frames along the time dimension to construct a temporal feature matrix. This temporal feature matrix is ​​input into the recurrent layers, which contain long short-term memory units. Through their internal memory units and gating mechanisms, the recurrent layers model the temporal dependencies, capturing the sequential patterns of footsteps across different time frames, thus distinguishing footsteps from other transient noises. The fully connected layers map the temporal features output from the recurrent layers to the gain and quality factor parameters of a parametric equalizer (PEQ). The PEQ filter's gain and Q-value parameters are then output through a multilayer perceptron (fully connected layer). The solved PEQ parameters are used as the supervised training target. The convolutional layer performs sliding convolution operations on the frequency axis using multiple convolutional kernels, with each kernel learning a different frequency domain pattern. Footsteps have energy concentrated in a narrow mid-to-high frequency band (e.g., 2kHz-4kHz), while gunshots are wide-bandwidth and high-energy, explosions have low-frequency impact characteristics, and environmental noise exhibits a stable, wide-bandwidth energy distribution. Through training, the convolutional layer can automatically learn the differentiated characteristics of these different audio events in the frequency domain. For example, footsteps exhibit regular energy peaks in specific frequency bands, while gunshots show energy bursts across the entire frequency range. The recurrent layer receives the multi-frame temporal feature matrix output by the convolutional layer and models the time series using a long short-term memory (LSM) gating mechanism (input gate, forget gate, output gate). Footsteps are a periodic sequence composed of multiple steps, with fixed time intervals between adjacent steps (depending on movement speed), while transient noises such as gunshots and explosions are random and sudden in time. The recurrent layer, through its internal memory units, can retain information from longer historical time steps, thus distinguishing the periodic temporal patterns of footsteps from the random temporal patterns of other transient noises. For example, when multiple consecutive frames exhibit similar frequency domain characteristics and the time intervals conform to the step point pattern, the recurrent layer identifies it as footsteps and outputs the corresponding equalizer parameters; when a frame exhibits wideband high-energy characteristics with no pattern between preceding and following frames, it is identified as gunshots or explosions, and the corresponding frequency band is suppressed. After training, the gain parameters and Q-values ​​output by the neural network are converted into bisecond-order filter coefficients. For a center frequency of... Sampling rate The gain is The quality factor is The peak equalizer has the following conversion formula: Where A is an intermediate quantity for gain conversion, used to convert decibel gain to linear gain: in, To normalize the angular frequency, the actual frequency is converted into a digital frequency: Where α is the bandwidth control parameter, which determines the bandwidth of the filter: The corresponding second-order filter coefficients are obtained: in These are the filter numerator coefficients. These are the denominator coefficients of the filter. The calculated numerator coefficients... and denominator coefficient Normalization is performed, using a0 as the divisor to make the constant term in the denominator equal to 1, resulting in normalized coefficients (bi-second-order filter coefficients) that can be directly used for difference equation calculations and are directly applicable to real-time audio filtering. The gain and quality factor parameters are then converted into bi-second-order filter coefficients. Bi-second-order filters are a fundamental filtering structure in digital signal processing; each bi-second-order filter corresponds to a second-order difference equation, requiring only a few multiplication and addition operations for calculation. By converting the parametric equalizer into bi-second-order filter coefficients, real-time filtering can be achieved on embedded audio processing chips with extremely low computational overhead, meeting the requirements of low latency and low power consumption. The neural network model obtained during offline training is deployed in the audio processing device. The trained neural network model and its parameters are stored in the storage medium of the audio processing device, including but not limited to flash memory, read-only memory, or programmable read-only memory of the embedded chip. The storage method includes hard-coding the model parameters as an array in the chip firmware or loading them from external storage into memory during chip initialization. The audio processing device can be an embedded audio product such as a gaming headset or gaming sound card. The system receives real-time audio streams from the gaming device via a digital audio interface (including Bluetooth audio interface, USB Audio interface, SPDIF interface, HDMI interface, or I²S digital audio interface). The real-time audio stream is buffered and divided into fixed-length audio frames, creating a multi-frame buffer to store multiple consecutive audio frames. The length of the multi-frame buffer is preset according to the recurrent layer's temporal receptive field, providing sufficient temporal context information to enable the recurrent layer to capture the temporal characteristics of footsteps. The audio frames are used as input to the convolutional layer, which performs a short-time Fourier transform on each audio frame. The short-time Fourier transform uses an analysis window of preset length and slides along the time axis with a preset overlap rate, converting the time-domain audio signal into a complex spectrum matrix. The rows of the complex spectrum matrix correspond to frequency sampling points, and the columns correspond to time frames. The complex spectrum matrix is ​​moduloed to obtain an amplitude spectrum matrix, which reflects the energy distribution of the audio signal at different times and frequencies. The convolutional layer performs sliding convolution on the frequency axis using a convolutional kernel to extract local frequency domain features from the amplitude spectrum matrix, capturing the differences in frequency domain between footsteps and other audio events such as gunshots and explosions, and outputting amplitude spectrum features. The amplitude spectrum features of multiple consecutive frames are concatenated along the time dimension to construct a temporal feature matrix, which serves as the input to the recurrent layer. This temporal feature matrix is ​​then input to the deployed neural network model, which infers and outputs the PEQ parameters (gain and Q value for each frequency band) corresponding to the current audio environment. The gain and Q value parameters output by the neural network, combined with a preset center frequency and sampling rate, are converted into bisecond-order filter coefficients according to the conversion formula after the neural network model training. The bisecond-order filter has a simple structure and high computational efficiency, requiring only a few multiplication and addition operations to complete the filtering calculation, thus reducing the computational load for real-time processing.The real-time audio signal is filtered based on these coefficients to enhance the footstep-related frequency bands and suppress interference. The filtered audio signal is then output to the playback device. The neural network model obtained in step S1 only requires forward inference in real-time processing and does not need to be retrained, ensuring the possibility of low-latency real-time processing.

[0036] Reference Figure 2 Here is a structural block diagram of a footsteps enhancement system in a game scene according to an embodiment of the present invention, comprising: The model training unit is used to pre-construct mixed audio samples containing footsteps and background sounds and generate corresponding enhanced target audio signals. Based on the mixed audio samples and the enhanced target audio signals, training equalizer parameters for frequency response correction are determined. The frequency domain features of the mixed audio samples are mapped to the equalizer parameters to form training data. The training data is used to train a neural network model. The neural network model includes a convolutional layer for extracting frequency domain features, a recurrent layer for analyzing temporal correlation, and a fully connected layer for outputting equalizer parameters. A solidified storage unit is used to solidify the trained neural network model and parameters in the storage medium of an audio processing device, wherein the audio processing device is an embedded audio processing chip. The audio acquisition unit uses Yahoo to acquire real-time audio signals of the game scene through the neural network model, performs time-frequency transformation on the real-time audio signals through the convolutional layer to obtain amplitude spectrum features, analyzes the time series correlation of the amplitude spectrum features through the recurrent layer, and outputs equalizer parameters through the fully connected layer. The equalization processing unit performs equalization processing on the real-time audio signal according to the equalizer parameters to enhance the game footsteps, and outputs the equalized audio signal to the audio playback device.

[0037] Specifically, the audio acquisition unit includes an audio input interface and an audio buffer. The audio input interface supports various digital audio interfaces, including Bluetooth audio, USB Audio, SPDIF, HDMI, or I²S digital audio interfaces, connecting to gaming devices to acquire the original game audio stream. The audio buffer is connected to the audio input interface and is used to buffer the real-time audio signal and divide it into audio frames. This module is configured with a multi-frame buffer to store multiple consecutive audio frames, providing contextual information for subsequent temporal feature analysis. The convolutional layer of the neural network model is used to perform short-time Fourier transform on the audio frames and take the modulus of the transform result to obtain amplitude spectrum features. The recurrent layer of the neural network model is used to analyze the temporal correlation of the amplitude spectrum features, and the fully connected layer of the neural network model is used to output equalizer parameters. The neural network model includes sequentially connected convolutional layers, recurrent layers, and fully connected layers, with parameter optimization completed through offline training. The equalizer parameters include gain parameters and quality factor parameters for each frequency band. The equalization processing unit includes a parameter converter and a digital filter. The parameter converter, connected to the fully connected layer, converts the gain and quality factor parameters output by the fully connected layer into bisecond-order filter coefficients, combining them with a preset center frequency and sampling rate. The preset center frequency is a multi-segment PEQ center frequency determined during the offline phase, such as 31.25Hz, 62.5Hz, 125Hz, 250Hz, 500Hz, 1000Hz, 2000Hz, 4000Hz, 8000Hz, and 16000Hz. The digital filter, connected to the parameter converter and the audio buffer, filters the real-time audio signal according to the bisecond-order filter coefficients, implementing the difference equation of the bisecond-order filter to dynamically equalize the real-time audio stream and enhance footstep noise suppression. The equalization processing unit also includes an audio output interface for outputting the filtered audio signal to audio playback devices, including headphones, speakers, and sound cards.

[0038] The aforementioned units are integrated into an embedded audio processing chip, such as a DSP, MCU, or FPGA, to achieve low-power, low-latency real-time footstep enhancement processing. The neural network model, trained offline, is pre-programmed into the chip. During real-time processing, only forward inference calculations are required, eliminating the need for online training and ensuring the system's real-time performance and low power consumption.

[0039] In summary, this invention pre-constructs a mixed audio sample containing footsteps and background sounds and generates a corresponding enhanced target audio signal. Based on the mixed audio sample and the enhanced target audio signal, it determines training equalizer parameters for frequency response correction. The frequency domain features of the mixed audio sample are mapped to the equalizer parameters to form training data, which is then used to train a neural network model. The trained neural network model and its parameters are stored in the storage medium of an audio processing device, which is an embedded audio processing chip. The neural network model acquires real-time audio signals from the game scene. The convolutional layer performs time-frequency transformation on the real-time audio signal to obtain amplitude spectrum features. The recurrent layer analyzes the time-series correlation of the amplitude spectrum features, and the fully connected layer outputs equalizer parameters. Based on the equalizer parameters, the real-time audio signal is equalized to enhance the game footsteps. The equalized audio signal is then output to an audio playback device, achieving selective enhancement of weak footstep signals in complex game sound fields while maintaining spatial positioning accuracy and a natural listening experience.

[0040] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for enhancing footsteps in a game scene, characterized in that, Includes the following steps: A mixed audio sample containing footsteps and background sounds is pre-constructed and a corresponding enhanced target audio signal is generated. Based on the mixed audio sample and the enhanced target audio signal, training equalizer parameters for frequency response correction are determined. The frequency domain features of the mixed audio sample are mapped to the equalizer parameters to form training data. A neural network model is trained using the training data. The neural network model includes a convolutional layer for extracting frequency domain features, a recurrent layer for analyzing temporal correlation, and a fully connected layer for outputting equalizer parameters. The trained neural network model and parameters are stored in the storage medium of the audio processing device, which is an embedded audio processing chip. The neural network model acquires real-time audio signals of the game scene, performs time-frequency transformation on the real-time audio signals through the convolutional layer to obtain amplitude spectrum features, analyzes the time-series correlation of the amplitude spectrum features through the recurrent layer, and outputs equalizer parameters through the fully connected layer. The real-time audio signal is processed according to the equalizer parameters to enhance the game footsteps, and the equalized audio signal is output to the audio playback device.

2. The method for enhancing footsteps in a game scene according to claim 1, characterized in that, The step of pre-constructing a mixed audio sample containing footsteps and background sounds includes: By controlling variables, footstep and background sound audio were collected in multiple game scenes, covering various ground materials, movement methods, collection distances, and environmental noise conditions. The collected audio data was divided into a footsteps sub-dataset and a background noise sub-dataset; Footstep audio is randomly selected from the footstep audio subset, background audio is randomly selected from the background sound subset, and a signal-to-noise ratio parameter is randomly generated. The amplitude of the background sound audio is adjusted proportionally according to the signal-to-noise ratio parameter and then superimposed with the footstep sound audio to obtain a mixed audio sample.

3. The method for enhancing footsteps in a game scene according to claim 2, characterized in that, The step of generating the corresponding enhanced target audio signal includes: Separate the footsteps component and the background sound component from the mixed audio samples; The footstep component is amplified by a fixed factor, which is preset according to the enhancement requirements to increase the energy proportion of the footstep in the mixed audio sample; The amplified footsteps component is then superimposed on the background sound component to obtain the enhanced target audio signal.

4. The method for enhancing footsteps in a game scene according to claim 1, characterized in that, The step of determining the training equalizer parameters for frequency response correction based on the mixed audio samples and the enhanced target audio signal includes: The time-frequency transformations of the mixed audio sample and the enhanced target audio signal are performed respectively to obtain the amplitude spectra of the mixed audio sample and the enhanced target audio signal; The amplitude spectra of the mixed audio samples and the enhanced target audio signal are normalized based on a preset reference frequency. Calculate the difference between the two normalized amplitude spectra at each frequency point to obtain the error curve; The error curve is subjected to mean compensation processing within a preset frequency band; Invert the compensated error curve to obtain the reverse compensation curve; The reverse compensation curve is subjected to bidirectional slope limiting, high-frequency gain limiting, and global gain clipping in sequence to obtain the physically constrained compensation curve. The compensation curve after the physical constraints are fitted to a finite number of training equalizer parameters.

5. The method for enhancing footsteps in a game scene according to claim 4, characterized in that, The processing steps for the bidirectional slope limitation include: Define the maximum permissible slope of attenuation as the frequency sampling point changes; The reverse compensation curve is scanned point by point from low frequency to high frequency, and the rise amplitude between adjacent frequency sampling points is limited to not exceed the maximum allowable slope to obtain the first limiting curve. The reverse compensation curve is scanned point by point from high frequency to low frequency, and the rise amplitude between adjacent frequency sampling points is limited to the maximum allowable slope to obtain the second limiting curve. The minimum value of the first limiting curve and the second limiting curve are combined to obtain the compensation curve after slope limitation.

6. The method for enhancing footsteps in a game scene according to claim 4, characterized in that, The step of fitting the physically constrained compensation curve to a finite number of training equalizer parameters includes: Set up a multi-band peak equalizer, each of which includes a center frequency, gain parameter, and quality factor parameter. The center frequency is preset according to the frequency band distribution to cover the frequency band where footstep energy is concentrated. Construct the overall frequency response of the multi-segment peak equalizer combination; A target function is established with the goal of minimizing the error between the overall frequency response and the compensation curve obtained after sequential bidirectional slope limiting, high-frequency gain attenuation, and maximum gain limiting. Within the preset range of gain parameters and quality factor parameters, determine the gain parameters and quality factor parameters that minimize the objective function.

7. The method for enhancing footsteps in a game scene according to claim 1, characterized in that, The step of acquiring real-time audio signals of the game scene through the neural network model includes: Receive real-time audio streams output by the game device via a digital audio interface; The real-time audio stream is cached and divided into audio frames of fixed length, and a multi-frame buffer is constructed to store multiple consecutive audio frames. The audio frames are used to construct multi-frame temporal information input to the convolutional layer.

8. The method for enhancing footsteps in a game scene according to claim 7, characterized in that, The step of obtaining amplitude spectrum features by performing time-frequency transformation on the real-time audio signal through the convolutional layer includes: The audio frame is used as the input to the convolutional layer; The convolutional layer performs time-frequency transformation on the audio frame, converting the time-domain audio signal into a frequency-domain representation; The energy distribution of the audio frame at each frequency point is extracted by taking the modulus of the time-frequency transformation result to obtain the amplitude spectrum features; The amplitude spectrum features of multiple consecutive frames are concatenated along the time dimension to construct a temporal feature matrix, which is then used as the input to the recurrent layer of the neural network model.

9. The method for enhancing footsteps in a game scene according to claim 1, characterized in that, The step of equalizing the real-time audio signal according to the equalizer parameters includes: The gain parameters and quality factor parameters output by the fully connected layer are combined with the preset center frequency and sampling rate to convert them into bi-second-order filter coefficients. The real-time audio signal is filtered according to the coefficients of the dual second-order filter. The filtered audio signal is output to the audio playback device.

10. A footstep sound enhancement system for a game scene, used to implement the footstep sound enhancement method for a game scene as described in any one of claims 1 to 9, characterized in that, include: The model training unit is used to pre-construct mixed audio samples containing footsteps and background sounds and generate corresponding enhanced target audio signals. Based on the mixed audio samples and the enhanced target audio signals, training equalizer parameters for frequency response correction are determined. The frequency domain features of the mixed audio samples are mapped to the equalizer parameters to form training data. The training data is used to train a neural network model. The neural network model includes a convolutional layer for extracting frequency domain features, a recurrent layer for analyzing temporal correlation, and a fully connected layer for outputting equalizer parameters. A solidified storage unit is used to solidify the trained neural network model and parameters in the storage medium of an audio processing device, wherein the audio processing device is an embedded audio processing chip. The audio acquisition unit uses Yahoo to acquire real-time audio signals of the game scene through the neural network model, performs time-frequency transformation on the real-time audio signals through the convolutional layer to obtain amplitude spectrum features, analyzes the time series correlation of the amplitude spectrum features through the recurrent layer, and outputs equalizer parameters through the fully connected layer. The equalization processing unit performs equalization processing on the real-time audio signal according to the equalizer parameters to enhance the game footsteps, and outputs the equalized audio signal to the audio playback device.

Images