An adaptive sound rendering method and system

Abstract

The application relates to the technical field of spatial audio rendering, and discloses a self-adaptive sound effect rendering method and system, which comprises the following steps: obtaining three-dimensional grid data and surface texture data of a target visual scene and extracting environmental acoustic characteristics; performing spatial ray tracing and propagation time delay analysis according to the environmental acoustic characteristics to obtain sound source propagation time delay; performing acoustic statistical analysis on the environmental acoustic characteristics and the sound source propagation time delay to obtain reverberation decay time; performing multi-dimensional feature splicing and numerical standardization processing on the reverberation decay time and the sound source propagation time delay to obtain an acoustic adaptation factor; performing nonlinear mapping and dimension matching processing on the acoustic adaptation factor to obtain a target reverberation convolution kernel; performing convolution rendering on a to-be-rendered audio signal according to the target reverberation convolution kernel, and then performing binaural positioning rendering to obtain final spatial audio. The application realizes that the rendered sound effect matches the current environmental acoustic characteristics.

Description

Technical Field

[0001] This invention relates to the field of spatial audio rendering technology, and in particular to an adaptive sound effect rendering method and system. Background Technology

[0002] With the rapid development of digital content industries such as virtual reality, gaming, and immersive interaction, users' demands for the immersion and realism of virtual scenes are gradually increasing. Among these, sound, as an important dimension of spatial perception, especially in changing scenes or complex spatial environments, directly impacts the quality of user experience, making spatial audio rendering technology a key support for user experience.

[0003] Existing audio rendering technologies commonly employ preset reverberation parameters or offline calculated acoustic models to generate spatial sound effects. Based on the listener's position, hardware such as delay units or signal processing algorithms are used to statically adjust the audio signal, including frequency adjustment and reverberation adaptation, to create an immersive sound field, immersive sound effects, and immersive sound, thereby improving the quality of the immersive experience in virtual reality scenes. However, these methods cannot generate matching acoustic parameters based on dynamically changing environments, and the generated reverberation effects cannot be correlated with the current visual scene in real time. Furthermore, when rendering spatial sound effects, this approach can only resolve the straight-line distance between the current sound source and the listener, lacking the ability to dynamically model the energy distribution and temporal structure of reflected and reverberated sound formed by multiple reflections of sound waves, making it difficult to adjust convolution kernel parameters in real time according to changes in the visual scene. This lack of technology at both the spatial feature perception and dynamic mapping of acoustic parameters leads to a disconnect between the sound presentation and the user's visual scene when existing spatial audio rendering systems handle dynamically changing scenes.

[0004] Therefore, existing technologies suffer from a disconnect between acoustic parameters and actual spatial characteristics. Summary of the Invention

[0005] This invention provides an adaptive sound effect rendering method and system to extract acoustic adaptation factors in a dynamically changing environment, so that the rendered sound effect matches the current acoustic characteristics of the environment.

[0006] Firstly, in order to solve the above-mentioned technical problems, the present invention provides an adaptive sound effect rendering method, comprising: Acquire 3D mesh data and surface texture data of the target visual scene and extract scene features to obtain environmental acoustic features; Spatial ray tracing is performed based on the environmental acoustic characteristics to obtain sound wave propagation characteristics. Propagation delay analysis is performed based on the sound wave propagation characteristics to obtain sound source propagation delay. Acoustic statistical analysis is performed based on the environmental acoustic characteristics and the sound source propagation delay to obtain reverberation decay time. Multidimensional features are spliced ​​together from the reverberation decay time and the sound source propagation delay to obtain acoustic splicing features. The acoustic splicing features are then numerically standardized to obtain the acoustic adaptation factor. The acoustic adaptation factor is nonlinearly mapped using a pre-defined convolutional neural network to obtain an initial reverberation convolution kernel. The initial reverberation convolution kernel is then subjected to dimension matching to obtain a target reverberation convolution kernel. Read the audio signal to be rendered, perform convolution rendering on the audio signal to be rendered according to the target reverberation convolution kernel to obtain convolutional spatial audio, and perform binaural localization rendering on the convolutional spatial audio to obtain the final spatial audio.

[0007] Secondly, the present invention provides an adaptive sound effect rendering method and system, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described above.

[0008] Compared with existing technologies, the advantages of the present invention are as follows: (1) By extracting the environmental acoustic features in the target visual scene, the present invention enables the calculation of acoustic parameters to match the scene geometry data and scene material data of the current visual scene, effectively solving the problem that traditional methods cannot capture the dynamic changes of the environment by relying solely on preset acoustic parameters, and realizing real-time perception of environmental acoustic features.

[0009] (2) The present invention calculates the acoustic adaptation factor by performing multi-dimensional feature splicing and numerical standardization processing on the reverberation decay time and the sound source propagation delay, so that the calculation of reverberation parameters can be adjusted in real time according to the changes in the current scene and complex material information, thus constructing a real-time sensing and response mechanism for environmental acoustic features, and solving the sense of disconnect between the sound performance and the current environment caused by the offline acoustic model in the traditional method.

[0010] (3) The present invention determines the initial reverberation convolution kernel by using a preset convolutional neural network to perform nonlinear mapping based on the acoustic adaptation factor, and performs dimension matching based on the initial reverberation convolution kernel to obtain the target reverberation convolution kernel, so that the acoustic adaptation factor is effectively converted into the parameter configuration of the reverberation model. This solves the problem that it is difficult to map the environmental acoustic feature data into the specific acoustic parameters of the reverberation model in the traditional technology, realizes the processing of audio signals driven in real time, and effectively avoids audio overload distortion. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the adaptive sound effect rendering method provided in the first embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] Reference Figure 1 The first embodiment of the present invention provides an adaptive sound effect rendering method, including the following steps: S11: Acquire the 3D mesh data and surface texture data of the target visual scene and extract scene features to obtain environmental acoustic features; S12, Spatial ray tracing is performed based on the environmental acoustic characteristics to obtain sound wave propagation characteristics. Propagation delay analysis is performed based on the sound wave propagation characteristics to obtain sound source propagation delay. Acoustic statistical analysis is performed based on the environmental acoustic characteristics and the sound source propagation delay to obtain reverberation decay time. S13, perform multi-dimensional feature splicing on the reverberation decay time and the sound source propagation delay to obtain acoustic splicing features, and perform numerical standardization on the acoustic splicing features to obtain acoustic adaptation factors; S14, a preset convolutional neural network is used to perform nonlinear mapping processing on the acoustic adaptation factor to obtain an initial reverberation convolution kernel, and the initial reverberation convolution kernel is subjected to dimension matching processing to obtain a target reverberation convolution kernel; S15, read the audio signal to be rendered, perform convolution rendering on the audio signal to be rendered according to the target reverberation convolution kernel to obtain convolutional spatial audio, perform binaural localization rendering on the convolutional spatial audio to obtain the final spatial audio.

[0014] In step S11, it is necessary to acquire the 3D mesh data and surface texture data of the target visual scene and extract scene features to obtain environmental acoustic features, including: The vertex coordinates and facet topological relationships in the 3D mesh data are analyzed to obtain scene geometry data; texture coordinate mapping is performed on the surface texture data based on the scene geometry data to obtain texture pixel data; Extract the color channel values ​​of the texture pixel data to obtain texture color features, and perform material matching on the texture color features according to the preset material parameter correspondence to obtain scene material data; The scene geometric data and the scene material data are feature-stitched to obtain a scene acoustic matrix. The scene acoustic matrix is ​​then subjected to feature dimensionality reduction to obtain environmental acoustic features.

[0015] In one implementation, for a 3D visual scene of an indoor living room, the scene's geometric data is first extracted, including a set of geometric features containing spatial information and normal vector information, providing spatial location information for determining subsequent scene material data. The vertex information in the extracted 3D spatial geometric data is accurately mapped to specific pixel positions in the surface texture data. Then, the color vector at the corresponding pixel position is extracted, and the texture color features, including the diffuse reflectance and roughness value at that pixel position, are calculated using a physically based rendering material conversion model. It should be noted that the physically based rendering material conversion model is an end-to-end material property mapping model, using microsurface theory and the law of conservation of energy as constraints for model construction. The RGB color channel count is used as the input vector, and the model is set based on a lookup mapping table in a material spectral database, determining the material color sample with the smallest color difference as the output. The material spectral database collects standard color samples of various common indoor materials, such as wood, stone, metal, and painted walls. A spectrophotometer is used to measure the spectral reflectance curves of each material's standard color sample in the visible light band, and the RGB color values ​​of the standard color samples are simultaneously collected under standard D65 lighting conditions. The diffuse reflectance and roughness values ​​corresponding to each material are recorded to construct the material spectral database. The constructed database index fields include four fields: RGB color vector, material type, diffuse reflectance, and roughness value. When performing material matching operations, the color difference between the input color vector and the RGB color vector of each record in the database is calculated. The diffuse reflectance and roughness value corresponding to the database record with the smallest color difference are determined as the output for that pixel location, thus obtaining the texture color feature.

[0016] Furthermore, to eliminate high-frequency noise during texture sampling, a bilinear interpolation algorithm is used to perform neighborhood smoothing calculations on the diffuse reflectance and roughness values ​​to obtain high-precision scene material data. The corresponding 3D vertex group is then extracted from the geometric feature set to serve as the scene spatial geometry. The scene spatial geometry and the high-precision scene material data (including spatial coordinates, normal vectors, target sound source coordinates, receiver coordinates, diffuse reflectance, and roughness values) are then matrix-concatenated to construct the scene acoustic matrix. The target sound source coordinates, receiver coordinates, and reflective surface geometry are retained as positioning fields. Principal component analysis is performed on the remaining continuous features. By calculating the eigenvalues ​​and eigenvectors of the covariance matrix, the first three eigenvectors are selected as principal components and, together with the positioning fields, form the environmental acoustic features. The material parameters are provided by a material spectral database; diffuse reflectance and roughness use normalized parameters recorded in the database; and the RGB color vector is used only for material type matching.

[0017] In step S12, spatial ray tracing is performed based on the environmental acoustic characteristics to obtain sound wave propagation characteristics. Propagation delay analysis is performed based on the sound wave propagation characteristics to obtain sound source propagation delay. Acoustic statistical analysis is performed based on the environmental acoustic characteristics and the sound source propagation delay to obtain reverberation decay time.

[0018] The step of performing spatial ray tracing based on the environmental acoustic characteristics to obtain sound wave propagation characteristics includes: The target sound source coordinates, receiver point coordinates, and reflector surface data are extracted from the environmental acoustic features to obtain scene positioning data; Based on the scene positioning data, a ray is emitted with the target sound source coordinates as the starting point, and the intersection of the emitted spatial ray and the scene grid is calculated to obtain the ray collision intersection point. The spatial ray is mirrored based on the normal vector of the reflecting surface at the intersection of the ray collision points to obtain the reflected line segments, and the reflected line segments that reach the coordinates of the receiving point are selected to obtain the reflection path; Calculate the straight-line distance between the target sound source coordinates and the receiving point coordinates, and accumulate the lengths of the reflection segments in the reflection path to obtain the propagation distance. Construct sound wave propagation characteristics based on the propagation distance and the reflection path.

[0019] For example, in an acoustic simulation scene of an indoor living room, the center position of the TV speaker is extracted from the environmental acoustic features as the target sound source coordinates, with coordinates set to (100, 200, 50), and the center of the sofa is extracted as the receiving point coordinates, with coordinates set to (400, 200, 50). The Euclidean distance between the two coordinate points is calculated, yielding a straight-line distance of 300. This straight-line distance is in scene grid coordinates, and the conversion between grid units and physical distance is determined by the scene resolution; in this example, the grid resolution is 0.115 meters per grid unit. Simultaneously, the plane equations and boundary coordinates of the surrounding walls and ceiling of the living room are calculated to obtain the position of the reflecting surface. Based on the obtained straight-line distance of 300 and the position of the reflecting surface, a Monte Carlo random sampling algorithm is used to emit a set of 1000 spatial rays into space, starting from the target sound source coordinates. A bounding box hierarchy algorithm is then used to perform mesh intersection calculations between the emitted spatial rays and the scene mesh to obtain the coordinates of the ray collision intersection point. It should be noted that after obtaining the coordinates of the ray collision intersection point, its validity needs to be verified. When it is determined that the coordinates of the ray collision intersection point are within the physical boundary of the wall polygon, the normal vector of the reflecting surface at the ray collision intersection point is mirrored using the Snell's law of reflection algorithm, and a reflection line segment with the same incident angle as the ray and the wall normal vector is output. The reflection line segment is then subjected to spatial ray tracing, and the propagation distance from the target sound source to the receiving point is obtained by accumulating the lengths of the reflection line segments in the reflection path. The positions of the reflecting surfaces along the reflection path are recorded sequentially, and the complete reflection path is output to obtain the sound wave propagation characteristics.

[0020] The step of performing propagation delay analysis based on the sound wave propagation characteristics to obtain the sound source propagation delay includes: The environmental physical parameters in the sound wave propagation characteristics are analyzed to obtain scene environment data; The sound wave propagation speed is obtained by processing the scene environment data using a preset empirical function of sound speed. The propagation time of the sound wave is obtained by dividing the propagation speed by the propagation distance; the propagation time of the sound wave is then converted into a time value to obtain the propagation delay of the sound source.

[0021] For example, in an indoor living room acoustic simulation scenario, environmental data within the living room space is extracted along the reflection path, including the ambient temperature and relative humidity of the current scene. When the living room is in air-conditioned cooling mode, the extracted ambient temperature is 25 degrees Celsius and the relative humidity is 60%. Using the ambient temperature and relative humidity as input parameters, an empirical formula algorithm for the speed of sound is applied to calculate the sound wave propagation speed in the current environment as 346 meters per second. The empirical formula for the speed of sound is c = 331.3 + 0.606 × T, where T is the temperature in degrees Celsius. The propagation distance tracked per unit time is approximately 34.6 meters after multiplying the straight-line distance by the grid resolution. The propagation distance is then divided according to the sound wave propagation speed, resulting in a calculated sound wave propagation time of 0.1 seconds.

[0022] Simultaneously, the frequency of the sound source emitted by the television speakers in the living room is extracted and set to 4000 Hz. Then, an atmospheric acoustic attenuation model algorithm is applied to the television speaker sound source frequency and the extracted scene environment data to obtain a sound wave air absorption coefficient of 3 dB per 100 meters. Multiplying this air absorption coefficient by the propagation distance of 34.6 meters yields a sound wave attenuation of approximately 1 dB. Since sound wave propagation involves losses, the degree of sound wave attenuation needs to be evaluated. It should be noted that in this example, the sound wave attenuation is a sound level quantity, used only to assess the energy loss of the sound wave along its propagation path, and not used to calculate the sound source propagation delay, which is determined by the propagation distance and the sound wave propagation speed. Converting the sound wave propagation time from seconds to milliseconds yields a sound source propagation delay of 100 milliseconds.

[0023] It should be noted that the time conversion for sound wave propagation refers to unit conversion. The sound wave propagation time is obtained by dividing the sound wave propagation speed by the propagation distance. The physical unit is seconds, and the measurement unit seconds is converted to milliseconds. The conversion method between milliseconds and seconds is 1 second = .

[0024] The step of performing acoustic statistical analysis based on the environmental acoustic characteristics and the sound source propagation delay to obtain the reverberation decay time includes: The sound absorption coefficient of the scene is obtained by weighted averaging the scene geometry data and the scene material data. The acoustic constants of the scene are obtained by statistically analyzing the sound absorption coefficient and geometric data of the scene using a preset acoustic statistical function. Based on the acoustic constants of the scene and the propagation delay of the sound source, an exponential decay fitting process is performed to obtain the decay time constant; The reverberation decay time is obtained by linearly mapping the scene geometry data according to the decay time constant.

[0025] For example, the 3D visual scene of the living room is analyzed, and the vertex coordinates of the spatial geometric mesh are extracted. A polyhedral volume integral algorithm is used to process the data, calculating the enclosed space volume of the living room to be 60 cubic meters. Simultaneously, a surface accumulation algorithm is used to calculate the boundary reflective area of ​​the living room to be 100 square meters. Next, visual recognition technology is used to determine the material type of the living room. The identified material sound absorption rate and the 100 square meter boundary reflective area are then input into an area-weighted average algorithm for processing, calculating the current scene sound absorption coefficient of the living room to be 20%.

[0026] Furthermore, using the obtained 20% scene sound absorption coefficient and a 60 cubic meter enclosed space volume as input parameters, acoustic statistical analysis was performed using the Sabine acoustic statistical model. It should be noted that the basic expression of the function is T60 = 0.161V / A, where V is the room volume, A represents the total sound absorption, and T60 represents the time required for the sound pressure level to decrease by 60 dB after the sound source stops. The total sound absorption A is equal to the product of the scene sound absorption coefficient and the boundary reflection area, i.e., A = 20% × 100 = 20 square meters. Substituting V = 60 cubic meters and A = 20 square meters into the Sabine acoustic function expression, the reverberation time T60 is calculated to be 0.161 × 60 / 20 = 0.483 seconds, or 483 milliseconds. Simultaneously, the obtained total sound absorption A and the scene sound absorption coefficient... Substituting into the scene acoustic constant R, the expression is R=A / (1- The scene acoustic constant R = 20 / (1-0.2) = 25 square meters was calculated, indicating that the scene acoustic constant characterizes the sound absorption capacity of the room's internal sound field. Next, based on the reverberation radius calculation formula in the Sabine acoustic statistical model, the reverberation radius at which the direct sound energy equals the reverberant sound energy is calculated to be 0.7 meters. The reverberation radius is used to determine whether the listener's distance from the sound source is within the reverberation dominance region. If the distance is greater than the reverberation radius, the listener is within the reverberation dominance region. In this example, the distance between the listener and the sound source, after conversion using scene grid coordinates, is approximately 34.6 meters, which is greater than the reverberation radius of 0.7 meters; therefore, the listener is determined to be within the reverberation dominance region. When the listener is within the reverberation dominance region, the scene acoustic constant R = 25 and the sound source propagation delay of 100 milliseconds are input into the exponential decay fitting algorithm. The reverberation energy sequence is formed by sorting the effective reflection paths obtained in step S12 according to their arrival time delay and combining the cumulative sound level attenuation of each path; the reverberation energy sequence is fitted using an exponential decay model to obtain a decay time constant of 5 milliseconds. The core of the exponential decay fitting algorithm is the nonlinear least squares method, and its basic mathematical expression is: ; in Let be the reverberation energy at time t. For the initial energy amplitude, The attenuation coefficient is... This is the steady-state offset constant.

[0027] In the Sabine acoustic statistical model, the reverberation radius is determined by the following formula: ; Where Q is the directivity factor of the sound source, and Q=1 is taken for an omnidirectional sound source; V is the volume of the enclosed space of the room. This refers to the reverberation time.

[0028] Finally, the obtained decay time constant is used to perform linear mapping on the 60 cubic meter enclosed space volume, and the reverberation decay time T = ×(V / V0), where V0 is the normalized reference volume, taken as V0=1 cubic meter, calculates T=5×60=300 milliseconds, obtaining the reverberation decay time of the target sound source in the living room as 300 milliseconds. Here, 483 milliseconds is the base reverberation time obtained from the Sabine model, used to verify the scene's sound absorption statistics; 300 milliseconds is the target sound source reverberation decay time obtained based on the effective reflection path fitting, serving as input for subsequent acoustic feature extraction.

[0029] In step S13, the reverberation decay time and the sound source propagation delay are multidimensionally spliced ​​to obtain acoustic splicing features. The acoustic splicing features are then numerically standardized to obtain acoustic adaptation factors, including: The frequency band energy distribution of the reverberation decay time is analyzed to obtain the energy distribution numbers covering the low-frequency, mid-frequency, and high-frequency bands; The sound source propagation delay is extracted to obtain the reflected sound level. The reflected sound level and the energy distribution number are then used for vector splicing to obtain the acoustic splicing feature. Numerical translation and scaling operations are performed on the acoustic splicing features to obtain the acoustic normalized feature matrix. Dimensionality reduction processing is then performed on the acoustic normalized feature matrix to obtain the acoustic core features. The acoustic core features are spatially acoustically mapped according to preset sound field distribution parameters to obtain acoustic mapping weights; The acoustic adaptation factor is obtained by weighted summation based on the acoustic mapping weights and the acoustic core features.

[0030] It should be noted that the vector concatenation process refers to combining the vectors according to a preset dimensional order. This can be achieved using deep learning concatenation methods, such as the Concat method which concatenates the feature maps along the channel dimension to increase the number of channels, the Add method which adds the feature maps element by element while keeping the number of channels unchanged, or the concatenate function built into NumPy, to obtain the acoustic concatenated features. The numerical translation and scaling operations on the acoustic concatenated features refer to applying a zero-mean normalization algorithm to the energy distribution number and the reflected sound level to obtain an acoustically normalized feature matrix that eliminates the influence of dimensions. Zero-mean normalization transforms the data into a distribution with a mean of 0 and a standard deviation of 1. The normalization formula is: ; in, It is the raw data. It is the mean of the data. It is the standard deviation of the data.

[0031] Furthermore, the dimensionality reduction processing of the acoustically standardized feature matrix refers to using principal component analysis (PCA) to project high-dimensional data into a low-dimensional space, thereby simplifying the data. First, the original data is standardized so that the mean of each variable is 0 and the standard deviation is 1. Then, the eigenvalues ​​and eigenvectors of the covariance matrix of the standardized sample matrix are calculated. Based on the magnitude of the eigenvalues, the first three eigenvectors are selected as principal components, transforming the original data into a new principal component space.

[0032] It is worth noting that the spatial acoustic mapping employs a multilayer perceptron mapping model for nonlinear transformation. The multilayer perceptron is a typical feedforward neural network, consisting of an input layer, a hidden layer, and an output layer. The input layer contains the obtained acoustic core features, with an input dimension of 3, corresponding to the three principal components of the acoustic core features. The hidden layer is a fully connected layer containing 64 neurons, using the ReLU activation function for nonlinear transformation. The output layer is also a fully connected layer containing 3 neurons, using the softmax activation function, and outputs acoustic mapping weights corresponding to the low-frequency, mid-frequency, and high-frequency bands, with the three weights being non-negative and summing to 1. The model is trained using supervised learning, with training data derived from acoustic simulation samples. For 3D scenes of different types of enclosed spaces, including concert halls, conference rooms, living rooms, and theaters, corresponding acoustic core feature vectors are collected. Sound field distribution parameters for each scene in the low, mid, and high frequency bands are calculated using acoustic simulation software. Simultaneously, the sound field distribution parameters for each frequency band are normalized and used as supervision labels for the corresponding acoustic core features, i.e., the true values ​​of the acoustic mapping weights. The total number of training samples is no less than 2000 sets, composed of different sizes and materials of various spatial scenes. Model training involves forward propagation layer by layer from the input feature sequence to calculate the output. Mean squared error is used as the loss function, and the Adam optimizer is used to adjust the network weights and biases with a learning rate of 0.001. The batch size is set to 32, and the training is conducted for 200 epochs until the loss function converges and the mean absolute error on the validation set is less than 0.05. The validation set uses 200 randomly selected independent samples that did not participate in the training for generalization performance evaluation. The output acoustic core feature vector of length 3 is mapped to the corresponding weight distribution of the three frequency bands, resulting in the acoustic mapping weights. Next, a weighted summation calculation is performed, which sums the acoustic core features according to the weight ratio to obtain the acoustic adaptation factor.

[0033] In step S14, a preset convolutional neural network is used to perform nonlinear mapping processing on the acoustic adaptation factor to obtain an initial reverberation convolution kernel. The initial reverberation convolution kernel is then subjected to dimension matching processing to obtain a target reverberation convolution kernel.

[0034] Specifically, a pre-defined convolutional neural network is used to perform nonlinear mapping processing on the acoustic adaptation factors to obtain an initial reverberation convolution kernel. For example, in an indoor concert hall scenario, the aforementioned weighted summation of the acoustic adaptation factors is input into a fully connected mapping layer, mapped to a 64-dimensional feature vector, and then reshaped into an 8x8 dynamic feature tensor, which serves as the input to the convolutional neural network. The 8×8 dynamic feature tensor is input into a preset two-dimensional convolutional neural network structure for convolution. First, it is input into the first convolutional layer using 32 3×3 convolutional kernels with a stride of 1 and SAME padding, resulting in an 8×8×32 feature map. Then, the output feature map is processed non-linearly using a linear activation function. Next, the 8×8×32 feature map is input into the second convolutional layer using 64 3×3 convolutional kernels with a stride of 1 and SAME padding, resulting in an 8×8×64 feature map. Again, the resulting feature map is processed non-linearly using a linear activation function and then fed into a max-pooling layer for 2×2 max-pooling. The output feature map after pooling is 4×4×64. Finally, the 4×4×64 feature map is input into the third convolutional layer using 128 3×3 convolutional kernels. With a stride of 1 and padding mode of SAME, the output feature map is 4×4×128. The resulting feature map is then processed non-linearly using a linear activation function, followed by a 2×2 max pooling operation. The output feature map after pooling is 2×2×128. Finally, the pooled feature map is flattened into a one-dimensional feature vector with a length of 2×2×128=512. To reduce the subsequent convolution kernel numerical processing, the features need further compression. This 512-dimensional feature vector is then input into a fully connected layer, mapping the 512 dimensions to a 256-dimensional spatial reflection feature sequence. The resulting 256-dimensional spatial reflection feature sequence is then processed by a linear rectified activation function, setting negative values ​​to zero and retaining positive values. The output is a 256-dimensional impulse response matrix, which is the initial reverberant convolution kernel containing 256 discrete values.

[0035] It should be noted that the training of the two-dimensional convolutional neural network model is supervised learning. A set of acoustic adaptation factors is used as the input training data, and 256 sampling points from the simulated actual impulse response matrix are used as the output labels. The total number of training samples is no less than 2000 sets, divided into training and validation sets in an 8:2 ratio. Mean squared error is used as the loss function, and the Adam optimizer is used to adjust the network weights and biases with a learning rate of 0.001. The batch size is set to 32, and the training is conducted for 200 epochs until the loss function converges and the mean absolute error on the validation set is less than 0.03.

[0036] The step of performing dimension matching processing on the initial reverberation convolution kernel to obtain the target reverberation convolution kernel includes: Frequency band analysis is performed on the initial reverberant convolution kernel to obtain the frequency band distribution matrix; The frequency band distribution matrix is ​​resampled to obtain an acoustic transition sequence, and the acoustic transition sequence is then subjected to dimensionality expansion processing to obtain a response compensation matrix. The target reverberation convolution kernel is obtained by performing dimension completion operation based on the response compensation matrix and the initial reverberation convolution kernel.

[0037] For example, in the spatial audio processing flow of a concert hall, the feature dimension of the initial reverberation convolution kernel with a sequence length of 256 is first read. When the feature dimension of the initial reverberation convolution kernel is inconsistent with the standard dimension of the matrix corresponding to the preset sampling rate identifier (1024), a short-time Fourier transform is applied to the initial reverberation convolution kernel with a window function length of 32 and an overlap step of 16. For an input sequence of length 256, the corresponding number of time frames is (256-32) / 16+1=15 frames, and the number of frequency points is 32 / 2+1=17, resulting in an initial frequency band distribution matrix of 17×15. This matrix is ​​then uniformly processed, with the frequency dimension reduced from 17 to 16 through frequency domain mean pooling, and the time dimension expanded from 15 to 16 through linear interpolation. The output is a frequency band distribution matrix of 16×16, where each matrix element represents the energy amplitude of the corresponding time frequency point. The mean of the 16×16 frequency band distribution matrix is ​​calculated along the time dimension to obtain a node weight vector of length 16. Resampling is performed based on the node weight vector using a bilinear interpolation algorithm. Linear interpolation is performed between adjacent discrete nodes according to a linear ratio, with an interpolation factor of 4, meaning three new values ​​are evenly inserted between every two adjacent nodes. This results in an acoustic transition sequence of length 16 × 4 = 64. Tensor product operations are then used to expand this one-dimensional acoustic transition sequence into a 64 × 64 response compensation matrix. The response compensation matrix is ​​then mapped to the initial reverberation convolution kernel to fill in missing high-frequency reflection details. The 256-dimensional initial reverberation convolution kernel is zero-padded to expand it into a 1024-dimensional temporary sequence, where the first 256 positions are filled with the original 256 values, and the remaining 768 positions are filled with zeros. Simultaneously, the 64 × 64 response compensation matrix is ​​expanded row-wise into a 4096-dimensional vector. Then, 768 values ​​are extracted from the expanded vector at fixed equal intervals as the compensation increment vector. The 768-dimensional compensation increment vector is added element-wise to the initial 768 zero values ​​to obtain a target reverberation convolution kernel with a dimension of 1024.

[0038] In step S15, the audio signal to be rendered is read, and the audio signal to be rendered is convolved and rendered according to the target reverberation convolution kernel to obtain convolutional spatial audio. The convolutional spatial audio is then rendered using binaural localization to obtain the final spatial audio.

[0039] The step of performing convolution rendering on the audio signal to be rendered based on the target reverberation convolution kernel to obtain convolutional spatial audio includes: The audio signal to be rendered is segmented to generate a signal frame sequence. The target reverberation convolution kernel is multiplied and added with the signal frame sequence to obtain the frame convolution feature. The matrix elements of the frame-convolutional features that exceed a preset saturation threshold are subjected to nonlinear amplitude adjustment processing to obtain amplitude compensation features. The amplitude compensation features are then spliced ​​and smoothed to obtain acoustic waveform features. The convolutional spatial audio is obtained by performing high-order coding processing based on the acoustic waveform features.

[0040] For example, in a concert hall acoustic simulation scenario, the audio signal to be rendered, which is a violin solo, is acquired, and it is divided into a signal frame sequence of length 2048. The signal frame sequence is then subjected to sliding multiplication and addition operation with the target reverberation convolution kernel using a fast convolution algorithm to obtain the frame convolution feature.

[0041] After obtaining the frame-by-frame convolutional features, the nonlinear amplitude adjustment refers to extracting the acoustic energy profile of volume changes in the frame-by-frame convolutional features using the Hilbert transform algorithm. The preset saturation threshold is predetermined based on the full-amplitude range of the digital audio and the distortion-free dynamic range of the playback device. Next, the peak value in the frame-by-frame convolutional features is compared with the preset saturation threshold. When the set saturation threshold is less than the peak value in the frame-by-frame convolutional features, the nonlinear amplitude adjustment mechanism is triggered. Based on the excess difference, a nonlinear compression gain mapping function based on a logarithmic curve is constructed to obtain the amplitude compensation feature.

[0042] The generated nonlinear compression gain mapping function is used to proportionally attenuate specific frequency band matrix elements in the framed convolutional features that exceed a preset saturation threshold. Multiple amplitude compensation features are then concatenated and smoothly cross-transformed according to a 50% overlap rate between adjacent frames to obtain the acoustic waveform features. The higher-order coding refers to using a three-dimensional spherical harmonic function to project the one-dimensional acoustic waveform features into three-dimensional space, obtaining the convolutional spatial audio with three-dimensional spatial orientation information. In this embodiment, the order of the spherical harmonic function expansion is set to [value missing]. =3, at this time the number of channels is ( +1)²=16, corresponding to 16 independent output channels, sufficient for high-precision sound field reconstruction in both horizontal and vertical directions. The polar angle in the direction parameters... and azimuth Derived from the scene positioning data described in step S12, polar angle The azimuth angle is determined by calculating the vertical height difference and horizontal distance between the receiving point (sofa center, coordinates (400, 200, 50)) and the target sound source (TV speaker center, coordinates (100, 200, 50)). The orientation of the two coordinate points on the horizontal plane is determined by their relative directions. The spherical harmonic functions of each order are calculated sequentially from the one-dimensional acoustic waveform characteristics. The value in the corresponding direction, where =0,1,2,3,m=-l,0,l and the obtained mono signals are multiplied and accumulated to perform projection decomposition of the one-dimensional signal onto the three-dimensional spatial sound field. The mono time-domain waveform is expanded into a 16-channel spatial sound field matrix, with each channel corresponding to a specific directional component in three-dimensional space, resulting in a convolutional spatial audio output with three-dimensional orientation information.

[0043] It should be noted that the Hilbert transform algorithm converts a real-valued signal into a complex-valued signal, thus obtaining an analytical representation of the signal. Its mathematical definition is as follows: ; The nonlinear compression gain mapping function, based on the logarithmic curve, is grounded in the Weber-Fechner law. It states that the change in loudness perceived by the human ear has an approximately linear relationship with the sound pressure level in decibels. The scale is logarithmic for decibels themselves, constructing a dynamic range control mapping relationship that compresses large signals and preserves or slightly expands small signals. Its mathematical expression is as follows: ; in, The difference is the magnitude of the value exceeding the threshold, and k is the compression curvature adjustment coefficient. To the maximum extent of the difference, This represents the maximum compression ratio, ranging from (0,1).

[0044] The mathematical definition of the three-dimensional spherical harmonic function is as follows: ; in, Let m be the order of the spherical harmonic function. Polar angle, It is the azimuth angle. For the Legendre function.

[0045] The step of performing binaural localization rendering on the convolutional spatial audio to obtain the final spatial audio includes: Spatial matching processing is performed between the convolutional spatial audio and the preset azimuth coordinates to obtain azimuth acoustic features; Based on the azimuth acoustic features and the convolutional spatial audio, the time difference and sound level difference on the left and right ear signals are calculated to obtain the binaural time difference and binaural sound level difference. Based on the binaural time difference and the binaural sound level difference, the convolutional spatial audio is subjected to binaural fast convolution processing to obtain a binaural synthesized signal; The binaural synthesized signal is subjected to acoustic crosstalk cancellation processing to obtain the final spatial audio.

[0046] In one implementation, the preset azimuth coordinates and the convolutional spatial audio are matched and interpolated using a spherical trigonometric interpolation algorithm within a pre-constructed head-related transfer function (HRTF) database. This process extracts the acoustic features corresponding to the azimuth, yielding the azimuth acoustic features. The HRTF database uses the publicly available standard CIPIC_HRTF database. This database contains 1250 head-related impulse response (HRIR) measurements from 45 test samples at 25 different azimuth angles and 50 different pitch angles, with an angle increment of approximately 5 degrees. Azimuth and pitch angles are used as spatial query indices. To eliminate individual differences, the HRIR values ​​from the 45 subjects in the same direction are averaged. First, the preset azimuth coordinates are converted from general spherical coordinates to the polar coordinate system used by the CIPIC database. Then, the spherical angular distance between the target azimuth and each index point in the database is calculated. The three index points closest to the target azimuth are selected on the sphere, and the target azimuth must be located within the spherical triangle formed by these three points. If the target azimuth is outside the spherical triangle, the nearest index point is selected. For the three selected index points, calculate the spherical centroid coordinates of the target orientation within the spherical triangle, and perform a weighted average with the corresponding left and right ear impulse responses point by point to obtain the left and right ear impulse responses of the target orientation, which are the directional acoustic features.

[0047] Next, the azimuth acoustic features and convolutional spatial audio are analyzed using a cross-correlation function analysis algorithm and a frequency band energy comparison algorithm. By calculating the cross-correlation peak values ​​of the left and right ear signals in the time domain, the binaural time difference is extracted. Simultaneously, the energy distribution of the left and right ear signals in different frequency bands is compared in the frequency domain to obtain the binaural sound level difference. The binaural time difference is used to generate a sound wave delay sequence that controls the phase shift through an all-pass filter design algorithm, and the binaural sound level difference is used to generate an amplitude attenuation matrix using a frequency band equalization algorithm. Based on the sound wave delay sequence and the amplitude attenuation matrix, a fast convolution algorithm is applied to the convolutional spatial audio to jointly modulate the audio signal in the time and frequency domains, outputting the binaural synthesized signal. It should be noted that the fast convolution algorithm described in this embodiment adopts the overlap-preserving method, which divides the audio signal to be rendered into signal frames of fixed length. Each segmented signal frame is multiplied with the target reverberation convolution kernel in the frequency domain, and then converted into a time-domain convolution result by inverse Fourier transform. Adjacent signal frames adopt a 50% overlap rate with those in step S15 and are overlap-preserving and spliced. The frame length is 2048 sampling points.

[0048] In this embodiment, a loudspeaker is used as the final playback terminal, therefore, acoustic crosstalk cancellation is required for the binaural synthesized signal. A crosstalk cancellation filter is generated from the binaural synthesized signal using a recursive least squares adaptive filtering algorithm. The filter order is set to 128, and the forgetting factor is set to 0.99. By mixing a specific proportion of the right channel inverted signal into the left channel signal, and simultaneously mixing the left channel inverted signal into the right channel, the acoustic crosstalk components from the left channel signal to the right ear and from the right channel signal to the left ear are canceled, resulting in the final spatial audio suitable for loudspeaker playback.

[0049] In summary, this invention discloses an adaptive sound effect rendering method, comprising: acquiring three-dimensional mesh data and surface texture data of a target visual scene and extracting environmental acoustic features; performing spatial ray tracing and propagation delay analysis based on the environmental acoustic features to obtain the sound source propagation delay; performing acoustic statistical analysis on the environmental acoustic features and the sound source propagation delay to obtain the reverberation decay time; performing multi-dimensional feature splicing and numerical standardization on the reverberation decay time and the sound source propagation delay to obtain an acoustic adaptation factor; performing nonlinear mapping and dimension matching on the acoustic adaptation factor to obtain a target reverberation convolution kernel; performing convolution rendering on the audio signal to be rendered based on the target reverberation convolution kernel, followed by binaural localization rendering to obtain the final spatial audio.

[0050] A second embodiment of the present invention provides an adaptive sound effect rendering system, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described above.

[0051] It should be noted that the adaptive sound effect rendering system provided in this embodiment of the invention is used to execute all the process steps of the adaptive sound effect rendering method in the above embodiment. The working principle and beneficial effects of the two are one-to-one, so they will not be described again.

[0052] It should be noted that the system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0053] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. An adaptive sound effect rendering method, characterized in that, include: Acquire 3D mesh data and surface texture data of the target visual scene and extract scene features to obtain environmental acoustic features; Spatial ray tracing is performed based on the environmental acoustic characteristics to obtain sound wave propagation characteristics. Propagation delay analysis is performed based on the sound wave propagation characteristics to obtain sound source propagation delay. Acoustic statistical analysis is performed based on the environmental acoustic characteristics and the sound source propagation delay to obtain reverberation decay time. Multidimensional features are spliced ​​together from the reverberation decay time and the sound source propagation delay to obtain acoustic splicing features. The acoustic splicing features are then numerically standardized to obtain the acoustic adaptation factor. The acoustic adaptation factor is nonlinearly mapped using a pre-defined convolutional neural network to obtain an initial reverberation convolution kernel. The initial reverberation convolution kernel is then subjected to dimension matching to obtain a target reverberation convolution kernel. Read the audio signal to be rendered, perform convolution rendering on the audio signal to be rendered according to the target reverberation convolution kernel to obtain convolutional spatial audio, and perform binaural localization rendering on the convolutional spatial audio to obtain the final spatial audio.

2. The adaptive sound effect rendering method according to claim 1, characterized in that, The process of acquiring 3D mesh data and surface texture data of the target visual scene and extracting scene features to obtain environmental acoustic features includes: The vertex coordinates and facet topological relationships in the 3D mesh data are analyzed to obtain the scene geometry data; Based on the scene geometry data, the surface texture data is mapped to texture coordinates to obtain texture pixel data; Extract the color channel values ​​of the texture pixel data to obtain texture color features, and perform material matching on the texture color features according to the preset material parameter correspondence to obtain scene material data; The scene geometric data and the scene material data are feature-stitched to obtain a scene acoustic matrix. The scene acoustic matrix is ​​then subjected to feature dimensionality reduction to obtain environmental acoustic features.

3. The adaptive sound effect rendering method according to claim 1, characterized in that, The step of performing spatial ray tracing based on the environmental acoustic characteristics to obtain sound wave propagation characteristics includes: The target sound source coordinates, receiver point coordinates, and reflector surface data are extracted from the environmental acoustic features to obtain scene positioning data; Based on the scene positioning data, a ray is emitted with the target sound source coordinates as the starting point, and the intersection of the emitted spatial ray and the scene grid is calculated to obtain the ray collision intersection point. The spatial ray is mirrored based on the normal vector of the reflecting surface at the intersection of the ray collision points to obtain the reflected line segments, and the reflected line segments that reach the coordinates of the receiving point are selected to obtain the reflection path; The straight-line distance between the target sound source coordinates and the receiving point coordinates is calculated, and the lengths of the reflection segments in the reflection path are accumulated to obtain the propagation distance. The sound wave propagation characteristics are constructed based on the propagation distance and the reflection path.

4. The adaptive sound effect rendering method according to claim 3, characterized in that, The step of performing propagation delay analysis based on the sound wave propagation characteristics to obtain the sound source propagation delay includes: The environmental physical parameters in the sound wave propagation characteristics are analyzed to obtain scene environment data; The sound wave propagation speed is obtained by processing the scene environment data using a preset empirical function of sound speed. The propagation time of the sound wave is obtained by dividing the propagation speed by the propagation distance. The propagation time of the sound wave is converted into time to obtain the propagation delay of the sound source.

5. The adaptive sound effect rendering method according to claim 2, characterized in that, The step of performing acoustic statistical analysis based on the environmental acoustic characteristics and the sound source propagation delay to obtain the reverberation decay time includes: The sound absorption coefficient of the scene is obtained by weighted averaging the scene geometry data and the scene material data. The acoustic constants of the scene are obtained by statistically analyzing the sound absorption coefficient and geometric data of the scene using a preset acoustic statistical function. Based on the acoustic constants of the scene and the propagation delay of the sound source, an exponential decay fitting process is performed to obtain the decay time constant; The reverberation decay time is obtained by linearly mapping the scene geometry data according to the decay time constant.

6. The adaptive sound effect rendering method according to claim 1, characterized in that, The reverberation decay time and the sound source propagation delay are multidimensionally spliced ​​to obtain acoustic splicing features. These acoustic splicing features are then numerically standardized to obtain acoustic adaptation factors, including: The frequency band energy distribution of the reverberation decay time is analyzed to obtain the energy distribution numbers covering the low-frequency, mid-frequency, and high-frequency bands; The sound source propagation delay is extracted to obtain the reflected sound level. The reflected sound level and the energy distribution number are then used for vector splicing to obtain the acoustic splicing feature. Numerical translation and scaling operations are performed on the acoustic splicing features to obtain the acoustic normalized feature matrix. Dimensionality reduction processing is then performed on the acoustic normalized feature matrix to obtain the acoustic core features. The acoustic core features are spatially acoustically mapped according to preset sound field distribution parameters to obtain acoustic mapping weights; The acoustic adaptation factor is obtained by weighted summation based on the acoustic mapping weights and the acoustic core features.

7. The adaptive sound effect rendering method according to claim 1, characterized in that, The step of performing dimension matching processing on the initial reverberation convolution kernel to obtain the target reverberation convolution kernel includes: Frequency band analysis is performed on the initial reverberant convolution kernel to obtain the frequency band distribution matrix; The frequency band distribution matrix is ​​resampled to obtain an acoustic transition sequence, and the acoustic transition sequence is then subjected to dimensionality expansion processing to obtain a response compensation matrix. The target reverberation convolution kernel is obtained by performing dimension completion operation based on the response compensation matrix and the initial reverberation convolution kernel.

8. The adaptive sound effect rendering method according to claim 1, characterized in that, The step of convolutional rendering the audio signal to be rendered based on the target reverberation convolution kernel to obtain convolutional spatial audio includes: The audio signal to be rendered is segmented to generate a signal frame sequence. The target reverberation convolution kernel is multiplied and added with the signal frame sequence to obtain the frame convolution feature. The matrix elements of the frame-convolutional features that exceed a preset saturation threshold are subjected to nonlinear amplitude adjustment processing to obtain amplitude compensation features. The amplitude compensation features are then spliced ​​and smoothed to obtain acoustic waveform features. The convolutional spatial audio is obtained by performing high-order coding processing based on the acoustic waveform features.

9. The adaptive sound effect rendering method according to claim 1, characterized in that, The step of performing binaural localization rendering on the convolutional spatial audio to obtain the final spatial audio includes: Spatial matching processing is performed between the convolutional spatial audio and the preset azimuth coordinates to obtain azimuth acoustic features; Based on the azimuth acoustic features and the convolutional spatial audio, the time difference and sound level difference on the left and right ear signals are calculated to obtain the binaural time difference and binaural sound level difference. Based on the binaural time difference and the binaural sound level difference, the convolutional spatial audio is subjected to binaural fast convolution processing to obtain a binaural synthesized signal; The binaural synthesized signal is subjected to acoustic crosstalk cancellation processing to obtain the final spatial audio.

10. An adaptive sound rendering system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method according to any one of claims 1 to 9.

Images